From 8321af538e17ad0e1762c7c7e2f88ee76f5b3a8f Mon Sep 17 00:00:00 2001 From: kinjelom Date: Thu, 21 Sep 2023 10:28:06 +0200 Subject: [PATCH] Ops to configure external blobstore to use AWS S3 Compatible Object Storage (e.g. Min.IO) --- operations/README.md | 1 + operations/use-s3-blobstore-custom.yml | 52 ++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 operations/use-s3-blobstore-custom.yml diff --git a/operations/README.md b/operations/README.md index c7282e2f3..d515e026a 100644 --- a/operations/README.md +++ b/operations/README.md @@ -57,6 +57,7 @@ This is the README for Ops-files. To learn more about `cf-deployment`, go to the | [`use-blobstore-cdn.yml`](use-blobstore-cdn.yml) | Adds support for accessing the `droplets` and `resource_pool` blobstore buckets via signed urls over a cdn. | This assumes that you are using the same keypair for both buckets. Introduces [new variables](example-vars-files/vars-use-blobstore-cdn.yml) | **NO** | | [`use-compiled-releases.yml`](use-compiled-releases.yml) | Instead of having your BOSH Director compile each release, use this ops-file to use pre-compiled releases for a deployment speed improvement. | These releases are compiled against a specific stemcell version that is listed in the opsfile. Note that no Windows releases are currently compiled. | **YES** | | [`use-external-blobstore.yml`](use-external-blobstore.yml) | Removes the singleton-blobstore instance group, and adds `fog_connection` properties for components that use the blobstore. **Warning**: this does not migrate data, and will delete any existing singleton-blobstore groups. | This requires an external data store. Introduces [new variables](example-vars-files/vars-use-external-blobstore.yml) for blobstore connection details which will need to be provided at deploy time. | **YES** | +| [`use-s3-blobstore-custom.yml`](use-s3-blobstore-custom.yml) | Configures external blobstore to use AWS S3 Compatible Object Storage (e.g. Min.IO). | Requires [`use-external-blobstore.yml`](use-external-blobstore.yml) and [`use-s3-blobstore.yml`](use-s3-blobstore.yml). Introduces new variables: `blobstore_endpoint` (e.g. `http://s3-blobstore.service.bosh:9001`), `blobstore_path_style` (e.g. `true`). | **NO** | | [`use-external-dbs.yml`](use-external-dbs.yml) | Removes the `database` instance group, `pxc` release, and all MySQL variables. **Warning**: this does not migrate data, and will delete existing database instance groups. | This requires an external data store. Introduces [new variables](example-vars-files/vars-use-external-dbs.yml) for DB connection details which will need to be provided at deploy time. This must be applied _before_ any ops files that removes jobs that use a database, such as the ops file to remove the routing API. | **YES** | | [`use-haproxy.yml`](use-haproxy.yml) | Deploys a single haproxy instance to be used as a load balancer. | This opsfile doesn't depend on use of an IaaS VIP and doesn't use `keepalived` property of the [haproxy-boshrelease](https://github.com/cloudfoundry-incubator/haproxy-boshrelease). | **NO** | | [`use-haproxy-public-network.yml`](use-haproxy-public-network.yml) | Puts haproxy instance on a public network with a static IP assigned to it. | Requires `use-haproxy.yml`. This ops file also requires your BOSH cloud-config to have a `vm_extension` called `cf-haproxy-network-properties`, which configures firewall rules to allow public traffic on the necessary ports (You will need to allow at least the default HTTP and HTTPS ports (`80` and `443`), port `4443` for `doppler`, as well as the [port range configured for the TCP Routing](https://github.com/cloudfoundry/cf-deployment/blob/a6983a1b3345cd9f5af0f26d5c10265de0c7851f/cf-deployment.yml#L726)). | **NO** | diff --git a/operations/use-s3-blobstore-custom.yml b/operations/use-s3-blobstore-custom.yml new file mode 100644 index 000000000..d8bf8dc46 --- /dev/null +++ b/operations/use-s3-blobstore-custom.yml @@ -0,0 +1,52 @@ +--- +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/buildpacks/fog_connection/provider + error: "Please apply 'use-external-blobstore.yml' and 'use-s3-blobstore.yml' before applying 'use-s3-blobstore-custom.yml'." + value: AWS + +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/buildpacks?/fog_connection?/endpoint? + value: ((blobstore_endpoint)) + +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/buildpacks?/fog_connection?/path_style? + value: ((blobstore_path_style)) + +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/droplets?/fog_connection?/endpoint? + value: ((blobstore_endpoint)) + +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/droplets?/fog_connection?/path_style? + value: ((blobstore_path_style)) + +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/packages?/fog_connection?/endpoint? + value: ((blobstore_endpoint)) + +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/packages?/fog_connection?/path_style? + value: ((blobstore_path_style)) + +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/resource_pool?/fog_connection?/endpoint? + value: ((blobstore_endpoint)) + +- type: replace + path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/resource_pool?/fog_connection?/path_style? + value: ((blobstore_path_style)) + +# +# You can use a bosh deployment with a bosh-dns-alias as the endpoint host: +# +# - type: replace +# path: /addons/name=bosh-dns-aliases/jobs/name=bosh-dns-aliases/properties/aliases/- +# value: +# domain: ((blobstore_host)) # for example: s3-blobstore.service.bosh +# targets: +# - query: 'q-s0' +# deployment: s3-blobstore +# instance_group: minio +# network: default +# domain: bosh +# \ No newline at end of file