diff --git a/content/en/docs/demos/certificate_rotation.md b/content/en/docs/demos/certificate_rotation.md new file mode 100644 index 00000000..9b6b7b72 --- /dev/null +++ b/content/en/docs/demos/certificate_rotation.md @@ -0,0 +1,181 @@ +--- +title: "Rotating the MeshRootCertificate" +description: "Using the OSM CLI to rotate the MeshRootCertificate" +type: docs +weight: 22 +--- + +This guide demonstrates how to use the [MeshRootCertificate](../guides/certificate_management/certificate-rotation.md) preview feature with the OSM CLI. + +> WARNING: This feature is currently in preview and requires the `EnableMeshRootCertificate` feature flag enabled in the MeshConfig. + +## Prerequisites + +- Kubernetes cluster running Kubernetes {{< param min_k8s_version >}} or greater. +- Have `kubectl` available to interact with the API server. +- Have `osm` CLI available for installing and managing the service mesh. +- Have `openssl` available to view certificate information. + +## Demo + +The following demo shows how to use the OSM CLI to initiate a root certificate rotation using the Tresor certificate provider. Learn more about this process in the [Certificate Rotation documentation](../guides/certificate_management/certificate-rotation.md). + +1. Install OSM with the `EnableMeshRootCertificate` [feature flag](../api_reference/config/v1alpha2.md#featureflags) set to true. + ```console + osm install --set="osm.featureFlags.enableMeshRootCertificate=true" + ``` + +1. Check that the default `MeshRootCertificate` was created and the role is `active`: + ```console + kubectl get mrc -n osm-system # Replace osm-system with the namespace where OSM is installed + NAME ROLE + osm-mesh-root-certificate active + ``` + + Confirm that a Secret was created with the root certificate: + ```console + kubectl get secrets -n osm-system + NAME TYPE DATA AGE + osm-ca-bundle Opaque 2 18h + ``` + + Take note of the modulus of the original certificate: + ```console + # Replace osm-system with the namespace where OSM is installed + kubectl get secret -n osm-system osm-ca-bundle -o jsonpath='{.data.ca\.crt}' | + base64 -d | + openssl x509 -modulus -noout + Modulus=BC79C8202E8EE11DC27335E7313BB9466DEC870367F57E9BA56F1E83296B9872FCDB924DC2A9B8C012EA72F7F76356E895D56A9D2DFAF8A62E768156A81F3787A526CAA0858FD1BC854E4BFE2D27C10F7DCC4586B7AB71572406A5A9648AB317C77DF56ED74F970A007D95FD11C1A51170A2579B3AE7CFE257714611A0C1FE11D2F8F3A6499CA94536B18E751D013B283A4331C1E6830B834CEC14A2A58B11537E2AFDB2A01BD5F606237F2DB25387F2954161AF0EC99D377D0CD3813B61BE39DF427C5B412D38207A0744E4B40869294B05171E1CE45253B8B4BEA15C009D8AF3DE7832E12D668636CF2D13D8F363A79F53CEECA4B3718FE557E136B6162E27 + ``` + +3. Enable permissive traffic policy mode to set up automatic application connectivity. + > Note: This is not a requirement to use `MeshRootCertificate`, but simplifies the demo by not requiring explicit traffic policies for application connectivity. + ```bash + kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":true}}}' --type=merge + ``` + +4. Deploy the `httpbin` service into the `httpbin` namespace after enrolling its namespace to the mesh. The `httpbin` service runs on port `14001`. + + ```bash + # Create the httpbin namespace + kubectl create namespace httpbin + + # Add the namespace to the mesh + osm namespace add httpbin + + # Deploy httpbin service in the httpbin namespace + kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/{{< param osm_branch >}}/manifests/samples/httpbin/httpbin.yaml -n httpbin + ``` + + Confirm the `httpbin` service and pods are up and running. + + ```console + kubectl get svc -n httpbin + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + httpbin ClusterIP 10.96.198.23 14001/TCP 20s + ``` + + ```console + kubectl get pods -n httpbin + NAME READY STATUS RESTARTS AGE + httpbin-5b8b94b9-lt2vs 2/2 Running 0 20s + ``` + +5. Deploy the `curl` client into the `curl` namespace after enrolling its namespace to the mesh. + + ```bash + # Create the curl namespace + kubectl create namespace curl + + # Add the namespace to the mesh + osm namespace add curl + + # Deploy curl client in the curl namespace + kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/{{< param osm_branch >}}/manifests/samples/curl/curl.yaml -n curl + ``` + + Confirm the `curl` client pod is up and running. + + ```console + kubectl get pods -n curl + NAME READY STATUS RESTARTS AGE + curl-54ccc6954c-9rlvp 2/2 Running 0 20s + ``` + +6. Confirm the `curl` client is able to access the `httpbin` service on port `14001`. + + ```console + kubectl exec -n curl -ti "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- curl -I http://httpbin.httpbin:14001 + HTTP/1.1 200 OK + server: envoy + date: Mon, 15 Mar 2021 22:45:23 GMT + content-type: text/html; charset=utf-8 + content-length: 9593 + access-control-allow-origin: * + access-control-allow-credentials: true + x-envoy-upstream-service-time: 2 + ``` + + A `200 OK` response indicates the HTTP request from the `curl` client to the `httpbin` service was successful. The traffic between the application sidecar proxies is encrypted and authenticated using mutual TLS (mTLS) by leverging the initial root certificate. + +7. Rotate the root certificate `MeshRootCertificate` using the OSM command: + ```console + osm alpha certificate rotate -w 20s + + *** This command is in Preview. Only run in dev/test environments *** + Found Active MeshRootCertificate [osm-mesh-root-certificate] for mesh [osm] in namespace [osm-system] + + Created new MRC [osm-mesh-root-certificate-11546fc67c238b0de2d91ca8f8331173] -> Passive role + waiting for propagation... + Moving MRC [osm-mesh-root-certificate-11546fc67c238b0de2d91ca8f8331173] -> Active role + waiting for propagation... + Moving MRC [osm-mesh-root-certificate] -> Passive role + waiting for propagation... + Moving MRC [osm-mesh-root-certificate] -> Inactive role + waiting for propagation... + + OSM successfully rotated root certificate for mesh [osm] in namespace [osm-system] + ``` + +8. Confirm that there is only one active MRC and the certificates are different. + ```console + kubectl get mrc -n osm-system + NAME ROLE + osm-mesh-root-certificate inactive + osm-mesh-root-certificate-11546fc67c238b0de2d91ca8f8331173 active + ``` + + Confirm that a Secret was created with the root certificate: + ```console + kubectl get secrets -n osm-system + NAME TYPE DATA AGE + osm-ca-bundle-11546fc67c238b0de2d91ca8f8331173 Opaque 2 18h + ``` + + Take note of the modulus of the new certificate (it will be different from the original): + ```console + kubectl get secret -n osm-system osm-mesh-root-certificate-11546fc67c238b0de2d91ca8f8331173 -o jsonpath='{.data.ca\.crt}' | + base64 -d | openssl x509 -modulus -noout + Modulus=D538E9513830028C19173CA653C7609AEC1D75E110F74C4C73BA6CD6C9C8F047C99DD3ECC3A66EA874510377B631C10C2E16521D7FF18725A525D87AC2072AFDA8DEDA8802558D2A93FC5DE9678F97BCFFBA854962E105C41DF64E2748AE31952833B3F8A37C06E1D1DBBA1967ADA0EFEE26634DE19A9174E106461DDE0F84B3311D204F6F0AC5C9A6A86572320A6B37C2CFF98DD8351FFBE13E47660E30F3EB703457E424DCEAF0CD9A92EE0CE3E47F578DB7399D001409712C9605D8897CD823690D51EE6FB1F2237291D19D3A546770350A1D6403D0466D3C5F2953DCC46056F320D53A5F34B4249F5E4BCD8F1B202D1374F5B9F01D39947DF5908E59FE49 + ``` + + View the certificate modulus being used by the `httpbin` service (it will be the same as the new certificate): + ```console + osm proxy get config_dump httpbin-b8c7bc4d9-zpm8m -n httpbin | jq -r '.configs[] | select(."@type"=="type.googleapis.com/envoy.admin.v3.SecretsConfigDump") | .dynamic_active_secrets[] | select(.name == "root-cert-for-mtls-outbound:httpbin/httpbin").secret.validation_context.trusted_ca.inline_bytes' | base64 -d | openssl x509 -modulus -noout + Modulus=D538E9513830028C19173CA653C7609AEC1D75E110F74C4C73BA6CD6C9C8F047C99DD3ECC3A66EA874510377B631C10C2E16521D7FF18725A525D87AC2072AFDA8DEDA8802558D2A93FC5DE9678F97BCFFBA854962E105C41DF64E2748AE31952833B3F8A37C06E1D1DBBA1967ADA0EFEE26634DE19A9174E106461DDE0F84B3311D204F6F0AC5C9A6A86572320A6B37C2CFF98DD8351FFBE13E47660E30F3EB703457E424DCEAF0CD9A92EE0CE3E47F578DB7399D001409712C9605D8897CD823690D51EE6FB1F2237291D19D3A546770350A1D6403D0466D3C5F2953DCC46056F320D53A5F34B4249F5E4BCD8F1B202D1374F5B9F01D39947DF5908E59FE49 + ``` + +9. Confirm the `curl` client is able to access the `httpbin` service on port `14001`. + ```console + kubectl exec -n curl -ti "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- curl -I http://httpbin.httpbin:14001 + HTTP/1.1 200 OK + server: envoy + date: Mon, 15 Mar 2021 22:45:23 GMT + content-type: text/html; charset=utf-8 + content-length: 9593 + access-control-allow-origin: * + access-control-allow-credentials: true + x-envoy-upstream-service-time: 2 + ``` + + A `200 OK` response indicates the HTTP request from the `curl` client to the `httpbin` service was successful. The traffic between the application sidecar proxies is encrypted and authenticated using mTLS by leveraging the new root certificate. \ No newline at end of file diff --git a/content/en/docs/guides/certificate_management/_index.md b/content/en/docs/guides/certificate_management/_index.md new file mode 100644 index 00000000..dee1514a --- /dev/null +++ b/content/en/docs/guides/certificate_management/_index.md @@ -0,0 +1,6 @@ +--- +title: "Certificate Management" +description: "How OSM uses certificates in the mesh" +type: docs +weight: 11 +--- diff --git a/content/en/docs/guides/certificate_management/certificate-rotation.md b/content/en/docs/guides/certificate_management/certificate-rotation.md new file mode 100644 index 00000000..3dea8389 --- /dev/null +++ b/content/en/docs/guides/certificate_management/certificate-rotation.md @@ -0,0 +1,209 @@ +--- +title: "Root Certificate Rotation" +description: "Rotating the root certificate in OSM" +type: docs +weight: 10 +--- + +# Root Certificate Rotation + +The mesh's root certificate is a long-lived certificate that is used by OSM to issue leaf certificates to services in the mesh thereby enabling mTLS. This document focuses on the steps to rotate a certificate, to learn more about OSM's certificates see [Certificate Provider Options](certificates.md). + +The certificate can be rotated either through a [manual process](#manual-root-certificate-rotation) or with OSM v1.3 we have introduced a preview feature to help [automate rotation](#certificate-rotation-with-meshrootcertificate) and minimize downtime using a new API called the `MeshRootCertificate`. + +### Manual Root Certificate Rotation + +#### Tresor + +> WARNING: Rotating root certificates will incur downtime between any services as they transition their mTLS certificate from one issuer to the next. + +The self-signed root certificate, which is created via the Tresor package within OSM, will expire in a decade. To rotate the root certificate, the following steps should be followed: + +1. Delete the `osm-ca-bundle` certificate in the OSM namespace + ```console + export osm_namespace=osm-system # Replace osm-system with the namespace where OSM is installed + kubectl delete secret osm-ca-bundle -n $osm_namespace + ``` + +2. Restart the control plane components + ```console + kubectl rollout restart deploy osm-controller -n $osm_namespace + kubectl rollout restart deploy osm-injector -n $osm_namespace + kubectl rollout restart deploy osm-bootstrap -n $osm_namespace + ``` + +When the components gets re-deployed, you should be able to eventually see the new `osm-ca-bundle` secret in `$osm_namespace`: + +```console +kubectl get secrets -n $osm_namespace +``` + +``` +NAME TYPE DATA AGE +osm-ca-bundle Opaque 3 74m +``` + +The new expiration date can be found with the following command: + +```console +kubectl get secret -n $osm_namespace $osm_ca_bundle -o jsonpath='{.data.ca\.crt}' | + base64 -d | + openssl x509 -noout -dates +``` + +For Envoy services and validation certificates to be rotated the data plane components must be restarted. + +#### Other certificate providers +The process of rotating the root certificate will be different for certificate providers other than Tresor. For Hashicorp Vault and cert-manager.io, users will need to rotate the root certificate themselves outside of OSM. + +### Certificate Rotation with MeshRootCertificate + +> WARNING: This feature is currently in preview and requires the `EnableMeshRootCertificate` feature flag enabled in the MeshConfig. + +The `MeshRootCertificate` is introduced in OSM v1.3. This custom resource can be enabled by using the `EnableMeshRootCertificate` feature flag when deploying OSM. + +```console +osm install --set="osm.featureFlags.enableMeshRootCertificate=true" +``` + +> Important: If OSM is installed with `EnableMeshRootCertificate` disabled, but the feature flag is then later enabled in the MeshConfig, the control plane components must be restarted. This allows the flag to be picked up and begin using the `MeshRootCertificate` for certificate management. There will be no change in the root certificate, but it will now be managed by the `MeshRootCertificate`. + +### MeshRootCertificate + +Each `MeshRootCertificate` represents a certificate that can be used in the mesh. There can be only two `MeshRootCertificate`'s that are used in the mesh at a given time and there must always be one `MeshRootCertificate` in the **active** `role`. + +The `MeshRootCertificate` role has the following options: + +- `active`: The certificate is used to sign **and** validate certificates in the mesh. There must always be one active certificate. +- `passive`: The certificate is used to validate certificates in the mesh. +- `inactive`: The certificate is not used in the mesh. + +Run `kubectl get meshrootcertificate` to see the default `MeshRootCertificate`: + +```yaml +apiVersion: config.openservicemesh.io/v1alpha2 +kind: MeshRootCertificate +metadata: + name: osm-mesh-root-certificate + namespace: osm-system # Replace osm-system with the namespace where OSM is installed +spec: + role: active + provider: + tresor: + ca: + secretRef: + name: osm-ca-bundle + namespace: osm-system # Replace osm-system with the namespace where OSM is installed + spiffeEnabled: false + trustDomain: cluster.local +``` + +The `provider` field can be `tresor`, `certManager`, or `vault`. Before using the providers for [CertManager](certificates.md#using-cert-manager) or [Vault](certificates.md#using-hashicorp-vault), additional pre-configuration steps are required. Refer to the [API documentation](../../api_reference/) on how to configure the `MeshRootCertificate` fields for each provider. + +The only field that can be modified after initial creation is the `role` field. Other fields will need to be changed by creating a new MeshRootCertificate and going through the [rotation process](#meshrootcertificate-rotation). On a new install, the initial `MeshRootCertificate` will be created by OSM and the certificate will be in the **active** `role`. Fields such as `trustDomain` and `spiffeEnabled` can be configured by passing options to the install command: + +```bash +osm install --set="osm.featureFlags.enableSPIFFE=true --set="osm.trustDomain=cluster.example" +``` + +### MeshRootCertificate Rotation + +To rotate the root certificate, a second `MeshRootCertificate` (MRC) will be created. The second `MeshRootCertificate` will be created in a `passive` role and rotated into the `active` role. Once the new certificate is `active`, the old certificate will be moved to the `passive` role then finally `inactive`. + +Below is a table representing the full process for rotating the certificate: + +| Step | original mrc1 role | new mrc2 role | Signing MRC | Validating MRC | +| ----- | ----------------------- | ------------------- | ------------ | -------------- | +| 1 | active | (not created) | mrc1 | mrc1 | +| 2 | active | passive | mrc1 | mrc1 and mrc2 | +| 3 | active | active | mrc1 or mrc2 | mrc1 and mrc2 | +| 4 | passive | active | mrc2 | mrc1 and mrc2 | +| 5 | inactive | active | mrc2 | mrc2 | +| 5 | (deleted) | active | mrc2 | mrc2 | + + +Each step requires time for the certificate information to be propagated to all the components in the mesh. The amount of time required will depend on your mesh. + +To verify if a certificate has been propagated, you can review the OSM controller logs and optionally run the following commands. If both certificates are in use, then you will see two certificates from the following commands, otherwise you will see the `active` certificate. + +We are working on improving this experience as we move the `MeshRootCertificate` feature out of preview. + +Example commands: + +- logs: + ``` + kubectl logs osm-controller-67f9fd585d-ccr4b | grep "in progress root certificate rotation" + ``` + +- osm validation and mutating webhooks certificates: + ``` + kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io osm-validator-mesh-osm -ojson | jq -r '.webhooks[] | .clientConfig.caBundle' | base64 --decode + kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io osm-validator-mesh-osm -ojson | jq -r '.webhooks[] | .clientConfig.caBundle' | base64 --decode + ``` + +- service certificates: + ```osm proxy get config_dump bookbuyer-b8c7bc4d9-zpm8m -n bookbuyer | jq -r '.configs[] | select(."@type"=="type.googleapis.com/envoy.admin.v3.SecretsConfigDump") | .dynamic_active_secrets[] | select(.name == "root-cert-for-mtls-outbound:bookstore/bookstore").secret.validation_context.trusted_ca.inline_bytes' | base64 -d + ``` + +- xds certificates: + ``` + osm proxy get config_dump bookbuyer-b8c7bc4d9-zpm8m -n bookbuyer | jq -r '.configs[] | select(."@type"=="type.googleapis.com/envoy.admin.v3.SecretsConfigDump") | .dynamic_active_secrets[] | select(.name == "validation_context_sds").secret.validation_context.trusted_ca.inline_bytes' | base64 -d + ``` + +There is an experimental CLI command that can automate this process using a time-based method. Check out the [certificate rotation demo](../../demos/certificate_rotation.md) for more information. Please provide feedback through the [OSM GitHub repository](https://github.com/openservicemesh/osm) on the rotation process to help improve the experience. + +#### Rotating with OSM's Tresor certificate + +Tresor requires no additional configuration to rotate the certificates. Create a second `MeshRootCertificate` and move it through the [rotation process](#meshrootcertificate-rotation). OSM will create a secret for the values in `secretRef` and store the certificate there. + +#### Rotating with Hashicorp Vault + +Root certificate rotation for Hashicorp Vault requires Vault version 1.11. Learn how to rotate a root certificate in Vault through the [Hashicorp documentation](https://learn.hashicorp.com/tutorials/vault/pki-engine#step-7-rotate-root-ca). + +Once the root certificate is rotated in Hashicorp you can configure a `MeshRootCertificate`, then move it through the [rotation process](#meshrootcertificate-rotation): + +```yaml +apiVersion: config.openservicemesh.io/v1alpha2 +kind: MeshRootCertificate +metadata: + name: osm-mesh-root-certificate-2 + namespace: osm-system # Replace osm-system with the namespace where OSM is installed +spec: + role: passive + provider: + vault: + host: vault.osm-system.svc.cluster.local + port: 8200 + protocol: http + role: + token: + secretKeyRef: + key: "osm-key" + name: "osm-vault-token" + namespace: osm-system # Replace osm-system with the namespace where OSM is installed + spiffeEnabled: false + trustDomain: cluster.local +``` + +#### Rotating with cert-manager + +Create a new issuer following the [cert-manager documentation](https://release-v1-2.docs.openservicemesh.io/docs/guides/certificates/#configure-cert-manger-for-osm-signing). + +Once you have a new issuer you can configure a `MeshRootCertificate` with the issuer information, then move it through the [rotation process](#meshrootcertificate-rotation): + +```yaml +apiVersion: config.openservicemesh.io/v1alpha2 +kind: MeshRootCertificate +metadata: + name: osm-mesh-root-certificate-2 + namespace: osm-system # Replace osm-system with the namespace where OSM is installed +spec: + role: passive + provider: + certManager: + IssuerName: , + IssuerKind: "Issuer", + IssuerGroup: "cert-manager.io", + spiffeEnabled: false + trustDomain: cluster.local +``` diff --git a/content/en/docs/guides/certificates.md b/content/en/docs/guides/certificate_management/certificates.md similarity index 88% rename from content/en/docs/guides/certificates.md rename to content/en/docs/guides/certificate_management/certificates.md index 0a8935aa..4e609768 100644 --- a/content/en/docs/guides/certificates.md +++ b/content/en/docs/guides/certificate_management/certificates.md @@ -52,53 +52,9 @@ kubectl get secret -n $osm_namespace $osm_ca_bundle -o jsonpath='{.data.ca\.crt} This will provide valuable certificate information, such as the expiration date and the issuer. -### Root Certificate Rotation +#### Root Certificate Rotation -#### Tresor - -> WARNING: Rotating root certificates will incur downtime between any services as they transition their mTLS certs from one issuer to the next. - -We are currently working on a zero-downtime root cert rotation mechanism that we expect to announce in one of our upcoming releases. - -The self-signed root certificate, which is created via the Tresor package within OSM, will expire in a decade. To rotate the root cert, the following steps should be followed: - -1. Delete the `osm-ca-bundle` certificate in the osm namespace - ```console - export osm_namespace=osm-system # Replace osm-system with the namespace where OSM is installed - kubectl delete secret osm-ca-bundle -n $osm_namespace - ``` - -2. Restart the control plane components - ```console - kubectl rollout restart deploy osm-controller -n $osm_namespace - kubectl rollout restart deploy osm-injector -n $osm_namespace - kubectl rollout restart deploy osm-bootstrap -n $osm_namespace - ``` - -When the components gets re-deployed, you should be able to eventually see the new `osm-ca-bundle` secret in `$osm_namespace`: - -```console -kubectl get secrets -n $osm_namespace -``` - -``` -NAME TYPE DATA AGE -osm-ca-bundle Opaque 3 74m -``` - -The new expiration date can be found with the following command: - -```console -kubectl get secret -n $osm_namespace $osm_ca_bundle -o jsonpath='{.data.ca\.crt}' | - base64 -d | - openssl x509 -noout -dates -``` - -For the Envoy service and validation certificates to be rotated the data plane components must restarted. - -#### Hashicorp Vault and Certmanager - -For certificate providers other than Tresor, the process of rotating the root certificate will be different. For Hashicorp Vault and cert-manager.io, users will need to rotate the root certificate themselves outside of OSM. +With OSM v1.3 root certificate rotation is in preview. Refer to [certificate rotation](certificate-rotation.md) for more details. ## Issuing Certificates