diff --git a/README.md b/README.md index 0df06ab..e267d13 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,11 @@ This project adheres to [Semantic Versioning][SemVer] and tries to not break fun The purpose of this project is to allow developers to generate state-of-the-art Kubernetes manifests without the need to have a degree in Kubernetes-Manifestology. -Just by describing the components of their application they should be able to get a set of Kubernetes manifests that adhere to industry best practices. +Just by describing the components of their application they should be able to get a set of Kubernetes manifests that adhere to best practices. -The spiritual prototype for k8ify is [Kompose][] and k8ify tries to do things similarly to Kompose if possible. Unfortunately Kompose does not provide the flexibility we need, hence the custom implementation. +The spiritual prototype for k8ify is [Kompose][] and k8ify tries to do things similarly to Kompose. Unfortunately Kompose does not provide the flexibility we need, hence the custom implementation. -We choose the [Compose][] format as many developers already use Docker Compose for local development environments and are familiar with the format. +We chose the [Compose][] format as many developers already use Docker Compose for local development and are familiar with it. ### Non-Goals @@ -34,7 +34,7 @@ This results in flexibility to support various modes of deployment, be it plain ## Mode of Operation -`k8ify` takes compose files in the current working directory and converts them to Kubernetes manfests. The manifests are written to the `manifests` directory. +`k8ify` takes Compose files in the current working directory and converts them to Kubernetes manfests. The manifests are written to the `manifests` directory. ### Command Line Arguments @@ -48,21 +48,21 @@ This results in flexibility to support various modes of deployment, be it plain #### `environment` -`k8ify` supports different environments (e.g. "prod", "test" and "dev") by merging compose files. A possible setup could look like this: +`k8ify` supports different environments (e.g. "prod", "test" and "dev") by merging Compose files. A setup could look like this: - `compose.yml` - The global default, the base used by all environments. - `compose-prod.yml` - Additional information about the `prod` environment. Used by `k8ify` when asked to generate manifests for the `prod` environment. - `compose-test.yml` - Additional information about the `test` environment. Used by `k8ify` when asked to generate manifests for the `test` environment. -- `compose-dev.yml` - Additional information about the developers local `dev` environment. Never used by `k8ify` but used by developers for local development. +- `compose-dev.yml` - Additional information about the developer's local `dev` environment. Never used by `k8ify` but used by developer for running everything locally. -`k8ify` will choose the correct compose files and merge them based on the selected environment. +`k8ify` will choose the correct Compose files and merge them based on the selected environment. #### `ref` - Multiple deployments in the same environment `k8ify` supports multiple deployments in the same environment, e.g. to deploy different branches of an application into the same `test` environment. It does so by adding a `-$ref` suffix to the name of all generated resources. -Each compose service is, by default, deployed for each `ref`. If you want to deploy a service only once per environment (e.g. a single shared database for all deployments) you can do so by adding the `k8ify.singleton: true` label to the service. +Each Compose service is, by default, deployed for each `ref`. If you want to deploy a service only once per environment (e.g. a single shared database for all deployments) you can do so by adding the `k8ify.singleton: true` label to the service. A resulting deployment might look like this: @@ -82,14 +82,14 @@ This parameter is generally set by the CI/CD pipeline, because the pipeline know #### `--shell-env-file [FILENAME]` - Load additional shell environment variables from file -k8ify relies on the shell environment to fill placeholders in the compose files. This argument can be used to load additional variables. The files have the usual "KEY=VALUE" format and they support quoted values. +k8ify relies on the shell environment to fill placeholders in the Compose files. This argument can be used to load additional variables. The files have the usual "KEY=VALUE" format and they support quoted values. A use case could be to protect your secrets. Instead of loading them into the shell environment you could put them into a file and use this argument to load said file. ### Labels -`k8ify` supports configuring services and volumes by using compose labels. All labels are optional. +`k8ify` supports configuring services and volumes by using Compose labels. All labels are optional. #### General @@ -103,7 +103,7 @@ Service Labels | `k8ify.converter: $script` | Call `$script` to convert this service into a K8s object, expecting YAML on `$script`'s stdout. Used for plugging additional functionality into k8ify. The first argument sent to `$script` is the name of the resource, after that all the parameters follow (next row) | | `k8ify.converter.$key: $value` | Call `$script` with parameter `--$key $value` | | `k8ify.serviceAccountName: $name` | Set this service's pod(s) spec.serviceAccountName to `$name`, which tells the pod(s) to use ServiceAccount `$name` for accessing the K8s API. This does not set up the ServiceAcccount itself. | -| `k8ify.partOf: $name` | This compose service will be combined with another compose service (resulting in a deployment or statefulSet with multiple containers). Useful e.g. for sidecars or closely coupled services like nginx & php-fpm. | +| `k8ify.partOf: $name` | This Compose service will be combined with another Compose service (resulting in a deployment or statefulSet with multiple containers). Useful e.g. for sidecars or closely coupled services like nginx & php-fpm. | | `k8ify.annotations.$key: $value` | Add annotation(s) to all resources generated by k8ify | | `k8ify.$kind.annotations.$key: $value` | Add annotation(s) to specific resource types generated by k8ify. $kind uses the default case used by k8s and is always singular (e.g. "StatefulSet") | | `k8ify.exposePlain.$port: true` | Set up a k8s Service which exposes this port directly instead of using the cluster-wide reverse proxy/load balancer, useful for non-HTTP applications (for HTTP always use `k8ify.expose`). Allocated public IP is visible in the k8s Service's `.status` field. | @@ -122,7 +122,7 @@ Volume Labels #### Health Checks -For each compose service k8ify will set up a basic TCP based health check (liveness and startup) by default. +For each Compose service k8ify will set up a basic TCP based health check (liveness and startup) by default. For all services providing HTTP endpoints you should provide at least a basic health check path and point `k8ify.liveness` to it. This replaces the TCP based health check by a more specific HTTP(S) check. @@ -148,33 +148,33 @@ This replaces the TCP based health check by a more specific HTTP(S) check. #### Target Cluster Configuration -There are some cases in which the output of k8ify needs to be different based on the target cluster's configuration. To make this work some properties of the target cluster can be configured via the `x-targetCfg` root key in the compose file. +There are some cases in which the output of k8ify needs to be different based on the target cluster's configuration. To make this work some properties of the target cluster can be configured via the `x-targetCfg` root key in the Compose file. | Key | Effect | | ---- | ------- | | `appsDomain: $domain` | A cluster may have a wildcard certificate for apps to use. If you configure this option and expose a service using `$domain`, the resulting Ingress uses this wildcard certificate (instead of e.g. Let's Encrypt). | | `maxExposeLength: $length` | k8ify does a length check on the exposed domain names, because if they're too long the Ingress will not work. Default is 63. | -| `encryptedVolumeScheme: $provider` | The implementation of encrypted volumes is provider specific. Use this to enable support for a provider. See [Provider](.docs/provider.md) for more information. | +| `encryptedVolumeScheme: $provider` | The implementation of encrypted volumes is provider specific. Use this to enable support for a provider. See [Provider](./docs/provider.md) for more information. | ## Conversion -The conversion process is documented in-depth in [Conversion](./docs/conversion.md). +The conversion process is documented in depth in [Conversion](./docs/conversion.md). ## Storage -Storage support is documented in-depth in [Storage](./docs/storage.md). +Storage support is documented in depth in [Storage](./docs/storage.md). ## Testing -In order to validate that `k8ify` does what we expect it to do, we use the concept of "golden tests": a predefined set of inputs (compose files) and outputs (Kubernetes manifests) are added to the repository. During the testing process, we run `k8ify` against each of the inputs, and verify that the outputs match the expected outputs. +In order to validate that `k8ify` does what we expect it to do, we use the concept of "golden tests": a predefined set of inputs (Compose files) and outputs (Kubernetes manifests) are added to the repository. During the testing process we run `k8ify` against each of the inputs, and verify that the outputs match the expected outputs. To set up a golden test named `$NAME`, you need to create two things in the `tests/golden/` directory: 1. A file called `$NAME.yml`, and -2. A directory called `$NAME` containing compose files. +2. A directory called `$NAME` containing Compose files. The structure of the YAML file should look like this: @@ -193,9 +193,9 @@ environments: Note that both the `refs` and `vars` fields are optional, but allow you to control the tests: - `refs` will make the test run `k8ify` for each of the provided values. If no `refs` is defined, `k8ify` will be run once for this environment with an empty `ref` value. -- `vars` can contain environment variables that are ADDED to the ones that are already set within the testing environment. If your compoose file makes use of any environment variables, make sure to add them here for reproducability. +- `vars` can contain environment variables that are ADDED to the ones that are already set within the testing environment. If your compoose file makes use of any environment variables, make sure to add them here for reproducibility. -To actually run the tests, run `go test` in the root of the repository. +To actually run the tests run `go test` in the root of the repository. ## License diff --git a/docs/conversion.md b/docs/conversion.md index 3ba5784..bf98334 100644 --- a/docs/conversion.md +++ b/docs/conversion.md @@ -1,65 +1,65 @@ # Conversion -This document describes the conversion process `k8ify` applies to convert Compose files to Kubernetes manifests. +This document describes the conversion process applied by `k8ify` to convert Compose files to K8s manifests. -Compose files define "compose services". Every compose service is translated into Kubernetes resources individually. +Compose files define "Compose services". Every Compose service is translated into K8s resources individually (some exceptions apply). -In general, every compose service is implemented as a Deployment (or StatefulSet, see below), exposed ports are exposed as Services, volumes are mapped to Persistent Volume Claims (which make Kubernetes provide Persistent Volumes) and environment variables are saved into secrets. Services may further be exposed via Ingresses. +In general, every Compose service is implemented as a Deployment (or StatefulSet, see below), exposed ports are exposed as Services, volumes are mapped to PersistentVolumeClaims (which make K8s provide PersistentVolumes) and environment variables are saved into Secrets. Services may further be exposed via Ingresses. -This results in the following list of Kubernetes resource for each compose service: +This results in the following list of K8s resource for each Compose service: -* 1 Workload ([`Deployment`](#k8s-deployment) or [`StatefulSet`](#k8s-statefulset)) -* 0-1 [`Service`](#k8s-service) (a single service can cover multiple ports; if no ports are exposed no service is created) -* 1 [`Secret`](#k8s-secret) (may be empty) -* 0-n [`PersistentVolumeClaim`](#k8s-persistentvolumeclaim) (optionally one per volume) -* 0-n [`Ingress`](#k8s-ingress) (one per port, IF enabled via `k8ify.expose` label on the compose service) +* 0-1 Workloads ([`Deployment`](#k8s-deployment) or [`StatefulSet`](#k8s-statefulset)) (if `k8ify.partOf` is used then the Compose service is merged into another workload) +* 0-1 [`Services`](#k8s-service) (a single Service can cover multiple ports; if no ports are exposed no Service is created) +* 0-1 [`Secrets`](#k8s-secret) +* 0-n [`PersistentVolumeClaims`](#k8s-persistentvolumeclaim) (optionally one per volume) +* 0-n [`Ingresses`](#k8s-ingress) (one per port, IF enabled via `k8ify.expose` label on the Compose service) -### Special considerations +### Special Considerations -Some compose concepts don't match neatly onto Kubernetes concepts, so some special care must be taken. +Some Compose concepts don't translate neatly to K8s concepts, therefore some special care must be taken. #### Ingress -In order to make a service available to the outside world, we need to support Ingresses. However, compose files have no notion of "available to the outside world", hence there is no direct way of generating an Ingress from the data in a compose file. Hence, setting up Ingresses is implemented via compose service labels (see [Labels](../README.md#labels)). +In order to make a Service available to the outside world, we need to support Ingresses. However, Compose files have no notion of "available to the outside world", hence there is no direct way of generating an Ingress from the data in a Compose file. Hence setting up Ingresses is implemented via Compose service labels (see [Labels](../README.md#labels)). -#### Environment variables and Secrets +#### Environment Variables and Secrets -Compose supports secrets and environment variables. However, the "secret" support is limited to file-based secrets, which are inherently incompatible with the Twelve-Factor Application principles, thus we don't want to use these. +Compose supports secrets and environment variables. However the "secret" support is limited to file-based secrets, which are inherently incompatible with the Twelve-Factor Application principles, thus we don't want to use these. -Instead, we only use the "environment" functionality of compose. However, environment variables can contain sensitive information, and we don't know which ones do and which ones don't. Thus, the conversion does not store any environment variables in the deployment, but puts a secretRef in there and then writes all environment variables to one secret per compose service. +Instead we only use the "environment" functionality of Compose. But environment variables can contain sensitive information, and we don't know which ones do and which ones don't. Thus the conversion does not store any environment variables in the Deployment or StatefulSet, but puts a secretRef in there and then writes all environment variables to one Secret per Compose service. #### Volumes -Only volumes defined in the `volumes` topl level section of the Compose files are taken into consideration. Local bind mounts or `tmpfs` mounts are ignored. +Only volumes defined in the `volumes` top level section of the Compose files are taken into consideration. Local bind mounts or `tmpfs` mounts are ignored. By default Volumes will be assigned the `ReadWriteOnce` access mode to prevent multiple instances of an application writing to the same storage location. -This chan be changed to `ReadWriteMany` by adding the label `k8ify.shared: true` to the volume. +This can be changed to `ReadWriteMany` by adding the label `k8ify.shared: true` to the volume. +See [Storage](./storage.md) for a more detailed explanation. #### Deployments vs. StatefulSets -By default, a compose service will be translated into a `Deployment`. +By default a Compose service will be translated to a `Deployment`. If a Compose service only uses shared (`ReadWriteMany`) volumes, it will still be translated to a `Deployment`. -If the compose service has non-shared (`ReadWriteOnce`) volumes mounted, a `StatefulSet` is used instead. This results in every replica getting its own `ReadWriteOnce` PersistentVolume. - -If a compose service only uses shared (`ReadWriteMany`) volumes, it will still be translated into a `Deployment`. +If the Compose service has non-shared (`ReadWriteOnce`) volumes mounted, a `StatefulSet` is used instead. This results in every replica getting its own `ReadWriteOnce` PersistentVolume. +See [Storage](./storage.md) for a more detailed explanation. #### Labels -Note that K8ify works with both Labels on Compose services (set in Compose files under `services.$name.labels`) and Kubernetes resource labels (set on individual resources under `metadata.labels`). -Labels are not automatically copied from Compose files to Kubernetes manifests. -Instead, the two concepts are used for different purposes: +Note that k8ify works with both Labels on Compose services (set in Compose files under `services.$name.labels`) and K8s resource labels (set on individual resources under `metadata.labels`). +A user of k8ify will normally only work with the Compose service labels and not with the K8s resource labels. +Labels are not automatically copied from Compose files to K8s manifests. Instead the two concepts are used for different purposes: -Labels on **Compose services** and **volumes** are used to **configure** and customize manifest generation. +Labels on **Compose services** and **Compose volumes** are used to **configure** and customize manifest generation. They are used to extend the functionality of the Compose file format. -Labels on the generated **Kubernetes resources** are used to **identify** resources managed via K8ify. +Labels on the generated **K8s resources** are used to **identify** K8s resources. They are used to set up relationships between the generated K8s resources. -The following set of labels is applied to all generated Kubernetes resources: +The following set of labels is applied to all generated K8s resources: ```yaml labels: @@ -74,23 +74,18 @@ labels: The following variables will be used in the table below: -* `$name` - Name of the Compose service (eg the "key" in the `services` map in the Compose file), e.g. **myapp** +* `$name` - Name of the Compose service (e.g. the "key" in the `services` map in the Compose file), e.g. **myapp** * `$ref` - Name of the `ref` passed to `k8ify`, see [Parameters](../README.md#parameters), e.g. **feat/foo** -* `$refSlug` - Normalized version of `ref` that is a valid DNS label (and hence can be used as a Kubernetes label value), eg **feat-foo** +* `$refSlug` - Normalized version of `ref` that is a valid DNS label (and hence can be used as a K8s label value), e.g. **feat-foo** -See [Example input](#example-input) below on how a Compose file would have to look to generat the following example manifests. +See [example input](#example-input) below on how a Compose file would have to look to generate the following example manifests. Note that some fields were omited here for brevity. -### Common - -Labels are documented [above](#labels) and not repeated in the examples below. - - ### K8s Deployment -See [Deployments vs. Statefulsets](#deployments-vs-statefulsets) for a documentation when a Deployment is used, and when a [StatefulSet](#k8s-statefulset). +See [Deployments vs. Statefulsets](#deployments-vs-statefulsets) for a documentation when a Deployment is used vs. a [StatefulSet](#k8s-statefulset). ```yaml apiVersion: apps/v1 @@ -125,8 +120,8 @@ spec: # If singleton or no ref given: `$name`, otherwise: `$name-$refSlug` - name: "myapp-feat-foo" # or "myapp" # `services.$name.image` - # Note: We support compose files per environment, so the image can be - # configured there. These compose files also support substitution of + # Note: We support Compose files per environment, so the image can be + # configured there. These Compose files also support substitution of # env vars that are set by the CI system, e.g. to fill in the correct # tag name. image: "docker.io/mycorp/myapp:v0.5.7" @@ -134,8 +129,8 @@ spec: - secretRef: # `$name(-$refSlug)-env` name: "myapp-feat-foo-env" - # To reference a value in a secret you need to use a special syntax in `services.$name.environment`: - # If an environment value starts with a literal '$_ref_:', it is interpreted as a secret reference. + # To reference a value in a Secret you need to use a special syntax in `services.$name.environment`: + # If an environment value starts with a literal '$_ref_:', it is interpreted as a Secret reference. # Example which would generate the secretRef shown below: # `DATABASE_PASSWORD=$_ref_:database-credentials-secret:password` env: @@ -235,8 +230,8 @@ spec: # If singleton or no ref given: `$name`, otherwise: `$name-$refSlug` - name: "myapp-feat-foo" # or "myapp" # `services.$name.image` - # Note: We support compose files per environment, so the image can be - # configured there. These compose files also support substitution of + # Note: We support Compose files per environment, so the image can be + # configured there. These Compose files also support substitution of # env vars that are set by the CI system, e.g. to fill in the correct # tag name. image: "docker.io/mycorp/myapp:v0.5.7" @@ -244,8 +239,8 @@ spec: - secretRef: # `$name(-$refSlug)-env` name: "myapp-feat-foo-env" - # To reference a value in a secret you need to use a special syntax in `services.$name.environment`: - # If an environment value starts with a literal '$_ref_:', it is interpreted as a secret reference. + # To reference a value in a Secret you need to use a special syntax in `services.$name.environment`: + # If an environment value starts with a literal '$_ref_:', it is interpreted as a Secret reference. # Example which would generate the secretRef shown below: # `DATABASE_PASSWORD=$_ref_:database-credentials-secret:password` env: @@ -398,7 +393,7 @@ kind: Ingress metadata: # `$name(-$ref)-$portString` # where `$portString` is the same as `spec.ports.$i.name` in the referenced - # K8s service + # K8s Service name: myapp-feat-foo-8001 # Whatever is configured in the config file (`.k8ify.default.yaml`) under # `ingressPatch.addAnnotations` @@ -439,15 +434,7 @@ The manifests above were generated using the following command: k8ify test feat/foo ``` -K8ify configuration - -```yaml -ingressPatch: - addAnnotations: - cert-manager.io/cluster-issuer: letsencrypt-production -``` - -Compose file +Compose file: ```yaml services: @@ -455,6 +442,7 @@ services: labels: k8ify.expose.8001: myapp.example.com k8ify.liveness: /health + k8ify.Ingress.annotations.cert-manager.io/cluster-issuer: letsencrypt-production image: docker.io/mycorp/myapp:v0.5.7 deploy: replicas: 2 diff --git a/docs/storage.md b/docs/storage.md index 9c41a8b..433b009 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -1,47 +1,51 @@ # Storage -Principles: +## Basics -* Storage is never shared by default -* Ignore any bind-mounts +K8s volumes can be "RWO" ("Read Write Once") or "RWX" ("Read Write Many"), although the latter isn't supported on all K8s instances. "RWO" volumes can only be used by one replica of an application (Pod) at a time. +K8s supports "Deployments" which can consist of multiple replicas of the same application (Pods), all sharing the same volume, which means that "Deployments" can only be used with "RWX" volumes. The alternative is a "StatefulSet", which works differently: In a StatefulSet each replica of the application (Pod) gets its own dedicated volume, i.e. if you have two replicas you also have two volumes. Therefore StatefulSets can be used with "RWO" volumes because volume sharing doesn't occur. -## Concepts +Volumes can also be shared between entirely different applications. Also, Compose volumes and Compose services can be labeled as singletons. k8ify needs to handle all those cases correctly. -Relevant for both volumes and services is the `k8ify.singleton` label. -Relevant for the volume is the `k8ify.shared` label. +## PersistentVolumeClaims vs PersistentVolumeClaim templates +A PersistentVolumeClaim ("PVC") is a request to K8s to provide a Volume, and the existence (or absence) of a PersistentVolumeClaim triggers code to create or delete Volumes. Deployments and StatefulSets work very differently when it comes to PersistentVolumeClaims, resp. who creates them. -## Volumes +With a Deployment the volumes needed don't change if you scale the deployment. A Deployment with 10 replicas (Pods) has exactly the same volume(s) as a Deployment with 1 replica because the volumes are shared between all replicas. Therefore the PersistentVolumeClaims are static and have to be created by the K8s user (or in this case k8ify), just like any other resource. -By default, create no PVC and set the AccessMode to `ReadWriteOnce`. +With a StatefulSet the volumes change when it scales; a StatefulSet with 10 replicas needs all the volumes for each replica separately. Therefore the PersistentVolumeClaims used by a StatefulSet can't be static, they must be created dynamically as the StatefulSet scales. Therefore the PersistentVolumeClaims for StatefulSets are defined as a template inside the StatefulSet, just like a container template. (Note that a StatefulSet can use both RWO volumes whose PVCs are created via template and RWX vvolumes whose PVCs were created directly at the same time) -### `k8ify.shared` +k8ify handles all of these cases. -If `true`, create a PVC and set its AccessMode to `ReadWriteMany`. +## Principles of Operation -### `k8ify.singleton` +* Ignore any bind mounts (used for development and not relevant in K8s) +* Volumes are RWO by default (not shared) +* If a Compose service uses one or more non-shared volume(s) (RWO), the service will be translated to a StatefulSet +* If a Compose service uses no volumes or all of them are marked as shared (RWX), the service will be translated to a Deployment +* Impossible combinations (e.g. RWO volume used by multiple Compose services) are detected and reported to the user -If `true`, and a PVC is created, omit the `refSlug` from its name. +## Error Cases -### `k8ify.storageClass` +- Compose service uses a volume that doesn't exist +- Multiple Compose services use same volume but the volume is `ReadWriteOnce` +- `k8ify.singleton` label differs between a Compose service and its volumes -This sets the `spec.storageClassName` field of the PVC. This is useful in some cases to choose between ssd and hdd storage or to enable encryption. +## Volume Labels -## Services +### `k8ify.shared` -By default, create a Deployment. +If `true` create a PVC and set its AccessMode to `ReadWriteMany` (RWX). -If any `ReadWriteOnce` volumes are attached, create a StatefulSet instead and include all `ReadWriteOnce` volumes in the volume templates. +### `k8ify.singleton` +If `true` and a PVC is created then omit the `refSlug` from its name. -## Error cases +### `k8ify.storageClass` -- Service uses a volume that doesn't exist -- Service is singleton, but volume isn't. -- Multiple services use same volume, but it's `ReadWriteOnce` -- `k8ify.singleton` label differs between service and its volumes. +This sets the `spec.storageClassName` field of the PVC. This is useful in some cases to choose between ssd and hdd storage or to enable encryption.