-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Update README.md * added mention of the controller in our docs
- Loading branch information
1 parent
84742ad
commit be10d86
Showing
1 changed file
with
22 additions
and
20 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,37 +1,39 @@ | ||
| # filer-sidecar-injector | ||
| # Kerberos Sidecar Injector | ||
|
|
||
| This is forked off of the repo [kube-sidecar-injector](https://github.com/morvencao/kube-sidecar-injector) and was a repo used for [a tutorial at Medium](https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74) to create a Kubernetes [MutatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#mutatingadmissionwebhook-beta-in-19) that injects a nginx sidecar container into pod prior to persistence of the object. | ||
| This repo was initiated as a copy of our [filer-sidecar-injector](https://github.com/StatCan/filer-sidecar-injector) repo as they both share a very similar design. | ||
| This is used to create a Kubernetes [MutatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#mutatingadmissionwebhook-beta-in-19) that injects a sidecar container into a pod prior to the persistence of the object. This sidecar is an alpine image based on the [kerberos sidecar image](https://gitlab.k8s.cloud.statcan.ca/cloudnative/docker/kerberos-sidecar) initially designed by our Cloud Native team. The only difference we had to make was to change from `USER 1001` to `USER 1000`. | ||
|
|
||
| ## Forked for Filer usage | ||
| Specifically for use with Filers. | ||
| The main goal of the kerberos sidecar is to allow for authentication to services(mainly currently SQL IaaS Databases) demanding something compliant with Windows Authentication. | ||
|
|
||
| ## Mutating the Pod | ||
| This webhook intercepts `pod` create requests and will `jsonpatch` the spec. Whether or not to mutate the pod spec happens in the `mutate` function. If the user namespace does not have an `existing-shares` configmap, or the pod does not have the `notebook-name` label present then it will not mutate and `inject` the filer sidecars. | ||
| This webhook intercepts `pod` create requests and will `jsonpatch` the spec. Whether or not to mutate the pod spec happens in the `mutate` function. If the user namespace does not have the `kerberos-sidecar-injection` label, does not have a `kerberos-keytab` secret, or the pod does not have the `notebook-name` label present, then it will **not** mutate and `inject` the kerberos sidecars. | ||
|
|
||
| ## Creating the jsonpatch | ||
| This is all done in the `createPatch` function. The very first thing that is done is determine if the pod _already has a volume_. This is because we do not want to overwrite any existing volumes in the `jsonpatch` as adding a new entry is very different from appending one. | ||
| From there the `existing-shares` configmap is iterated through in order to create a new sidecar spec for each entry. | ||
| From there, we don't really do any manipulations to the template stored in the `kerberos-sidecar-injector-json` configMap. Since every pod that gets injected will all share the exact same specs for the sidecar, there is no need for any configurations to change from the default. | ||
|
|
||
| ### Getting the values to use in the json patch | ||
| For the sidecar spec we have to use the `filer-sidecar-injector-json` configmap object from the `das` namespace, this works as the model / structure that we will modify and inject into the `pod` spec. | ||
|
|
||
| We also need to get the bucket url, name, secret and access key for the particular svm. The bucket url is taken from the `filers-list` cm in the `das` namespace that is read in at injector startup. The bucket name is taken from the `existing-shares` cm and is put through the same hashing function used when creating the bucket in order to mount to the correct one. The secret and access keys are read in from a secret in the user's namespace. If any of these key values are empty then the sidecar will skip as the mounting would just fail. | ||
| For the sidecar spec, we have to use the `kerberos-sidecar-injector-json` configmap object from the `das` namespace. This works as the model / structure that we inject into the `pod` spec. | ||
|
|
||
| ### Inserting the values | ||
| Using the json template that we deep copy (to avoid modifying the same one over and over again and thus only getting the last changes), we substitute the values in and call various `addX` or `updateX` functions whose only purpose is to return a `[]patchOperation`. These are self explanatory but we will note any exceptional cases below. | ||
| Using the json template that we deep copy (to avoid modifying the same one over and over again and thus only getting the last changes), we call various `addX` or `updateX` functions whose only purpose is to return a `[]patchOperation`. These are self explanatory but we will note any exceptional cases below. | ||
|
|
||
| ### updateWorkingVolumeMounts | ||
| This function adds a volume mount to the user container spec. To determine if we are modifying the correct container we use the `NB_PREFIX` environment variable which is present and if you are testing with a podspec ensure this variable is there. | ||
|
|
||
| ### updateUserEnvVars | ||
| The sidecar injector will also insert environment variables that users can leverage with the `minio client` or `mc`. These are inserted at the notebook container level so they may use them from within their notebook. This also relies on the `NB_PREFIX` environment variable. | ||
| This function adds the necessary volume mounts to the user container spec. To determine if we are modifying the correct container, we use the `NB_PREFIX` environment variable that should be present in the container specs. If you are testing with a podspec, ensure this variable is present. | ||
|
|
||
| ### Extra Quirks | ||
| When retrieving the secrets, we need to replace any `_`'s with `-`'s as k8s resources do not like underscores, however, when creating the user environment variables those need to have underscores, as linux works better with underscores. | ||
| ## For testing | ||
| Create a PR with the `auto-deploy` label, and after the image has successfully pushed to the ACR, go to the `kerberos` application in das argocd, turn off autosync (if it is on) and update the `kerberos-sidecar-injector` image tag with the pushed image tag. Start up a notebook and ensure it patches correctly. | ||
|
|
||
| We also must limit the length of the names in the spec. There is a hard 64 character limit and currently to avoid collision we hash the name and use it. A better fix for this will be tracked in [BTIS-528](https://jirab.statcan.ca/browse/BTIS-528) | ||
| ## Building and Deploying | ||
| Once your PR is merged and the workflows are completed, take the SHA tag for the newly built image, and update the tag for the deployment for the kerberos-sidecar-injector in our gitlab [aaw-argocd-manifests](https://gitlab.k8s.cloud.statcan.ca/business-transformation/aaw/aaw-argocd-manifests) repo. | ||
|
|
||
| We built in a [retry on failure](https://github.com/StatCan/filer-sidecar-injector/pull/13) command for `goofys` as it is currently unclear why goofys sometimes fails to mount, but in general having it retry a few times seems to fix things with more investigation being done in [BTIS-523](https://jirab.statcan.ca/browse/BTIS-523) | ||
| ## Kerberos Sidecar Volumes | ||
| There are 3 volume mounts needed for the kerberos sidecar to function as expected: | ||
| - `kerberos-credential-cache` which is set up as an `emptyDir` and is used to store the cached results of the authentication | ||
| - `kerberos-conf` which is generated from the `kerberos-sidecar-config` configMap, which gets created in the user's namespace by our kerberos controller. It stores the configurations for the sidecar | ||
| - `kerberos-keytab` which is generated from the secret of the same name in the user's namespace which gets created by running the `ktutil-keytab` script from a running notebook. It gets used by the sidecar as the authentication token | ||
|
|
||
| ### Building and Deploying for testing | ||
| Create a PR with the `auto-deploy` label, and after the image has successfully pushed to the ACR, go to the `netapp` application in das argocd and turn off autosync and update the `filer-sidecar-injector` image tag with the pushed image. Start up a notebook and ensure it patches correctly. | ||
| ## Related Components | ||
| The [kerberos controller](https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/profiles-controller-aaw2.0/cmd/kerberos.go) is setup to create the necessary components for the kerberos sidecar container to work as expected. When a user creates a "kerberos-keytab" secret in their namespace, with the use of our custom `ktutil-keytab` script defined in our [container images repo](https://github.com/StatCan/zone-kubeflow-containers/blob/master/resources/common/ktutil-keytab.sh), the controller should pick up this change in the user's namespace and create two resources: | ||
| - a `kerberos-sidecar-config` configMap to store the configurations for the sidecar | ||
| - a `allow-egress-to-kerberos-kdc` networkPolicy to allow connections from the sidecar to the needed service(Keytab Distribution Center). |