Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tilt provider support for CAPI #358

Closed
jds9090 opened this issue Aug 24, 2023 · 5 comments · Fixed by #359
Closed

Tilt provider support for CAPI #358

jds9090 opened this issue Aug 24, 2023 · 5 comments · Fixed by #359
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@jds9090
Copy link
Contributor

jds9090 commented Aug 24, 2023

Most people who use CAPI provision clusters using clusterctl(https://cluster-api.sigs.k8s.io/clusterctl/overview.html). When we want to use kamaji as control plane with CAPI, we can use cluster-api-control-plane-provider-kamaji.(https://github.com/clastix/cluster-api-control-plane-provider-kamaji). But it still needs kamaji to be installed(clastix/cluster-api-control-plane-provider-kamaji#49).

Recently, from CAPI v1.5.0(https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.5.0), we can use the addon option of clusterctl and I suggest we deploy both kamaji and cluster-api-control-plane-provider-kamaji with the option.

Here is my suggestion using the addon option of clusterctl.

ubuntu@kamaji-tilt:~/.cluster-api$ clusterctl init --control-plane kamaji --infrastructure openstack --addon kamaji-core
Fetching providers
Skipping installing cert-manager as it is already installed
Installing Provider="cluster-api" Version="v1.5.0" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.5.0" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kamaji" Version="v0.3.0" TargetNamespace="kamaji-system"
Installing Provider="infrastructure-openstack" Version="v0.7.3" TargetNamespace="capo-system"
Installing Provider="addon-kamaji-core" Version="v0.0.1" TargetNamespace="kamaji-core-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

ubuntu@kamaji-tilt:~/.cluster-api$ k get po -A
NAMESPACE                       NAME                                                         READY   STATUS    RESTARTS   AGE
capi-kubeadm-bootstrap-system   capi-kubeadm-bootstrap-controller-manager-7f7545c44f-w8kjs   1/1     Running   0          24s
capi-system                     capi-controller-manager-6c8c59f4-8x29l                       1/1     Running   0          25s
capo-system                     capo-controller-manager-6cb6565fd9-5jrlp                     1/1     Running   0          23s
cert-manager                    cert-manager-5b8f9b9d96-jb4qh                                1/1     Running   0          4m22s
cert-manager                    cert-manager-cainjector-54f68bfb64-z872n                     1/1     Running   0          4m22s
cert-manager                    cert-manager-webhook-f6c8487d6-d9b75                         1/1     Running   0          4m21s
kamaji-core-system              capi-kamaji-core-controller-manager-b58d8997b-59bhk          1/1     Running   0          22s
kamaji-system                   capi-kamaji-controller-manager-7bbd5c8c95-cwsdq              1/1     Running   0          24s
kube-system                     coredns-565d847f94-96xnv                                     1/1     Running   0          10m
kube-system                     coredns-565d847f94-cg5f4                                     1/1     Running   0          10m
kube-system                     etcd-kind-control-plane                                      1/1     Running   0          10m
kube-system                     kindnet-pj62f                                                1/1     Running   0          10m
kube-system                     kube-apiserver-kind-control-plane                            1/1     Running   0          10m
kube-system                     kube-controller-manager-kind-control-plane                   1/1     Running   0          10m
kube-system                     kube-proxy-9hhkx                                             1/1     Running   0          10m
kube-system                     kube-scheduler-kind-control-plane                            1/1     Running   0          10m
local-path-storage              local-path-provisioner-684f458cdd-6x4bn                      1/1     Running   0          10m

You can also create a pre-configured kind cluster (if you have not already done so) and launch your development environment, run

cd cluster-api
make tilt-up

스크린샷 2023-08-24 오전 8 57 42

@jds9090
Copy link
Contributor Author

jds9090 commented Aug 24, 2023

~/.cluster-api/clusterctl.yaml file example as follows:

providers:
  - name: "kamaji"
    url: "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/releases/v0.3.0/control-plane-components.yaml"
    type: "ControlPlaneProvider"
  - name: "kamaji-core"
    url: "https://github.com/clastix/kamaji/releases/v0.3.x/install.yaml"
    type: "AddonProvider"

@prometherion
Copy link
Member

Thanks for the really helpful feature request, @jds9090: TIL about CAPI addons, thanks for sharing!

Definitely a green light to set up a TILT-based dev environment scaffolding, it seems a popular project and I definitely should give it a try.

In regard to the CAPI addons, I think there's a pre-requisite issue, #292, since manifests would be required to get exposed using a GitHub release, and we're working on there.

However, just installing Kamaji is not enough since along with that a Datastore must be created, and configured properly.

The preferred way to install Kamaji is via Helm which offers hooks to automate some tasks using Kubernetes jobs, like the creation of an etcd cluster, properly configured with multi-tenancy. By default, when installing Kamaji, a default one is created, although I would suggest a more fine-grained control using the kamaji-etcd chart which offers a more robust lifecycle management (such as backup).

I'd like to start a discussion on how to address this, just sharing my thoughts.

  1. Let's implement the TILT dev environment
  2. Address the release of Kamaji manifests using plain YAML files exposed on the GitHub release page
    • This means we're removing the etcd manifests from those files
  3. Consider if the same approach, such as CAPI Addons exposed as a single YAML manifest in the GH release, could be adopted for kamaji-etcd, although this should be tracked in its repository

Tagging for visibility @maxgio92 @ptx96 @bsctl and @dierbei who did the Go releaser implementation.

@jds9090
Copy link
Contributor Author

jds9090 commented Aug 24, 2023

Thanks for the really helpful feature request, @jds9090: TIL about CAPI addons, thanks for sharing!

Definitely a green light to set up a TILT-based dev environment scaffolding, it seems a popular project and I definitely should give it a try.

In regard to the CAPI addons, I think there's a pre-requisite issue, #292, since manifests would be required to get exposed using a GitHub release, and we're working on there.

However, just installing Kamaji is not enough since along with that a Datastore must be created, and configured properly.

The preferred way to install Kamaji is via Helm which offers hooks to automate some tasks using Kubernetes jobs, like the creation of an etcd cluster, properly configured with multi-tenancy. By default, when installing Kamaji, a default one is created, although I would suggest a more fine-grained control using the kamaji-etcd chart which offers a more robust lifecycle management (such as backup).

I'd like to start a discussion on how to address this, just sharing my thoughts.

  1. Let's implement the TILT dev environment

  2. Address the release of Kamaji manifests using plain YAML files exposed on the GitHub release page

    • This means we're removing the etcd manifests from those files
  3. Consider if the same approach, such as CAPI Addons exposed as a single YAML manifest in the GH release, could be adopted for kamaji-etcd, although this should be tracked in its repository

Tagging for visibility @maxgio92 @ptx96 @bsctl and @dierbei who did the Go releaser implementation.

I agree with you. I also had to install datastore with this command for each tenant:

helm install kamaji-etcd clastix/kamaji-etcd --set datastore.enabled=true

In my case, I wanted tenant clusters to be deployed with their own datastore for security. Please, consider also this case because some people are more interested in this case for some specific policy of companies .

I hope you will come up with more fine-grained control.

@jds9090
Copy link
Contributor Author

jds9090 commented Aug 24, 2023

@prometherion In the view of CAPI, we can use this addon to install datastore(https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/blob/main/docs/quick-start.md) per tenant. If tenant uses the default, it may say "already installed".
Let me test it.

@jds9090
Copy link
Contributor Author

jds9090 commented Aug 24, 2023

Unfortunately, the addon-provider is not what I want to find.. this is a replacement of ClusterResourceSet(https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220712-cluster-api-addon-orchestration.md#motivation)

@prometherion prometherion added the enhancement New feature or request label Aug 25, 2023
@prometherion prometherion added this to the v0.3.4 milestone Aug 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants