Flux installation without CNI

[eminaktas]

Hi folks,

We wanted to share an idea with you. We have been working on the FluxCD to deploy without any CNI installed. At first look, it might be seen as a crazy idea, but the idea was to manage Cilium installation with FluxCD.

We could also install FluxCD after CNI and have it take over the installation, but this use case could be risky since if CNI is broken, FluxCD could be broken. So, we wanted to eliminate this risk.

We found the discussion here. We took as an example @maxpain's repository and extended it to overcome some problems since the deployment must be in the same node, and the HA structure couldn't be provided.

Steps we followed:

• With the flux install command, we generate the flux resources and save the output in a file.
• We used the kubectl-slice command to split the resources to avoid having issues with the CRD installation due to generated resources not being ordered.
• We also need to take care of the ports for the application. With kustomization YAML file, we are manipulating ports and some application arguments. We selected some ports that could not overlap with other applications. (22570-22602).
• Some components require communication with other controllers. To provide it, we deploy nginx-proxy for notification and source controllers.
• Then, we are ready to apply resources; however, we are using resources in two parts since kustomization causes a race condition, and pods cannot detect the CRD in a cluster in the first two restarts. So, we first apply CRDs and wait a couple of seconds, then apply deployment resources with kustomization.

We need your feedback and want to write the blog as a case study.

/cc @developer-guy @Dentrax @yasintahaerol @ugur99 @alialperak @dholbach

[stefanprodan]

Installing Flux without a CNI is possible using a custom Deployment where all Flux controllers run inside the same pod and talk to each other over loopback. I've been experimenting with this here https://github.com/stefanprodan/flux-local-dev/tree/flux-xs/flux-xs

[eminaktas]

Thanks for sharing @stefanprodan. This is an excellent idea.

Is it possible to provide a leader election with your method? You disabled the leader election. 🤔

[stefanprodan]

The idea is to use this at bootstrap, setup the CNI only, then get rid of it and deploy Flux the normal way. Hence this deployment doesn't need leader election because it's just a temporary workload.

[eminaktas]

When the cilium is broken, the connectivity between controllers might get hurt, too, since a service-to-service connection might be needed for some components. This could happen if you replace the Kube-proxy with cilium. So, we want to keep FluxCD serviceable at all times.

[stefanprodan]

There are many tradeoff when running all Flux controllers in the same pod, for example you can't run them with least privileges: even if source-controller doesn't need cluster-admin you must run it like so for kustomize and helm controller to be able to install CRDs and the CNI. I'm considering creating an experimental Flux distribution for clusters without CNI but I'm not sure if the tradeoffs are acceptable. WDYT?

[stefanprodan]

@eminaktas can you please give this a try and let me know if it works on clusters without a CNI https://github.com/stefanprodan/flux-aio

Thanks!

[eminaktas]

There are many tradeoff when running all Flux controllers in the same pod, for example you can't run them with least privileges: even if source-controller doesn't need cluster-admin you must run it like so for kustomize and helm controller to be able to install CRDs and the CNI. I'm considering creating an experimental Flux distribution for clusters without CNI but I'm not sure if the tradeoffs are acceptable. WDYT?

As you mentioned before, it will be a short period. So I guess it shouldn't be a problem.

I had a different approach. You can find my work here.

@eminaktas can you please give this a try and let me know if it works on clusters without a CNI https://github.com/stefanprodan/flux-aio

Thanks!

I will test it in our cluster setup. Thanks!