Flannel VXLAN Security - Firewall Requirements

[mjrist]

Describe the solution you'd like

This is a question and request to provide more details in the documentation about the security implications of using the default flannel CNI with the vxlan backend.

The documentation (https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/#networking) states: "IMPORTANT: The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472."

We are looking to better understand the security implications of using flannel vxlan. For example, what can someone do from outside the cluster if they do have access to UDP port 8472 on one of the nodes? Is it possible to access the overlay and communicate with pods from outside the cluster? Can someone please point us to more information related to this?

We would like the default network solution to be secure by default (I know that statement is a little vague and I am open to discussion here). However, in short, we want to be able to set up a cluster without the need manually configure a firewall blocking UDP port 8472.

It's worth noting that we tried using the ipsec backend as it appears to be more inherently secure (and we also would like the benefit of encrypted node-to-node traffic). However, we ran into bugs seemly related to strongswan creating duplicate SAs and consequently breaking routing. Then after looking at flannel's documentation we realized that the ipsec backend is marked experimental and unsupported, and decided to switch back to vxlan. (If that is truly the case, this documentation - https://rancher.com/docs/k3s/latest/en/installation/network-options/#flannel-options - should probably be updated to reflect that.)

Any guidance that could provided related to the security implications of using the vxlan backend would be greatly appreciated. Thanks!

(Related question: #1252 I'm also interested in the answer to that one.)

[brandond]

https://blog.ipspace.net/2018/11/omg-vxlan-is-still-insecure.html is a good intro to vxlan security concerns.

I personally would recommend trying out wireguard over ipsec.

[mjrist]

Thanks!

I had actually read that article before. Just looked at the referenced presentation which goes into more details. I am going to take a look at some of the other referenced articles as well. This is definitely helpful.

Here are some thoughts (and please don't take any of this the wrong way - I think you guys/girls are doing fantastic work! And I understand the challenges of software development.)

I agree with the author that (with any tool) we should "understand how it works, and what its advantages and limitations are". We will be more successful and secure if we do. That's why I started this conversation - to understand the implications of using vxlan.

However, I think we also have to be realistic. Not everyone is a network engineer and has the same level of competence as the author. The author's bio indicates that he "has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced internetworking technologies since 1990". A lot of engineers I work with haven't even been alive that long.

I'm not against the use of vxlan. It can work for us as long as we know how make it secure and can ensure that it is. However, I am against surprises when it comes to using software tools or libraries.

The first page of the k3s docs states "Secure by default with reasonable defaults for lightweight environments." (https://rancher.com/docs/k3s/latest/en/) My understanding of secure by default is that an out of the box installation will be secure without additional configuration (I realize "out of the box installation" may be loosely defined here). The fact that failing to set up a firewall rule leaves my cluster exposed does not seem to be secure by default (again I'm not trying to be contentious). I understand that there are different ways of setting up a cluster and different network options - and of course some of those options will be more secure than others. However, currently the default CNI and backend is flannel vxlan - and if I understand correctly it is "insecure" without additional firewall configuration to block access to UDP 8472.

Again the point is not to rant - so I hope it doesn't come across that way.

Some questions are:

• Is work being done to switch over to a network backend that is "secure by default"?
• Is there more that could be done to automatically ensure that the default use of the vxlan backend is secure?
• As a bare minimum is it possible to better expose these security implications to the end user?

Also, thanks for the recommendation to try wireguard. We are going to evaluate that. A little context here might be helpful. We are using k3s to build an application platform which our customers use to deploy our software. So ease of setup and secure by default are high priorities for us. If these were just on-site clusters, I would not be so concerned as I would have more control over the setup/firewalls. By the way, you guys have done a phenomenal job at making k3s lightweight and easy to deploy! (Which is one of the main reasons we are using it.)

[ChristianCiach]

We are using k3s to build an application platform which our customers use to deploy our software. So ease of setup and secure by default are high priorities for us.

This sound very similar to what we are doing. We are also using flannel with wireguard-backend for this use-case. Fortunately, K3s configures wireguard automatically. You just need to make sure to use a wireguard-capable kernel. This is the default on Ubuntu, but when using RHEL you have to install a kernel module from ELRepo. You may also have to install the the wireguard-tools package. The rest is handled automatically by K3s/Flannel.

By the way, you guys have done a phenomenal job at making k3s lightweight and easy to deploy!

I would like to echo that! I've evaluated many distributions and ways to install Kubernetes, but nothing is as simple and reliable as K3s. We are deploying our stuff on-premise, mostly in air-gapped environments, and I have very little influence regarding the configuration (like OS) of the hosts that are provided by our customers. K3s just works everywhere. My K3s-Ansible-Playbook doesn't even need to distinguish between operating systems. It's just awesome!

[manuelbuil]

Note that flannel has just merged wireguard with dual-stack support. As soon as flannel releases it, we will integrate it into k3s :)

[mjrist]

@manuelbuil can you expound on this a bit? I would like to make sure I understand the significance of this.

My understanding is that K3s already supports using flannel with wireguard as the backend. However - if I understand correctly - one must install wireguard separately.

Are you saying that flannel will now automatically include and install wireguard and the necessary kernel modules?

Or did you mean that it is just adding dual-stack support?

[brandond]

K3s's current support for wireguard is just done via extension plugin shell commands:

k3s/pkg/agent/flannel/setup.go

Lines 68 to 75 in 2f3bfc2

	wireguardBackend = `{
	"Type": "extension",
	"PreStartupCommand": "wg genkey | tee %flannelConfDir%/privatekey | wg pubkey",
	"PostStartupCommand": "export SUBNET_IP=$(echo $SUBNET | cut -d'/' -f 1); ip link del flannel.1 2>/dev/null; echo $PATH >&2; wg-add.sh flannel.1 && wg set flannel.1 listen-port 51820 private-key %flannelConfDir%/privatekey && ip addr add $SUBNET_IP/32 dev flannel.1 && ip link set flannel.1 up && ip route add $NETWORK dev flannel.1",
	"ShutdownCommand": "ip link del flannel.1",
	"SubnetAddCommand": "read PUBLICKEY; wg set flannel.1 peer $PUBLICKEY endpoint $PUBLIC_IP:51820 allowed-ips $SUBNET persistent-keepalive 25",
	"SubnetRemoveCommand": "read PUBLICKEY; wg set flannel.1 peer $PUBLICKEY remove"
	}`

When available, we will switch over to using the actual wireguard flannel plugin, which (besides dropping the shell commands hack) will also enable ipv6 support.

[mjrist]

I see. Thanks for the explanation.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[mcarbonneaux]

there a solution to force flannel to listen on internal private interface and not on the default public interface ?
flannel-io/flannel#2113