Configure multicard interfaces #1952
Conversation
| # EKS already forces amazon-ec2-net-utils to manage the primary ENI only (0/0): | ||
| # https://github.com/awslabs/amazon-eks-ami/pull/1738 | ||
| # | ||
| # When VPC CNI is installed it creates another ENI from network card with index 0 (0/1). | ||
| # VPC CNI will continue adding additional ENIs until max number of interfaces for the intance | ||
| # is reached (0/2, 0/3, etc). | ||
| # | ||
| # This script configures IP and routing for Elastic Network Interfaces that are part of | ||
| # non-zero indexed EC2 network cards (1/0, 1/1, 2/1, 3/1, etc). The way we find out whether | ||
| # we need to configure the interface is by checking IMDS: | ||
| # /latest/meta-data/network/interfaces/macs/${mac_address}/network-card/ | ||
| # This script will skip any interfaces that are part of the 0 indexed card. |
There was a problem hiding this comment.
A+++++ comment, thx.
I'm wondering if we can leverage systemd-networkd here instead of setting all the routes up ourselves. We modify the default networkd config so that it only matches the primary ENI (by using PermanentMACAddress). Can we add more MAC's there for the other interfaces on other cards? Or create additional networkd configs for those interfaces?
There was a problem hiding this comment.
If we add the mac addresses to the current drop-in file, the default routes for them will end up in the main route table, which I think it's not a good idea.
If we want to leverage systemd-networkd we'll have to create separate .network file for the non zero cards populate the mac addresses and configure the policy routing. I was thinking about that too, but figured it is probably easier/clear to setup everything with iproute package commands. I'm fine with changing direction, also asked somebody more familiar with networking to take a look, let's see what their take will be too :)
There was a problem hiding this comment.
it is probably easier/clear to setup everything with iproute package commands
I would much rather defer to networkd 😂 we have to phone a friend every time we touch this script
There was a problem hiding this comment.
+1 to seeing what we can do to avoid adding this into our AMIs for AL2023. I think we need to figure out how amazon-ec2-net-utils can support management of just the primary ENI and let others (like VPC CNI) handle all secondary ENI management.
We're also going through this much effort to bring up network interfaces that aren't on network card 0, but those interfaces aren't actually used by K8s pods in any case since we only assign IPs from network card 0 right? This is something we've got to chase with the VPC CNI to make it network card aware, but until then I wonder how much value this PR brings us other than consistency with our AL2 AMI.
There was a problem hiding this comment.
+1 to seeing what we can do to avoid adding this into our AMIs for AL2023. I think we need to figure out how amazon-ec2-net-utils can support management of just the primary ENI and let others (like VPC CNI) handle all secondary ENI manageme
We already "tell" ec2-net-utils to manage the primary ENI, by using drop-in. When checking with VPC CNI people I was told that there is backlog item about handling non primary interfaces, but I didn't get anything more than that. I can ask further.
We're also going through this much effort to bring up network interfaces that aren't on network card 0, but those interfaces aren't actually used by K8s pods in any case since we only assign IPs from network card 0 right?
Potential use case is if someone runs the pod in the host network namespace and specifically binds their app to one of non zero card interfaces.
I wonder how much value this PR brings us other than consistency with our AL2 AMI.
My main goal was parity with AL2 AMI.
There was a problem hiding this comment.
+1 to seeing what we can do to avoid adding this into our AMIs for AL2023. I think we need to figure out how amazon-ec2-net-utils can support management of just the primary ENI and let others (like VPC CNI) handle all secondary ENI management
Making VPC CNI aware of network cards other than card 0 is going to be a much bigger change and currently not prioritized by the networking team. We went ahead with same approach as AL2 AMI to not block the AL2023 AMI on this.
cc: @jayanthvn on setting up routes via iproute vs using networkd by creating separate .network files
There was a problem hiding this comment.
The CNI plugin will not manage the ENI setup and routes for the primary ENIs on non-zero network cards, even if the CNI plugin manages the secondary ENIs on the non-zero network cards.
| else | ||
| echo "handling interface $if_name" | ||
|
|
||
| if_ip_addr=$(imds "/latest/meta-data/network/interfaces/macs/$trimmed_mac/local-ipv4s" | head -1) |
There was a problem hiding this comment.
Is there any plans to support IPv6?
|
We synced on the side, Yang is in favor of creating separate |
|
Thanks @M00nF1sh, agree. @nkvetsinski We already modify the existing networkd config in |
|
I'm going to open a new PR for this guys, since we're changing the approach. |
Issue #, if available:
Description of changes:
In the AMI we force
systemd-networkdto only manage the primary interface. In this commit I'm adding a script that will configure ip addresses and routes for interfaces that are not managed bysystemd-networkd.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Testing Done
I verified that:
iperf, starting a server on one pod and binding it to non zero card interface. On another pod (running on different machine), usingiperfclient (bound to non zero card interface). Runningtcpdumpon both machines and verifying traffic is flowing thru the correct interfaces.See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.