Documents and source codes for the deprecated domain
cogadvisor.ioare moved to cogadvisor-net branch
Attaching secondary network interfaces that is linked to different network interfaces on host (NIC) to pod provides benefits of network segmentation and top-up network bandwidth in the containerization system.
Multi-NIC CNI is the CNI plugin operating on top of Multus CNI. However, unlike Multus, instead of defining and handling each secondary network interface one by one, this CNI automatically discovers all available secondary interfaces and handles them as a NIC pool. With this manner, it can provide the following benefits.
i) Common secondary network definition: User can manage only one network definition for multiple secondary interfaces with a common CNI main plugin such as ipvlan, macvlan, and sr-iov.
ii) Common NAT-bypassing network solution: All secondary NICs on each host can be assigned with non-conflict CIDR and non-conflict L3 routing configuration that can omit an overlay networking overhead. Particularyly, the CNI is built-in with L3 IPVLAN solution composing of the following functionalities.
- Interface-host-devision CIDR Computation: compute allocating CIDR range for each host and each interface from a single global subnet with the number of bits for hosts and for interface.
- L3 Host Route Configuration: configure L3 routes (next hop via dev) in host route table according to the computed CIDR.
- Distributed IP Allocation Management: manage IP allocation/deallocation distributedly via the communication between CNI program and daemon at each host.
iii) Policy-based secondary network attachment: Instead of statically set the desired host's master interface name one by one, user can define a policy on attaching multiple secondary network interfaces such as specifying only the number of desired interfaces, filtering only highspeed NICs.
The Multi-NIC CNI architecture can be found here.
The Multi-NIC operator operates over a custom resource named MultiNicNetwork defined by users. This definition will define a Pod global subnet, common network definition (main CNI and IPAM plugin), and attachment policy. After deploying MultiNicNetwork, NetworkAttachmentDefinition with the same name will be automatically configured and created respectively.
# network.yaml
apiVersion: multinic.fms.io/v1
kind: MultiNicNetwork
metadata:
name: multi-nic-sample
spec:
subnet: "192.168.0.0/16"
ipam: |
{
"type": "multi-nic-ipam",
"hostBlock": 6,
"interfaceBlock": 2,
"vlanMode": "l3"
}
multiNICIPAM: true
plugin:
cniVersion: "0.3.0"
type: ipvlan
args:
mode: l3
attachPolicy:
strategy: none
namespaces:
- default| Argument | Description | Value | Remarks |
|---|---|---|---|
| subnet | cluster-wide subnet for all hosts and pods | CIDR range | currently support only v4 |
| hostBlock | number of address bits for host indexing | int (n) | the number of assignable host = 2^n |
| ipam | ipam plugin config | string | ipam can be single-NIC IPAM (e.g., whereabouts, VPC-native IPAM) or multi-NIC IPAM (e.g., Multi-NIC IPAM Plugin) |
| multiNicIPAM | indicator of ipam type | bool | true if ipam returns multiple IPs from masters key of NetworkAttachmentDefinition config at once, false if ipam returns only single IP from static config in ipam block |
| plugin | main plugin config | NetConf + plugin-specific arguments | main plugin integration must implement Plugin with GetConfig function |
| attachPolicy | attachment policy | policy | strategy with corresponding arguments to select host NICs to be master of secondary interfaces on Pod |
| namespaces | (optional) limit network definition application to list of namespaces (i.e., to create NetworkAttachmentDefinition resource) | []string | if not specified, network definitions will be applied to all namespaces. new item can be added to the list by kubectl edit to create new NetworkAttachmentDefinition. the created NetworkAttachmentDefinition must be deleted manually if needed. |
- Secondary interfaces attached to worker nodes, check terraform script here
- Multus CNI installation; compatible with networkAttachmentDefinition and pod annotation in multus-cni v3.8
- For IPVLAN L3 CNI, the following configurations are additionally required
- enable allowing IP spoofing for each attached interface
- set security group to allow IPs in the target container subnet
- IPVLAN support (kernel version >= 4.2)
- Kubernetes with OLM:
- Openshift Container Platform:
- Search for
multi-nic-cni-operatorin OperatorHub
- Search for
kubectl apply -f deploy/operator-sdk run bundle ghcr.io/foundation-model-stack/multi-nic-cni-bundle:v1.0.2-
Prepare
network.yamlas shown in the example -
Deploy
kubectl apply -f network.yaml
After deployment, the operator will create NetworkAttachmentDefinition of Multus CNI from MultiNicNetwork as well as dependent resource such as SriovNetworkNodePolicy, SriovNetwork for sriov plugin.
-
To attach additional interfaces, annotate the pod with the network name
metadata: annotations: k8s.v1.cni.cncf.io/networks: multi-nic-sample
- Deploy concheck driver
kubectl create -f connection-check/concheck.yaml
- Check log
expected log:
kubectl logs job/multi-nic-concheck
########################################### ## Connection Check: multinic-ipvlanl3 ########################################### FROM TO CONNECTED/TOTAL IPs BANDWIDTHs multi-nic-n7zf6-worker-2-dbjpg multi-nic-n7zf6-worker-2-zt5l5 2/2 [192.168.0.65 192.168.64.65] [ 6.10Gbits/sec 10.2Gbits/sec] multi-nic-n7zf6-worker-2-zt5l5 multi-nic-n7zf6-worker-2-dbjpg 2/2 [192.168.0.1 192.168.64.1] [ 7.81Gbits/sec 12.4Gbits/sec] ###########################################
- Clean up
kubectl delete pod -n default --selector multi-nic-concheck kubectl delete job -n default --selector multi-nic-concheck kubectl delete -f connection-check/concheck.yaml
kubectl delete -f deploy/operator-sdk cleanup multi-nic-cni-operator