api: add support for reading Pod IPs #119

aojea · 2024-10-26T09:16:52Z

Allow NRI plugins to read the allocated IPs to the pod on the PodSandbox hooks. This is specially useful for networking daemons that depend on the Pod IPs assigned via CNI by the container runtime (service mesh, network policies, ... ) to create seamless integrations without having to modify the existing environment.

Existing network integrations use techniques like:

CNI chaining (invasive) as the implementation needs to modify the existing deployment and can cause disruption due to mutations on the existing configurations [Feature] Early podIP notification to better co-operate with Calico in policy-only mode Azure/AKS#4385
Read Pod Status, that is too late in the process, since the pod lifecycle does not update the pod status until the containers are running, but there are applications like network policy implemetnations or service meshes that needs this information before the pod is completely created (ref: Pod.Status.PodIP not updated during postStart lifecycle hook kubernetes/kubernetes#85966)
Inspect the host namespaces to gather the information, that is very intrusive and not portable, creating a tighly coupling between the implementations.

Add a nri plugin that can be used as reference implementations for this integration, the nri plugin just log the interesting networking details of the pod (ips and network namespace) on the different hooks of the pod lifecucle.

Container runtime dependencies

This feature requires the container runtime to pass to the nri plugin the IP information.

Containerd patch

diff --git a/internal/cri/nri/nri_api_linux.go b/internal/cri/nri/nri_api_linux.go
index 0adcd9d11..598e656b2 100644
--- a/internal/cri/nri/nri_api_linux.go
+++ b/internal/cri/nri/nri_api_linux.go
@@ -633,6 +633,14 @@ func (p *criPodSandbox) GetPid() uint32 {
        return p.pid
 }

+func (p *criPodSandbox) GetIPs() []string {
+       if p.IP == "" {
+               return nil
+       }
+       ips := append([]string{p.IP}, p.AdditionalIPs...)
+       return ips
+}
+
 //
 // NRI integration wrapper for CRI Containers
 //
diff --git a/internal/nri/sandbox.go b/internal/nri/sandbox.go
index 85d6178aa..15ad9538d 100644
--- a/internal/nri/sandbox.go
+++ b/internal/nri/sandbox.go
@@ -34,6 +34,7 @@ type PodSandbox interface {
        GetLinuxPodSandbox() LinuxPodSandbox

        GetPid() uint32
+       GetIPs() []string
 }

 type LinuxPodSandbox interface {
@@ -55,6 +56,7 @@ func commonPodSandboxToNRI(pod PodSandbox) *nri.PodSandbox {
                Annotations:    pod.GetAnnotations(),
                RuntimeHandler: pod.GetRuntimeHandler(),
                Pid:            pod.GetPid(),
+               Ips:            pod.GetIPs(),
        }
 }

You need to build containerd with this NRI patch. In the meantime, for development you can modify the go.mod file in containerd with a replace statement to use the local code and update the vendor folder in containerd.

diff --git a/go.mod b/go.mod
index b7a8aa29f..07366db7c 100644
--- a/go.mod
+++ b/go.mod
@@ -152,3 +152,5 @@ require (
 )

 replace github.com/containerd/containerd/api => ./api
+
+replace github.com/containerd/nri => ../nri

Tested

This was tested with kind but can be tested in any kubernetes cluster just enabling the NRI plugins functionality.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.nri.v1.nri"]
    disable = false
nodes:
- role: control-plane

Build a new containerd binary with the patches indicated in the previous section and run it on the nodes:

~/src/containerd$ make
~/src/containerd$ docker cp bin/containerd kind-control-plane:/usr/local/bin/containerd
~/src/containerd$ docker exec kind-control-plane systemctl restart containerd
~/src/containerd$ docker exec kind-control-plane systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/etc/systemd/system/containerd.service; enabled; preset: enabled)
     Active: active (running) since Sat 2024-10-26 09:09:26 UTC; 7s ago
... snipped ...
Oct 26 09:09:26 kind-control-plane containerd[265194]: time="2024-10-26T09:09:26.908843468Z" level=info msg="Synchronizing NRI (plugin) with current runtime state"
Oct 26 09:09:26 kind-control-plane containerd[265194]: time="2024-10-26T09:09:26.916135536Z" level=info msg="containerd successfully booted in 0.268150s"
Oct 26 09:09:26 kind-control-plane systemd[1]: Started containerd.service - containerd container runtime.

Build the network-logger plugin, copy it into a node and run it, it will output the network parameters of the Pods in each of the lifecycle hooks

~/src/nri$ cd plugins/network-logger/
~/src/nri/plugins/network-logger$ go build
~/src/nri/plugins/network-logger$ docker cp network-logger kind-control-plane:.
~/src/nri/plugins/network-logger$  docker exec kind-control-plane /network-logger --idx 10
time="2024-10-26T09:14:32Z" level=info msg="Created plugin 10-network-logger (network-logger, handles RunPodSandbox,StopPodSandbox,RemovePodSandbox)"
time="2024-10-26T09:14:32Z" level=info msg="Registering plugin 10-network-logger..."
time="2024-10-26T09:14:32Z" level=info msg="Configuring plugin 10-network-logger for runtime v2/v2.0.0-rc.5-110-g5e084bdc6..."
time="2024-10-26T09:14:32Z" level=info msg="Started plugin 10-network-logger..."
time="2024-10-26T09:14:32Z" level=info msg="Synchronized state with the runtime (10 pods, 8 containers)..."
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/kube-controller-manager-kind-control-plane: namespace=<host-network> ips=[]"
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/etcd-kind-control-plane: namespace=<host-network> ips=[]"
time="2024-10-26T09:14:32Z" level=info msg="pod local-path-storage/local-path-provisioner-57c5987fd4-pjdhz: namespace=/var/run/netns/cni-b830a016-210f-2979-a786-a4f50d3b2b19 ips=[10.244.0.2]"
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/kube-scheduler-kind-control-plane: namespace=<host-network> ips=[]"
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/kube-proxy-hlr5m: namespace=<host-network> ips=[]"
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/kube-apiserver-kind-control-plane: namespace=<host-network> ips=[]"
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/kube-network-policies-762xk: namespace=<host-network> ips=[]"
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/coredns-7bf5d5c7b4-4zgdf: namespace=/var/run/netns/cni-65bdb00b-7b50-f785-a79a-5e9dd73f409a ips=[10.244.0.5]"
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/kube-apiserver-kind-control-plane: namespace=<host-network> ips=[]"
time="2024-10-26T09:14:32Z" level=info msg="pod kube-system/etcd-kind-control-plane: namespace=<host-network> ips=[]"

If you create now new Pods it will show up in the log output

INFO   [0027] Started pod default/test-nri-1: namespace=/var/run/netns/cni-2b46640c-8073-214b-77ae-fdebdcd7c223 ips=[10.244.0.7]
INFO   [0038] Stopped pod default/test-nri-1: ips=[10.244.0.7]
INFO   [0047] Removed pod default/test-nri-1: ips=[10.244.0.7]

aojea · 2024-10-26T09:17:01Z

/assign @samuelkarp

Allow NRI plugins to read the allocated IPs to the pod on the PodSandbox hooks. This is specially useful for networking daemons that depend on the Pod IPs assigned via CNI by the container runtime (service mesh, network policies, ... ) to create seamless integrations without having to modify the existing environment. Existing network integrations use techniques like: - CNI chaining (invasive) as the implementation needs to modify the existing deployment and can cause disruption due to mutations on the existing configurations. - Read Pod Status, that is too late in the process, since the pod lifecycle does not update the pod status until the containers are running, but there are applications like network policy implemetnations or service meshes that needs this information before the pod is completely created. - Inspect the host namespaces to gather the information, that is very intrusive and not portable, creating a tighly coupling between. Add a nri plugin that can be used as reference implementations for this integration, the nri plugin just log the interesting networking details of the pod (ips and network namespace) on the different hooks of the pod lifecucle. Change-Id: Idb9605c719903d0df51c9a1947e0f838870b5c0b Signed-off-by: Antonio Ojea <[email protected]>

kad

looks as reasonable addition to Pod status struct.
lgtm.

mikebrow

LGTM
suggested followups:
might want to also consider passing cniresult
might want to also consider passing cni config detail
might want to consider adding a verbose mode on the plugin to show detail for the pod's netns / cni status detail - could check on a timer and/or if there is a delta from the last status check etc..

mikebrow · 2024-10-28T16:50:41Z

side note with the listed patch .. https://github.com/containerd/nri/blob/main/plugins/logger/nri-logger.go#L92 should also show the new content albeit not as focused.

aojea · 2024-10-30T09:32:29Z

naive question, I'm not familiar with this repository workflow, what is the next step to be able to get this merged?

Thanks

mikebrow · 2024-10-31T19:37:25Z

you assigned samuel so wasn't sure if you wanted to wait on his review or not

otherwise we can go ahead and merge..

samuelkarp · 2024-10-31T20:08:45Z

I was away for a few days. I can look today.

aojea force-pushed the podiups branch from c390b31 to f4f9905 Compare October 26, 2024 09:20

aojea force-pushed the podiups branch from f4f9905 to eaf78a9 Compare October 26, 2024 09:31

This was referenced Oct 26, 2024

[Feature] Early podIP notification to better co-operate with Calico in policy-only mode Azure/AKS#4385

Open

Make SRIOV network device plugin independent of CNI k8snetworkplumbingwg/sriov-network-device-plugin#549

Open

klihub requested review from klihub, mikebrow and fuweid October 28, 2024 08:43

kad approved these changes Oct 28, 2024

View reviewed changes

klihub approved these changes Oct 28, 2024

View reviewed changes

mikebrow approved these changes Oct 28, 2024

View reviewed changes

samuelkarp approved these changes Oct 31, 2024

View reviewed changes

samuelkarp merged commit 6d486ac into containerd:main Oct 31, 2024
8 checks passed

This was referenced Nov 1, 2024

expose Pod assigned IPs to NRI plugins containerd/containerd#10921

Merged

expose Pod assigned IPs to NRI plugins cri-o/cri-o#8731

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

api: add support for reading Pod IPs #119

api: add support for reading Pod IPs #119

aojea commented Oct 26, 2024 •

edited

Loading

aojea commented Oct 26, 2024

kad left a comment

mikebrow left a comment

mikebrow commented Oct 28, 2024

aojea commented Oct 30, 2024

mikebrow commented Oct 31, 2024 •

edited

Loading

samuelkarp commented Oct 31, 2024

api: add support for reading Pod IPs #119

api: add support for reading Pod IPs #119

Conversation

aojea commented Oct 26, 2024 • edited Loading

Container runtime dependencies

Tested

aojea commented Oct 26, 2024

kad left a comment

Choose a reason for hiding this comment

mikebrow left a comment

Choose a reason for hiding this comment

mikebrow commented Oct 28, 2024

aojea commented Oct 30, 2024

mikebrow commented Oct 31, 2024 • edited Loading

samuelkarp commented Oct 31, 2024

aojea commented Oct 26, 2024 •

edited

Loading

mikebrow commented Oct 31, 2024 •

edited

Loading