kubeadm config images pull support for Amazon ECR with containerd

[aaroniscode]

FEATURE REQUEST

Would like to be able to pre pull container images when using Amazon ECR container registry and containerd as the CRI.

containerd does not have native support for authenticating to Amazon ECR. This is an open ticket: containerd/containerd#6637.

This capability might best be a feature of containerd. However, this authentication is already built into kubelet, maybe it could be easily reused (as a library) for kubeadm?

Versions

v1.15.4

[timothysc]

This really doesn't seem like a kubeadm issue.

[aaroniscode]

@timothysc That may be the case. However, it appears from the comment containerd/containerd#6637 that the containerd team is considering not adding the logic to authenticate with ECR.

If that's the case, where should the logic live to authenticate with ECR?

[neolit123]

@aaroniscode

If that's the case, where should the logic live to authenticate with ECR?

ideally we should not include registry authorizations mechanisms as part of kubeadm.

as per:
containerd/containerd#6637

crictl pull --creds aaa:bbb gcr.io/google-containers/busybox:latest

does crictl support the --creds as part of it's config or is this only supported from a CLI flag?

[neolit123]

cc @bart0sh

[neolit123]

@aaroniscode

for the kubeadm side to work, crictl needs to support authorization files in a default location, instead of only accepting --creds on the CLI.

this way any crictl pull call would read credentials from a config and not require the explicit --creads flag which kubeadm lacks. we have the option to expose the USER:PASSWORD using environment variables, but it feels like this would be a hack / workaround for the lack of a feature in critctl.

related:
kubernetes-sigs/cri-tools#482

/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

@aaroniscode

for the kubeadm side to work, crictl needs to support authorization files in a default location, instead of only accepting --creds on the CLI.

this way any crictl pull call would read credentials from a config and not require the explicit --creads flag which kubeadm lacks. we have the option to expose the USER:PASSWORD using environment variables, but it feels like this would be a hack / workaround for the lack of a feature in critctl.

related:
kubernetes-sigs/cri-tools#482

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aaroniscode]

@neolit123 Is it at all possible that the kubeadm could leverage the credential provider inside Kubernetes?

https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/aws/aws_credentials.go

Or maybe I misunderstood that the Kubernetes ECR credential support is limited only to Docker and doesn't work with containerd?

The Kubernetes docs around ECR Support don't mention a limitation of only supporting Docker: https://kubernetes.io/docs/concepts/containers/images/#using-amazon-elastic-container-registry

[neolit123]

@neolit123 Is it at all possible that the kubeadm could leverage the credential provider inside Kubernetes?

kubeadm should be cloud provider agnostic.

Or maybe I misunderstood that the Kubernetes ECR credential support is limited only to Docker and doesn't work with containerd?

The Kubernetes docs around ECR Support don't mention a limitation of only supporting Docker: https://kubernetes.io/docs/concepts/containers/images/#using-amazon-elastic-container-registry

i think this only works for docker currently, even if the docs don't mention it.

https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/aws/aws_credentials.go

this comment gives some explanations about the above package:

containerd/containerd#6637

Finally, the future roadmap for Kubernetes has the ECR credential fetch moving from the Kubelet (--cloud=provider=aws) into the control plane (--cloud-provider=external), so a standalone Kubelet would need to fully rely on containerd for ECR pull credentials.

i have not information for what is the future of pkg/credentialprovider/aws and credential providers.
most likely moving outside of k/k and decoupling from the kubelet.

The Kubernetes level provides credentials for pulling Pod images, but these credentials are not used by containerd when it pulls the sandbox image.

this is true. credentials for pulling the sandbox image still have to be provided outside of Kubernetes and into raw crictl pull calls.

[aaroniscode]

thanks @neolit123 . This is really helpful and makes sense. So once containerd has native support for ECR pull credentials, kubeadm will "just work".

[roshankakad]

Hello Team,

Getting crictl pul image error ""401 Unauthorized "
root@ip-X-X-X-X:~# crictl pull X.dkr.ecr.us-east-2.amazonaws.com/netq-app-air:4.7.0-SNAPSHOT

E0703 04:20:12.150068 4026085 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image ".dkr.ecr.us-east-2.amazonaws.com/netq-app-air:4.7.0-SNAPSHOT": failed to resolve reference ".dkr.ecr.us-east-2.amazonaws.com/netq-app-air:4.7.0-SNAPSHOT": pulling from host .dkr.ecr.us-east-2.amazonaws.com failed with status code [manifests 4.7.0-SNAPSHOT]: 401 Unauthorized" image=".dkr.ecr.us-east-2.amazonaws.com/netq-app-air:4.7.0-SNAPSHOT"
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image ".dkr.ecr.us-east-2.amazonaws.com/netq-app-air:4.7.0-SNAPSHOT": failed to resolve reference "498186410471.dkr.ecr.us-east-2.amazonaws.com/netq-app-air:4.7.0-SNAPSHOT": pulling from host X.dkr.ecr.us-east-2.amazonaws.com failed with status code [manifests 4.7.0-SNAPSHOT]: 401 Unauthorized.

k8s version: 1.27.2
containerd://1.6.21

Please share your comments