registry.k8s.io/pause:3.6 does not use the address specified by --image-repository when executing kubeadm init #2851

Max-Qiu · 2023-03-29T10:46:58Z

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

What happened?

use kubeadm init ...... --image-repository registry.aliyuncs.com/google_containers , it show :
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory /etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
then use systemctl status -l kubelet.service , it show :
remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image "registry.k8s.io/pause:3.6": failed to pull image "registry.k8s.io/pause:3.6": failed to pull and unpack image "registry.k8s.io/pause:3.6": failed to resolve reference "registry.k8s.io/pause:3.6": failed to do request: Head "https://registry.k8s.io/v2/pause/manifests/3.6\": net/http: TLS handshake timeout"

What did you expect to happen?

I hope that registry.k8s.io/pause:3.6 will be downloaded through the address specified by --image-repository

How can we reproduce it (as minimally and precisely as possible)?

Block registry.k8s.io
Execute kubeadm init --control-plane-endpoint=192.168.220.101 --kubernetes-version v1.26.3 --pod-network-cidr=10.244.0.0/16 --image-repository registry.aliyuncs.com/google_containers

Anything else we need to know?

I need to execute the following code on each node to solve the problem

ctr -n k8s.io i pull registry.aliyuncs.com/google_containers/pause:3.6
ctr -n k8s.io i tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6

Kubernetes version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

OS version

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ uname -a
Linux node1 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

pacoxu · 2023-03-29T11:22:20Z

Check your containerd configuration.

neolit123 · 2023-03-29T11:32:22Z

see

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#override-pause-image-containerd

/kind support

LxxxSec · 2023-05-05T06:23:38Z

我也遇到了和你类似的问题，我的解决方案如下：

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

接着编辑/etc/containerd/config.toml文件，把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可，如下方代码所示（pause版本因人而异）：

sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

接着重启服务，重新init即可：

systemctl restart containerd.service

luyajin · 2023-07-24T08:00:42Z

我修改了这个配置怎么一直都不生效呢

luyajin · 2023-07-24T08:00:52Z

sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

linonetwo · 2023-11-14T07:35:25Z

Is this a bug? i'm using docker, not containerd , how to fix it on user side?

chendave · 2023-11-14T08:31:41Z

Is this a bug? i'm using docker, not containerd , how to fix it on user side?

this is not a bug, it's just a matter of configuration, shouldn't docker use containerd as the runtime by default?

neolit123 · 2023-11-14T09:02:36Z

Is this a bug? i'm using docker, not containerd , how to fix it on user side?

k8s maintainers recommend to move to containerd or cri-o.
docker (via cri-dockerd) is no longer well supported.

kingwingfly · 2023-12-02T23:58:04Z

systemctl restart containerd.service

义父！！！！！！！！！！！！！！！！！

zhkmxx9302013 · 2024-02-13T15:18:22Z

我也遇到了和你类似的问题，我的解决方案如下：
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
接着编辑/etc/containerd/config.toml文件，把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可，如下方代码所示（pause版本因人而异）：
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"
接着重启服务，重新init即可：
systemctl restart containerd.service

困扰了半个月...换了一堆源，原来在这里。。

jiangxiaoqiang · 2024-02-19T07:49:53Z

我也遇到了和你类似的问题，我的解决方案如下：
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
接着编辑/etc/containerd/config.toml文件，把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可，如下方代码所示（pause版本因人而异）：
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"
接着重启服务，重新init即可：
systemctl restart containerd.service

我也是这个问题，不过是在集群运行过程中无法拉取。修改了配置不生效不知道是什么原因。

Kensleycc · 2024-08-28T09:42:36Z

我也是这个问题，不过是在集群运行过程中无法拉取。修改了配置不生效不知道是什么原因。

给后来的朋友记录一下我的情况

【环境】

使用cri为cri-dockerd，已经修改/usr/lib/systemd/system/cri-docker.service

cat /usr/lib/systemd/system/cri-docker.service
...
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=<Repository Path>/pause:3.8

【问题】

masterkubeadm init没问题，worker kubeadm join 之后，对应的 kube-proxy pod 一直处于FailedCreatePodSandBox .

Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               6m33s                 default-scheduler  Successfully assigned kube-system/kube-proxy-h2k9q to kendev2
  Warning  FailedCreatePodSandBox  80s (x26 over 6m42s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "registry.k8s.io/pause:3.9": Error response from daemon: Get "https://registry.k8s.io/v2/": dial tcp 34.96.108.209:443: connect: connection refused

【Solution】

最后定位原因是worker未重启cri-dockerd进程使配置生效:

# systemctl status cri-docker.service 

● cri-docker.service - CRI Interface for Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-08-27 11:44:00 UTC; 21h ago
TriggeredBy: ● cri-docker.socket
       Docs: https://docs.mirantis.com
   Main PID: 103962 (cri-dockerd)
      Tasks: 11
     Memory: 18.6M
     CGroup: /system.slice/cri-docker.service
             └─103962 /usr/bin/cri-dockerd --container-runtime-endpoint fd://

重启后worker重新join即可:

systemctl restart cri-docker.service

systemctl status cri-docker.service  
● cri-docker.service - CRI Interface for Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-08-28 09:19:19 UTC; 2s ago
TriggeredBy: ● cri-docker.socket
       Docs: https://docs.mirantis.com
   Main PID: 2045345 (cri-dockerd)
      Tasks: 8
     Memory: 10.5M
     CGroup: /system.slice/cri-docker.service
             └─2045345 /usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=<Repository Path>/pause:3.8

neolit123 · 2024-08-28T10:25:23Z

you probably need to tell cri-dockerd to use exactly that image.
maybe there is a cri-dockerd config for that. try asking the cri-docker maintainers.

WQL782795 · 2024-10-11T02:35:03Z

我也遇到了和你类似的问题，我的解决方案如下：
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
接着编辑/etc/containerd/config.toml文件，把[plugins."io.containerd.grpc.v1.cri"]块下的sandbox_image值改为国内镜像即可，如下方代码所示（pause版本因人而异）：
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"
接着重启服务，重新init即可：
systemctl restart containerd.service

义父+1！！！

neolit123 closed this as completed Mar 29, 2023

k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Mar 29, 2023

qiref mentioned this issue Sep 21, 2023

failed to pull image when images already exists in local #2936

Closed

zhaozhong mentioned this issue Dec 12, 2023

v1.28 国内 kubeadm init 失败的问题 xiaopeng163/learn-k8s-from-scratch#22

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

registry.k8s.io/pause:3.6 does not use the address specified by --image-repository when executing kubeadm init #2851

registry.k8s.io/pause:3.6 does not use the address specified by --image-repository when executing kubeadm init #2851

Max-Qiu commented Mar 29, 2023

pacoxu commented Mar 29, 2023 •

edited

Loading

neolit123 commented Mar 29, 2023

LxxxSec commented May 5, 2023

luyajin commented Jul 24, 2023

luyajin commented Jul 24, 2023

linonetwo commented Nov 14, 2023

chendave commented Nov 14, 2023

neolit123 commented Nov 14, 2023

kingwingfly commented Dec 2, 2023

zhkmxx9302013 commented Feb 13, 2024

jiangxiaoqiang commented Feb 19, 2024

Kensleycc commented Aug 28, 2024 •

edited

Loading

neolit123 commented Aug 28, 2024

WQL782795 commented Oct 11, 2024

registry.k8s.io/pause:3.6 does not use the address specified by --image-repository when executing kubeadm init #2851

registry.k8s.io/pause:3.6 does not use the address specified by --image-repository when executing kubeadm init #2851

Comments

Max-Qiu commented Mar 29, 2023

Is this a BUG REPORT or FEATURE REQUEST?

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Kubernetes version

OS version

pacoxu commented Mar 29, 2023 • edited Loading

neolit123 commented Mar 29, 2023

LxxxSec commented May 5, 2023

luyajin commented Jul 24, 2023

luyajin commented Jul 24, 2023

linonetwo commented Nov 14, 2023

chendave commented Nov 14, 2023

neolit123 commented Nov 14, 2023

kingwingfly commented Dec 2, 2023

zhkmxx9302013 commented Feb 13, 2024

jiangxiaoqiang commented Feb 19, 2024

Kensleycc commented Aug 28, 2024 • edited Loading

neolit123 commented Aug 28, 2024

WQL782795 commented Oct 11, 2024

pacoxu commented Mar 29, 2023 •

edited

Loading

Kensleycc commented Aug 28, 2024 •

edited

Loading