Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container / #1025

leoyer · 2018-07-27T07:04:21Z

OS : Ubuntu 17.10
CPU: Intel(R) Core(TM) i5-6400 CPU @ 2.70GHz
architecture x64
Docker version 17.03.0-ce
sudo systemctl status kubelet output

   kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadmin.conf
   Active: active (running) since Fri 2018-07-27 09:22:33 CST; 10min ago
     Docs: http://kubernetes.io/docs/
 Main PID: 26082 (kubelet)
    Tasks: 20 (limit: 4915)
   Memory: 39.8M
      CPU: 10.489s
   CGroup: /system.slice/kubelet.service
           └─26082 /usr/bin/kubelet
7月 27 09:31:05 leoyer kubelet[26082]: I0727 09:31:05.385977   26082 kubelet_node_status.go:269] Setti
7月 27 09:31:15 leoyer kubelet[26082]: I0727 09:31:15.403479   26082 kubelet_node_status.go:269] Setti
7月 27 09:31:25 leoyer kubelet[26082]: I0727 09:31:25.419190   26082 kubelet_node_status.go:269] Setti
7月 27 09:31:35 leoyer kubelet[26082]: I0727 09:31:35.435863   26082 kubelet_node_status.go:269] Setti
7月 27 09:31:45 leoyer kubelet[26082]: I0727 09:31:45.453341   26082 kubelet_node_status.go:269] Setti
7月 27 09:31:55 leoyer kubelet[26082]: I0727 09:31:55.475820   26082 kubelet_node_status.go:269] Setti
7月 27 09:32:05 leoyer kubelet[26082]: I0727 09:32:05.493034   26082 kubelet_node_status.go:269] Setti
7月 27 09:32:15 leoyer kubelet[26082]: I0727 09:32:15.510566   26082 kubelet_node_status.go:269] Setti
7月 27 09:32:25 leoyer kubelet[26082]: I0727 09:32:25.535113   26082 kubelet_node_status.go:269] Setti
7月 27 09:32:35 leoyer kubelet[26082]: I0727 09:32:35.551342   26082 kubelet_node_status.go:269] Setti

and i hava the internet ,
docker images

REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
etcd-integration-test                      latest              2317207e4579        8 days ago          931 MB
k8s.gcr.io/etcd-amd64                      3.2.18-0            062f5eb5dde8        8 days ago          219 MB
k8s.gcr.io/kube-proxy-amd64                v1.11.1             d5c25579d0ff        9 days ago          97.8 MB
k8s.gcr.io/kube-controller-manager-amd64   v1.11.1             52096ee87d0e        9 days ago          155 MB
k8s.gcr.io/kube-apiserver-amd64            v1.11.1             816332bd9d11        9 days ago          187 MB
k8s.gcr.io/kube-scheduler-amd64            v1.11.1             272b3a60cd68        9 days ago          56.8 MB
busybox                                    latest              22c2dd5ee85d        10 days ago         1.16 MB
k8s.gcr.io/coredns                         1.1.3               b3b94275d97c        2 months ago        45.6 MB
k8s.gcr.io/etcd-amd64                      3.2.18              b8df3b177be2        3 months ago        219 MB
k8s.gcr.io/debian-hyperkube-base-amd64     0.10                7812d248bfc9        4 months ago        398 MB
golang                                     1.8.7               0d283eb41a92        5 months ago        713 MB
k8s.gcr.io/pause                           3.1                 da86e6ba6ca1        7 months ago        742 kB
k8s.gcr.io/pause                           latest              350b164e7ae1        4 years ago         240 kB

kubeadm config images pull --kubernetes-version=1.11.1

[config/images] Pulled k8s.gcr.io/kube-apiserver-amd64:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-controller-manager-amd64:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-scheduler-amd64:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-proxy-amd64:v1.11.1
[config/images] Pulled k8s.gcr.io/pause:3.1
[config/images] Pulled k8s.gcr.io/etcd-amd64:3.2.18
[config/images] Pulled k8s.gcr.io/coredns:1.1.3

and i just use kubeadm init
now the kubelet service active but no container be run
ps aux | grep kubelet

root      4783  1.8  1.0 1472896 83672 ?       Ssl  11:11   0:08 /usr/bin/kubelet
root      6216  0.0  0.0  16104   972 pts/1    S+   11:18   0:00 grep --color=auto kubelet

docker ps -a

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

journalctl -u kubelet | grep -i garbage

7月 27 11:11:20 leoyer kubelet[4783]: E0727 11:11:20.750974    4783 kubelet.go:1261] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
7月 27 11:22:47 leoyer kubelet[6701]: E0727 11:22:47.854424    6701 kubelet.go:1261] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
7月 27 14:18:41 leoyer kubelet[18984]: E0727 14:18:41.768492   18984 kubelet.go:1261] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
7月 27 14:47:31 leoyer kubelet[20950]: E0727 14:47:31.229252   20950 kubelet.go:1261] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /

I don't know why it looks like this. Everything looks normal.

chuckha · 2018-08-07T21:35:46Z

I think it's ok to ignore that error. Are those messages from before kubeadm init is run? Does the error persist after kubeadm init succeeds?

fanux · 2018-08-20T08:32:25Z

E0820 16:11:20.882842   34288 kubelet.go:1261] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
F0820 16:11:21.883008   34288 server.go:262] failed to run Kubelet: runonce failed: no pod manifest update after 1s

kubelet runonce can't start static pod..

jasonwangm · 2019-04-10T07:33:54Z

I also met with the same issue: "Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /"

lancedang · 2019-04-16T14:14:13Z

does this problem have serious influence？I met this today and some other problem which together lead to that I can not stop certain container and some pod hang on Terminating status. The following is the scene

E0416 20:07:05.161512   40706 kubelet.go:1261] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
E0416 20:07:09.815915   40706 remote_runtime.go:278] ContainerStatus "937e8efe208a616de87513c0f1273f0c5d143c2d4f6b883549a71d8cfb52fd31" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 937e8efe208a616de87513c0f1273f0c5d143c2d4f6b883549a71d8cfb52fd31
E0416 20:07:09.816512   40706 remote_runtime.go:278] ContainerStatus "e035d27d15a3a62ed4ffead8b0629427323534b2774f6231044078332309f352" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: e035d27d15a3a62ed4ffead8b0629427323534b2774f6231044078332309f352
E0416 20:07:18.108922   40706 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to start sandbox container for pod "stargate.1000000175.s750.g3.1": Error response from daemon: grpc: the connection is unavailable
E0416 20:07:18.108992   40706 kuberuntime_sandbox.go:56] CreatePodSandbox for pod "stargate.1000000175.s750.g3.1_default-mobileloan(6b8f2228-6038-11e9-a2b5-faa5219b9900)" failed: rpc error: code = Unknown desc = failed to start sandbox container for pod "stargate.1000000175.s750.g3.1": Error response from daemon: grpc: the connection is unavailable

sbezverk · 2020-04-01T12:30:15Z

Have the same issue and the node where it happens went into NotReady state:

Image garbage collection failed multiple times in a row: failed to get imageFs info: unable to find data in memory cache

Running 1.18.0

dhawal55 · 2020-04-03T03:45:09Z

I see the same error followed by skipping pod synchronization - PLEG is not healthy

neolit123 · 2020-04-03T13:46:06Z

you might want to search for the same error message reported in the kubernetes/kubernetes repository and if there is no ticket for that - create one and mark it with /sig node.

yuripastushenko · 2020-04-07T13:40:18Z

We have the same error.
Images are not garbage collected from node.

takakawa · 2020-04-16T02:53:46Z

I have the same error.kubeadm init failed because of a kubelet failure.

Kubernetes v1.18.1
Docker: 19.03.8

mapmiss · 2021-07-20T13:19:46Z

我有同样的错误。由于 kubelet 失败，kubeadm init 失败。

Kubernetes v1.18.1
Docker：19.03.8

Has your problem been solved? I have the same problem.

32328254 · 2023-07-12T06:33:09Z

i have same error

huanggze · 2023-11-18T14:37:04Z

hi, ppl! do you find a solution for this? thanks

huanggze · 2023-11-18T15:27:22Z

Finally, I solved it by re-installing containerd

neolit123 added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jul 27, 2018

leoyer closed this as completed Aug 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container / #1025

Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container / #1025

leoyer commented Jul 27, 2018

chuckha commented Aug 7, 2018

fanux commented Aug 20, 2018

jasonwangm commented Apr 10, 2019

lancedang commented Apr 16, 2019 •

edited

Loading

sbezverk commented Apr 1, 2020

dhawal55 commented Apr 3, 2020

neolit123 commented Apr 3, 2020 •

edited

Loading

yuripastushenko commented Apr 7, 2020

takakawa commented Apr 16, 2020 •

edited

Loading

mapmiss commented Jul 20, 2021

32328254 commented Jul 12, 2023

huanggze commented Nov 18, 2023

huanggze commented Nov 18, 2023

Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container / #1025

Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container / #1025

Comments

leoyer commented Jul 27, 2018

chuckha commented Aug 7, 2018

fanux commented Aug 20, 2018

jasonwangm commented Apr 10, 2019

lancedang commented Apr 16, 2019 • edited Loading

sbezverk commented Apr 1, 2020

dhawal55 commented Apr 3, 2020

neolit123 commented Apr 3, 2020 • edited Loading

yuripastushenko commented Apr 7, 2020

takakawa commented Apr 16, 2020 • edited Loading

mapmiss commented Jul 20, 2021

32328254 commented Jul 12, 2023

huanggze commented Nov 18, 2023

huanggze commented Nov 18, 2023

lancedang commented Apr 16, 2019 •

edited

Loading

neolit123 commented Apr 3, 2020 •

edited

Loading

takakawa commented Apr 16, 2020 •

edited

Loading