With containerd, I pulled images debian, ubuntu, nginx all latest, using --platform=linux/arm64/v8 and multiple strange observations.

[chetanshivaji]

Description

docker pull --platform=linux/arm64/v8 nginx:latest && docker pull --platform=linux/arm64/v8 debian:latest && docker pull --platform=linux/arm64/v8 ubuntu:latest
------------------------------------------------------------------------------------------------------------------
[root@localhost tryNested]# docker images
REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
nginx        latest    d2eb56950b84   6 days ago     278MB
debian       latest    27586f460943   12 days ago    383MB
ubuntu       latest    b359f1067efa   54 years ago   28.9MB
------------------------------------------------------------------------------------------------------------------
[root@localhost tryNested]# docker images --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE                   ID             DISK USAGE   CONTENT SIZE   USED
nginx:latest            d2eb56950b84        278MB         69.6MB       
├**─ linux/arm64/v8       96c43ba31637        278MB         69.6MB**       
├─ linux/amd64          396c6e925f28           0B             0B       
├─ linux/arm/v5         583c1fe635d5           0B             0B       
├─ linux/arm/v7         0c07ab67bdea           0B             0B       
├─ linux/386            1608b0773886           0B             0B       
├─ linux/mips64le       767d520bfd3e           0B             0B       
├─ linux/ppc64le        5d944fb59cfb           0B             0B       
└─ linux/s390x          708aae785214           0B             0B       

debian:latest           27586f460943        383MB         99.1MB    ✔  
**├─ linux/arm64/v8       4466c6813e1d        203MB         49.6MB       
├─ linux/amd64          e225d70fafe8        180MB         49.6MB    ✔**  
├─ linux/arm/v5         baabbe93525e           0B             0B       
├─ linux/arm/v7         926de0d2617f           0B             0B       
├─ linux/386            f0d9b867cc9d           0B             0B       
├─ linux/mips64le       a198b97962c6           0B             0B       
├─ linux/ppc64le        47585ee2d5a4           0B             0B       
└─ linux/s390x          5245f9d6107f           0B             0B       

**ubuntu:latest           b359f1067efa       28.9MB             0B**   

------------------------------------------------------------------------------------------------------------------

[root@localhost tryNested]# docker inspect d2eb56950b84|grep -ani arch
54:        "Architecture": "arm64",
[root@localhost tryNested]# docker inspect 27586f460943|grep -ani arch
38:        "Architecture": "amd64",
[root@localhost tryNested]# docker inspect b359f1067efa|grep -ani arch
41:        "Architecture": "arm64",
------------------------------------------------------------------------------------------------------------------

Questions
1. why ubuntu shows nothing in --tree command.
2. why arch of debian shows amd64?
4. why for debian both arch got pulled i.e. arm64 and amd64 but for nginx and ubuntu only linux/arm64/v8 got pulled?

Reproduce

.

Expected behavior

Questions
**1. why ubuntu shows nothing in --tree command.
Should show it downloaded.

2. why arch of debian shows amd64?
   It should show arm64 because we pulled arm64.

3. why for debian both arch got pulled i.e. arm64 and amd64 but for nginx and ubuntu only linux/arm64/v8 got pulled?**
   it should output in same way for all images, why it's different for different images.?

docker version

[root@localhost tryNested]# docker version
Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:39:44 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:02 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[root@localhost tryNested]# 

docker info

[root@localhost tryNested]# docker info
Client:
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 0
  Paused: 0
  Stopped: 5
 Images: 3
 Server Version: 27.3.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.4.17-2136.323.8.2.el7uek.x86_64
 Operating System: Oracle Linux Server 7.9
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.544GiB
 Name: localhost.localdomain
 ID: 6fdc46df-4874-45fd-94bd-7b4a1295ffc0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTPS Proxy: http://10.115.27.54:3128/
 No Proxy: localhost,127.0.0.1,localaddress,.localdomain.com
 Username: [email protected]
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[root@localhost tryNested]# 

Additional Info

I see same behavior on docker desktop

[chetanshivaji]

@thaJeztah do you see this same on your setup.?

[chetanshivaji]

Question 4. why we see 0B at multiple places

#docker pull debian:latest

#docker images  --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE                   ID             DISK USAGE   CONTENT SIZE   USED
debian:latest           27586f460943        232MB         99.1MB    Γ£ö
Γö£ΓöÇ linux/arm64/v8       4466c6813e1d       49.6MB         49.6MB
Γö£ΓöÇ linux/amd64          e225d70fafe8        182MB         49.6MB    Γ£ö
Γö£ΓöÇ linux/arm/v5         baabbe93525e           **0B             0B**
Γö£ΓöÇ linux/arm/v7         926de0d2617f           **0B             0B**
Γö£ΓöÇ linux/386            f0d9b867cc9d           **0B             0B**
Γö£ΓöÇ linux/mips64le       a198b97962c6           0B             0B
Γö£ΓöÇ linux/ppc64le        47585ee2d5a4           0B             0B
ΓööΓöÇ linux/s390x          5245f9d6107f           0B             0B

[thaJeztah]

I tried reproducing this issue, but didn't get the same problem;

docker pull --platform=linux/arm64/v8 nginx:latest && docker pull --platform=linux/arm64/v8 debian:latest && docker pull --platform=linux/arm64/v8 ubuntu:latest

latest: Pulling from library/nginx
14c9d9d19932: Download complete
c62fa0108604: Download complete
783efa720071: Download complete
66b98279bfb9: Download complete
e9346e20ffd2: Download complete
e0917afa5d8b: Download complete
e905ecdf6023: Download complete
Digest: sha256:d2eb56950b84efe34f966a2b92efb1a1a2ea53e7e93b94cdf45a27cf3cd47fc0
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

latest: Pulling from library/debian
6d11c181ebb3: Download complete
Digest: sha256:27586f4609433f2f49a9157405b473c62c3cb28a581c413393975b4e8496d0ab
Status: Downloaded newer image for debian:latest
docker.io/library/debian:latest

latest: Pulling from library/ubuntu
25a614108e8d: Download complete
Digest: sha256:b359f1067efa76f37863778f7b6d0e8d911e3ee8efa807ad01fbf5dc1ef9006b
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

docker image ls --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE                   ID             DISK USAGE   CONTENT SIZE   USED
nginx:latest            d2eb56950b84        281MB         69.6MB
├─ linux/arm64/v8       96c43ba31637        281MB         69.6MB
├─ linux/amd64          396c6e925f28           0B             0B
├─ linux/arm/v5         583c1fe635d5           0B             0B
├─ linux/arm/v7         0c07ab67bdea           0B             0B
├─ linux/386            1608b0773886           0B             0B
├─ linux/mips64le       767d520bfd3e           0B             0B
├─ linux/ppc64le        5d944fb59cfb           0B             0B
└─ linux/s390x          708aae785214           0B             0B

debian:latest           27586f460943        205MB         49.6MB
├─ linux/arm64/v8       4466c6813e1d        205MB         49.6MB
├─ linux/amd64          e225d70fafe8           0B             0B
├─ linux/arm/v5         baabbe93525e           0B             0B
├─ linux/arm/v7         926de0d2617f           0B             0B
├─ linux/386            f0d9b867cc9d           0B             0B
├─ linux/mips64le       a198b97962c6           0B             0B
├─ linux/ppc64le        47585ee2d5a4           0B             0B
└─ linux/s390x          5245f9d6107f           0B             0B

ubuntu:latest           b359f1067efa        139MB         28.9MB
├─ linux/arm64/v8       1d392dca9b70        139MB         28.9MB
├─ linux/amd64          74f92a6b3589           0B             0B
├─ linux/arm/v7         4192b2c3b431           0B             0B
├─ linux/ppc64le        294858195461           0B             0B
├─ linux/riscv64        08d0f41b17da           0B             0B
└─ linux/s390x          48459f9f4d0b           0B             0B

1. why ubuntu shows nothing in --tree command.

Not sure about that one; did an error show up during the pull? Is there information in the daemon logs that may be relevant?

I do notice some things in your setup;

• It looks like you're running on Oracle Linux; we don't have official packages for that Linux distribution; while it's possible packages built for another distro "work", it's definitely not tested.
• I see you have a proxy configured; this could be irrelevant, but does add an extra moving part where things could go wrong.

Was this the first time the ubuntu:latest image was pulled, or was the image already present before you pulled? (possibly for a different architecture)? Trying to narrow-down what could've happened (i.e., pull that appends a platform to an image that's already in the store vs a fully new multi-platform image being pulled).

2. why arch of debian shows amd64?

That is currently a limitation; various commands were designed before multi-platform images existed, and only handle a single architecture. Currently that means that (e.g.) inspect defaults to showing information for a single architecture; if multiple platforms are pulled, then it picks the "best matching" platform, which in your case is the daemon's native platform (linux/amd64).

Work is in progress to improve this;

• c8d: Add --platform flag to history, save and load #5331
• c8d/inspect: Add Manifests field moby/moby#48264

3. why for debian both arch got pulled i.e. arm64 and amd64 but for nginx and ubuntu only linux/arm64/v8 got pulled?

In both cases, the platform that was specified for the pull (linux/arm64/v8) was pulled. For the debian:latest image, you likely already had the linuxx/amd64 variant pulled before you did this pull. In that case, if the image index (the "multi-platform" image, which is the top-level "wrapper" around the platform variants) was still up-to-date, the existing variant is kept, and the newly pulled variant is added in the local image store.

Multi-platform images are effectively a "bundle" of multiple variants of an image. When you pull the image, the registry resolves the name of the image to the wrapper, after which the Docker Engine pulls the platform variant(s) of the image (by default it pulls the daemon's "native" platform/architecture). By passing the --platform flag, you can override that default, and ask it to pull an image-variant that may not match the daemon's native architecture.

Question 4. why we see 0B at multiple places

This is to indicate that the given variant of the image is not present in the daemon's image store.

For example, in the output below, the debian:latest image on Docker Hub provides 8 platform variants, but only the linux/arm64/v8 was pulled (so only that variant is present in the daemon's image store). The other variants were not pulled, so show as 0B in the output.

IMAGE                   ID             DISK USAGE   CONTENT SIZE   USED
debian:latest           27586f460943        205MB         49.6MB
├─ linux/arm64/v8       4466c6813e1d        205MB         49.6MB
├─ linux/amd64          e225d70fafe8           0B             0B
├─ linux/arm/v5         baabbe93525e           0B             0B
├─ linux/arm/v7         926de0d2617f           0B             0B
├─ linux/386            f0d9b867cc9d           0B             0B
├─ linux/mips64le       a198b97962c6           0B             0B
├─ linux/ppc64le        47585ee2d5a4           0B             0B
└─ linux/s390x          5245f9d6107f           0B             0B

After pulling another variant (I pulled linux/s390x), that variant will now also show up with the size consumed in the daemon's image store;

IMAGE                   ID             DISK USAGE   CONTENT SIZE   USED
debian:latest           27586f460943        383MB         97.5MB
├─ linux/s390x          5245f9d6107f        178MB         47.9MB
├─ linux/arm64/v8       4466c6813e1d        205MB         49.6MB
├─ linux/amd64          e225d70fafe8           0B             0B
├─ linux/arm/v5         baabbe93525e           0B             0B
├─ linux/arm/v7         926de0d2617f           0B             0B
├─ linux/386            f0d9b867cc9d           0B             0B
├─ linux/mips64le       a198b97962c6           0B             0B
└─ linux/ppc64le        47585ee2d5a4           0B             0B

The output currently shows size that's consumed locally, as that's been the information that's been relevant for most users (how much space does this image occupy on my machine?).

We may be able to show the size of variants in the registry (i.e., "how much data would be downloaded if I pull this variant"), which may be relevant for some use-cases. There's some limitations to that because the registry only holds information about the compressed size of images ("content size"), but not the size used after extracting ("disk usage").

These are things that are still being discussed though; our goal is to provide sensible defaults (information most users are looking for), which may not always be all information. Some additional information may be added as an optional / customisable option, but also through other commands. For example, we're looking at providing a docker image info command as alternative to docker image inspect, which can provide more in-depth information about an image, but in a human-readable (non-JSON) output.

☝️ feedback on the above is welcome; we're still designing some of these bits (which is also why the "WARNING: This is an experimental feature. The output may change and shouldn't be depended on." is shown).

[chetanshivaji]

@thaJeztah is there a way to pick up arch specific image using imageId and not --platform flag while doing docker save?
eg. for this image
debian:latest 27586f460943 383MB 99.1MB ✔
├─ linux/arm64/v8 4466c6813e1d 203MB 49.6MB
├─ linux/amd64 e225d70fafe8 180MB 49.6MB ✔

Is there a way to docker save using image id as below,
docker save 4466c6813e1d -o debianArm.tar
docker save e225d70fafe8 -o debianAmd64.tar

I know I can do,
docker save -o 27586f460943 -o debian.tar

[chetanshivaji]

@thaJeztah Thanks for the details above, I agree to your points, and I tried steps I mentioned in fresh Ubuntu and seems no issue.
As I have been experimenting with building multiplatform images, and pulling them from docker hub, can't remember sequence of pulling deleting of images and archs. So no track record of it.

I understand --platform feature is on they way to be released in 28.0.0
but it is not possible to save arch specific image with default containerd overlayfs coming as default enabled on Docker Desktop.
It is like overlayfs is coming default, but saving arch specific image code is yet to be released.

No other solution than building image one by one platform specific and do docker save. or wait for 28.0.0 moby.

[thaJeztah]

Is there a way to docker save using image id as below,

I need to check up on that; I agree that is something that ideally would be supported. It's possible that some additional logic is needed on the daemon side to handle resolving variants based on that). Currently, resolving images by their ID is only done for the top-level image. If no match is found, it falls back to assuming it's an image reference;

docker image save -o foo.tar 5245f9d6107f
Error response from daemon: No such image: 5245f9d6107f:latest

I did a quick check as well if it would do better with the full digest, but doesn't look like it either;

docker image save -o foo.tar 5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b
Error response from daemon: No such image: sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b

docker image save -o foo.tar sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b
Error response from daemon: No such image: sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b

docker image save -o foo.tar docker.io/library/debian@sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b
Error response from daemon: No such image: debian@sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b

docker image save -o foo.tar sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b
Error response from daemon: No such image: sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b

I should probably check if there's a tracking ticket for that already, and if not, create one.

[thaJeztah]

No other solution than building image one by one platform specific and do docker save. or wait for 28.0.0 moby.

I'm curious if there's a specific reason you need to save each variant separately; it should still be possible to save the image as a whole (multiple architectures), and load those into another engine (pushing to a registry is probably the more conventional way though).

It's worth noting that saving an individual variant of an image "dismantles" the multi-platform image, which may also discard attestations / SBOMS (if present). We currently don't show those in the tree view, but some discussion on that is happening, and a PoC pull request was opened;

• display presence of attestations in docker images --tree #5451