Can't pull container. failed to register layer: lsetxattr user.overlay.impure /etc: operation not supported

[Gabgobie]

Title pretty much says everything. As it seems this issue is with the container.

They seem to have found a solution here

Other containers pull fine for me.

[dfandrich]

What command-line are you running that gives you this error?

[Gabgobie]

I'm using Bash inside an Ubuntu 23.04 LXC. I uninstalled all native docker tools and installed the official engine according to the docker installation guide

[dfandrich]

Please tell us what you typed in bash before seeing that error.

[Gabgobie]

docker pull <image>

root@docker-temp:~# docker pull curlimages/curl
Using default tag: latest
latest: Pulling from curlimages/curl
4abcf2066143: Already exists
1113023ab841: Extracting [==================================================>]  4.308MB/4.308MB
4ca545ee6d5d: Download complete
failed to register layer: lsetxattr user.overlay.impure /etc: operation not supported
root@docker-temp:~# docker pull quay.io/curl/curl:latest
latest: Pulling from curl/curl
b6f1a98f98c5: Already exists
1113023ab841: Extracting [==================================================>]  4.308MB/4.308MB
4ca545ee6d5d: Download complete
failed to register layer: lsetxattr user.overlay.impure /etc: operation not supported
root@docker-temp:~#

[dfandrich]

Are you running Docker inside a Docker container? I could see that causing problems. In any case, this looks like a local configuration problem to me. It works fine on my system: $ docker pull curlimages/curl Using default tag: latest latest: Pulling from curlimages/curl Digest: sha256:25d29daeb9b14b89e2fa8cc17c70e4b188bca1466086907c2d9a4b56b59d8e21 Status: Image is up to date for curlimages/curl:latest docker.io/curlimages/curl:latest $ docker pull quay.io/curl/curl:latest latest: Pulling from curl/curl Digest: sha256:1e136d717aab088c1f157ff816cf4219fe457a1687f11580e6a127488954f67f Status: Downloaded newer image for quay.io/curl/curl:latest quay.io/curl/curl:latest

[Gabgobie]

Hmm interesting. I didn't touch a thing about the container.

It's a LXC Ubuntu 23.04 on Proxmox VE. I changed nothing besides running an apt update && apt full-upgrade -y and following the docker engine install guide to get a clean system -> using the docker.com repo.

Which version of the docker engine are you running? I'm on 25.0.2

root@docker-temp:~# docker -v
Docker version 25.0.2, build 29cf629

The link I provided in my first message suggests that it's an issue of Docker v25 breaking something and the fix is on the container side.

[Gabgobie]

Btw thanks a lot for your quick help

[Gabgobie]

https://docs.docker.com/engine/release-notes/25.0/#extended-attributes-for-tar-files

Do you use tar archives that were created on MacOS? The docker release notes seem to blame the MacOS tar implementation

[xquery]

cannot repro ... you might also try running podman

[BrandonStudio]

Same issue.
I'm on Ubuntu 22.04 with rootless docker.
It works fine when I pull 7.88.1 but fails when I pull latest.
I have not tested other version.

[Furisto]

I see the same issue. The latest release that works is 8.1.1. Every release that I have tested after that has this problem.

[xquery]

are you (@Furisto ) also on ubuntu ?

[aquarta]

I see the same issue, I'm using a brand new workspace in gitpod
Linux fiware-tutorialsiotsens-r1iaycrgx85 6.1.75-060175-generic #202402010024 SMP PREEMPT_DYNAMIC Thu Feb 1 01:44:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"

[xquery]

so ubuntu it is then, will take a deeper look

[aquarta]

Thanks, I forgot to report the docker version

docker --version
Docker version 26.0.1, build d260a54

The pull is working fine with 7.88.1 version

[xquery]

This issue is between docker (some version) and ubuntu (some version) ... we have not made any changes in terms of the container itself and more about how buildah does things ... the problem is that latest docker no longer ignores a certain class of errors and we need to rebuild using an updated version of buildah (containers/buildah@662908f).

In the short term, one can rollback to some older version of docker or use podman ... will refresh the buildah toolchain and arrange a new release which will hopefully address.

[Gabgobie]

Thanks a lot for taking the time to look into it. Feel free to ping me so I can tell you if it's fixed when you're ready

[xquery]

does the latest release fix things for you ?

[BrandonStudio]

$ docker pull curlimages/curl:latest
latest: Pulling from curlimages/curl
4abcf2066143: Pull complete
e52d86b3391d: Extracting [==================================================>]  7.111MB/7.111MB
4ca545ee6d5d: Download complete
failed to register layer: lsetxattr user.overlay.origin /bin: operation not supported

It seems something has changed? Is that a good news?

[chadlwilson]

Is there any reason to believe this would be a problem specifically with buildx / buildkit?

I have issues with the curl image inside a multi-stage Dockerfile - but only since buildkit 0.17.0 released a few days ago (i.e 0.16.0 is OK) (edit: the problem seemed to start when the rootless image indirectly started using fuse-overlayfs 0.14 instead of 0.13)

Running within a dind environment with rootless buildkit builder, host kernel Linux 5.10.219-208.866.amzn2.x86_64 amd64.

[chadlwilson]

In any case, there seems to be still be a problem here when using the curl image alongside rootless dind to build images (e.g ones using the curl image in a multi-stage build file) in particular when using fuse-overlayfs with buildx/buildkit.

It seems right now we'd need the curl image to be built with buildah 1.35+ to avoid this problem as it appears no fix has been backported to older buildah versions. Currently this project is using 1.23.1 off Ubuntu Jammy repositories. Even Ubuntu 24.04 Noble doesn't have a fixed buildah I think, as the issue hasn't been backported to the 1.33.x line.

Other than building an arbitrary buildah version from source to use to then build curl images, I've tried seeing if a backport can be done at containers/storage#2166 (which could then be pulled into buildah 1.33.x with a patch update of the storage libraries)