nerdctl push images fails with unsupported media type error

[nx2804]

Description

Hi Team,

i am using nerdctl tag and push images to the repository it fails with following error. Kindly suggest on this
FATA[0000] failed commit on ref "index-sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53": unexpected status from PUT request to http://serviceserver:88/v2/library/pause/manifests/3.9: 415 Unsupported Media Type

Steps to reproduce the issue

1. use nerdctl tag
2. use nerdctl push
3. it fails with unsupported media type error. instead if i use docker push it works fine

Describe the results you received and expected

1. use nerdctl tag
2. use nerdctl push
3. it fails with unsupported media type error. instead if i use docker push it works fine
   FATA[0000] failed commit on ref "index-sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53": unexpected status from PUT request to http://serviceserver:88/v2/library/pause/manifests/3.9: 415 Unsupported Media Type

What version of nerdctl are you using?

2.1

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

Hey @nx2804

Can you nerdctl info and nerdctl version?

Also, what registry is that? Harbor? Distribution? Artifactory?

Also, if you could rerun the command you are running with --debug-full.

[nx2804]

Both harbor and artifactory

…

On Fri, 24 Jan 2025 at 1:19 PM, Apostasie ***@***.***> wrote: Hey @nx2804 <https://github.com/nx2804> Can you nerdctl info and nerdctl version? Also, what registry is that? Harbor? Distribution? Artifactory? — Reply to this email directly, view it on GitHub <#3838 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCAEUMGXM7VW35KDEB6FFND2MHV7TAVCNFSM6AAAAABVZCJBROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJRHA3TMMBVGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[apostasie]

Hey @nx2804

Can you nerdctl info and nerdctl version?

Also, if you could rerun the command you are running with --debug-full.

Both of the above ^ would help.

Thanks.

[nx2804]

Client:
Namespace: k8s.io
Debug Mode: false

Server:
Server Version: v1.7.11
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.14.0-427.28.1.el9_4.x86_64
Operating System: Rocky Linux 9.4 (Blue Onyx)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 90.97GiB
Name: pps-k8-ctl-01
ID: 03d9a9d2-1bd2-444e-9af9-432008178e40

DEBU[0000] pushing digest="sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53" image="serviceserver:88/library/pause:3.9"
DEBU[0000] push digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=317616
DEBU[0000] checking and pushing to digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=317616 url="http://serviceserver:88/v2/library/pause/blobs/sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007"
DEBU[0000] do request digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, /" request.header.user-agent=containerd/1.7.10+unknown request.method=HEAD size=317616 url="http://serviceserver:88/v2/library/pause/blobs/sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007"
DEBU[0000] push digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" mediatype=application/vnd.docker.container.image.v1+json size=973
DEBU[0000] checking and pushing to digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" mediatype=application/vnd.docker.container.image.v1+json size=973 url="http://serviceserver:88/v2/library/pause/blobs/sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c"
DEBU[0000] do request digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, /" request.header.user-agent=containerd/1.7.10+unknown request.method=HEAD size=973 url="http://serviceserver:88/v2/library/pause/blobs/sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c"
DEBU[0000] fetch response received digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.connection=keep-alive response.header.content-length=156 response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 24 Jan 2025 09:14:57 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx response.header.set-cookie="sid=39796cc549c65bcae09ae8819cb83bc6; Path=/; HttpOnly" response.header.set-cookie.1="_xsrf=VGhHSTVMcW1ZVjVoQUhQNHBKcUduY2J5SFNCNTlTUDQ=|1737710097550744045|a39c292d322d97c7dc6fc0a199e4c827787f173d235b571d6932568592f5c7ac; Expires=Fri, 24 Jan 2025 10:14:57 UTC; Max-Age=3600; Path=/" response.header.www-authenticate="Bearer realm="http://serviceserver:88/service/token\",service=\"harbor-registry\",scope=\"repository:library/pause:pull\"" response.status="401 Unauthorized" size=317616 url="http://serviceserver:88/v2/library/pause/blobs/sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007"
DEBU[0000] Unauthorized digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" header="Bearer realm="http://serviceserver:88/service/token\",service=\"harbor-registry\",scope=\"repository:library/pause:pull\"" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=317616
DEBU[0000] do request digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, /" request.header.user-agent=containerd/1.7.10+unknown request.method=HEAD size=317616 url="http://serviceserver:88/v2/library/pause/blobs/sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007"
DEBU[0000] fetch response received digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" mediatype=application/vnd.docker.container.image.v1+json response.header.connection=keep-alive response.header.content-length=156 response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 24 Jan 2025 09:14:57 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx response.header.set-cookie="sid=ab0d819ba89e3da7d2c93f06eaab12c1; Path=/; HttpOnly" response.header.set-cookie.1="_xsrf=M2RESXZHWUxxVUF2MDc0WU1PTjJ1UlRNQ2tOejRnTEc=|1737710097551083269|d077b3b1195dd4283753d4aaa5d8860eab275822d6918bb83c066413207fd411; Expires=Fri, 24 Jan 2025 10:14:57 UTC; Max-Age=3600; Path=/" response.header.www-authenticate="Bearer realm="http://serviceserver:88/service/token\",service=\"harbor-registry\",scope=\"repository:library/pause:pull\"" response.status="401 Unauthorized" size=973 url="http://serviceserver:88/v2/library/pause/blobs/sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c"
DEBU[0000] Unauthorized digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" header="Bearer realm="http://serviceserver:88/service/token\",service=\"harbor-registry\",scope=\"repository:library/pause:pull\"" mediatype=application/vnd.docker.container.image.v1+json size=973
DEBU[0000] do request digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, /" request.header.user-agent=containerd/1.7.10+unknown request.method=HEAD size=973 url="http://serviceserver:88/v2/library/pause/blobs/sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c"
DEBU[0000] fetch response received digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=973 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Fri, 24 Jan 2025 09:14:57 GMT" response.header.docker-content-digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag=""sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c"" response.header.server=nginx response.header.set-cookie="sid=f8ad6648a1b813583a5844cb7eb03872; Path=/; HttpOnly" response.header.set-cookie.1="_xsrf=ZnQ2U0IwU0VSU1F4MVlVRWNOb0xqb3c3QVUxU0pTdUM=|1737710097572010039|b33dfe250e7c897bc6027f96cc75245963e75e78bfa117a2e0286ab3476afc35; Expires=Fri, 24 Jan 2025 10:14:57 UTC; Max-Age=3600; Path=/" response.header.x-frame-options=DENY response.status="200 OK" size=973 url="http://serviceserver:88/v2/library/pause/blobs/sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c"
DEBU[0000] fetch response received digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=317616 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/octet-stream response.header.date="Fri, 24 Jan 2025 09:14:57 GMT" response.header.docker-content-digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" response.header.docker-distribution-api-version=registry/2.0 response.header.etag=""sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007"" response.header.server=nginx response.header.set-cookie="sid=c57e1eba578d75358eb730b9a36b6a67; Path=/; HttpOnly" response.header.set-cookie.1="_xsrf=ZlViamRsUDNobjhlVE5GT2V4RXFMYTdIR1hYaUxURG0=|1737710097571947558|139d44c1cf61af267781260044e92974016d49a6239efb0ac6527f5efc7578f4; Expires=Fri, 24 Jan 2025 10:14:57 UTC; Max-Age=3600; Path=/" response.header.x-frame-options=DENY response.status="200 OK" size=317616 url="http://serviceserver:88/v2/library/pause/blobs/sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007"
DEBU[0000] push digest="sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10" mediatype=application/vnd.docker.distribution.manifest.v2+json size=526
DEBU[0000] checking and pushing to digest="sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10" mediatype=application/vnd.docker.distribution.manifest.v2+json size=526 url="http://serviceserver:88/v2/library/pause/manifests/sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10"
DEBU[0000] do request digest="sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, /" request.header.user-agent=containerd/1.7.10+unknown request.method=HEAD size=526 url="http://serviceserver:88/v2/library/pause/manifests/sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10"
DEBU[0000] fetch response received digest="sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.connection=keep-alive response.header.content-length=526 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Fri, 24 Jan 2025 09:14:57 GMT" response.header.docker-content-digest="sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10" response.header.docker-distribution-api-version=registry/2.0 response.header.etag=""sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10"" response.header.server=nginx response.header.set-cookie="sid=45030e158d51831e66f83fc6fd4a7328; Path=/; HttpOnly" response.header.set-cookie.1="_xsrf=MzZKUk9HWEsxc1dpSGlBWkdhVXQwTkZvb2NmcndYVW4=|1737710097577915504|20d6be9f357197a05405d93441965c695d5f5c2b773d85ba7779294ab1adae2d; Expires=Fri, 24 Jan 2025 10:14:57 UTC; Max-Age=3600; Path=/" response.header.x-frame-options=DENY response.status="200 OK" size=526 url="http://serviceserver:88/v2/library/pause/manifests/sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10"
DEBU[0000] push digest="sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=317
DEBU[0000] checking and pushing to digest="sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=317 url="http://serviceserver:88/v2/library/pause/manifests/3.9"
DEBU[0000] do request digest="sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, /" request.header.user-agent=containerd/1.7.10+unknown request.method=HEAD size=317 url="http://serviceserver:88/v2/library/pause/manifests/3.9"
DEBU[0000] fetch response received digest="sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.connection=keep-alive response.header.content-length=2817 response.header.content-security-policy="frame-ancestors 'none'" response.header.content-type=application/vnd.docker.distribution.manifest.v1+prettyjws response.header.date="Fri, 24 Jan 2025 09:14:57 GMT" response.header.docker-content-digest="sha256:876bd64e218370081d192782899a2ea47125e0a7b3805a23890db6fba82b1bc1" response.header.docker-distribution-api-version=registry/2.0 response.header.etag=""sha256:876bd64e218370081d192782899a2ea47125e0a7b3805a23890db6fba82b1bc1"" response.header.server=nginx response.header.set-cookie="sid=f15963058d758f769a4b8bc73f59dc53; Path=/; HttpOnly" response.header.set-cookie.1="_xsrf=Wm1vazhHQzNVNElaT0plbmFuRXJ1ejhsTFZPWkUybUQ=|1737710097581138514|f8c3b5aca0ef87227bcdf338ce380c0ef7399d66945bf93ce32f7da463073403; Expires=Fri, 24 Jan 2025 10:14:57 UTC; Max-Age=3600; Path=/" response.header.x-frame-options=DENY response.status="200 OK" size=317 url="http://serviceserver:88/v2/library/pause/manifests/3.9"
DEBU[0000] do request digest="sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json request.header.user-agent=containerd/1.7.10+unknown request.method=PUT size=317 url="http://serviceserver:88/v2/library/pause/manifests/3.9"
DEBU[0000] fetch response received digest="sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.connection=keep-alive response.header.content-length=134 response.header.content-type="text/plain; charset=utf-8" response.header.date="Fri, 24 Jan 2025 09:14:57 GMT" response.header.server=nginx response.header.set-cookie="sid=2c2dad83120a1070f035a432f095dc68; Path=/; HttpOnly" response.header.set-cookie.1="_xsrf=MlVlNzJVSEFBUHhmYTREcHYxd0c2MUlZOUJtbEZGN00=|1737710097593801565|a7bd20df5531788e079d3e3cc20e72a382b5055aab610a97e966ea59ca4170a3; Expires=Fri, 24 Jan 2025 10:14:57 UTC; Max-Age=3600; Path=/" response.header.x-content-type-options=nosniff response.status="415 Unsupported Media Type" size=317 url="http://serviceserver:88/v2/library/pause/manifests/3.9"
DEBU[0000] unexpected response body="{"errors":[{"code":"UNSUPPORTED_MEDIA_TYPE","message":"Manifest.list is not supported.","detail":"Manifest.list is not supported."}]}\n" digest="sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53" mediatype=application/vnd.docker.distribution.manifest.list.v2+json resp="&{415 Unsupported Media Type 415 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Length:[134] Content-Type:[text/plain; charset=utf-8] Date:[Fri, 24 Jan 2025 09:14:57 GMT] Server:[nginx] Set-Cookie:[sid=2c2dad83120a1070f035a432f095dc68; Path=/; HttpOnly _xsrf=MlVlNzJVSEFBUHhmYTREcHYxd0c2MUlZOUJtbEZGN00=|1737710097593801565|a7bd20df5531788e079d3e3cc20e72a382b5055aab610a97e966ea59ca4170a3; Expires=Fri, 24 Jan 2025 10:14:57 UTC; Max-Age=3600; Path=/] X-Content-Type-Options:[nosniff]] 0xc00072f400 134 [] false false map[] 0xc000978100 }" size=317
index-sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53: waiting |--------------------------------------|
manifest-sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10: waiting |--------------------------------------|
config-sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c: waiting |--------------------------------------|
elapsed: 0.1 s total: 0.0 B (0.0 B/s)
FATA[0000] failed commit on ref "index-sha256:25e2e7fb27c4444cee9cccc0643befd31dbcc14821ce595943bc10304da28c53": unexpected status from PUT request to http://serviceserver:88/v2/library/pause/manifests/3.9: 415 Unsupported Media Type
Client:
Namespace: k8s.io
Debug Mode: false

Server:
Server Version: v1.7.11
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.14.0-427.28.1.el9_4.x86_64
Operating System: Rocky Linux 9.4 (Blue Onyx)

[apostasie]

From a glance: Manifest.list is not supported - your registry does not support manifest list.
Is your registry version very outdated?

Alternatively, I would suggest you ask about that on your registry bug tracker.

Meanwhile, I guess you can just push single platform images instead.

[nx2804]

I tried pushing only single images

…

On Sat, 25 Jan 2025 at 1:59 AM, Apostasie ***@***.***> wrote: From a glance: Manifest.list is not supported - your registry does not support manifest list. Is your registry version very outdated? Alternatively, I would suggest you ask about that on your registry bug tracker. Meanwhile, I guess you can just push single platform images instead. — Reply to this email directly, view it on GitHub <#3838 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCAEUMF6SU3PTI4HZDADYKL2MKPEFAVCNFSM6AAAAABVZCJBROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJTGMZDOOJVGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[apostasie]

I tried pushing only single images

It does not look like it.
Which exact command did you run?

[nx2804]

nerdctl push serviceserver:88/nginx:latest

…

On Sat, 25 Jan 2025 at 11:31 AM, Apostasie ***@***.***> wrote: I tried pushing only single images It does not look like it. Which exact command did you run? — Reply to this email directly, view it on GitHub <#3838 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCAEUMHVRC5Y5RA2BQD35BL2MMSB3AVCNFSM6AAAAABVZCJBROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJTG44TSMJXGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[apostasie]

nerdctl push serviceserver:88/nginx:latest
…

I believe that ^ will push an OCI index.

You should figure out if your registry version is outdated and/or contact the project making it to figure out why they would not accept oci index.

[apostasie]

(you might also want to give a try to containerd v2 instead of the v1 you are using)