pull from quay.io

[andrewrothstein]

2023-07-14 09:09:58 2023-07-14T13:09:58.621737Z  WARN oci_registry::api: /jetstack/cert-manager-approver-policy/manifests/v0.7.0 not found at manifests/quay.io/jetstack/cert-manager-approver-policy/v0.7.0 in repository (I/O error: No such file or directory (os error 2)); pulling from upstream
2023-07-14 09:09:58 2023-07-14T13:09:58.894248Z  INFO actix_web::middleware::logger: 172.19.0.5 "HEAD /v2/jetstack/cert-manager-approver-policy/manifests/v0.7.0?ns=quay.io HTTP/1.1" 200 7467 "-" "containerd/v1.7.1" 0.272743    
2023-07-14 09:09:58 2023-07-14T13:09:58.924871Z  WARN oci_registry::api: blobs/sha256/e5/83f427138ee7404105ba037a1cb0a31beb20e4b403e0e4f85eb3d11a8cdcd1 not found in repository (I/O error: No such file or directory (os error 2)); pulling from upstream
2023-07-14 09:09:59 2023-07-14T13:09:59.022850Z ERROR oci_registry::api::error: 500: Error with upstream registry: request failed with status 404 Not Found

the same process pulled other jetstack cert-manager images from quay.io alright.

[mcronce]

Same issue here with that image. I'll take a look and figure out why we're getting a 404.

Thanks!

[mcronce]

This ended up being the same issue as #6; fixed with 3c62c24. I'll spin a release today - hoping to fix #6 as well first

[mcronce]

@andrewrothstein v0.3.12 images and chart are released with these fixes

[deliantwo]

Hello,
it seems I'm getting the same error when doing a pull of quay.io/kiwigrid/k8s-sidecar:1.25.1

2023-09-16T13:42:28.106079Z  INFO oci_registry: Aged out 0 objects
2023-09-16T13:42:53.864345Z  WARN oci_registry::api: blobs/sha256/de/5edccb4484e1d6027a46ce2f159ac1d12239ee707cd36a399be7c5f24ba496 not found in repository (Failed to get object from S3: Service(NoSuchKey("The specified key does not exist."))); pulling from upstream
2023-09-16T13:42:54.536735Z ERROR oci_registry::api::error: 404: Error with upstream registry: request failed with status 404 Not Found
2023-09-16T13:42:54.536873Z  INFO actix_web::middleware::logger: 10.42.6.43 "GET /v2/kiwigrid/k8s-sidecar/blobs/sha256:de5edccb4484e1d6027a46ce2f159ac1d12239ee707cd36a399be7c5f24ba496?ns=quay.io HTTP/1.1" 404 70 "-" "containerd/v1.6.15-k3s1" 0.675772

I was on oci-registry 0.3.8. I've upgraded to 0.3.20 but I'm getting the same error.
A direct pull works as expected.

[mcronce]

@deliantwo I'll take a look

[mcronce]

@deliantwo hmm...I'm not able to reproduce. Trying to grab that specific blob URL fails, but running a crictl pull, the manifest points me to /v2/kiwigrid/k8s-sidecar/blobs/sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de?ns=quay.io

If you manually delete the cached manifest for that tag, then rerun the pull, do you get the same result?

Also, what OS and architecture are you running?

[deliantwo]

Thank you @mcronce !
After deleting the cached manifest for that tag, it worked. What could have been the issue?

Just for the record I'm on k3s 1.26.1 (agents and master on debian)
Here the logs after deleting the manifest:

2023-09-19T16:39:37.213601Z  INFO actix_web::middleware::logger: 10.42.6.43 "HEAD /v2/kiwigrid/k8s-sidecar/manifests/1.25.1?ns=quay.io HTTP/1.1" 200 16977 "-" "containerd/v1.6.15-k3s1" 0.022141
2023-09-19T16:41:15.790159Z  WARN oci_registry::api: /kiwigrid/k8s-sidecar/manifests/sha256:415d07ee1027c3ff7af9e26e05e03ffd0ec0ccf9f619ac00ab24366efe4343bd not found at manifests/quay.io/kiwigrid/k8s-sidecar/sha256:415d07ee1027c3ff7af9e26e05e03ffd0ec0ccf9f619ac00ab24366efe4343bd in repository (Failed to get object from S3: Service(NoSuchKey("The specified key does not exist."))); pulling from upstream
2023-09-19T16:41:17.100112Z  INFO actix_web::middleware::logger: 10.42.6.43 "GET /v2/kiwigrid/k8s-sidecar/manifests/sha256:415d07ee1027c3ff7af9e26e05e03ffd0ec0ccf9f619ac00ab24366efe4343bd?ns=quay.io HTTP/1.1" 200 3893 "-" "containerd/v1.6.15-k3s1" 1.324734    
2023-09-19T16:41:17.791713Z  WARN oci_registry::api: blobs/sha256/e8/dde4861b1f6050f4c0414278304279bc9e163196fb4cd69b0976763512c140 not found in repository (Failed to get object from S3: Service(NoSuchKey("The specified key does not exist."))); pulling from upstream
2023-09-19T16:41:18.684926Z  INFO actix_web::middleware::logger: 10.42.6.43 "GET /v2/kiwigrid/k8s-sidecar/blobs/sha256:e8dde4861b1f6050f4c0414278304279bc9e163196fb4cd69b0976763512c140?ns=quay.io HTTP/1.1" 200 8189 "-" "containerd/v1.6.15-k3s1" 0.897401    
2023-09-19T16:41:18.794701Z  WARN oci_registry::api: blobs/sha256/66/e1d5e70e420aa86a23bd8b4eebf2a6eb60b4aff9ee8a6ca52e27f51f57b1be not found in repository (Failed to get object from S3: Service(NoSuchKey("The specified key does not exist."))); pulling from upstream
2023-09-19T16:41:19.523374Z  INFO actix_web::middleware::logger: 10.42.6.43 "GET /v2/kiwigrid/k8s-sidecar/blobs/sha256:66e1d5e70e420aa86a23bd8b4eebf2a6eb60b4aff9ee8a6ca52e27f51f57b1be?ns=quay.io HTTP/1.1" 200 622306 "-" "containerd/v1.6.15-k3s1" 0.730652    
2023-09-19T16:41:19.711794Z  WARN oci_registry::api: blobs/sha256/ef/ebc2e683d297ae71acae9230d17af8b95684d9a4f9b7f601a8e1e9b103bda3 not found in repository (Failed to get object from S3: Service(NoSuchKey("The specified key does not exist."))); pulling from upstream
2023-09-19T16:41:21.282280Z  INFO actix_web::middleware::logger: 10.42.6.43 "GET /v2/kiwigrid/k8s-sidecar/blobs/sha256:efebc2e683d297ae71acae9230d17af8b95684d9a4f9b7f601a8e1e9b103bda3?ns=quay.io HTTP/1.1" 200 3109434 "-" "containerd/v1.6.15-k3s1" 1.653176    
2023-09-19T16:41:21.288482Z  WARN oci_registry::api: blobs/sha256/76/f9849b1b85eb87c718d5c72dc08fbef0d2cd9502cf528fb9d0a4dc6acad9fb not found in repository (Failed to get object from S3: Service(NoSuchKey("The specified key does not exist."))); pulling from upstream
2023-09-19T16:41:22.806523Z  INFO actix_web::middleware::logger: 10.42.6.43 "GET /v2/kiwigrid/k8s-sidecar/blobs/sha256:76f9849b1b85eb87c718d5c72dc08fbef0d2cd9502cf528fb9d0a4dc6acad9fb?ns=quay.io HTTP/1.1" 200 5501593 "-" "containerd/v1.6.15-k3s1" 1.526743
2023-09-19T16:42:30.517504Z  INFO oci_registry: Aged out 0 objects

[mcronce]

My hypothesis is that the manifest was updated with new blobs at some point after the manifest was stored in your cache - so oci-registry was returning the old manifest to the container client, which was then trying to pull the old blob, which no longer existed upstream. I believe docker.io keeps old blobs around for quite a long time, but I'm not sure what quay.io does.

I cannot, unfortunately, prove it...but I am surprised that this is the first I've seen/heard of the case. I've had the possibility in mind for a while - it seems like something that would happen at least reasonably frequently if it happens at all.

I'll see if I can come up with a solution to handle it by automatically invalidating the manifest, but I don't think blob requests have sufficient information in them to identify the specific manifest being requested. The URLs include kiwigrid/k8s-sidecar but not 1.26.1 - I'll look at the headers to make sure.

If not, I can at least add something useful to the logs when a blob 404s. I've also been kicking around the idea of an API endpoint to force-invalidate something from cache, which will at least make manually handling this easier - you won't have to interact directly with the cache storage.

[mcronce]

Looks like a no go on identifying which manifest a blob request belongs to

I'll at least move forward with the manual deletion endpoint and improved logs on upstream 404s for blobs, though

[mcronce]

mcronce/oci-registry:latest was just pushed up with manual deletion endpoints. It's not in a tagged release yet, as I'm testing that and other changes in my lab. I'll get improved blob 404 logging added before releasing as well.

[mcronce]

v0.4.4 was released tonight and includes the new endpoint. I missed the logging change, will get it in ASAP and get it released