Configuring private registry credentials for on-demand sync

[kennyiv]

Hello,

I am trying to follow the documentation to set-up a on-demand sync for a private registry specifically Azure Container Registry. Is the example for configuring authentication credentials still valid? examples/sync-auth-filepath.json . I manually created that file and added it to the externalSecrets helm values so that it is mounted. Running the application in debug or trace mode doesn't seem to provide additional insight into why the credentials are failing against the private registry. Any recommendations or next steps would be appreciated.

Thanks

[rchincha]

@kennyiv Can you share your zot configuration (anonymized) and relevant logs? Will help triage.

[kennyiv]

@rchincha attached below is the anonymized config. The logs indicate that the credentials fail or are otherwise incorrect:
kustomization.txt
zot-values.txt
auth.json

[eusebiu-constantin-petu-dbk]

Hello, in the sync auth file you should put only the host[:port] like in the examples. It should work after that.

[kennyiv]

I have implemented the feedback by defining https://index.docker.io for the URL and the address for credentials to be index.docker.io. Zot still throws an invalid username/password. I am using the same credentials that work with ContainerD the docker username & password. Does the username and password for Zot within the auth.json file need to be in a special format like base64 encoded?

[eusebiu-constantin-petu-dbk]

No, it doesn't need to be base64.
Can we see some logs?

[kennyiv]

Our main concern here is to get the Azure private registry working since we can already pull dockerhub images without authentication. I get the same errors for Azure as well:
{"level":"info","module":"http","component":"session","clientIP":"192.168.2.169:55116","method":"GET","path":"/v2/library/busybox/manifests/sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629 │ │ {"level":"info","repository":"library/busybox","reference":"sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a4410784edb00a9","goroutine":912,"caller":"zotregistry.dev/zot/pkg/api/routes.go:2023","ti │ │ {"level":"info","goroutine":946,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:508","time":"2025-02-04T19:22:21.248409522Z","message":"getting available client"} │ │ {"level":"info","remote":"https://[redacted].azurecr.io/","repository":"library/busybox","reference":"sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a4410784edb00a9","goroutine":946,"caller":"zotre │ │ {"level":"error","error":"unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e1e0a26 │ │ {"level":"info","goroutine":946,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:508","time":"2025-02-04T19:22:21.287708856Z","message":"getting available client"} │ │ {"level":"info","remote":"https://index.docker.io","repository":"library/busybox","reference":"sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a4410784edb00a9","goroutine":946,"caller":"zotregistry. │ │ {"level":"error","error":"unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password","repository":"library/busybox","reference":"sha256:e004c2cc521c95383aebb1fb5 │ │ {"level":"info","goroutine":946,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:508","time":"2025-02-04T19:22:21.343527513Z","message":"getting available client"} │ │ {"level":"info","remote":"https://quay.io/","repository":"library/busybox","reference":"sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a4410784edb00a9","goroutine":946,"caller":"zotregistry.dev/zot │ │ {"level":"error","error":"reading manifest sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a4410784edb00a9 in quay.io/library/busybox: unauthorized: access to the requested resource is not authorize │ │ {"level":"info","goroutine":946,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:508","time":"2025-02-04T19:22:21.578950895Z","message":"getting available client"} │ │ {"level":"info","remote":"https://registry.k8s.io/","repository":"library/busybox","reference":"sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a4410784edb00a9","goroutine":946,"caller":"zotregistry │ │ {"level":"error","error":"reading manifest sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a4410784edb00a9 in registry.k8s.io/library/busybox: manifest unknown: Requested entity was not found.","rep │ │ {"level":"info","goroutine":946,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:508","time":"2025-02-04T19:22:21.678039632Z","message":"getting available client"} │ │ {"level":"info","remote":"https://ghcr.io/","repository":"library/busybox","reference":"sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a4410784edb00a9","goroutine":946,"caller":"zotregistry.dev/zot │ │ {"level":"error","error":"Requesting bearer token: invalid status code from registry 403 (Forbidden)","repository":"library/busybox","reference":"sha256:e004c2cc521c95383aebb1fb5893719aa7a8eae2e7a71f316a441 │ │ {"level":"info","goroutine":946,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:508","time":"2025-02-04T19:22:21.739216221Z","message":"getting available client"

[kennyiv]

Hello I have some better logs to attach. I reran after verifying our Azure ACR credentials and ensuring the Azure address is in the form of (name).azurecr.io in the auth.json file and the URL in the config file is "https://(name).azureacr.io/".

        "sync": {
          "credentialsFile": "/secrets/auth/auth.json",
          "enable": true,
          "registries": [
            {
              "urls": ["https://name.azurecr.io/"],
              "onDemand": true,
              "tlsVerify": true
            },

redacted-zot-logs-feb-6.txt

[kennyiv]

The secret is mounted using:

externalSecrets:
- secretName: "zot-auth"
  mountPath: "/secrets/auth"

Created using

secretGenerator:
- name: zot-auth
  files:
  - auth.json
generatorOptions:
  disableNameSuffixHash: true
namespace: kube-system

With auth.json containing:

{
    "name.azurecr.io": {
        "username": "username",
        "password": "password"
    }
}

Please let me know if you see anything incorrect above. I really appreciate the help

[rchincha]

@kennyiv looks like we need to give the azure path a fresh look. Others are reporting issues with azure as well.