Unable to access Container Registry images despite running docker-credential-gcr configure-docker using docker-compose in Container Optimized OS

[ernsheong]

I am running Google's container optimized OS, with the docker-compose tool as documented by https://cloud.google.com/community/tutorials/docker-compose-on-container-optimized-os (docker-compose runs in a container, accessed by an alias)

I am getting the issue referenced here: docker/compose#4885, that is supposedly resolved.

I have already run the initialization command: docker-credential-gcr configure-docker

However, as per my comment there (docker/compose#4885 (comment)), I am unable to pull in container registry's images via the aforementioned docker-compose alias. docker pull gcr.io/PROJECT_ID/IMAGE works though.

Any help would be greatly appreciated.

[ernsheong]

Because the docker-compose command is actually a container, I suspect there is additional volume mapping that I need to do in addition to the current alias in order for this to work?

It is currently:

echo alias docker-compose="'"'docker run --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v "$PWD:/rootfs/$PWD" \
    -w="/rootfs/$PWD" \
    docker/compose:1.16.1'"'" >> ~/.bashrc

[dekkagaijin]

I'm currently in Copenhagen (taking some vacation after DockerCon), but I believe the issue may be that ~/.docker/config.json needs to be in a volume which is shared between docker-credential-gcr, docker-compose, and docker itself. Does adding something like -v "$HOME/.docker:$HOME/.docker" help?
I don't have good access to my workstation, currently, otherwise I'd try and be more help debugging.

[ernsheong]

Thanks for the reply. -v "$HOME/.docker:$HOME/.docker" does not seem to work. In fact, $PWD is my $HOME directory in the server. -v "$HOME:$HOME" also did not work.

(I did remember to source my ~/.bashrc :))

Not sure how popular docker-compose is right now, but it would be much much less painful to have it natively installed, or some kind of opt-in...

[dekkagaijin]

Ok. I'll look into it in more depth when I'm back home (next week). Until then, you could try removing docker-credential-gcr from your docker config, setting an environment variable like $GCPTOKEN containing your access token: https://cloud.google.com/container-registry/docs/advanced-authentication and logging in manually, e.g.: `docker login -u _token -p $GCPTOKEN https://gcr.io`

…

________________________________ From: Jonathan ES Lin <[email protected]> Sent: Sunday, October 22, 2017 4:11:07 PM To: GoogleCloudPlatform/docker-credential-gcr Cc: Jake Sanders; Comment Subject: Re: [GoogleCloudPlatform/docker-credential-gcr] Unable to access Container Registry images despite running `docker-credential-gcr configure-docker` using docker-compose in Container Optimized OS (#10) Thanks for the reply. -v "$HOME/.docker:$HOME/.docker does not seem to work. In fact, $PWD is my $HOME directory in the server. -v "$HOME:$HOME also did not work. (I did remember to source my ~/.bashrc :)) — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#10 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABJSvchvk2FhI7gCiIbOYj5nufreILsdks5su0z7gaJpZM4P8UXy>.

[ernsheong]

docker pull works and is able to pull in GCR images, and would be the simplest fallback for now :)

[beatcracker]

I've the the same issue on Windows using Docker Toolbox. Steps to reproduce:

1. Install and configure Google Cloud SDK
2. Run gcloud components install docker-credential-gcr
3. Run docker-machine env default and configure shell according to its output.
   If you skip this step, docker-credential-gcr configure-docker will fail with ERROR: Unable to determine Docker version: exit status 1
4. Run docker-credential-gcr configure-docker

After this, I'm able to docker pull gcr.io/project/image:latest but docker-compose fails with:

Pulling somecontainer (gcr.io/project/image:latest)...
ERROR: denied: Permission denied for "latest" from request "/v2/gcr.io/project/image/manifests/latest".

[Maher4Ever]

I'm having the exact same issue as well. Although inconvenient, I'm running docker pull every time before running docker-compose up for now as @ernsheong suggested.

[harmon]

This might have to do with the difference between running it as your user, or having root run it. The Google Container Optimized OS has /root/ locked down as read only, but your /home/<your_user> is writable, so running commands as your user would put .docker/config.json into /home/<your_user>/.docker/config.json, whereas having some boot script run as root would try and write that into /root/.docker. I'm not sure this is your exact problem, but see if it is. I'm having an issue around this with Google's Datalab using custom Docker images.

[syhol]

I had the same issue as OP, I ended up with:

SECRET="$(echo "https://gcr.io" | docker-credential-gcr get | jq '.Secret')"
docker login -u _token -p "${SECRET}" https://gcr.io
function docker-compose()
{
    docker run \
        -i --rm \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v ~/.docker:/root/.docker \
        docker/compose:1.16.1 \
        "${@}"
}

Make sure the machine has access to jq. I'm not sure if this is a recommended or secure practice, but its working for me.

[astleychen]

Is this still a known issue here? I failed to pull images from gcr when using docker-compose build.

docker-compose version 1.11.2, build dfed245

[jonjohnsonjr]

@astleychen on COS specifically? This might be what's going on: https://stackoverflow.com/questions/51236449

[rupurt]

I'm on COS and also had similar problems. The documentation for COS makes it seem like it should be as simple as running 2 commands.

$ docker-credential-gcr configure-docker
$ docker run --rm gcr.io/<your-project>/<your-image>

The file gets created in ~/.docker/config.json. But I couldn't pull the private image to run it. I can successfully pull public images.

After bashing my head against the wall for most of the day I tried the login command docker-credential-gcr gcr-login. After I followed those instructions I can now successfully pull images from the private registry. This doesn't seem scriptable so I'm unsure how I should proceed with my infrastructure setup via Terraform. Any insight would be greatly appreciated.

[rupurt]

When I run the example from @syhol I get a better error message. It looks like docker-credential-gcr configure-docker doesn't report errors.

docker login -u _token -p 'my_secret_abc123...' https://gcr.io
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://gcr.io/v2/: unauthorized: GCR login failed. You may have invalid credentials. To login successfully, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

[rupurt]

I managed to crack the case! I needed to use the devstorage.read_only scope for the service account. I'm using terraform so it was as simple as:

  # ...
  service_account {
    scopes = [
      "https://www.googleapis.com/auth/compute.readonly",
      # The next line was all I needed to add
      "https://www.googleapis.com/auth/devstorage.read_only"
    ]
  }
}

[langri-sha]

I managed to find https://cloud.google.com/container-optimized-os/docs/how-to/run-container-instance#starting_a_docker_container_via_cloud-config which describes how to run the service as root:

• create a custom user
• make sure to configure the service environment to use the custom user's home directory
• this should now fix issues when running docker-credential-gcr as root

[MrBlenny]

I've been doing a docker pull for each file in the docker compose as suggested by ernsheong.

The following script will automate that:

PATH_TO_DOCKER_COMPOSE='./docker/docker-compose.yaml'

cat $PATH_TO_DOCKER_COMPOSE | grep '    image: ' | while read -r line ; do
    IMAGE_NAME="${line/'image: '/''}" 
    docker pull $IMAGE_NAME
done

[cosmos-sajal]

Did anyone try -
gcloud auth login

Worked for me, I feel my kubectl setup deleted the creds.

[fpgaminer]

You can use a docker-compose container with docker-credential-gcr added to it, ala: https://hub.docker.com/r/cryptopants/docker-compose-gcr

It's a drop-in replacement for the containerized docker-compose suggested by COS docs, and can pull from private gcr.io seamlessly.

alias docker-compose='docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD:$PWD" -w="$PWD" cryptopants/docker-compose-gcr'

[SandeepGolluri]

the problem is Python 3 is not supported by the Google Cloud SDK. we need to use python 2
install python2 and run below command
export CLOUDSDK_PYTHON=python2
run the compose file it worked for me.