Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configuring Docker Shim aborts with Docker invocation on ARC #68

Open
zickzackv opened this issue Jan 31, 2024 · 9 comments
Open

Configuring Docker Shim aborts with Docker invocation on ARC #68

zickzackv opened this issue Jan 31, 2024 · 9 comments

Comments

@zickzackv
Copy link

I wanted to try out the installer action on our recently setup actions-runner-controller. Sadly it failed while setting up the docker shim.

docker-runner: 2.312.0
actions-runner-controller: 0.27.5

I could't trace the line in src/main.ts where docker is failing.

Nix-installer   Run docker info 2024-01-31T08:47:51.8813172Z ##[group]Run docker info
Nix-installer   Run docker info 2024-01-31T08:47:51.8813666Z docker info
Nix-installer   Run docker info 2024-01-31T08:47:51.8821211Z shell: /usr/bin/bash -e {0}
Nix-installer   Run docker info 2024-01-31T08:47:51.8821625Z ##[endgroup]
Nix-installer   Run docker info 2024-01-31T08:47:51.9301635Z Client:
Nix-installer   Run docker info 2024-01-31T08:47:51.9311729Z  Version:    24.0.7
Nix-installer   Run docker info 2024-01-31T08:47:51.9312458Z  Context:    default
Nix-installer   Run docker info 2024-01-31T08:47:51.9313060Z  Debug Mode: false
Nix-installer   Run docker info 2024-01-31T08:47:51.9313666Z  Plugins:
Nix-installer   Run docker info 2024-01-31T08:47:51.9314287Z   compose: Docker Compose (Docker Inc.)
Nix-installer   Run docker info 2024-01-31T08:47:51.8821625Z ##[endgroup]
Nix-installer   Run docker info 2024-01-31T08:47:51.9301635Z Client:
Nix-installer   Run docker info 2024-01-31T08:47:51.9311729Z  Version:    24.0.7
Nix-installer   Run docker info 2024-01-31T08:47:51.9312458Z  Context:    default
Nix-installer   Run docker info 2024-01-31T08:47:51.9313060Z  Debug Mode: false
Nix-installer   Run docker info 2024-01-31T08:47:51.9313666Z  Plugins:
Nix-installer   Run docker info 2024-01-31T08:47:51.9314287Z   compose: Docker Compose (Docker Inc.)
Nix-installer   Run docker info 2024-01-31T08:47:51.9315073Z     Version:  v2.23.0
Nix-installer   Run docker info 2024-01-31T08:47:51.9316190Z     Path:     /usr/libexec/docker/cli-plugins/docker-compose
Nix-installer   Run docker info 2024-01-31T08:47:51.9317057Z
Nix-installer   Run docker info 2024-01-31T08:47:51.9317367Z Server:
Nix-installer   Run docker info 2024-01-31T08:47:51.9317955Z  Containers: 0
Nix-installer   Run docker info 2024-01-31T08:47:51.9318640Z   Running: 0
Nix-installer   Run docker info 2024-01-31T08:47:51.9319240Z   Paused: 0
Nix-installer   Run docker info 2024-01-31T08:47:51.9319833Z   Stopped: 0
Nix-installer   Run docker info 2024-01-31T08:47:51.9320459Z  Images: 0
Nix-installer   Run docker info 2024-01-31T08:47:51.9321076Z  Server Version: 24.0.7
Nix-installer   Run docker info 2024-01-31T08:47:51.9321827Z  Storage Driver: overlay2
Nix-installer   Run docker info 2024-01-31T08:47:51.9322592Z   Backing Filesystem: xfs
Nix-installer   Run docker info 2024-01-31T08:47:51.9323465Z   Supports d_type: true
Nix-installer   Run docker info 2024-01-31T08:47:51.9324210Z   Using metacopy: false
Nix-installer   Run docker info 2024-01-31T08:47:51.9324971Z   Native Overlay Diff: true
Nix-installer   Run docker info 2024-01-31T08:47:51.9325748Z   userxattr: false
Nix-installer   Run docker info 2024-01-31T08:47:51.9326650Z  Logging Driver: json-file
Nix-installer   Run docker info 2024-01-31T08:47:51.9327474Z  Cgroup Driver: cgroupfs
Nix-installer   Run docker info 2024-01-31T08:47:51.9328192Z  Cgroup Version: 1
Nix-installer   Run docker info 2024-01-31T08:47:51.9328985Z  Plugins:
Nix-installer   Run docker info 2024-01-31T08:47:51.9329483Z   Volume: local
Nix-installer   Run docker info 2024-01-31T08:47:51.9330199Z   Network: bridge host ipvlan macvlan null overlay
Nix-installer   Run docker info 2024-01-31T08:47:51.9332044Z   Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Nix-installer   Run docker info 2024-01-31T08:47:51.9332950Z  Swarm: inactive
Nix-installer   Run docker info 2024-01-31T08:47:51.9333324Z  Runtimes: io.containerd.runc.v2 runc
Nix-installer   Run docker info 2024-01-31T08:47:51.9333782Z  Default Runtime: runc
Nix-installer   Run docker info 2024-01-31T08:47:51.9334203Z  Init Binary: docker-init
Nix-installer   Run docker info 2024-01-31T08:47:51.9334700Z  containerd version: 091922f03c2762540fd057fba91260237ff86acb
Nix-installer   Run docker info 2024-01-31T08:47:51.9335354Z  runc version: v1.1.9-0-gccaecfc
Nix-installer   Run docker info 2024-01-31T08:47:51.9335775Z  init version: de40ad0
Nix-installer   Run docker info 2024-01-31T08:47:51.9336394Z  Security Options:
Nix-installer   Run docker info 2024-01-31T08:47:51.9336718Z   seccomp
Nix-installer   Run docker info 2024-01-31T08:47:51.9337102Z    Profile: builtin
Nix-installer   Run docker info 2024-01-31T08:47:51.9337565Z  Kernel Version: 5.10.197-186.748.amzn2.x86_64
Nix-installer   Run docker info 2024-01-31T08:47:51.9338156Z  Operating System: Alpine Linux v3.19 (containerized)
Nix-installer   Run docker info 2024-01-31T08:47:51.9338689Z  OSType: linux
Nix-installer   Run docker info 2024-01-31T08:47:51.9338994Z  Architecture: x86_64
Nix-installer   Run docker info 2024-01-31T08:47:51.9339327Z  CPUs: 2
Nix-installer   Run docker info 2024-01-31T08:47:51.9339619Z  Total Memory: 7.544GiB
Nix-installer   Run docker info 2024-01-31T08:47:51.9340061Z  Name: k8s-action-runner-gxfqk-zlcm8
Nix-installer   Run docker info 2024-01-31T08:47:51.9340615Z  ID: 5a8303f3-5736-4edf-b365-dc5eccd07f41
Nix-installer   Run docker info 2024-01-31T08:47:51.9341099Z  Docker Root Dir: /var/lib/docker
Nix-installer   Run docker info 2024-01-31T08:47:51.9341500Z  Debug Mode: false
Nix-installer   Run docker info 2024-01-31T08:47:51.9341834Z  Experimental: false
Nix-installer   Run docker info 2024-01-31T08:47:51.9342186Z  Insecure Registries:
Nix-installer   Run docker info 2024-01-31T08:47:51.9342504Z   127.0.0.0/8
Nix-installer   Run docker info 2024-01-31T08:47:51.9342820Z  Live Restore Enabled: false
Nix-installer   Run docker info 2024-01-31T08:47:51.9343250Z  Product License: Community Engine
Nix-installer   Run docker info 2024-01-31T08:47:51.9343556Z
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9464264Z ##[group]Run DeterminateSystems/nix-installer-action@main
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9464958Z with:
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9465403Z   start-daemon: false
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9465921Z   flakehub: false
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9466357Z   force-docker-shim: false
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9467126Z   github-token: ***
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9467615Z   github-server-url: https://github.com
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9468117Z   kvm: true
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9468548Z   modify-profile: true
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9468989Z   reinstall: false
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9469624Z   diagnostic-endpoint: https://install.determinate.systems/nix/diagnostic
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9470651Z   trust-runner-user: true
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:51.9471199Z ##[endgroup]
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1182349Z ##[group]Enabling the Docker shim for running Nix on Linux in CI without Systemd.
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1184798Z Changing init from 'null' to 'none'
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1186482Z Changing planner from 'null' to 'linux'
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1188400Z ##[endgroup]
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1189952Z ##[group]Configuring KVM
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1361384Z [command]/usr/bin/sudo rm -f /etc/udev/rules.d/99-determinate-nix-installer-kvm.rules
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1497965Z ##[endgroup]
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1499260Z KVM is not available.
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1501217Z ##[group]Installing Nix
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:52.1504333Z Fetching binary from https://install.determinate.systems/nix/nix-installer-x86_64-linux?ci=github&correlation=GH-cc60063f-c695-40bf-81c0-
33cbaaf6d95b
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:53.7755374Z [command]/runner/_work/_temp/5c79da76-e711-44c1-b2d3-425f8ef55409 install linux
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:53.7837472Z `nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:53.7994300Z  INFO Step: Create directory `/nix`
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:53.8016743Z  INFO Step: Provision Nix
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:56.1587105Z  INFO Step: Create build users (UID 30001-30032) and group (GID 30000)
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.0002951Z  INFO Step: Configure Nix
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.1275128Z  INFO Step: Create directory `/etc/tmpfiles.d`
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.1276616Z  INFO Step: Leave the Nix daemon unconfigured
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.1278257Z  INFO Step: Remove directory `/nix/temp-install-dir`
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.7742640Z Nix was installed successfully!
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.7746050Z To get started using Nix, open a new shell or run `. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh`
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.7747686Z
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.7767801Z ##[endgroup]
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.7770572Z ##[group]Configuring the Docker shim as the Nix Daemon's process supervisor
Nix-installer   Run DeterminateSystems/nix-installer-action@main        2024-01-31T08:47:57.8499036Z ##[error]Error: The process '/usr/local/bin/docker' failed with exit code 125
Nix-installer   Post Run DeterminateSystems/nix-installer-action@main   2024-01-31T08:47:57.8740941Z Post job cleanup.
Nix-installer   Post Run DeterminateSystems/nix-installer-action@main   2024-01-31T08:47:58.0345782Z ##[group]Enabling the Docker shim for running Nix on Linux in CI without Systemd.
Nix-installer   Post Run DeterminateSystems/nix-installer-action@main   2024-01-31T08:47:58.0347304Z Changing init from 'null' to 'none'
Nix-installer   Post Run DeterminateSystems/nix-installer-action@main   2024-01-31T08:47:58.0348239Z Changing planner from 'null' to 'linux'
Nix-installer   Post Run DeterminateSystems/nix-installer-action@main   2024-01-31T08:47:58.0349455Z ##[endgroup]
Nix-installer   Post Run DeterminateSystems/nix-installer-action@main   2024-01-31T08:47:58.0378552Z Added `/nix/var/nix/profiles/default/bin` and `/home/runner/.nix-profile/bin` to `$GITHUB_PATH`
Nix-installer   Post Run DeterminateSystems/nix-installer-action@main   2024-01-31T08:47:58.0380150Z Nix was already installed, using existing install
@zickzackv
Copy link
Author

We now setup our arc (Kubernetes Runners) with a DIND setup. So the runner container has a docker binary and a docker socket available and one can start other containers with that docker daemon and socket. Sadly the socket's path is specified within an environment variable; that environment is lost when executing sudo (without -E). Root (with sudo) will not find the docker socket to start an nix daemon.

@zickzackv
Copy link
Author

Even if the nix daemon could be started within our ARC/DIND setup I would suspect that a nix deamon running in another pod/container will not have access to /nix/store in the original container/pod where the installer run.

@remi-gelinas
Copy link

Yep, same ARC Kubernetes setup for us and same issue and logs/error code. Until this is solved we'll have to go back to Cachix's install action

@lucperkins
Copy link
Member

@zickzackv Could you give it another try with a more recent commit or on plain old main? We've overhauled this Action pretty significantly in recent weeks.

@grahamc
Copy link
Member

grahamc commented May 15, 2024

As an aside, we'd really like to support this better -- but we don't have anyone that we're regularly interacting with to do design & implementation reviews with. If someone is able to do that, I'd be glad to be in touch: [email protected].

@grahamc
Copy link
Member

grahamc commented May 17, 2024

@remi-gelinas / @zickzackv -- if possible, can you paste sanitized env dumps? I'm curious what environment variables are present that we might use to turn on and off relevant behavior.

@zickzackv
Copy link
Author

@grahamc Sadly I can't tests the installer under ARC. I'm not working anymore with ARC :-(

@remi-gelinas
Copy link

@grahamc I think I can grab a sanitized dump for you. No clue if it'll be helpful, but I'll see what I can extract today

@remi-gelinas
Copy link

@grahamc Here is a heavily sanitized env dump from one of our runners attempting to use v11 of the action:

KUBERNETES_SERVICE_PORT_HTTPS=443
GITHUB_WORKSPACE=/runner/_work/runner/runner
KUBERNETES_SERVICE_PORT=443
HOSTNAME=<redacted>
RUNNER_WORKDIR=/runner/_work
GITHUB_ACTION=__run
DOCKERD_IN_RUNNER=false
GITHUB_RUN_NUMBER=186
RUNNER_NAME=<redacted>
GITHUB_REPOSITORY_OWNER_ID=<redacted>
GITHUB_URL=https://github.com/
RUNNER_ORG=<redacted>
AWS_DEFAULT_REGION=us-west-2
ACTIONS_RUNNER_HOOK_JOB_COMPLETED=/etc/arc/hooks/job-completed.sh
GITHUB_TRIGGERING_ACTOR=<actor>
GITHUB_REF_TYPE=branch
AWS_REGION=us-west-2
DOCKER_ENABLED=true
GITHUB_ACTIONS=true
_=/usr/bin/env
RUNNER_ENVIRONMENT=self-hosted
GITHUB_REF=refs/pull/72/merge
RUNNER_OS=Linux
GITHUB_REF_PROTECTED=false
RUNNER_STATUS_UPDATE_HOOK=false
HOME=/home/runner
GITHUB_API_URL=https://api.github.com
RUNNER_TRACKING_ID=<trackingid>
RUNNER_ARCH=X64
RUNNER_TEMP=/runner/_work/_temp
GITHUB_EVENT_PATH=/runner/_work/_temp/_github_workflow/event.json
GITHUB_EVENT_NAME=pull_request
GITHUB_ACTIONS_RUNNER_EXTRA_USER_AGENT=actions-runner-controller/v0.27.5
RUNNER_ASSETS_DIR=/runnertmp
GITHUB_RUN_ID=<runid>
DISABLE_RUNNER_UPDATE=true
GITHUB_ACTOR=<actorname>
RUNNER_EPHEMERAL=true
GITHUB_RUN_ATTEMPT=1
GITHUB_GRAPHQL_URL=https://api.github.com/graphql
RUNNER_GROUP=
GITHUB_SERVER_URL=https://github.com
SHLVL=4
KUBERNETES_PORT_443_TCP_PROTO=tcp
GITHUB_ACTOR_ID=101587823
RUNNER_TOOL_CACHE=/opt/hostedtoolcache
GITHUB_JOB=<jobname>
AWS_STS_REGIONAL_ENDPOINTS=regional
ACTIONS_RUNNER_HOOK_JOB_STARTED=/etc/arc/hooks/job-started.sh
DOCKER_HOST=unix:///run/docker.sock
GITHUB_REPOSITORY=<reponame>
GITHUB_RETENTION_DAYS=30
RUNNER_WORKSPACE=/runner/_work/runner
GITHUB_ACTION_REPOSITORY=
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/runner/.local/bin/
GITHUB_BASE_REF=main
CI=true
ImageOS=ubuntu20
GITHUB_REPOSITORY_OWNER=<redacted>
GITHUB_HEAD_REF=<branchname>
GITHUB_ACTION_REF=
RUNNER_LABELS=self-hosted,np,default
RUNNER_ENTERPRISE=
GITHUB_WORKFLOW=<workflowname>
DEBIAN_FRONTEND=noninteractive
OLDPWD=/

Maybe of note are the DOCKER_ENABLED and DOCKERD_IN_RUNNER vars, but I have no idea. Let me know if I can help further, or if you would like a guinea pig to test a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@grahamc @zickzackv @lucperkins @remi-gelinas