writing /etc/resolv.conf caused: open /etc/resolv.conf: permission denied: unknown without "userns-remap":"sysbox"; kernel version is fine

[dimikot]

Despite it's said in https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md#docker-userns-remap that "Starting with Sysbox v0.5.0, configuring Docker in userns-remap mode is no longer required, even if the host has no support for shiftfs. However, without shiftfs, you will need a host with kernel >= 5.12 so that Sysbox can use the kernel's ID-mapped mounts feature." - it doesn't work unfortunately.

Full error message:

$ docker compose up ci-runner
[+] Running 1/0
 ✔ Container ci-runner-ci-runner-1  Recreated                                                                                                                                                                               Attaching to ci-runner-1
Error response from daemon: failed to create task for container: failed to create shim task:
OCI runtime create failed: container_linux.go:439: starting container process caused:
process_linux.go:608: container init caused: switching Docker DNS: rootfs_linux.go:1420:
writing /etc/resolv.conf caused: open /etc/resolv.conf: permission denied: unknown

# uname -a
Linux ip-10-40-83-20 5.15.0-1009-aws #11-Ubuntu SMP Thu May 26 19:39:49 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

# dpkg -l | grep sysbox
ii  sysbox-ce                         0.6.4.linux

# dpkg -l | grep docker
ii  docker-buildx-plugin              0.14.0-1~ubuntu.22.04~jammy             arm64
ii  docker-ce                         5:26.1.1-1~ubuntu.22.04~jammy           arm64
ii  docker-ce-cli                     5:26.1.1-1~ubuntu.22.04~jammy           arm64
ii  docker-ce-rootless-extras         5:26.1.1-1~ubuntu.22.04~jammy           arm64
ii  docker-compose-plugin             2.27.0-1~ubuntu.22.04~jammy             arm64

# cat /etc/docker/daemon.json
{
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/bin/sysbox-runc"
        }
    },
    "default-runtime": "sysbox-runc",
    "bip": "172.20.0.1/16",
    "default-address-pools": [
        {
            "base": "172.25.0.0/16",
            "size": 24
        }
    ]
}

When I add "userns-remap": "sysbox", it starts working.

The container image is based on "ubuntu:22.04".

I wanted some minimally changed configuration to work, since it's said that sysbox version and kernel version are new enough for it all to work without userns-remap.

[ctalledo]

Hi @dimikot,

Thanks for giving Sysbox a shot!

Can you share your docker-compose YAML so I can recreate locally please?

With Kernel 5.15 you shouldn't need to run Docker in userns-remap mode, as Sysbox will leverage the kernel's ID-mapped-mounts feature to ensure files show up with proper ownership inside the rootless Sysbox container. However we still recommend shiftfs until kernel 5.19 which adds overlayfs on ID-mapped-mounts support.

Thanks!

[dimikot]

@ctalledo Thanks! Try using the following compose.yml, I think it's self-sufficient:

services:
    ci-storage:
        image: ghcr.io/dimikot/ci-storage:main
        ports:
            - 26022:22
        environment:
            - TZ
        volumes:
            - ci-storage-mnt:/mnt
volumes:
    ci-storage-mnt:
        external: false

See my linux kernel version in the above uname -a listing. It's 5.15.0-1009-aws.

I actually don't want shiftfs in sake of just performance, since its installation is not so trivial (compiling from source code etc.); I want the setup to be as vanilla as possible.

[ctalledo]

Hi @dimikot,

Apologies for the late reply.

I am not able to repro on an AWS Ubuntu Focal host with kernel 5.15:

root@sysbox-test:~# docker compose up 
[+] Running 6/6
 ✔ ci-storage Pulled                                                                                                                                                                                                                                                       7.9s 
   ✔ a8b1c5f80c2d Pull complete                                                                                                                                                                                                                                            2.0s 
   ✔ c47bb6d4174f Pull complete                                                                                                                                                                                                                                            7.1s 
   ✔ cc39bbef79d9 Pull complete                                                                                                                                                                                                                                            7.1s 
   ✔ 85d1c32a4d0e Pull complete                                                                                                                                                                                                                                            7.1s 
   ✔ 4f4fb700ef54 Pull complete                                                                                                                                                                                                                                            7.2s 
[+] Running 3/3
 ✔ Network root_default          Created                                                                                                                                                                                                                                   0.0s 
 ✔ Volume "root_ci-storage-mnt"  Created                                                                                                                                                                                                                                   0.0s 
 ✔ Container root-ci-storage-1   Created                                                                                                                                                                                                                                   3.3s Attaching to ci-storage-1
ci-storage-1  | 2024-05-24 02:06:05 UTC
ci-storage-1  | To access this container over SSH, a secret CI_STORAGE_PUBLIC_KEY or a mounted file /run/secrets/CI_STORAGE_PUBLIC_KEY should exist. The container will start, but it's not accessible, which may be fine in dev environment.
ci-storage-1  | Starting SSH server...

root@sysbox-test:~/nestybox/sysbox# docker ps
CONTAINER ID   IMAGE                             COMMAND                 CREATED          STATUS          PORTS                   NAMES
07cd2f49e140   ghcr.io/dimikot/ci-storage:main   "/root/entrypoint.sh"   48 seconds ago   Up 44 seconds   0.0.0.0:26022->22/tcp   root-ci-storage-1

root@sysbox-test:~/nestybox/sysbox# docker inspect root-ci-storage-1 | grep Runtime
            "Runtime": "sysbox-runc",

root@sysbox-test:~/nestybox/sysbox# uname -a
Linux sysbox-test 5.15.0-1061-aws #67~20.04.1-Ubuntu SMP Wed Apr 17 15:09:54 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Not quite sure why if fails in your case ...

[ctalledo]

I also used Sysbox v0.6.4:

● sysbox.service - Sysbox container runtime
     Loaded: loaded (/lib/systemd/system/sysbox.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-05-24 02:10:57 UTC; 5min ago
       Docs: https://github.com/nestybox/sysbox
   Main PID: 26146 (sh)
      Tasks: 2 (limit: 8898)
     Memory: 348.0K
     CGroup: /system.slice/sysbox.service
             ├─26146 /bin/sh -c /usr/bin/sysbox-runc --version && /usr/bin/sysbox-mgr --version && /usr/bin/sysbox-fs --version && /bin/sleep infinity
             └─26165 /bin/sleep infinity

May 24 02:10:57 ip-172-31-39-41 systemd[1]: Started Sysbox container runtime.
May 24 02:10:57 ip-172-31-39-41 sh[26147]: sysbox-runc
May 24 02:10:57 ip-172-31-39-41 sh[26147]:         edition:         Community Edition (CE)
May 24 02:10:57 ip-172-31-39-41 sh[26147]:         version:         0.6.4
May 24 02:10:57 ip-172-31-39-41 sh[26147]:         commit:         085502643ea5281652c6984eed9797872f22698a
May 24 02:10:57 ip-172-31-39-41 sh[26147]:         built at:         Sat Apr  6 16:43:31 UTC 2024
May 24 02:10:57 ip-172-31-39-41 sh[26147]:         built by:         Cesar Talledo
May 24 02:10:57 ip-172-31-39-41 sh[26147]:         oci-specs:         1.1.0+dev
May 24 02:10:57 ip-172-31-39-41 sh[26154]: sysbox-mgr
May 24 02:10:57 ip-172-31-39-41 sh[26154]:         edition:         Community Edition (CE)
May 24 02:10:57 ip-172-31-39-41 sh[26154]:         version:         0.6.4
May 24 02:10:57 ip-172-31-39-41 sh[26154]:         commit:         03f5d7bc584fdcb2319b2c1831bd58581185fc1c
May 24 02:10:57 ip-172-31-39-41 sh[26154]:         built at:         Sat Apr  6 16:43:43 UTC 2024
May 24 02:10:57 ip-172-31-39-41 sh[26154]:         built by:         Cesar Talledo
May 24 02:10:57 ip-172-31-39-41 sh[26159]: sysbox-fs
May 24 02:10:57 ip-172-31-39-41 sh[26159]:         edition:         Community Edition (CE)
May 24 02:10:57 ip-172-31-39-41 sh[26159]:         version:         0.6.4
May 24 02:10:57 ip-172-31-39-41 sh[26159]:         commit:         1a678b72ac430009739fa6596b824f29b1f7fe2e
May 24 02:10:57 ip-172-31-39-41 sh[26159]:         built at:         Sat Apr  6 16:43:40 UTC 2024
May 24 02:10:57 ip-172-31-39-41 sh[26159]:         built by:         Cesar Talledo