Devcontainer CLI in GitHub Actions always fails

[jecolvin]

I see there's a closed issue describing almost exactly what we're experiencing: #606

However we are using CLI 0.50.2 and still seeing all commands past initializeCommand fail. Rolling back the CLI version hasn't helped (we've tried both 0.50.0 and 0.49.0).

Here's an example output:

2023-08-24T20:03:30.9990932Z ##[group]🏃 start container
2023-08-24T20:03:31.0004748Z About to run devcontainer up --workspace-folder /home/runner/work/dbt-analytics/dbt-analytics --remote-env SNOWFLAKE_DBT_PASSWORD=*** --remote-env GITHUB_OUTPUT=/mnt/github/output --remote-env GITHUB_ENV=/mnt/github/env --remote-env GITHUB_PATH=/mnt/github/path --remote-env GITHUB_STEP_SUMMARY=/mnt/github/step-summary --cache-from ghcr.io/brooklyn-data/devcontainers/dbt-snowflake --mount type=bind,source=/home/runner/work/_temp/_runner_file_commands/set_output_e038d2ae-6e71-436a-8450-acb48cec946c,target=/mnt/github/output --mount type=bind,source=/home/runner/work/_temp/_runner_file_commands/set_env_e038d2ae-6e71-436a-8450-acb48cec946c,target=/mnt/github/env --mount type=bind,source=/home/runner/work/_temp/_runner_file_commands/add_path_e038d2ae-6e71-436a-8450-acb48cec946c,target=/mnt/github/path --mount type=bind,source=/home/runner/work/_temp/_runner_file_commands/step_summary_e038d2ae-6e71-436a-8450-acb48cec946c,target=/mnt/github/step-summary
2023-08-24T20:03:31.2022348Z [2023-08-24T20:03:31.201Z] @devcontainers/cli 0.50.2. Node.js v18.17.1. linux 5.15.0-1041-azure x64.
2023-08-24T20:03:31.2022896Z 
2023-08-24T20:03:34.4521010Z [2023-08-24T20:03:34.450Z] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=/home/runner/work/dbt-analytics/dbt-analytics,target=/workspaces/dbt-analytics --mount source=/home/runner/.dbt,target=/home/vscode/.dbt,type=bind --mount type=bind,src=/home/runner/work/_temp/_runner_file_commands/set_output_e038d2ae-6e71-436a-8450-acb48cec946c,dst=/mnt/github/output --mount type=bind,src=/home/runner/work/_temp/_runner_file_commands/set_env_e038d2ae-6e71-436a-8450-acb48cec946c,dst=/mnt/github/env --mount type=bind,src=/home/runner/work/_temp/_runner_file_commands/add_path_e038d2ae-6e71-436a-8450-acb48cec946c,dst=/mnt/github/path --mount type=bind,src=/home/runner/work/_temp/_runner_file_commands/step_summary_e038d2ae-6e71-436a-8450-acb48cec946c,dst=/mnt/github/step-summary -l devcontainer.local_folder=/home/runner/work/dbt-analytics/dbt-analytics -l devcontainer.config_file=/home/runner/work/dbt-analytics/dbt-analytics/.devcontainer/devcontainer.json --entrypoint /bin/sh -l devcontainer.metadata=[{"id":"ghcr.io/devcontainers/features/common-utils:2"},{"id":"ghcr.io/devcontainers/features/git:1"},{"id":"ghcr.io/devcontainers/features/node:1","customizations":{"vscode":{"extensions":["dbaeumer.vscode-eslint"]}}},{"id":"ghcr.io/devcontainers/features/python:1","customizations":{"vscode":{"extensions":["ms-python.python","ms-python.vscode-pylance"],"settings":{"python.defaultInterpreterPath":"/usr/local/python/current/bin/python","python.formatting.autopep8Path":"/usr/local/py-utils/bin/autopep8","python.formatting.blackPath":"/usr/local/py-utils/bin/black","python.linting.flake8Path":"/usr/local/py-utils/bin/flake8","python.linting.flake8Enabled":false,"python.linting.mypyPath":"/usr/local/py-utils/bin/mypy","python.linting.mypyEnabled":false,"python.linting.pylintPath":"/usr/local/py-utils/bin/pylint","python.linting.pylintEnabled":false}}}},{"customizations":{"vscode":{"settings":{"python.defaultInterpreterPath":"/usr/local/bin/python"}}},"remoteUser":"vscode"},{"id":"ghcr.io/brooklyn-data/devcontainer-features/dbt:latest","entrypoint":"/usr/local/share/dbt-entrypoint.sh","customizations":{"vscode":{"extensions":["innoverio.vscode-dbt-power-user","samuelcolvin.jinjahtml","redhat.vscode-yaml"],"settings":{"files.associations":{"*.sql":"jinja-sql","*.yaml":"jinja-yaml","**/docs/**/*.md":"jinja-md"},"yaml.schemas":{"https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/dbt_yml_files.json":["**/models/**/*.yml"],"https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/dbt_project.json":["**/dbt_project.yml"],"https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/selectors.json":["**/selectors.yml"],"https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/packages.json":["**/packages.yml"]}}}}},{"customizations":{"vscode":{"settings":{"emeraldwalk.runonsave":{"commands":[{"match":"\\.sql","cmd":"sqlfluff fix ${file} --force"}]},"terminal.integrated.defaultProfile.linux":"zsh","python.formatting.provider":"none","python.linting.mypyEnabled":true,"python.analysis.typeCheckingMode":"basic","[python]":{"editor.defaultFormatter":"ms-python.black-formatter","editor.formatOnSave":true,"editor.codeActionsOnSave":{"source.organizeImports":true,"source.fixAll":true}},"black-formatter.importStrategy":"fromEnvironment","ruff.importStrategy":"fromEnvironment","ruff.path":["/usr/local/py-utils/bin/ruff"]},"extensions":["ms-python.python","ms-python.black-formatter","charliermarsh.ruff","ms-python.vscode-pylance","bungcip.better-toml","emeraldwalk.runonsave","EditorConfig.EditorConfig","eamodio.gitlens","github.vscode-pull-request-github","visualstudioexptteam.vscodeintellicode","GitHub.copilot"]}},"mounts":["source=${localEnv:HOME}/.dbt,target=/home/vscode/.dbt,type=bind"]},{"customizations":{"vscode":{"extensions":["esbenp.prettier-vscode","snowflake.snowflake-vsc","tamasfe.even-better-toml"]}}}] vsc-dbt-analytics-453ed9b86d2e8e601bbecdf50891fc14208adceeae247d72a5dbb43c3135946b-uid -c echo Container started
2023-08-24T20:03:34.4526857Z 
2023-08-24T20:03:34.7623519Z [2023-08-24T20:03:34.759Z] Container started
2023-08-24T20:03:34.7623832Z 
2023-08-24T20:03:35.1246380Z ##[endgroup]
2023-08-24T20:03:35.1256909Z ##[group]🚀 Run command in container
2023-08-24T20:03:35.1327117Z About to run devcontainer exec --workspace-folder /home/runner/work/dbt-analytics/dbt-analytics --remote-env SNOWFLAKE_DBT_PASSWORD=*** --remote-env GITHUB_OUTPUT=/mnt/github/output --remote-env GITHUB_ENV=/mnt/github/env --remote-env GITHUB_PATH=/mnt/github/path --remote-env GITHUB_STEP_SUMMARY=/mnt/github/step-summary bash -c dbt debug --target ci
2023-08-24T20:03:35.1328067Z dbt deps
2023-08-24T20:03:35.1328768Z dbt build --select state:modified+ --full-refresh --state target --target ci
2023-08-24T20:03:43.1214582Z ##[error]Dev container exec failed: (exit code: 2)
2023-08-24T20:03:43.1225298Z ##[error]Dev container exec failed: (exit code: 2)

[bamurtaugh]

Thanks for opening. @chrmarti curious to get your thoughts here with your work on #606.

[chrmarti]

@jecolvin What do you get for getent passwd $(id -un) in the container? Could you share the devcontainer.json and Dockerfile here?

[jecolvin]

@chrmarti

Output of getent passwd $(id -un):

vscode:x:1000:1000::/home/vscode:/bin/bash

devcontainer.json contents in the project, which pulls from a pre-compiled devcontainer we built:

{
    "name": "mfp-dbt-analytics",
    "image": "ghcr.io/brooklyn-data/devcontainers/dbt-snowflake:latest",
    "customizations": {
        "vscode": {
            "extensions": [
                "esbenp.prettier-vscode",
                "snowflake.snowflake-vsc",
                "tamasfe.even-better-toml"
            ]
        }
    }
}

devcontainer.json contents of pre-compiled image:

{
  // Base image configuration
  "name": "BDC dev container",
  "build": {
    "dockerfile": "./Dockerfile",
    "args": {
      "PYTHON_VERSION": "${localEnv:PYTHON_VERSION:3.10}"
    }
  },

  // Mount user's .dbt folder to access profiles.yml
  "mounts": ["source=${localEnv:HOME}/.dbt,target=/home/vscode/.dbt,type=bind"],

  "features": {
    // dbt configuration
    "ghcr.io/brooklyn-data/devcontainer-features/dbt:latest": {
      "version": "${localEnv:DBT_VERSION:latest}",
      "adapter": "${localEnv:DBT_ADAPTER:core}",
      "installDataframeLibrary": true,
      "installTools": true
    }
  },

  // Configure tool-specific properties.
  "customizations": {
    // Configure properties specific to VS Code.
    "vscode": {
      // Set *default* container specific settings.json values on container create.
      "settings": {
        "emeraldwalk.runonsave": {
          "commands": [
            { "match": "\\.sql", "cmd": "sqlfluff fix ${file} --force" }
          ]
        },
        "terminal.integrated.defaultProfile.linux": "zsh",
        "python.linting.enabled": true,
        "python.formatting.provider": "none",
        "python.linting.banditEnabled": true,
        "python.linting.mypyEnabled": true,
        "python.linting.pydocstyleEnabled": true,
        "python.analysis.typeCheckingMode": "basic",
        "[python]": {
          "editor.defaultFormatter": "ms-python.black-formatter",
          "editor.formatOnSave": true,
          "editor.codeActionsOnSave": {
            "source.organizeImports": true,
            "source.fixAll": true
          }
        },
        "black-formatter.importStrategy": "fromEnvironment",
        "ruff.importStrategy": "fromEnvironment",
        "ruff.path": ["/usr/local/py-utils/bin/ruff"]
      },

      // Add the IDs of extensions you want installed when the container is created.
      "extensions": [
        "ms-python.python",
        "ms-python.black-formatter",
        "charliermarsh.ruff",
        "ms-python.vscode-pylance",
        "tamasfe.even-better-toml",
        "emeraldwalk.runonsave",
        "EditorConfig.EditorConfig",
        "eamodio.gitlens",
        "github.vscode-pull-request-github",
        "visualstudioexptteam.vscodeintellicode",
        "GitHub.copilot"
      ]
    }
  },

  "postCreateCommand": "pre-commit install",

  "remoteUser": "vscode"
}

And its Dockerfile:

ARG PYTHON_VERSION=3

FROM mcr.microsoft.com/devcontainers/python:${PYTHON_VERSION}-bullseye

RUN pipx install --system-site-packages \
    --pip-args '--no-cache-dir --force-reinstall' \
    pre-commit; \
    pipx install --system-site-packages \
    --pip-args '--no-cache-dir --force-reinstall' \
    ruff;

[chrmarti]

Does it work with 0.48.0? (We switched to using getent in 0.49.0, though that seems to work in the container.)

[jecolvin]

No, it still fails in GHA. Everything works perfectly fine using the devcontainer locally, it's only the GHA that hits the error.

[chrmarti]

It seems to be unrelated to #606 then, that was introduced only with 0.49.0.

It looks like the exec fails with an exit code 2, is that correct? Is that exit code coming from the commands you run inside the container?

[jecolvin]

I guess so... I tried running with an echo command and that ran successfully. I'm not sure why though, we made no changes to this GHA workflow and it was running successfully 2 weeks ago. Here's a gist with the last successful run and then the next run two days later, which failed.

[jecolvin]

@chrmarti The recent update to the devcontainer CLI (v0.51.2) solved the issue. No changes in our GHA workflow, but all the commands are working again!