Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make docker sandbox compose up wait time configurable #1270

Closed
wants to merge 2 commits into from
Closed

Make docker sandbox compose up wait time configurable #1270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2fa097b
Allow configurable wait time for compose up via environment variable
tw-aisi Jan 26, 2025
f566d0d
Add documentation on INSPECT_EVAL_COMPOSE_UP_WAIT feature
tw-aisi Feb 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/sandboxing.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,3 +336,8 @@ services:
To diagnose sandbox execution issues (e.g. commands that don't terminate properly, contianer lifecylce issues, etc.) you should use Inspect's [Tracing](tracing.qmd) facility.

Trace logs record the beginning and end of calls to `subprocess()` (e.g. tool calls that run commands in sandboxes) as well as control commands sent to Docker Compose. The `inspect trace anomalies` subcommand then enables you to query for commands that don't terminate, timeout, or have errors. See the article on [Tracing](tracing.qmd) for additional details.

### Sandbox container startup wait time
For complicated sandboxes that take multiple minutes to fully spin up, you may run into an error of the form "One or more docker containers failed to start from [...]". This error occurs because by default, Inspect will only wait 120 seconds for all sandbox containers to start.

To override this 120 second default wait time, you can set the environment variable `INSPECT_EVAL_COMPOSE_UP_WAIT` which specifies in seconds how long inspect should wait for sandbox containers to spin up.
9 changes: 6 additions & 3 deletions src/inspect_ai/util/_sandbox/docker/compose.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,14 @@ async def compose_up(project: ComposeProject) -> None:
# passing the --wait flag (see https://github.com/docker/compose/issues/10596).
# In practice, we will catch any errors when calling compose_check_running()
# immediately after we call compose_up().
seconds_to_wait_str: str = os.environ.get(
"INSPECT_EVAL_COMPOSE_UP_WAIT", COMPOSE_WAIT
)
await compose_command(
["up", "--detach", "--wait", "--wait-timeout", COMPOSE_WAIT],
["up", "--detach", "--wait", "--wait-timeout", seconds_to_wait_str],
project=project,
# wait up to 5 minutes for container to go up (compose wait + 3 minutes)
timeout=300,
# wait up to (compose wait + 3 minutes) for container to go up
timeout=int(seconds_to_wait_str) + 180,
)


Expand Down