[Bug]: The missing runner binary leads to CREATING_CONTAINER_ERROR after clearing /tmp

[r4victor]

Steps to reproduce

1. Create an SSH fleet.
2. Clear /tmp on the fleet instance.
3. See runs fail with CREATING_CONTAINER_ERROR:

[11:49:54] WARNING  dstack._internal.server.background.tasks.process_running_jobs
                    :475 shim failed to execute job massive-pug-1-1-0:           
                    CREATING_CONTAINER_ERROR (createContainer error: Error       
                    response from daemon: invalid mount config for type "bind":  
                    bind source path does not exist:                             
                    /tmp/dstack-runner1354965780)

This is caused by the fact that the shim downloads the runner binary into /tmp at startup and then mounts the runner into the container. If the /tmp is cleared after the runner is downloaded, then mounting fails.

Actual behaviour

No response

Expected behaviour

No response

dstack version

master

Server logs

No response

Additional information

No response

[r4victor]

@un-def, any chance you get it fixed with the ongoing shim refactoring?

[un-def]

@r4victor is there any reason why we download runner to /tmp on each shim (re)start? Couldn't we download runner once to a well-known location (e.g., /usr/local/bin/shim-runner, as with shim)?

[r4victor]

@un-def, I don't think there is a good reason to download the shim to /tmp – just a historical quirk. So the fix should be straightforward as you described. Although, there are some things to keep in mind like the runner should be re-downloaded on shim version update / SSH fleet re-creation.