Fix: terminate hanging jobs

[hbruch]

This PR

• adds a script executed on startup to terminate runs still in started state.
• enables run monitoring to terminate jobs hanging on startup/cancellation (after 180s) or running for more than 6h

[the-infinity]

Everything else looks good.

[derhuerst]

The background job (sub process) started here might fail for any reason, including /opt/dagster/app/scripts/terminate_starting_and_started_runs.py missing. I'm afraid that this will cause further headaches down the road, at some inconvenient moment, in production. I strongly suggest that we change the script to crash (exit with non-0) the contain if this happens.

It seems very hard to get the following behaviour solely with a Bash script:

• If the background job fails, also quit the main process.
• If the main process exits, don't leave the background job running.
• If both run through cleanly, exit as usual.

After experimenting for too long (with many variants of 1, 2 & 3), I propose just using tini in combination with kill "-$$" as a workaround.

#!/bin/bash

set -eo pipefail

echo 'Running terminate_starting_and_started_runs.py as a background job.' >&2

set -x

# In case the background job fails, we kill the entire shell ($$ has its PID) and all its children (by negating the PID).
# This *does not* work if `$$` evaluates to 1 (our shell is the init process), so we *must* run this script with an "external" init command.
python /opt/dagster/app/scripts/terminate_starting_and_started_runs.py \
	|| kill -TERM -- "-$$" &

exec "$@"
``

[derhuerst]

I think it's likely that we'll want to change this soon, so let's make it configurable via an environment variable.

[derhuerst]

(See the comment about background jobs below.)

Suggested change

	COPY scripts/ /opt/dagster/app/scripts/
	ARG TARGETARCH
	ENV TINI_VERSION=v0.19.0
	ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-${TARGETARCH} /tini
	RUN chmod +x /tini && /tini --version
	COPY scripts/ /opt/dagster/app/scripts/
	ENTRYPOINT [ "/tini", "--", "/opt/dagster/app/scripts/start_runs_termination_script_in_background.sh" ]

[derhuerst]

I'm afraid that this change will cause more headaches than helping us. Effectively, we have two race conditions here:

1. If Dagster and its GraphQL API don't start quickly enough, we either don't clean hanging jobs (this PR's goal) or we crash (as proposed in my comments).
2. If the Python script for some reason is too slow act (right after Dagster has started and not started new jobs yet because of auto-materialization), it might accidentally kill legitimate jobs.