feat(spec): add completion deadline (#206)

When provisioning the pod, we have a 30min timeout, but this timeout is
not utilized in the pod spec in any way.

Provisioned pods on MP+ take quota from the non-terminating pool, but in
the case of sandbox environment we do have a deadline, since the pods
are one-shot and destroyed after running. Adding this deadline
explicitly to the spec results in OpenShift terminating the pods by
itself and additionally uses a different pool of quota, the terminating
one.

For the deadline in spec I have chosen to use our default 30min + 10min
for provisioning, because from the PoV of OpenShift it is included in
the deadline.

Related to the failed pull-from-upstream run from Apr 29 2024, as has
been reported on the Matrix channel on May 2 2024:


https://matrix.to/#/!ySjfdvNvPOsVtWHNDb:fedora.im/$FwhFOOhbbh1N2NXbVoA5oH3b97f8a2kzRSyhy9F2-F8?via=fedora.im&via=libera.chat&via=matrix.org

See details:


https://docs.openshift.com/container-platform/3.11/dev_guide/builds/advanced_build_operations.html#builds-setting-maximum-duration

files/container-cmd.sh

@@ -4,5 +4,6 @@ set -x
 
 source ./setup_env_in_openshift.sh
 
+# [WARN] When adjusting, don't forget the timeout in spec (sandcastle/api.py)
 # 30 minutes is a default timeout for the sandbox pod
 sleep 1800

sandcastle/api.py

@@ -233,6 +233,8 @@ def set_pod_manifest(self, command: Optional[List] = None):
             "containers": [container],
             "restartPolicy": "Never",
             "automountServiceAccountToken": False,
+            # [NOTE] 1800s which is the default time out + 600s for the provisioning
+            "completionDeadlineSeconds": 1800 + 600,
         }
         self.pod_manifest = {
             "apiVersion": "v1",

tests/unit/test_pod_and_env.py

@@ -222,6 +222,7 @@ def test_manifest(init_openshift_deployer):
                 }
             ],
             "restartPolicy": "Never",
+            "completionDeadlineSeconds": 1800 + 600,
         },
     }
     sandcastle.env_vars = {KEY: VALUE}