What do we want for the reproducer

[adrienbernede]

Introduction

RADIUSS Shared CI prints an almost(*) exact reproducer in the logs of each job.

The requirements were:

• Print a sequence of bash command anyone could run.
• Be as close as possible as possible to the CI job.
• Be standalone (no requirements like running in a pre-existing clone).

(*) The reproducer is almost exact, as it cannot capture the variables that where set in extra jobs. E.g., running address sanitizer may require to set some variables, those would not be know by the reproducer which is defined in RADIUSS Shared CI.

The current reproducer logic has defaults inherent to it’s initial specifications: It mimics the CI job and it’s optimizations, meaning it runs with the service account, in an allocation and using the shared memory (/dev/shm) to speed up the run.

Proposition

I suggest we improve this reproducer so that it becomes more convenient to use. In particular, I would like it to be easily usable by a developer to debug a CI job.

I would like to support 2 use cases

• Debugging a failure in Spack -> run spack with verbose / debug options on.
• Debugging a failure in the Build -> produce a build tree for dependencies and a build directory for the project that are persistent.

Options

1. Renounce to the exact reproducer and produce a list commands to run debug mode ? Both Spack in debug mode and the project build ?
2. Control the modes with variables ?
3. Print 3 commands, one for each mode (exact reproducer, spack debug, build debug).
4. Other options ?

[adrienbernede]

@rhornung67 @davidbeckingsale @mcfadden8

I am leaning toward option 3, printing something like:

xsu umdev
working_dir="/usr/workspace/${USER}/Umpire/develop-321704-$(date +%s)"
mkdir -p ${working_dir} && cd ${working_dir} 
git clone https://github.com/LLNL/Umpire.git --single-branch --depth=1 
cd Umpire
git fetch origin --depth=1 05d043ac7c73751ff5186ff2d476321c061974ca 
git checkout 05d043ac7c73751ff5186ff2d476321c061974ca 
git submodule update --init --recursive 
REPRODUCE_CMD="SPEC=\"~shared +fortran +tools tests=basic  %[email protected] \" USE_DEV_SHM=true MODULE_LIST=\"\" srun --overlap --reservation=ci --qos=ci_ruby --time=10 --nodes=1 ./scripts/gitlab/build_and_test.sh"
SPACK_DEBUG_CMD="SPEC=\"~shared +fortran +tools tests=basic  %[email protected] \" USE_DEV_SHM=false MODULE_LIST=\"\" srun --overlap --reservation=ci --qos=ci_ruby --time=10 --nodes=1 ./scripts/gitlab/build_and_test.sh --spack-debug"
BUILD_DEBUG_CMD="SPEC=\"~shared +fortran +tools tests=basic  %[email protected] \" USE_DEV_SHM=false MODULE_LIST=\"\" srun --overlap --reservation=ci --qos=ci_ruby --time=10 --nodes=1 ./scripts/gitlab/build_and_test.sh --build-debug"

Which prepares your shell for the 3 options, and all you have to do is, for example:

${SPACK_DEBUG_CMD}

[adrienbernede]

Since the reproducer is in the Shared CI, and we are trying to strip Spack out of the Shared CI, I just changed my mind:

My new preferred option is simply to default to USE_DEV_SHM=false in the build and test script. This keeps the reproducer simple.