Adding new environment variables to rocoto/job card?

[benjamin-cash]

There is a set of environment variables that I need to add to batch scripts on Frontera to run successfully. So far in my testing I have been adding them manually to the .xml file after setup_expt runs, but I would like to have them automatically included. Where should I add them so that they will be picked up and included in the task information in the .xml?

[KateFriedman-NOAA]

@benjamin-cash If they are settings for compute/runtime and are not configuration variables then they should go in the respective env file for the machine (e.g. env/HERCULES.env). Perhaps, could you share the variables you need to add and I can help determine where they should go, if not in the env file that is?

[benjamin-cash]

Hi @KateFriedman-NOAA - Thanks for the quick reply! The variables I need are:
BASH_COMPLETION
BASH_COMPLETION_DIR
BASH_COMPLETION_COMPAT_DIR

If these are not set when running on Frontera one of the modules will error out due to unbound variables. So these need to be set in the job script, at the same level as HOMEgfs, EXPDIR, etc. I tried adding one of them as a test to my FRONTERA.env but it didn't appear in the .xml or the batch script, so it seems like I didn't put it in the right location.

[KateFriedman-NOAA]

Ah I see, yeah, the env file isn't sourced until after modules are loaded and configs have been sourced. Question: do you need these variables for all jobs or just some? Will the values always be the same or will they change from cycle to cycle or job to job?

[benjamin-cash]

Any job that uses a built executable needs these, and the values can stay the same. I think the values are actually irrelevant, they just can't be unbound.

Getting into the weeds a bit, all of my executables are built using a containerized version of spack-stack. On Frontera, containers are handled via apptainer, and there is a script in the apptainer module that will error out with an unbound variable error if these are not set. This is the only machine where I have to do this, but I haven't found any other path past it than setting the variables in the job script.

[benjamin-cash]

Manually adding them to the xml works, but it is definitely tedious and error-prone.

[KateFriedman-NOAA]

Gotcha. Given the modules need them I suggest adding them to the module_base for Frontera. Add them near the top so they are set before the modules load. Set them like setenv("BASH_COMPLETION","VALUE_FOR_BASH_COMPLETION").

[DavidHuber-NOAA]

Putting these variables in module_base may work, but I'm not sure how lmod sets the environment variables -- if it load modules first or sets variables first. I ran into an issue on WCOSS2 where LMOD_TMOD_FIND_FIRST needed to be set to prevent default modules from being loaded, but adding it to module_base did not do the trick. Instead, I had to put it in module-setup.sh.

So if adding these BASH* variables to module_base does not do the trick, I would suggest module-setup.sh.

[benjamin-cash]

Thanks @KateFriedman-NOAA ! That did the trick. :) @DavidHuber-NOAA - appreciate you jumping in as well, I'll bear that it mind if I run into a scenario like that.

[DavidHuber-NOAA]

Great, glad to hear it!