Allow for alternative Singularity data mount or better mount point detection

[matthewfeickert]

Issue

Currently packtivity selects the root path of a user's $HOME to use as the data mount point

packtivity/packtivity/handlers/execution_handlers.py

Lines 424 to 438 in e7020f5

	def run_containers_in_singularity_runtime(config, state, log, metadata, race_spec):
	import tempfile
	import shutil
	tmpdir_home = tempfile.mkdtemp(prefix="_sing_home_")
	tmpdir_work = tempfile.mkdtemp(prefix="{}/".format(tmpdir_home))
	homemount = "/".join(os.path.expanduser("~").split("/")[:2])
	cmdline = singularity_execution_cmdline(
	state,
	log,
	metadata,
	race_spec,
	dirs={"work": tmpdir_work, "home": tmpdir_home, "datamount": homemount},
	)

This means somewhere like LXPLUS8 you get

$ echo $HOME
/afs/cern.ch/user/f/feickert

and so

datamount="/afs"

while if you're somewhere like the Analysis Facility at UChicago you'd get

$ echo $HOME
/home/feickert

and so

datamount="/home"

Example

This is fine, but can cause problems for the way that Singularity interacts with the local file system when mounting. For example, if a user makes a Python virtual environment and installs recast-atlas[local] and tries to run the examples/rome workflow at the UChicago AF with a script like

#!/bin/bash

export PACKTIVITY_CONTAINER_RUNTIME=singularity
export SINGULARITY_CACHEDIR="/tmp/$(whoami)/singularity"

mkdir -p "${SINGULARITY_CACHEDIR}"

# Confirm workflow
recast catalogue ls
recast catalogue describe examples/rome
recast catalogue check examples/rome

recast run examples/rome --backend local --tag examples-rome

it will fail as the steps that the eventselection stage runs through in the container include

source /home/atlas/release_setup.sh

With the data mount set to /home the command packtivity runs would be something like

singularity exec -C  -B /home:/home --pwd /tmp/_sing_home_82sbnqfs/r6cndyad -H /tmp/_sing_home_82sbnqfs docker://reanahub/reana-demo-atlas-recast-eventselection:1.0 sh -c bash

which given how Singularity handles bind mounts means that the path /home/atlas in the container doesn't exist anymore as it has gotten clobbered by the UChicago filesystem's /home, causing the workflow to fail.

Proposed Solution or Idea

There should either be some way to set an alternative datamount in

packtivity/packtivity/handlers/execution_handlers.py

Line 424 in e7020f5

def run_containers_in_singularity_runtime(config, state, log, metadata, race_spec):

(maybe via environmental variable?), or there should be an alternative method for setting the datamount (this seems hard to do in general). @lukasheinrich might have smarter ideas here.

Other Related Issues

• I think that this might be what @ntadej was getting at in Issue Singularity should not assume data is in home #69, but I'll let them confirm that.
• Cross posted with https://gitlab.cern.ch/recast-atlas/docs/-/issues/10