Generic Toil Utilities require cluster utilities

[lonbar]

Expected behaviour

At least according to the documentation, generic toil utilties don't require cluster utilities. The command toil clean should be available without installing extras.

Actual behaviour

Currently toil must be installed with [aws] or [google] extras if generic utilties are to be available. Consider the following workflow:

$ python -m venv venv && source venv/bin/activate
$ pip3 install toil
$ toil debug-job --help
Traceback (most recent call last):
  File "/home/me/venv/bin/toil", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/me/venv/lib/python3.12/site-packages/toil/utils/toilMain.py", line 14, in main
    modules = loadModules()
              ^^^^^^^^^^^^^
  File "/home/me/venv/lib/python3.12/site-packages/toil/utils/toilMain.py", line 55, in loadModules
    from toil.utils import toilLaunchCluster  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/venv/lib/python3.12/site-packages/toil/utils/toilLaunchCluster.py", line 22, in <module>
    from toil.lib.aws import build_tag_dict_from_env
  File "/home/me/venv/lib/python3.12/site-packages/toil/lib/aws/__init__.py", line 25, in <module>
    from botocore.exceptions import ClientError
ModuleNotFoundError: No module named 'botocore'
$ pip3 install toil[aws] && toil debug-job --help
usage: toil debug-job [-h] [--logCritical] [--logError] [--logWarning] [--logDebug] [--logInfo] [--logOff]
                      [--logLevel {Critical,Error,Warning,Debug,Info,critical,error,warning,debug,info,CRITICAL,ERROR,WARNING,DEBUG,INFO}] [--logFile LOGFILE] [--rotatingLogging]
                      [--logColors BOOL] [--version] [--tempDirRoot TEMPDIRROOT] [--printJobInfo] [--retrieveTaskDirectory RETRIEVE_TASK_DIRECTORY]
                      jobStore job

positional arguments:
  jobStore              The location of the job store for the workflow. A job store holds persistent information about the jobs, stats, and files in a workflow. If the workflow is
                        run with a distributed batch system, the job store must be accessible by all worker nodes. Depending on the desired job store implementation, the location
                        should be formatted according to one of the following schemes: file:<path> where <path> points to a directory on the file system aws:<region>:<prefix> where
                        <region> is the name of an AWS region like us-west-2 and <prefix> will be prepended to the names of any top-level AWS resources in use by job store, e.g. S3
                        buckets. google:<project_id>:<prefix> TODO: explain For backwards compatibility, you may also specify ./foo (equivalent to file:./foo or just file:foo) or
                        /bar (equivalent to file:/bar).
  job                   The job store id or job name of a job within the provided jobstore

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --tempDirRoot TEMPDIRROOT
                        Path to where temporary directory containing all temp files are created, by default generates a fresh tmp dir with 'tempfile.gettempdir()'. (default: /tmp)
  --printJobInfo        Dump debugging info about the job instead of running it (default: False)
  --retrieveTaskDirectory RETRIEVE_TASK_DIRECTORY
                        Download CWL or WDL task inputs to the given directory and stop. (default: None)

Logging Options:
  --logCritical         Turn on loglevel Critical. Default: DEBUG. (default: DEBUG)
  --logError            Turn on loglevel Error. Default: DEBUG. (default: DEBUG)
  --logWarning          Turn on loglevel Warning. Default: DEBUG. (default: DEBUG)
  --logDebug            Turn on loglevel Debug. Default: DEBUG. (default: DEBUG)
  --logInfo             Turn on loglevel Info. Default: DEBUG. (default: DEBUG)
  --logOff              Same as --logCRITICAL. (default: DEBUG)
  --logLevel {Critical,Error,Warning,Debug,Info,critical,error,warning,debug,info,CRITICAL,ERROR,WARNING,DEBUG,INFO}
                        Set the log level. Default: DEBUG. Options: ['Critical', 'Error', 'Warning', 'Debug', 'Info', 'critical', 'error', 'warning', 'debug', 'info', 'CRITICAL',
                        'ERROR', 'WARNING', 'DEBUG', 'INFO']. (default: DEBUG)
  --logFile LOGFILE     File to log in. (default: None)
  --rotatingLogging     Turn on rotating logging, which prevents log files from getting too big. (default: False)
  --logColors BOOL      Enable or disable colored logging. Default: True

The reason appears to be that utils/toilMain.py attempts to import both generic and cluster utilities, without checking if those utilities are available or not.

$ toil --version
7.0.0

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1675

[stxue1]

I fixed this a while ago in #4953. The latest version of toil from source has this working (7.1.0a1). We will do a new release pretty soon that should resolve this issue.