Ubuntu testing CI is flaky due to installation issues

[RastislavTuranyi]

The CI on the Ubuntu runner is flaky - it fails some of the time because the Install tox step errors out some of the time. I have not observed this happening on the Windows runner. Here's the error:

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
pkgs/main/linux-64                                          Using cache
pkgs/main/noarch                                            Using cache
pkgs/r/linux-64                                             Using cache
pkgs/r/noarch                                               Using cache
Transaction

  Prefix: /home/runner/miniconda3/envs/test

  Updating specs:

   - tox
   - ca-certificates
   - openssl


  Package              Version  Build            Channel          Size
────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────

  + colorama             0.4.6  pyhd8ed1ab_1     conda-forge      27kB
  + cachetools           5.5.1  pyhd8ed1ab_0     conda-forge      15kB
  + filelock            3.17.0  pyhd8ed1ab_0     conda-forge      18kB
  + packaging             24.2  pyhd8ed1ab_2     conda-forge      60kB
  + platformdirs         4.3.6  pyhd8ed1ab_1     conda-forge      20kB
  + pluggy               1.5.0  pyhd8ed1ab_1     conda-forge      24kB
  + tomli                2.2.1  pyhd8ed1ab_1     conda-forge      19kB
  + typing_extensions   4.12.2  pyha770c72_1     conda-forge      40kB
  + distlib              0.3.9  pyhd8ed1ab_1     conda-forge     274kB
  + pyproject-api        1.9.0  pyh29332c3_0     conda-forge      26kB
  + virtualenv         20.29.1  pyhd8ed1ab_0     conda-forge       4MB
  + python_abi            3.10  5_cp310          conda-forge       6kB
  + chardet              5.2.0  py310hff52083_2  conda-forge     243kB
  + tox                 4.24.1  pyh29332c3_0     conda-forge     190kB

  Summary:

  Install: 14 packages

  Total download: 4MB

────────────────────────────────────────────────────────────────────────


error    libmamba Error when extracting package: Could not stat site-packages/cachetools-5.5.1.dist-info/direct_url.json
cachetools-5.5.1-pyhd8ed1ab_0.conda extraction failed
Found incorrect download: cachetools. Aborting

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/runner/miniconda3/lib/python3.12/site-packages/conda/exception_handler.py", line 18, in __call__
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/miniconda3/lib/python3.12/site-packages/mamba/mamba.py", line 960, in exception_converter
        raise e
      File "/home/runner/miniconda3/lib/python3.12/site-packages/mamba/mamba.py", line 953, in exception_converter
        exit_code = _wrapped_main(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/miniconda3/lib/python3.12/site-packages/mamba/mamba.py", line 899, in _wrapped_main
        result = do_call(parsed_args, p)
                 ^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/miniconda3/lib/python3.12/site-packages/mamba/mamba.py", line 764, in do_call
        exit_code = install(args, parser, "install")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/miniconda3/lib/python3.12/site-packages/mamba/mamba.py", line 559, in install
        transaction.fetch_extract_packages()
    RuntimeError: Found incorrect download: cachetools. Aborting


`$ /home/runner/miniconda3/condabin/mamba install tox`

  environment variables:
                 CIO_TEST=<not set>
                    CONDA=/home/runner/miniconda3
        CONDA_DEFAULT_ENV=test
                CONDA_EXE=/home/runner/miniconda3/bin/conda
           CONDA_PKGS_DIR=/home/runner/conda_pkgs_dir
             CONDA_PREFIX=/home/runner/miniconda3/envs/test
           CONDA_PREFIX_1=/home/runner/miniconda3
    CONDA_PROMPT_MODIFIER=
         CONDA_PYTHON_EXE=/home/runner/miniconda3/bin/python
               CONDA_ROOT=/home/runner/miniconda3
              CONDA_SHLVL=2
           CURL_CA_BUNDLE=<not set>
      DEPLOYMENT_BASEPATH=/opt/runner
        GITHUB_EVENT_PATH=/home/runner/work/_temp/_github_workflow/event.json
              GITHUB_PATH=/home/runner/work/_temp/_runner_file_commands/add_path_4a2a04c4-91ec-
                          437c-9fe4-f6395f7b81d6
               LD_PRELOAD=<not set>
                     PATH=/home/runner/miniconda3/envs/test/bin:/home/runner/miniconda3/condabin
                          :/home/runner/miniconda3/condabin:/snap/bin:/home/runner/.local/bin:/o
                          pt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vend
                          or/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbi
                          n:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/g
                          ames:/snap/bin
       REQUESTS_CA_BUNDLE=<not set>
        SELENIUM_JAR_PATH=/usr/share/java/selenium-server.jar
            SSL_CERT_FILE=<not set>
               SWIFT_PATH=/usr/share/swift/usr/bin

     active environment : test
    active env location : /home/runner/miniconda3/envs/test
            shell level : 2
       user config file : /home/runner/.condarc
 populated config files : /home/runner/miniconda3/.condarc
                          /home/runner/.condarc
          conda version : 24.11.3
    conda-build version : not installed
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen2
                          __conda=24.11.3=0
                          __glibc=2.39=0
                          __linux=6.8.0=0
                          __unix=0=0
       base environment : /home/runner/miniconda3  (writable)
      conda av data dir : /home/runner/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/runner/conda_pkgs_dir
       envs directories : /home/runner/miniconda3/envs
                          /home/runner/.conda/envs
               platform : linux-64
             user-agent : conda/24.11.3 requests/2.32.3 CPython/3.12.8 Linux/6.8.0-1020-azure ubuntu/24.04.1 glibc/2.39 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.12
                UID:GID : 1001:118
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.
If you suspect this error is being caused by a malfunctioning plugin,
consider using the --no-plugins option to turn off plugins.

Example: conda --no-plugins install <package>

Alternatively, you can set the CONDA_NO_PLUGINS environment variable on
the command line to run the command without plugins enabled.

Example: CONDA_NO_PLUGINS=true conda install <package>

Looking for: ['tox']


Pinned packages:
  - python 3.10.*

[ajjackson]

That is quite annoying, especially as there doesn't seem to be much consistency to exactly what conda falls over on. (If so it might be resolved by pinning some packages?)

An option that could make the conda packaging a bit more consistent is to use the "cache" action to maintain the conda package cache and/or the main environment that contains tox. pytest-conda would still want to build new environments but would use that cache.

It's not clear if that would help/hinder performance as ultimately we are downloading the same packages (and adding an upload step!)

Let's hold off for a few more days and see if this keeps happening.