Add setuptools_scm for pyproject.toml, setup.cfg and setup.py

[casparvl]

Set up automatic versioning by setuptools_scm

[casparvl]

Trying with pip install git+https:

pip install git+https://github.com/casparvl/test-suite.git@setuptools_scm_versioning
...
Successfully installed eessi-testsuite-0.3.3

Trying with a clone, and pip install .:

git clone https://github.com/casparvl/test-suite.git
cd test-suite/
git checkout setuptools_scm_versioning
pip install . 
...
Successfully installed eessi-testsuite-0.3.3

Running python3 -m build on that same checkout:

python3 -m build
...
Successfully built eessi_testsuite-0.3.3.tar.gz and eessi_testsuite-0.3.3-py3-none-any.whl

Running python3 setup.py sdist under that same checkout:

python3 setup.py sdist
...
$ ls -al dist/eessi-testsuite-0.3.3.tar.gz
-rw-r----- 1 casparl casparl 1843862 Sep 21 11:21 dist/eessi-testsuite-0.3.3.tar.gz

Note that all of this is done with a Python 3.9.18. I've tried the same with a Python 3.6 (on Karolina), all had the same effect: it assigned version 0.3.3 to the build.

[boegel]

I'm mainly worried about the tarball that people would get via GitHub, like https://github.com/EESSI/test-suite/archive/v0.3.2.tar.gz (via https://github.com/EESSI/test-suite/releases/tag/v0.3.2).

There we'll get 0.0.0, unless we do something extra here.

Could we still retain a version.py file or something, and use the version in there instead of using 0.0.0 as default if setuptools_scm is not available, or if it fails?

I'm as annoyed as you are that we have the version in multiple places currently, but we can avoid forgetting to update those in CI. With the current approach, there's no way to avoid people hitting the 0.0.0, which is a step backwards I think

[casparvl]

Yes, it gets it from the git metadata, so you need to have that available. In fact, a

wget https://github.com/casparvl/test-suite/archive/v0.3.3.tar.gz
tar -xvzf v0.3.3.tar.gz
cd test-suite-0.3.3/
pip install .

Will fail with

        raise LookupError(
    LookupError: setuptools-scm was unable to detect version for /tmp/pip-req-build-flbmoflr.

    Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

    For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

And, clearly, if you run it directly from the tarball (by setting your paths), you'll get the v 0.0.0.

Ok, I'll need to think about. I'm not against checking it in CI, but I also want dynamically created versions for when someone is running from a git checkout, so that we see they are not on a plain release version.

[casparvl]

So... I'm learning a lot here. There is a config option fallback_version, which should be used when the version cannot otherwise be determined (e.g. when you're installing from a release tarball from github, as Kenneth mentioned). But... it's only supported from setuptools_scm version 8 onwards, and that requires python 3.8 or newer.

That would be the cleanest approach. But, I think we can still do it differently. We can use setup.py to dynamically read the version. According to https://packaging.python.org/en/latest/guides/single-sourcing-package-version/#single-sourcing-the-version that's one way of 'single-sourcing the package version' (even if that would then require a hardcoded version). The good part about this approach is that it will use the same fallback version I specified myself in eessi.testsuite.__init__.py which is used if all the imports fail. It would retain the possibility of having setuptools_scm also available at runtime on a git cloned feature branch, in which case that will be used to create a .dev version.

I'll experiment a bit. Maybe I make the setup.py such that for 3.8 and above, it uses fallback_version, and for all others it relies on setting version in setup.py directly.

Note that even nicer is just setting

dynamic = ["version", "readme"]
# ...
[tool.setuptools.dynamic]
version = {attr = "my_package.__version__"}

In pyproject.toml, but that's only supported from setuptools 61.0.0 according to https://packaging.python.org/en/latest/guides/single-sourcing-package-version/#single-sourcing-the-version . And that requires Python 3.7 or above.

[smoors]

But... it's only supported from setuptools_scm version 8 onwards, and that requires python 3.8 or newer.

maybe it's ok to require python >= 3.9 ? CentOS-7 is out of support for some time now, and python-3.9 is available as an RPM in Rocky-8

[casparvl]

@boegel probably has an opinion here... I mean, it is a fact that we typically run the test suite with system python's. And we don't control all of those (e.g. Vega still has python 3.6). Then again: I also just made new easyConfigs for ReFrame at the toolchain level, in which case we would use the python's from the module env (and thus much newer ones). In that case, it's no problem at all.

[casparvl]

Anyway, I've been fiddling a bit. This

python_version = sys.version_info
if python_version < (3, 8):
    scm_dict = {'write_to': 'eessi/testsuite/_version.py'}
    scm_require = ['packaging<=21.3', 'setuptools_scm<7']
#    scm_arg_key = "write_to"
else:
    scm_dict = {'version_file': 'eessi/testsuite/_version.py', 'fallback_version': '80.0.0'}
    scm_require = ['setuptools>=61', 'setuptools_scm>=8']
#    scm_arg_key = "version_file"

# sys.path.append('.')
# from eessi.testsuite import __version__

setuptools.setup(
#    use_scm_version={scm_arg_key: 'eessi/testsuite/_version.py', 'fallback_version': 80.0.0},
    use_scm_version=scm_dict,
    setup_requires=scm_require,
)

is an option, and as you can see from the commented part, we could even import it from eessi.testsuite.__version__ and have it only hardcoded there. It works if I remove the pyproject.toml. My plan was to then set version directly for the Python 3.7 and 3.8 case (also imported from eessi.testsuite.__version__).

The problem is that I'm running into this issue that if there is a pyproject.toml the fallback_version from setup.py isn't used. For some reason, it's respecting the arguments for version_file argument from the setup.py, but uses the fallback_version from pyproject.toml (which doesn't exist, and thus it falls back to 0.0.0 or something).

And in pyproject.toml, I cannot do dynamic stuff, unless I assume new enough Python. Ugh, what a mess.

Of course, I could unconditionally set version = __version__ in the setuptools.setup call. But then, I don't have the advantage of getting a proper development version if I pip install git+https ... from a feature branch (which would work in the conditional case). I feel like I'm running in circles...

Also: using {attr = "my_package.__version__"} only helps if I can then use that as the configured fallback_version for setuptools_scm. Which I'm not sure I can do... I guess I can only make those calls in a [tool.setuptools.dynamic] section (but I'm not sure. See https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html . Who knows, I might be able to do fallback_version = {attr = "my_package.__version__"} in the setuptools_scm section...

[casparvl]

Oh, some good news: fallback_version is a way older argument than I expected. I couldn't find it in the code prior to v8, but it probably moved. https://pypi.org/project/setuptools-scm/6.4.2/ states that version 6.4.2 already supported it. This version also supports python 3.6. Now, the only problem I still have is that things go wrong if I dynamically read it in setup.py, because this is ignored when pyproject.toml is used.

A solution there would be to let go of the 'single version' principle, and define it in two places: the pyproject.toml and in eessi.testsuite.__init__.py. Another solution is to make eessi.testsuite.__init__.py smarter and have it parse the pyproject.toml to specify the version. That's a little unusual and would make the pyroject.toml needed at runtime, but only for those cases where the first two methods (reading it from the eessi.testsuite._version.py) in eessi.testsuite.__init__.py failed to provide a version.

That means we have the following scenarios:

1. At installation time, when installing as a package, from a release tarball (e.g. https://github.com///archive/refs/tags/<tag_name>.tar.gz). Here, we don't have git (scm) info. Thus, we rely on the hard-coded value for fallback_version in pyproject.toml. In case of installing with setup.py and setup.cfg directly, we'd be relying on the from eessi.testsuite import __version__ (which in turn will parse it from the pyproject.toml). In this scenario, the version will always be a release version, but that's ok: we're installing from a release tarball, so it is a release.
2. At installation time, when installing as a package, from a feature branch tarball (e..g. https://github.com///archive/refs/heads/<branch_name>.tar.gz). Here, we don't have git (scm) info. Thus, we rely on the hard-coded value for fallback_version in pyproject.toml. Note that we won't get development versions in this case, even though we would want that ideally. There is however no information anymore to generate the development version from - that tarball simply doesn't contain anything.
3. Installed as a package. Here, setuptools_scm has generated the eessi.testsuite._version.py when performing the installation. It doesn't matter if this package was installed by cloning the repo and doing pip install ., or pip install git+https, or (I think) a pip install from pypi. The last one should also provide the right version, since we make the wheels using python setup.py sdist on a git clone of the release. That python setup.py sdist will invoke setuptools_scm to (correctly) generate the eessi.testsuite._version.py file, which will subsequently be used by the installation at runtime. It will also have that same version used as the version name for the installed package (i.e. the version you see with pip list). Note both a pip install . and a pip install git+https will correctly result in development versions being generated (for pypi releases, that's a non-issue, since those are always release versions anyway).
4. Cloned from git (and running from there). Here, setuptools_scm has all the scm info it needs to generate a version on the fly, based on the current commit and tag information from the repo. Note that this is how our daily runs of the testsuite are done, so they will benefit from development versioning.

I'll try this approach: specify the fallback_version in pyproject.toml, have eessi.testsuite.__init__.py parse that file to obtain it, and have setup.py get it from eessi.testsuite.__init__.py. That would allow us to get the correct versions in all but the case where you run from a non-release tarball (in which the information is fundamentally missing, so no matter how smart the tool, we cannot create a development version), and the version is only specified in one location (pyproject.toml).

[casparvl]

It only took 67 commits to get to a point where this works, and is also properly tested. 😅

[smoors]

approving this as the tests seem to work, although i didn't rigorously check the code.
if something breaks, i count on you to fix it :)