Ensure Huggingface, Torch Packages Up-to-Date with Necessary Improvements/Fixes

[fabianlim]

@hickeyma @Ssukriti @tedhtchang

The platform is build upon the torch and huggingface ecosystem whereby there are constant fixes that are pushed every release.

• sometimes we also require to be on nightly pre-releases as there is a biweekly-monthly lag for each release.
• The fear of upgrading packages comes with breaking APIs, which will severely impact any CI/CD testing. On the other hand, fixes improve memory consumption and training speed, and will be a good idea to regress the codebase with the latest nightlies.

Desiderata

1. Have a ground source of truth for package versions. A good choice would be pyproject.toml that can be used to also address Python package dependency management for building/development #56 , Build the Python wheel for release, for pip installs #57
2. Have a system whereby nightlies (or new releases) can be constantly regressed with new code changes (a good choice would be to have one such regression for each new PR).
   • will be a good idea to also consider some means to disable regressions, for PRs that do not update source code (e.g., documentation updates).
3. Have a means to freeze package versions during CD/CI testing, so that if a test fails, we know exactly on what versions it has failed for. And vice-versa, if a commit is known to work, we know exactly for what versions it worked for.

NPM addresses these desiderata for node

See here:

• npm install will occur from package.json with semvar versioning that tells what kind of upgrades (e.g., minor, patch) are allowed.
  • during npm install the packages will be resolved to install the latest packages that satisfy the constraints.
  • the final versions of packages installed will be output in package-lock.json.
  • the package-lock.json is typically checked to have a record of what exact package versions are working.
• during CI/CD, all tests are run only after npm ci; note that this differs from npm install, in that it installs from package-lock.json. Thus there is no package resolution during testing. This ensures if the tests pass, we know exactly what package versions they pass for.

Current Solution

Use dependabot to automatically check for new versions, and then raise a PR to upgrade the package version.

Other Solutions

Two possibilities:

1. Poetry has poetry.lock, that works together with pyproject.toml. The poetry flow as described here is completely analogue with the npm flow described above.
2. Using pip-compile to generate requirements.txt from pyproject.toml and checking that in as the lock file.

[Ssukriti]

@gkumbhat is thinking about how to do dependency management and will update when he has a solution

[VassilisVassiliadis]

@Ssukriti I was going to open a new issue for reproducible/deterministic virtual environments but I think that it would overlap quite a bit with this issue.

Let me know how you'd like to proceed. This is what I had in mind for a new issue:

Support creation of deterministic virtual environments for fms-hf-tuning

Is your feature request related to a problem? Please describe.

We are using fms-hf-tuning to profile LLMs when installing fms-hf-tuning from source we do not get reproducible virtual experiments. This means that if we install the same version of fms-hf-tuning at 2 different points in time we may end up installing wildly different versions of the python packages that fms-hf-tuning depends on. In turn this can influence the profiling information we collect as well as the stability of fms-hf-tuning itself.

Describe the solution you'd like

We would like the fms-hf-tuning codebase to support reproducible/deterministic creation of virtual environments. One way to achieve this is to add a requirements.txt file with the most recent versions of the python dependencies for which fms-hf-tuning is known/expected to work.

Then, fms-hf-tuning should:

1. update this requirements.txt file to reflect the most recent versions of python dependencies with which fms-hf-tuning is expected to work
2. use this requirements.txt file for CI/CD tests
3. use this requirements.txt file for building the virtual environment in the fms-hf-tuning image

Describe alternatives you've considered

Alternative, we could update the pyproject.toml file to use poetry. The poetry framework comes with a poetry.lock file that is a superset of requirements.txt and ships with a CLI tool for managing python dependencies. The price is that you need to install poetry in your virtual environment prior to installing fms-hf-tuning so that you get the benefits of a deterministic virtual environment. The poetry.lock file is not consistent with the schema of the requirements.txt file and therefore pip cannot use it directly.

Another option is to use poetry and have it generate an equivalent requirements.txt file which is also version controlled (e.g. poetry export --format requirements.txt)

Additional context

The goal here is to:

1. have the option for deterministic virtual environment
   1. blindly installing the package from pypi will get you whichever versions pip decides to install based on the python dependency constraints in pyproject.toml
   2. installing requirements.txt and then installing fms-hf-tuning will produce a deterministic virtual environment
2. the virtual environment in the image should be deterministic
3. CI/CD and other external runs (e.g. profiling measurements) should use deterministic virtual environments tied
   1. this way we can track the effects of changes to specific commits of fms-hf-tuning
4. maintainers consciously make the decision to update the versions of python dependencies of fms-hf-tuning

[VassilisVassiliadis]

@tedhtchang we have a couple of suggestions in this issue for ways to version control the versions of python dependencies.

For what it's worth, I started playing with poetry on a fork of fms-hf-tuning: https://github.com/VassilisVassiliadis/fms-hf-tuning/blob/poetry/pyproject.toml

The resulting poetry lock file is here: https://github.com/VassilisVassiliadis/fms-hf-tuning/blob/poetry/poetry.lock

[tedhtchang]

@VassilisVassiliadis The poetry framework is a good suggestion. Seems like we may need to change how we run test, release, and build packages. Would you like to create a PR ?

[VassilisVassiliadis]

I'll give it a go