Skip to content

PiPY Wheel Release Instructions

Jump to bottom
James Busche edited this page Jun 27, 2024 · 62 revisions

Table of Contents

Goals

Every 2 weeks we can create a fms-hf-tuning pypi release:

Ability to test changes before push to main and before release

  • pre-release candidate is created off the main branch, which allows us to test
  • When we're ready to consider the release done, We sync the release branch with the release candidate tag Then we create a release without the rc# convention.

Release branch that is stable and tested that RH pulls from

  • ensure that this is using the release wheel with semver tag --> all using same commit

Creating a new wheel pre-release:

Pre-reqs:

0.1 Need to be an admin in https://github.com/foundation-model-stack/fms-hf-tuning

0.2 Need to know what the next release will be... can check https://pypi.org/project/fms-hf-tuning/#history

Step 1. Create a pre-release candidate from main

1.1 Go to https://github.com/foundation-model-stack/fms-hf-tuning and on the lower right under "Releases" Click on "Create a new release"

1.2 Click on "Choose a tag" and type in the new tag name. For example, v0.1.0-rc.1 and press "create tag on publish"

1.3 For the other items:

For release title, I'm using: v0.2.0-rc.1
For Target - Leave it at main
Click on "Generate Release notes"
Don't Click on "Set as the latest release"
Click on "Set as pre-release" button.

1.4 Click on Publish release

Step 2. Check the github actions to make sure it runs to completion

https://github.com/foundation-model-stack/fms-hf-tuning/actions

It should show up as available as a release on pypi here: https://pypi.org/project/fms-hf-tuning/#history

Step 3. Initial verifying of the release:

3.1. See that you can clone the new tag:

git clone --branch <tag or release name> https://github.com/foundation-model-stack/fms-hf-tuning.git

for example:

git clone --branch v0.2.0-rc.1 https://github.com/foundation-model-stack/fms-hf-tuning.git

3.2. See if you can install the new pypi file:

python -m venv .env
source .env/bin/activate
pip install --upgrade pip
pip install fms-hf-tuning

and it should return the right version

pip list |grep fms
fms-hf-tuning            v0.2.0-rc.1

Step 4. Create the Ready for Red Hat release when the IBM release candidate testing is complete and we're ready to go:

4.1 Merge the release candidate into the release branch. I did this on my laptop with the following steps:

mkdir RELEASE ; cd RELEASE
git clone https://github.com/foundation-model-stack/fms-hf-tuning.git -b release
cd fms-hf-tuning
git branch
git remote -v
git fetch --tags origin
git tag -l
git merge <tag_name>  for example
git merge v0.2.0-rc.1
git push

Step 5. Create a new real release:

5.1 Go to https://github.com/foundation-model-stack/fms-hf-tuning and on the lower right under "Releases" Click on "Create a new release"

5.2 Click on "Choose a tag" and type in the new tag name. For example, v0.2.0 and press "create tag on publish"

5.3 For the other items:

For release title, I'm using: v0.2.0
For Target - Change it from main to release  
Click on "Generate Release notes"
Click on "Set as the latest release"  (If that option is there...)

5.4 Click on Publish release

5.5 It should show up as available as a release on pypi here: https://pypi.org/project/fms-hf-tuning/#history

Step 6. Check the github actions to make sure it runs to completion

https://github.com/foundation-model-stack/fms-hf-tuning/actions

Step 7. Make an artificatory release image:

7.1 Clone or pull the latest copy of sft-trainer-image

git clone  https://github.ibm.com/ai-foundation/sft-trainer-image.git
cd sft-trainer-image

7.2 make a new branch

git checkout -b release-v0.2.0

7.3 Clone the new tag into the sft-trainer repo:

git clone --branch v0.2.0 https://github.com/foundation-model-stack/fms-hf-tuning.git

7.4 git add the new folder with the right tag version of fms-hf-tuning

git add fms-hf-tuning

7.5 Commit it and push

git commit -sm "0.2.0 release for sft-trainer"
git push --set-upstream origin release-v0.2.0

7.6 Create a PR:

for example: https://github.ibm.com/ai-foundation/sft-trainer-image/pull/new/release-v0.2.0

Results in: https://github.ibm.com/ai-foundation/sft-trainer-image/pull/14

7.7 The interim image is created.

You can see it here: https://v3.travis.ibm.com/github/ai-foundation/sft-trainer-image/builds/20049215

It shows:

docker-na-private.artifactory.swg-devops.com/wcp-ai-foundation-team-docker-virtual/sft-trainer:release-v0.2.0_ubi9_py311

which should compute to:

docker-na-public.artifactory.swg-devops.com/wcp-ai-foundation-team-docker-virtual/sft-trainer:release-v0.2.0_ubi9_py311

7.8 Once the PR is merged, it'll result in an "official" release image. Can see that here:

https://v3.travis.ibm.com/github/ai-foundation/sft-trainer-image/builds/20051870

Which ended up with an image of:

docker-na-private.artifactory.swg-devops.com/wcp-ai-foundation-team-docker-virtual/sft-trainer:4304af0_ubi9_py311

which on the public side would be:

docker-na-public.artifactory.swg-devops.com/wcp-ai-foundation-team-docker-virtual/sft-trainer:4304af0_ubi9_py311

Step 8. Verifying the new real release:

8.1. See that you can clone the new tag:

git clone --branch <tag or release name> https://github.com/foundation-model-stack/fms-hf-tuning.git

for example:

git clone --branch v0.2.0 https://github.com/foundation-model-stack/fms-hf-tuning.git

8.2. See if you can install the new pypi file:

python -m venv .env
source .env/bin/activate
pip install --upgrade pip
pip install fms-hf-tuning

and it should return the right version

pip list |grep fms
fms-hf-tuning            v0.2.0

Step 9. Let people know on the slack channel that the release is ready. For example:

0.1.0: https://ibm-research.slack.com/archives/C0698T0FW3S/p1716242505819769 and 0.2.0 https://ibm-research.slack.com/archives/C0698T0FW3S/p1717112087215109

Step 10. Manual testing using GPUs on DevStg

WIKI is removed...

Clone this wiki locally