Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Pixi for dependency management #4

Merged
merged 30 commits into from
Aug 16, 2024
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
06a6407
Use pixi for dependency management
jayqi Jul 29, 2024
178364b
Fix tensorflow
jayqi Jul 29, 2024
bef25dc
Move shared dependencies into a base feature
jayqi Jul 30, 2024
c24358d
move pixi files
klwetstone Jul 30, 2024
e897d08
remove conda-lock files
klwetstone Jul 30, 2024
f27fc7d
dockerfile scrap work
klwetstone Jul 30, 2024
c355861
update dockerfile
klwetstone Jul 30, 2024
d1c728d
use pixi docker base image
klwetstone Jul 30, 2024
bbd31dc
work updating test_lockfile
klwetstone Aug 1, 2024
b2f40fb
update make update-lockfiles for pixi
klwetstone Aug 1, 2024
76be305
separate dockerfile to update pixi.lock
klwetstone Aug 5, 2024
33e22bb
update lockfile
klwetstone Aug 6, 2024
4a83b05
update makefile
klwetstone Aug 6, 2024
ea9b46a
remove test_lockfile.py adapted for pixi
klwetstone Aug 6, 2024
c2290a5
makefile logging
klwetstone Aug 6, 2024
b5dbb58
[Into Pixi PR #4] Fix Dockerfiles (#5)
jayqi Aug 9, 2024
8b97f0f
Use a cuda base image
jayqi Aug 9, 2024
07594a4
Hardcode python executable for test
jayqi Aug 9, 2024
8188f71
Remove extra Python
jayqi Aug 9, 2024
925ac16
pass CPU_OR_GPU to entrypoint.sh
klwetstone Aug 13, 2024
5ec5e64
log cpu or gpu in entrypoint
klwetstone Aug 14, 2024
3d6e80e
use jammy for locking
klwetstone Aug 14, 2024
91ddba1
test pixi.lock update
klwetstone Aug 15, 2024
aa6c968
hacky pixi dependency fix
klwetstone Aug 15, 2024
f1adf43
revert pixi.lock
klwetstone Aug 15, 2024
6070c3e
only log in smoke tests
klwetstone Aug 15, 2024
9d8dfec
fix mounting for lockfile generation
klwetstone Aug 15, 2024
a52df64
revert makefile changes for testing
klwetstone Aug 15, 2024
ef2d0ba
fix comment
klwetstone Aug 15, 2024
bd39cee
omit trailing slash
klwetstone Aug 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -119,18 +119,31 @@ build:
--build-arg CPU_OR_GPU=${CPU_OR_GPU} \
--tag ${LOCAL_IMAGE}:${LOCAL_TAG}

runtime/pixi.lock:
ifeq (,$(wildcard ./runtime/pixi.lock))
@echo Generating runtime/pixi.lock from scratch
@echo "version: 5" > runtime/pixi.lock
@echo "environments:" >> runtime/pixi.lock
@echo "packages:" >> runtime/pixi.lock
else
@echo Updating existing runtime/pixi.lock
endif

## Updates runtime environment lockfile using Docker
update-lockfile:
@echo Generating the lockfile for CPU and GPU within Docker
update-lockfile: runtime/pixi.lock
@echo Building Docker image to generate lockfile
docker build runtime \
--file runtime/Dockerfile-lock \
--tag pixi-lock:local
@echo Running lock container
docker run \
--mount type=bind,source="$(shell pwd)"/runtime,target=/tmp \
--mount type=bind,source="$(shell pwd)"/runtime/pixi.toml,target=/tmp/pixi.toml \
--mount type=bind,source="$(shell pwd)"/runtime/pixi.lock,target=/tmp/pixi.lock \
--rm \
pixi-lock:local



## Ensures that your locally built image can import all the Python packages successfully when it runs
test-container: _check_image _echo_image _submission_write_perms
docker run \
Expand Down
4 changes: 2 additions & 2 deletions runtime/Dockerfile-lock
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
FROM ghcr.io/prefix-dev/pixi:0.26.1-bookworm-slim
FROM ghcr.io/prefix-dev/pixi:0.26.1-jammy

USER root

RUN mkdir -p /tmp
WORKDIR /tmp
WORKDIR /tmp/

klwetstone marked this conversation as resolved.
Show resolved Hide resolved
ENTRYPOINT ["pixi", "tree", "--manifest-path", "pixi.toml", "--platform", "linux-64", "-v"]
9 changes: 7 additions & 2 deletions runtime/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,13 @@ main () {

ls -alh

echo "Running submission..."
pixi run -e $CPU_OR_GPU python main.py
if $IS_SMOKE_TEST; then
echo "Running smoke test"
pixi run -e $CPU_OR_GPU python main.py
else
echo "Running submission using $CPU_OR_GPU"
pixi run -e $CPU_OR_GPU python main.py &> "/code_execution/submission/private_log.txt"
fi

echo "Exporting submission.csv result..."

Expand Down
2 changes: 1 addition & 1 deletion runtime/pixi.lock
Original file line number Diff line number Diff line change
Expand Up @@ -5886,4 +5886,4 @@ packages:
license: BSD-3-Clause
license_family: BSD
size: 554846
timestamp: 1714722996770
timestamp: 1714722996770
Loading