Skip to content

Commit

Permalink
#883: Update packages (#404)
Browse files Browse the repository at this point in the history
1. Update pandas, numpy and pyarrow in template-Python3.10 flavor.
2. Deactivate conversion tests for pandas dataframes for `halffloat[pyarrow] + float` and `float16[pyarrow] + float` because in newer versions of pandas this conversion is fixed, but we cannot (yet) use the latest versions in all flavors (pandas 2.1 and later require >= Python3.9)
3. Update ignore list in import_modules.py (due to latest packages in new standard-all flavor)
4. Changed test import_modules.py: Instead of importing ALL python modules in one UDF, fetch first list of installed python packages, and then call UDF one by one. This is to avoid an out-of memory issue. For example, importing sagemaker already allocates ~150MB on the heap.
  • Loading branch information
tomuben authored May 10, 2024
1 parent ca3d4bc commit 1cc8270
Show file tree
Hide file tree
Showing 8 changed files with 84 additions and 147 deletions.
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
pandas|2.0.2
pandas|2.0.3
numpy|1.24.3
pyarrow|14.0.2
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
pandas|2.0.2
numpy|1.24.3
pyarrow|14.0.2
pandas|2.2.2
numpy|1.26.4
pyarrow|16.0.0
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
libxcrypt|4.4.36 # Needed for building the udfclient and because libcrypt got removed from conda sysroot, see https://github.com/conda-forge/linux-sysroot-feedstock/issues/52
python|3.8.13
numpy|1.24.3
pandas|2.0.2
pandas|2.0.3
pyarrow|14.0.2
libblas|3.9.0=15_linux64_mkl
mamba|1.3.1
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
libxcrypt|4.4.36 # Needed for building the udfclient and because libcrypt got removed from conda sysroot, see https://github.com/conda-forge/linux-sysroot-feedstock/issues/52
python|3.8.13
numpy|1.24.3
pandas|2.0.2
pandas|2.0.3
pyarrow|14.0.2=py38h296dbf9_9_cuda
libblas|3.9.0=15_linux64_mkl
mamba|1.3.1
Expand Down
Loading

0 comments on commit 1cc8270

Please sign in to comment.