Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data] [1/n] Fix dependency related Ray Data tests for Python 3.12 #46545

Closed
wants to merge 13 commits into from
Closed

[Data] [1/n] Fix dependency related Ray Data tests for Python 3.12 #46545

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
fd85e66
fixes
scottjlee Jul 10, 2024
22bbe6c
Merge branch 'ray-py312' into 0710-py312-datatests
scottjlee Jul 10, 2024
7cb1ebb
bump tfds for ml requirements
scottjlee Jul 10, 2024
a4152b3
forgot gpu requirements
scottjlee Jul 10, 2024
86d3b90
pin etils==1.9.1
scottjlee Jul 10, 2024
8e0b07d
update requirements_compiled.txt from BK
scottjlee Jul 10, 2024
82bb46d
Merge branch 'ray-py312' into 0710-py312-datatests
scottjlee Jul 11, 2024
aefdf5f
[py312] Fix `test_tensor.py` (#46669)
omatthew98 Jul 17, 2024
f1fc5b0
Merge branch 'ray-py312' into 0710-py312-datatests
scottjlee Jul 17, 2024
b4b51dc
update keras calls
scottjlee Jul 17, 2024
c9b3098
Merge branch 'ray-py312' of https://github.com/ray-project/ray into r…
scottjlee Jul 19, 2024
f9d22e8
Merge branch 'ray-py312' into 0710-py312-datatests
scottjlee Jul 19, 2024
499bdaa
update requirements-compiled.txt
scottjlee Jul 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/data/examples/stablediffusion_batch_prediction.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@
" \"transformers>=4.26.0\",\n",
" \"diffusers>=0.13.1\",\n",
" \"xformers>=0.0.16\",\n",
" \"torch<2\",\n",
" \"torch==2.3.0\",\n",
" ]\n",
" }\n",
")"
Expand Down
3 changes: 2 additions & 1 deletion python/ray/data/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,7 +390,8 @@ py_test(
name = "test_mars",
size = "medium",
srcs = ["tests/test_mars.py"],
tags = ["team:data", "exclusive"],
# mars does not support Python 3.11+, so mark as manual test.
tags = ["team:data", "exclusive", "manual"],
deps = ["//:ray_lib", ":conftest"],
)

Expand Down
4 changes: 2 additions & 2 deletions python/ray/data/tests/test_tf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,8 @@ def train_func():
multi_worker_model = build_model()
multi_worker_model.compile(
optimizer=tf.keras.optimizers.SGD(),
loss=tf.keras.losses.mean_absolute_error,
metrics=[tf.keras.metrics.mean_squared_error],
loss=tf.keras.losses.MeanAbsoluteError(),
metrics=[tf.keras.metrics.MeanSquaredError()],
)

dataset = train.get_dataset_shard("train").to_tf("X", "Y", batch_size=4)
Expand Down
5 changes: 4 additions & 1 deletion python/requirements/ml/data-test-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@
# https://github.com/ray-project/ray/pull/29448#discussion_r1006256498

python-snappy
tensorflow-datasets==4.9.3
# 4.9.6 fixes the following error with Python 3.12:
# `RecursionError: maximum recursion depth exceeded`
# See: https://github.com/tensorflow/datasets/issues/4666#issuecomment-2149200103
tensorflow-datasets==4.9.6
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

w00t, you would need to update here too https://github.com/ray-project/ray/blob/ray-py312/python/requirements_compiled.txt#L1894 otherwise pip will complain during installation

datasets
pytest-repeat
soundfile
Expand Down
7 changes: 5 additions & 2 deletions python/requirements/ml/dl-cpu-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,12 @@ tensorflow==2.16.1; sys_platform != 'darwin' or platform_machine != 'arm64'
tensorflow-macos==2.16.1; sys_platform == 'darwin' and platform_machine == 'arm64'
tensorflow-probability[tf]==0.24.0
tensorflow-io-gcs-filesystem==0.37.0; python_version <= '3.11'
tensorflow-datasets==4.9.3
tensorflow-datasets==4.9.6
array-record==0.5.1; sys_platform != 'darwin' and platform_system != "Windows"
etils==1.5.2

# Needed for Python 3.12+ support.
# `tensorflow-datasets 4.9.6 depends on etils>=1.9.1; python_version >= "3.11"`
etils==1.9.1

# If you make changes below this line, please also make the corresponding changes to `dl-gpu-requirements.txt`
# and to `install-dependencies.sh`!
Expand Down
2 changes: 1 addition & 1 deletion python/requirements/ml/dl-gpu-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
tensorflow==2.16.1; sys_platform != 'darwin' or platform_machine != 'arm64'
tensorflow-macos==2.16.1; sys_platform == 'darwin' and platform_machine == 'arm64'
tensorflow-probability[tf]==0.24.0
tensorflow-datasets==4.9.3
tensorflow-datasets==4.9.6

--extra-index-url https://download.pytorch.org/whl/cu121 # for GPU versions of torch, torchvision
--find-links https://data.pyg.org/whl/torch-2.3.0+cu121.html # for GPU versions of torch-scatter, torch-sparse, torch-cluster, torch-spline-conv
Expand Down
Loading
Loading