Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecation Warning for parallelism Argument in read_parquet with ray_args #2864

Closed
DimitarSirakov opened this issue Jun 19, 2024 · 0 comments · Fixed by #2876
Closed

Deprecation Warning for parallelism Argument in read_parquet with ray_args #2864

DimitarSirakov opened this issue Jun 19, 2024 · 0 comments · Fixed by #2876
Assignees
Labels
enhancement New feature or request

Comments

@DimitarSirakov
Copy link

There's a deprecation warning related to the parallelism argument in Ray 2.10. The warning suggests using override_num_blocks instead. This issue occurs in read_api.py at line 3087.

Steps to Reproduce:

  1. Set up an environment with Ray 2.10.
  2. Use the awswrangler.s3.read_parquet function with ray_args that includes the parallelism parameter.
  3. Observe the warning message.

Code Example:

import awswrangler as wr

# Example usage of read_parquet with ray_args including parallelism
df = wr.s3.read_parquet(
    path="s3://bucket/path/",
    ray_args={"parallelism": 10}
)

Expected Behavior:
The read_parquet function should accept the appropriate parameter for specifying parallelism without generating a deprecation warning.

Suggested Fix:
Update the read_parquet function to use override_num_blocks instead of parallelism when passing ray_args.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants