Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python): Add storage_options in write_parquet #9920

Closed
wants to merge 0 commits into from

Conversation

ion-elgreco
Copy link
Contributor

@ion-elgreco ion-elgreco commented Jul 16, 2023

#9770

You can now pass fsspec kwargs into write_parquet just as in read_parquet.

Example:

import polars as pl

df = pl.from_repr(
"""
┌─────┬─────┬─────┬────────┐
│ a   ┆ b   ┆ c   ┆ column │
│ --- ┆ --- ┆ --- ┆ ---    │
│ i64 ┆ i64 ┆ i64 ┆ str    │
╞═════╪═════╪═════╪════════╡
│ 1   ┆ 2   ┆ 3   ┆ hello  │
│ 2   ┆ 1   ┆ 3   ┆ world  │
│ 1   ┆ 2   ┆ 2   ┆ ...    │
└─────┴─────┴─────┴────────┘
""")

fp = "abfs://polarstest@<storage_account_name>.dfs.core.windows.net/test/new.parquet"
storage_options = {
    "account_key": <KEY>,
}
df.write_parquet(fp, storage_options=storage_options)

pl.read_parquet(fp, storage_options=storage_options)
shape: (3, 4)
a b c column
i64 i64 i64 str
1 2 3 "hello"
2 1 3 "world"
1 2 2 "..."

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars labels Jul 16, 2023
@ion-elgreco ion-elgreco requested a review from cnpryer July 17, 2023 06:35
@ion-elgreco ion-elgreco changed the title feat(python): Add storage_options in DataFrame.write_parquet feat(python): Add storage_options in write_parquet Jul 19, 2023
@ion-elgreco
Copy link
Contributor Author

@stinodego @cnpryer are there any changes I need to make for this PR to get merged?

Copy link
Contributor

@svaningelgem svaningelgem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ion-elgreco : I'd love to see the possibility to pass in the path directly, much as it is being done in the read_parquet. After that it's just opened with fsspec.

Additionally: why only parquet, why not the other write_* methods as well?

@ion-elgreco
Copy link
Contributor Author

Hi @ion-elgreco : I'd love to see the possibility to pass in the path directly, much as it is being done in the read_parquet. After that it's just opened with fsspec.

Additionally: why only parquet, why not the other write_* methods as well?

Was checking write_parquet at that time and that seemed easiest to add. I could add the other ones, but I need to know if I'm on the right track first^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants