Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameter description depreciated in newer awswrangler library #977

Open
1 task
marcinpurtak opened this issue Aug 19, 2024 · 0 comments
Open
1 task

Parameter description depreciated in newer awswrangler library #977

marcinpurtak opened this issue Aug 19, 2024 · 0 comments
Assignees
Labels
2.0 viadot 2.0 orchestration Issues related to the orchestration layer.

Comments

@marcinpurtak
Copy link
Collaborator

marcinpurtak commented Aug 19, 2024

Description - triaged

The description param, used by RedshiftSpectrum.from_df() when calling wr.s3.to_parquet(), is deprecated in awswrangler v3.0. Since users depend on this parameter, we should have unit tests failing when upgrading awswrangler.

Tasks

  • Add a unit test which catches this deprecation (in case eg. someone wants to upgrade awswrangler). In other words, the test should use the description param.

NOTE: WIP was added in #917 (075b333)

Original description

Currently in the viadot 2.0 requirements we have stiff version of the lib awsrangler==2.20.1 , however in the version newer than 3.0 parameter description get obsolete.

We have this parameter in the workflow beginning from the deployment file.

For example:

`from templates import extract_and_load_sharepoint

URL = "https:/xxx.sharepoint.com/sites/DACN/doku%20comp/General/Source_data/EUMapping.xlsx"
extract_and_load_sharepoint(
table="sp_eumapping",
sharepoint_url=URL,
sheet_name="Sheet1",
description="Sharepoint file: " + URL,
sharepoint_credentials_secret="sharepointsecret2",
schedule="0 8 * * *",
)`

@marcinpurtak marcinpurtak added 2.0 viadot 2.0 orchestration Issues related to the orchestration layer. labels Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.0 viadot 2.0 orchestration Issues related to the orchestration layer.
Projects
None yet
Development

No branches or pull requests

2 participants