Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for S3 Extra arguments in Seed file upload to s3 #397

Merged
merged 12 commits into from
Sep 11, 2023

Conversation

linkerzx
Copy link
Contributor

@linkerzx linkerzx commented Sep 5, 2023

Description

This M.R. (still in draft) adds a profile parameters adding predefined ExtraArgs when making a call to upload_to_s3, during the seed file upload.
resolve #389

Models used to test - Optional

Checklist

  • You followed contributing section
  • You kept your Pull Request small and focused on a single feature or bug fix.
  • You added unit testing when necessary
  • You added functional testing when necessary

@nicor88 nicor88 changed the title Draft - feat: add support for S3 Extra arguments in Seed file upload to s3 feat: add support for S3 Extra arguments in Seed file upload to s3 Sep 5, 2023
README.md Outdated Show resolved Hide resolved
@svdimchenko
Copy link
Contributor

overall the code looks good to me.
The parameter name should be definitely fixed, I like naming which @nicor88 has offered.
Also it would be nice to add boto3-stubs[s3]~=1.28 row to dev-requirements.txt.

BTW @nicor88 WDYT about migrating from dev-requirements.txt and dependencies in setup.py to poetry ? It may help with dependencies management.

@nicor88
Copy link
Contributor

nicor88 commented Sep 6, 2023

@svdimchenko I think that we discussed that with @Jrmyy back then - we decided to use pip with requirements to adapt a bit with the landscape of other dbt adapters - but I'm happy to reconsider to use it again.

@linkerzx
Copy link
Contributor Author

Hi @nicor88 , @svdimchenko - made some changes following the initial review:

  1. Changed the parameter name to seed_s3_upload_args
  2. Applied formatting to the markdown tables
  3. Added an example on how to use the arguments in readme
  4. Added boto3-stubs[s3]~=1.28 to dev-requirements
  5. Merged on top of current main

@svdimchenko svdimchenko added the enable-functional-tests Label to trigger functional testing label Sep 11, 2023
dev-requirements.txt Outdated Show resolved Hide resolved
@svdimchenko svdimchenko merged commit 9d029bb into dbt-labs:main Sep 11, 2023
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enable-functional-tests Label to trigger functional testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for seed file upload with enforced ACL/SSE KMS keys
3 participants