Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(snowflake-driver): support azure exports buckets #8730

Merged
merged 45 commits into from
Sep 30, 2024
Merged

feat(snowflake-driver): support azure exports buckets #8730

merged 45 commits into from
Sep 30, 2024

Commits on Sep 26, 2024

  1. wip: add azure export bucket options to Snowflake driver

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    78acec5 View commit details
    Browse the repository at this point in the history

  2. add env var for Azure SAS Token

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    02af37b View commit details
    Browse the repository at this point in the history

  3. add STORAGE_INTEGRATION variant to export bucket flows

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    d3f0593 View commit details
    Browse the repository at this point in the history

  4. Simplify isUnloadSupported()

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    3f28ea0 View commit details
    Browse the repository at this point in the history

  5. add extractFilesFromGCS() to BaseDriver Class

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    d34d448 View commit details
    Browse the repository at this point in the history

  6. implement extractFilesFromAzure() in SnowFlake Driver

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    3a630b9 View commit details
    Browse the repository at this point in the history

  7. some refactoring in extractFilesFromAzure()

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    2a7007d View commit details
    Browse the repository at this point in the history

  8. debug and fix extractFilesFromAzure()

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    748d507 View commit details
    Browse the repository at this point in the history

  9. remove error if no csv files were exported (that's ok)

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    187161b View commit details
    Browse the repository at this point in the history

  10. move extractFilesFromAzure to BaseDriver from Snowflake

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    77e3ed5 View commit details
    Browse the repository at this point in the history

  11. switch Athena Driver to use extractUnloadedFilesFromS3() from BaseDvi… 

    …er impl
    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    4d46a81 View commit details
    Browse the repository at this point in the history

  12. switch RedShift Driver to use extractUnloadedFilesFromS3() from BaseD… 

    …vier impl
    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    53e31c4 View commit details
    Browse the repository at this point in the history

  13. switch Databricks Driver to use extractUnloadedFilesFromS3() and extr… 

    …actFilesFromAzure() from BaseDvier impl
    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    4c79e3c View commit details
    Browse the repository at this point in the history

  14. Remove unused/cleanup from databricks driver

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    5ed6ca5 View commit details
    Browse the repository at this point in the history

  15. update extractFilesFromGCS() in BaseDriver to receive credentials ins… 

    …tead of Storage instance
    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    6ca19b3 View commit details
    Browse the repository at this point in the history

  16. remove dep on @google-cloud/storage in Databricks for export bucket flow

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    3ff0844 View commit details
    Browse the repository at this point in the history

  17. fix Databricks driver export bucket flows after refactoring

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    2073213 View commit details
    Browse the repository at this point in the history

  18. set up CI to test clickhouse with s3 export bucket

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    b801306 View commit details
    Browse the repository at this point in the history

  19. Rename databricks export bucket test to reflect the s3 type

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    9f92483 View commit details
    Browse the repository at this point in the history

  20. fix clickhouse-export-bucket-s3-full script in package.json

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    94fb026 View commit details
    Browse the repository at this point in the history

  21. set up CI to test Snowflake with s3 export bucket

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    01ea8cd View commit details
    Browse the repository at this point in the history

  22. rename databricks tests snapshot after renaming test job

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    99d2577 View commit details
    Browse the repository at this point in the history

  23. improve extractUnloadedFilesFromS3 flow to support different bucket n… 

    …aming schemes
    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    29d1d32 View commit details
    Browse the repository at this point in the history

  24. add snowflake test snapshots for export bucket test

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    95493ce View commit details
    Browse the repository at this point in the history

  25. add clickhouse test snapshots for export bucket test

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    0a21132 View commit details
    Browse the repository at this point in the history

  26. update databricks with bucket tests snapshots

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    6ffa16b View commit details
    Browse the repository at this point in the history

  27. rename athena test to reflect that it uses export bucket

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    4e31cf4 View commit details
    Browse the repository at this point in the history

  28. reformat CLOUD_DATABASES in drivers tests CI job + add clickhouse-exp… 

    …ort-bucket-s3
    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    7b0de55 View commit details
    Browse the repository at this point in the history

  29. add snowflake export bucket to azure driver test

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    ff50876 View commit details
    Browse the repository at this point in the history

  30. add databricks export bucket to azure driver test

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    a30d4b3 View commit details
    Browse the repository at this point in the history

  31. add snowflake export bucket to azure via storage integration driver test

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    68e660a View commit details
    Browse the repository at this point in the history

  32. add snowflake export bucket to gcs via storage integration driver test

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    6431e2a View commit details
    Browse the repository at this point in the history

  33. improve/refactor exported files filtering in extractFilesFromAzure()

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    3c64607 View commit details
    Browse the repository at this point in the history

  34. set env secrets for drivers tests

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    cd3507d View commit details
    Browse the repository at this point in the history

  35. Fix createExportBucket in Snowflake (making sasToken optional)

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    be8e473 View commit details
    Browse the repository at this point in the history

  36. Fix CUBEJS_DB_EXPORT_BUCKET env var for tests

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    6482174 View commit details
    Browse the repository at this point in the history

  37. Remove includeIncrementalSchemaSuite from every test suite besides on… 

    …e + add comment
    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    347cfc9 View commit details
    Browse the repository at this point in the history

  38. Fix CUBEJS_DB_EXPORT_BUCKET env var for tests in databricks

    @KSDaemon
    KSDaemon committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    27e0d8c View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. fix databricks export to azure flow + align test config

    @KSDaemon
    KSDaemon committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    fea2f7f View commit details
    Browse the repository at this point in the history

  2. Fix blob object filtering during unloading in azure

    @KSDaemon
    KSDaemon committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    1b714d5 View commit details
    Browse the repository at this point in the history

  3. sync @google-cloud/storage package used across packages

    @KSDaemon
    KSDaemon committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    67ea07a View commit details
    Browse the repository at this point in the history

  4. yarn lock sync

    @KSDaemon
    KSDaemon committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    277b996 View commit details
    Browse the repository at this point in the history

  5. remove not needed includeIncrementalSchemaSuite in some tests

    @KSDaemon
    KSDaemon committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    eb2aaf4 View commit details
    Browse the repository at this point in the history

  6. rename bigquery → bigquery-export-bucket-gcs to align with the rest t… 

    …ests
    @KSDaemon
    KSDaemon committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    0d02a3a View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Add comments for exporting urls from S3 in BigQueryDriver

    @KSDaemon
    KSDaemon committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    17ed037 View commit details
    Browse the repository at this point in the history