Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature(ingest/athena): introduce support for complex and nested schemas in Athena #8137

Merged
merged 32 commits into from
Oct 18, 2023
Merged

feature(ingest/athena): introduce support for complex and nested schemas in Athena #8137

merged 32 commits into from
Oct 18, 2023

Commits on May 26, 2023

  1. chore(deps): remove internal method reference & soften PyAthena versi… 

    …on requirements

Co-authored-by: dnks23 <[email protected]>
    @bossenti @dnks23
    bossenti and dnks23 committed May 26, 2023
    Configuration menu
    Copy the full SHA
    6af0633 View commit details
    Browse the repository at this point in the history

  2. feature(ingest/athena): implement custom SQLalchemy dialect for Athen… 

    …a to detect complex data types correctly

Co-authored-by: dnks23 <[email protected]>
    @bossenti @dnks23
    bossenti and dnks23 committed May 26, 2023
    Configuration menu
    Copy the full SHA
    cf595d1 View commit details
    Browse the repository at this point in the history

  3. feature(ingest/athena): implement schema field generation supporting … 

    …nested fields for SQLalchemy types and enable Athena source

Co-authored-by: dnks23 <[email protected]>
    @bossenti @dnks23
    bossenti and dnks23 committed May 26, 2023
    Configuration menu
    Copy the full SHA
    9c0b6ab View commit details
    Browse the repository at this point in the history

  4. Merge branch 'datahub-project:master' into feature/athena-improvements

    @bossenti
    bossenti authored May 26, 2023
    Configuration menu
    Copy the full SHA
    7750e1d View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into feature/athena-improvements

    @bossenti
    bossenti authored May 26, 2023
    Configuration menu
    Copy the full SHA
    a2100b7 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. Merge branch 'master' into feature/athena-improvements

    @bossenti
    bossenti authored May 30, 2023
    Configuration menu
    Copy the full SHA
    d2670a4 View commit details
    Browse the repository at this point in the history

  2. chore: make MapType discoverable

    @bossenti
    bossenti committed May 30, 2023
    Configuration menu
    Copy the full SHA
    c7f0bc7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into feature/athena-improvements

    @bossenti
    bossenti authored May 30, 2023
    Configuration menu
    Copy the full SHA
    ad2c608 View commit details
    Browse the repository at this point in the history

  4. backport to python 3.7

    @bossenti
    bossenti committed May 30, 2023
    Configuration menu
    Copy the full SHA
    f22aedd View commit details
    Browse the repository at this point in the history

  5. fix formatting

    @bossenti
    bossenti committed May 30, 2023
    Configuration menu
    Copy the full SHA
    5aaf348 View commit details
    Browse the repository at this point in the history

  6. improve linting

    @bossenti
    bossenti committed May 30, 2023
    Configuration menu
    Copy the full SHA
    7dfb376 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. fix import order

    @bossenti
    bossenti committed May 31, 2023
    Configuration menu
    Copy the full SHA
    c6f2ba0 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. Merge remote-tracking branch 'bossenti/master' into feature/athena-im… 

    …provements
    @bossenti
    bossenti committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    ee5011f View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2023

  1. Merge branch 'master' into feature/athena-improvements

    @treff7es
    treff7es authored Aug 2, 2023
    Configuration menu
    Copy the full SHA
    71ae39b View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. Merge remote-tracking branch 'datahub/master' into feature/athena-imp… 

    …rovements

# Conflicts:
#	metadata-ingestion/src/datahub/ingestion/source/sql/athena.py
#	metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py
    @bossenti
    bossenti committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    68673b6 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. Merge remote-tracking branch 'datahub/master' into feature/athena-imp… 

    …rovements
    @bossenti
    bossenti committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    c95b928 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. Merge branch 'master-dh' into feature/athena-improvements

    Tim authored and Tim committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    5cc0b82 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Merge branch 'master-dh' into feature/athena-improvements

    @bossenti
    bossenti committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    d2ef380 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. Merge branch 'master' into feature/athena-improvements

    @hsheth2
    hsheth2 authored Sep 12, 2023
    Configuration menu
    Copy the full SHA
    6f2b0cc View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2023

  1. Merge branch 'master-dh' into feature/athena-improvements

    @bossenti
    bossenti committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    6fe590a View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2023

  1. Merge branch 'master-dh' into feature/athena-improvements

    @bossenti
    bossenti committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    0c326de View commit details
    Browse the repository at this point in the history

  2. change type hints to be 3.7-compatible

    @bossenti
    bossenti committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    c2dc330 View commit details
    Browse the repository at this point in the history

  3. fix lint + tests

    @hsheth2
    hsheth2 committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    50e51ad View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. Merge branch 'master-dh' into feature/athena-improvements

    @bossenti
    bossenti committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    7884fc3 View commit details
    Browse the repository at this point in the history

  2. address linting issues

    @bossenti
    bossenti committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    b58b6d5 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/feature/athena-improvements' int… 

    …o feature/athena-improvements
    @bossenti
    bossenti committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    54f874d View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. Merge branch 'master' into feature/athena-improvements

    @hsheth2
    hsheth2 authored Oct 4, 2023
    Configuration menu
    Copy the full SHA
    5dc3d87 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. Merge remote-tracking branch 'datahub/master' into feature/athena-imp… 

    …rovements
    @bossenti
    bossenti committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    60b21e9 View commit details
    Browse the repository at this point in the history

  2. refactor: replace central definition of STRUCT by registering custom … 

    …type
    @bossenti
    bossenti committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    4c0da22 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/feature/athena-improvements' int… 

    …o feature/athena-improvements
    @bossenti
    bossenti committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    42d0114 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2023

  1. style: apply formatting

    @bossenti
    bossenti committed Oct 15, 2023
    Configuration menu
    Copy the full SHA
    92c8000 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'datahub/master' into feature/athena-imp… 

    …rovements
    @bossenti
    bossenti committed Oct 15, 2023
    Configuration menu
    Copy the full SHA
    f68c9e0 View commit details
    Browse the repository at this point in the history