Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest/airflow): airflow plugin v2 #8853

Merged
merged 64 commits into from
Oct 4, 2023
Merged

feat(ingest/airflow): airflow plugin v2 #8853

merged 64 commits into from
Oct 4, 2023

Commits on Sep 19, 2023

  1. fix(airflow): fix provider loading exception 

    Would previously see this from the plugin manager.
```
Exception: The package 'acryl-datahub' from setuptools and acryl-datahub-airflow-plugin do not match. Please make sure they are aligned
```

Also fixes + registers hook types
    @hsheth2
    hsheth2 committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    d4437de View commit details
    Browse the repository at this point in the history

  2. start refactoring backend 

    continue making tweaks

refactor ti + config

refactor get_lineage_config

refactoring

add in special case handling from backport

listener-based plugin

fix hooks + add log_level config

make logging work

more sql fixes

start working on combining lineage

work with old datahub lineage entities

support fine grained lineage

support enable_extractors

tweak stuff

preserve datajob info

fix bugs in emitter

support threaded exec

fix rendering

refactoring

more refactoring

tweak comment

review comments

mypy

fix mypy errors

drop python 3.7

fix lint

fix lint
    @hsheth2
    hsheth2 committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    72378e1 View commit details
    Browse the repository at this point in the history

  3. start updating docs

    @hsheth2
    hsheth2 committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    66c20c8 View commit details
    Browse the repository at this point in the history

  4. update workflow

    @hsheth2
    hsheth2 committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    b4545cc View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2023

  1. Merge branch 'master' into airflow-plugin

    @hsheth2
    hsheth2 committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    a1b43e6 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2023

  1. Merge branch 'master' into airflow-plugin

    @hsheth2
    hsheth2 committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    7faeb4c View commit details
    Browse the repository at this point in the history

  2. update docs

    @hsheth2
    hsheth2 committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    f875eeb View commit details
    Browse the repository at this point in the history

  3. plugin v1 extra

    @hsheth2
    hsheth2 committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    125e0d9 View commit details
    Browse the repository at this point in the history

  4. update ci

    @hsheth2
    hsheth2 committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    0423ff3 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. update mypy + build

    @hsheth2
    hsheth2 committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    e6dab06 View commit details
    Browse the repository at this point in the history

  2. fix(ingest): bump typing-extensions

    @hsheth2
    hsheth2 committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    aa84b23 View commit details
    Browse the repository at this point in the history

  3. unify on emit method

    @hsheth2
    hsheth2 committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    2ac887b View commit details
    Browse the repository at this point in the history

  4. add simple unit test

    @hsheth2
    hsheth2 committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    c711a7e View commit details
    Browse the repository at this point in the history

  5. start tests

    @hsheth2
    hsheth2 committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    f73a380 View commit details
    Browse the repository at this point in the history

  6. loosen airflow req

    @hsheth2
    hsheth2 committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    048b6e9 View commit details
    Browse the repository at this point in the history

  7. start testing setup

    @hsheth2
    hsheth2 committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    9c8538b View commit details
    Browse the repository at this point in the history

  8. add an emitter interface

    @hsheth2
    hsheth2 committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    1f8f7a7 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2023

  1. sync file emitter

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    8b0ef4c View commit details
    Browse the repository at this point in the history

  2. docs tweaks

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    7119008 View commit details
    Browse the repository at this point in the history

  3. most of test scaffolding

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    3436095 View commit details
    Browse the repository at this point in the history

  4. stuff

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    1b4f431 View commit details
    Browse the repository at this point in the history

  5. get basic plugin v1 test working

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    1ffa623 View commit details
    Browse the repository at this point in the history

  6. add another simple test

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    73e3acc View commit details
    Browse the repository at this point in the history

  7. set test ids

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    5c2ad60 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'master' into airflow-plugin

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    fe4e012 View commit details
    Browse the repository at this point in the history

  9. update tests more

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    9fddd68 View commit details
    Browse the repository at this point in the history

  10. snowflake test

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    09a0d17 View commit details
    Browse the repository at this point in the history

  11. update v2 goldens

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    ed55d67 View commit details
    Browse the repository at this point in the history

  12. fix mypy issues

    @hsheth2
    hsheth2 committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    6463500 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2023

  1. update v1 tests

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    c537166 View commit details
    Browse the repository at this point in the history

  2. tweak tests

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    f4639cc View commit details
    Browse the repository at this point in the history

  3. fix _F

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    a36a036 View commit details
    Browse the repository at this point in the history

  4. hack to inherit from module type

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    0952905 View commit details
    Browse the repository at this point in the history

  5. fix gradle

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    7878337 View commit details
    Browse the repository at this point in the history

  6. enable debug log in test

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    505f4ef View commit details
    Browse the repository at this point in the history

  7. list loaded plugins

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    13a5e9d View commit details
    Browse the repository at this point in the history

  8. attempt 1 at refactoring for airflow 2.3-2.4

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    d4b4161 View commit details
    Browse the repository at this point in the history

  9. make it a module for compat with <2.5

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    28c99d6 View commit details
    Browse the repository at this point in the history

  10. add tests for old plugin v2

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    9b0e866 View commit details
    Browse the repository at this point in the history

  11. fix mypy for 2.1.4

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    dcc2fab View commit details
    Browse the repository at this point in the history

  12. update test golden

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    27391ff View commit details
    Browse the repository at this point in the history

  13. fix build and test workflow

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    77c1133 View commit details
    Browse the repository at this point in the history

  14. handle subprocess kill timeout

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    18370cd View commit details
    Browse the repository at this point in the history

  15. setup tox tests

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    19a36ce View commit details
    Browse the repository at this point in the history

  16. fix env list for tox

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    83a2621 View commit details
    Browse the repository at this point in the history

  17. slightly more robust tests

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    7036c43 View commit details
    Browse the repository at this point in the history

  18. logging improvements

    @hsheth2
    hsheth2 committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    817c73e View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2023

  1. refactor get_platform_from_sqlalchemy_uri into dedicated file

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    0b04b30 View commit details
    Browse the repository at this point in the history

  2. close schema resolver

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    f787800 View commit details
    Browse the repository at this point in the history

  3. add support for arbitrary operators

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    1e8dafc View commit details
    Browse the repository at this point in the history

  4. update tests

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    9efb96d View commit details
    Browse the repository at this point in the history

  5. update desc generation

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    4815c1a View commit details
    Browse the repository at this point in the history

  6. fix sqlglot create table lineage

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    215b5a1 View commit details
    Browse the repository at this point in the history

  7. convert downstream set to list

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    841c22e View commit details
    Browse the repository at this point in the history

  8. update more goldens

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    e558cf1 View commit details
    Browse the repository at this point in the history

  9. one more

    @hsheth2
    hsheth2 committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    8c98aad View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2023

  1. fix issue with mcp ordering / dedup in test code

    @hsheth2
    hsheth2 committed Sep 30, 2023
    Configuration menu
    Copy the full SHA
    6db6a34 View commit details
    Browse the repository at this point in the history

  2. start fixing more tests

    @hsheth2
    hsheth2 committed Sep 30, 2023
    Configuration menu
    Copy the full SHA
    50e1130 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2023

  1. update v2 tests

    @hsheth2
    hsheth2 committed Oct 1, 2023
    Configuration menu
    Copy the full SHA
    e27729a View commit details
    Browse the repository at this point in the history

  2. update v2.4 tests

    @hsheth2
    hsheth2 committed Oct 1, 2023
    Configuration menu
    Copy the full SHA
    15f5122 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. review

    @hsheth2
    hsheth2 committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    5e2405b View commit details
    Browse the repository at this point in the history

  2. finishing touches

    @hsheth2
    hsheth2 committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    3b49da8 View commit details
    Browse the repository at this point in the history

  3. update example + deps

    @hsheth2
    hsheth2 committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    d61e5e9 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into airflow-plugin

    @hsheth2
    hsheth2 committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    bcdb3e6 View commit details
    Browse the repository at this point in the history