feat(ingest): support view lineage for all sqlalchemy sources #9039

mayurinehate · 2023-10-18T17:12:50Z

Additional Changes:

Support incremental lineage for all sqlalchemy sources
Keep column level lineage enabled and incremental lineage disabled by default
Monkey-patch hive dialect to extract hive view definitions to extract lineage
Fix incremental_lineage_helper for empty upstreams
Support postgres-like partial view definitions

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Additional Changes: 1. Support incremental lineage for all sqlalchemy sources 2. Keep column level lineage enabled and incremental lineage disabled by default 3. Monkey-patch hive dialect to extract hive view definitions to extract lineage 4. Fix incremental_lineage_helper for empty upstreams Pending Followup Changes: 1. Support postgres-like partial view definitions

mayurinehate · 2023-10-19T13:52:55Z

metadata-ingestion/src/datahub/configuration/common.py

@@ -283,7 +283,7 @@ class VersionedConfig(ConfigModel):

 class LineageConfig(ConfigModel):
    incremental_lineage: bool = Field(
-        default=True,
+        default=False,


Incremental lineage requires presence of DataHubGraph, which is available by default only when using DataHub rest sink. We plan to keep this default enabled in managed ingestion.

mayurinehate · 2023-10-19T13:55:19Z

metadata-ingestion/src/datahub/ingestion/api/incremental_lineage_helper.py

+                    graph, urn, lineage_aspect, wu.metadata.systemMetadata
+                )
+            elif lineage_aspect.upstreams:
+                yield _convert_upstream_lineage_to_patch(


If there is a table level upstream only aspect with empty upstreams, we ignore it, as part of incremental lineage.

mayurinehate · 2023-10-19T13:56:36Z

metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_lineage_v2.py

-        return self._create_upstream_lineage_workunit(
-            dataset_identifier, upstreams, fine_upstreams
-        )
+        if upstreams:


We should not emit upstream lineage if no upstreams are found.

mayurinehate · 2023-10-19T13:58:17Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive.py

@@ -90,19 +96,39 @@ def dbapi_get_columns_patched(self, connection, table_name, schema=None, **kw):
    logger.warning(f"Failed to patch method due to {e}")


+try:
+    from pyhive.sqlalchemy_hive import HiveDialect


Alternatively we can also move this code to acryl pyhive fork - https://github.com/acryldata/PyHive

This seemed simpler and easier to test this end to end. Open to suggestions here.

This seems fine for now, and we can fix up when we refactor sql common next week

metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py

mayurinehate · 2023-10-19T14:02:11Z

metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py

+            self._view_definition_cache = FileBackedDict[str]()
+        else:
+            self._view_definition_cache = {}
+


moved use_file_backed_cache related logic from teraform source here.

Now that we fixed the file backed dict to support windows, imo we can probably drop this config flag

Can you point me to the PR that fixes this ?

- primarily to reduce adverse effect on other sources, such as dbt which have their own flavour of incremental lineage implementation

hsheth2 · 2023-10-20T21:13:08Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive.py

+    HiveDialect.get_view_definition = get_view_definition_patched
+except ModuleNotFoundError:
+    pass
+except Exception as e:


Failure to patch should cause the source to fail to load right?

Right. Let me remove this exception handling.

metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py

hsheth2 · 2023-10-20T21:17:24Z

metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py

+            self._view_definition_cache = FileBackedDict[str]()
+        else:
+            self._view_definition_cache = {}
+


Now that we fixed the file backed dict to support windows, imo we can probably drop this config flag

hsheth2 · 2023-10-20T21:21:24Z

metadata-ingestion/src/datahub/ingestion/source/sql/two_tier_sql_source.py

@@ -71,6 +71,10 @@ def __init__(self, config, ctx, platform):
        super().__init__(config, ctx, platform)
        self.config: TwoTierSQLAlchemyConfig = config

+    def get_db_schema(self, dataset_identifier: str) -> Tuple[Optional[str], str]:


When we do the refactoring, let's put identifiers in a real data class instead of joining into a string in one place and then splitting in other places

metadata-ingestion/src/datahub/utilities/sqlglot_lineage.py

hsheth2 · 2023-10-23T21:52:07Z

There's one remaining thing around handling partial view definitions

hsheth2 · 2023-10-25T20:14:26Z

@mayurinehate Seeing a real issue in the smoke test - looks like sql_common needs a dependency on sqlparse

tests/test_stateful_ingestion.py:5: in <module>
    from datahub.ingestion.source.sql.mysql import MySQLConfig, MySQLSource
../metadata-ingestion/src/datahub/ingestion/source/sql/mysql.py:18: in <module>
    from datahub.ingestion.source.sql.sql_common import (
../metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py:34: in <module>
    from datahub.emitter.sql_parsing_builder import SqlParsingBuilder
../metadata-ingestion/src/datahub/emitter/sql_parsing_builder.py:11: in <module>
    from datahub.ingestion.source.usage.usage_common import BaseUsageConfig, UsageAggregator
../metadata-ingestion/src/datahub/ingestion/source/usage/usage_common.py:41: in <module>
    from datahub.utilities.sql_formatter import format_sql_query, trim_query
../metadata-ingestion/src/datahub/utilities/sql_formatter.py:4: in <module>
    import sqlparse
E   ModuleNotFoundError: No module named 'sqlparse'

fallback to native postgres view lineage extraction for failed views

mayurinehate · 2023-10-26T04:18:28Z

There's one remaining thing around handling partial view definitions

This is now added in this PR.

@mayurinehate Seeing a real issue in the smoke test - looks like sql_common needs a dependency on sqlparse

Fixed it.

metadata-ingestion/src/datahub/utilities/sqlglot_lineage.py

hsheth2 · 2023-10-26T16:14:00Z

The cypress test failure appears unrelated (glossary/glossary_navigation.js), so merging through

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Oct 18, 2023

vercel bot deployed to Preview October 18, 2023 17:47 View deployment

vercel bot deployed to Preview October 19, 2023 13:17 View deployment

mayurinehate force-pushed the master+ing-200-view-lineage-for-sql-sources branch from dd85c59 to 75f63a2 Compare October 19, 2023 13:47

mayurinehate marked this pull request as ready for review October 19, 2023 13:47

mayurinehate requested review from treff7es, asikowitz and hsheth2 October 19, 2023 13:47

vercel bot had a problem deploying to Preview October 19, 2023 13:59 Failure

mayurinehate commented Oct 19, 2023

View reviewed changes

revert incremental lineage helper refractor to source.py

93b7ba1

- primarily to reduce adverse effect on other sources, such as dbt which have their own flavour of incremental lineage implementation

vercel bot deployed to Preview October 20, 2023 14:06 View deployment

hsheth2 reviewed Oct 20, 2023

View reviewed changes

fix tests, address review comments

7c18697

vercel bot deployed to Preview October 23, 2023 08:30 View deployment

hsheth2 approved these changes Oct 23, 2023

View reviewed changes

hsheth2 added merge-pending-ci A PR that has passed review and should be merged once CI is green. and removed merge-pending-ci A PR that has passed review and should be merged once CI is green. labels Oct 23, 2023

Merge branch 'master' into master+ing-200-view-lineage-for-sql-sources

bf4ecb5

vercel bot deployed to Preview October 25, 2023 04:50 View deployment

hsheth2 added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Oct 25, 2023

maggiehays added the hacktoberfest-accepted Acceptance for hacktoberfest https://hacktoberfest.com/participation/ label Oct 26, 2023

mayurinehate added 2 commits October 26, 2023 09:19

feat(ingest): support view lineage extraction for postgres, trino

b74e794

fallback to native postgres view lineage extraction for failed views

fix tests

337ca34

vercel bot deployed to Preview October 26, 2023 04:12 View deployment

mayurinehate requested a review from hsheth2 October 26, 2023 04:18

Merge branch 'master' into master+ing-200-view-lineage-for-sql-sources

e60a7f4

vercel bot deployed to Preview October 26, 2023 04:40 View deployment

hsheth2 approved these changes Oct 26, 2023

View reviewed changes

metadata-ingestion/src/datahub/utilities/sqlglot_lineage.py Outdated Show resolved Hide resolved

mayurinehate added 2 commits October 26, 2023 12:15

add sqlglot dependency

1fe4b02

update comment

c5ca9b2

vercel bot deployed to Preview October 26, 2023 07:23 View deployment

hsheth2 merged commit f402090 into datahub-project:master Oct 26, 2023
53 of 54 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingest): support view lineage for all sqlalchemy sources #9039

feat(ingest): support view lineage for all sqlalchemy sources #9039

mayurinehate commented Oct 18, 2023 •

edited

Loading

mayurinehate Oct 19, 2023

mayurinehate Oct 19, 2023 •

edited

Loading

mayurinehate Oct 19, 2023

mayurinehate Oct 19, 2023

hsheth2 Oct 20, 2023

mayurinehate Oct 19, 2023

hsheth2 Oct 20, 2023

mayurinehate Oct 23, 2023

hsheth2 Oct 20, 2023

mayurinehate Oct 23, 2023

mayurinehate Oct 23, 2023

hsheth2 Oct 20, 2023

hsheth2 Oct 20, 2023

hsheth2 commented Oct 23, 2023

hsheth2 commented Oct 25, 2023

mayurinehate commented Oct 26, 2023

hsheth2 commented Oct 26, 2023

feat(ingest): support view lineage for all sqlalchemy sources #9039

feat(ingest): support view lineage for all sqlalchemy sources #9039

Conversation

mayurinehate commented Oct 18, 2023 • edited Loading

Checklist

Choose a reason for hiding this comment

mayurinehate Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsheth2 commented Oct 23, 2023

hsheth2 commented Oct 25, 2023

mayurinehate commented Oct 26, 2023

hsheth2 commented Oct 26, 2023

mayurinehate commented Oct 18, 2023 •

edited

Loading

mayurinehate Oct 19, 2023 •

edited

Loading