feat(ingest/snowflake): initialize schema resolver from datahub for l… #8903
Conversation
…ineage-only ingestion
github-actions
bot
added
the
ingestion
| def _init_schema_resolver(self) -> SchemaResolver: | ||
| if not self.config.include_technical_schema and self.config.parse_view_ddl: | ||
| if self.ctx.graph: | ||
| return self.ctx.graph.initialize_schema_resolver_from_datahub( |
There was a problem hiding this comment.
instead of initialize_schema_resolver_from_datahub returning Tuple[SchemaResolver, set[urns]], can we change it so that it only returns SchemaResolver, and in turn SchemaResolver has a method to get a set of loaded urns?
There was a problem hiding this comment.
makes sense. What would we call this method in SchemaResolver ? get_urns ?
| @@ -1004,8 +1017,6 @@ def _process_view( | |||
| yield from self._process_tag(tag) | |||
|
|
|||
| yield from self.gen_dataset_workunits(view, schema_name, db_name) | |||
| elif self.config.parse_view_ddl: | |||
There was a problem hiding this comment.
these changes confuse me a bit - are we dropping parse_view_ddl, or is that functionality handled elsewhere?
There was a problem hiding this comment.
These conditions were part of schema extraction process, such that - even if schema ingestion is disabled but parse_view_ddl is enabled, the code to fetch and generate schema metadata from snowflake would run, in order for that to be used during view definitions cll extraction.
Actual view definition parsing logic is in snowflake_lineage_v2.py and the only thing it requires is view_definitions to be populated. This code is still intact.
There was a problem hiding this comment.
ok that broadly makes sense
mayurinehate
force-pushed
the
master+ing-334-snowflake-initialize-schema-resolver
branch
from
d28b157
to
04ea7cb
Compare
mayurinehate
force-pushed
the
master+ing-334-snowflake-initialize-schema-resolver
branch
from
04ea7cb
to
3eb8189
Compare
hsheth2
added
the
merge-pending-ci
…ineage-only ingestion
Checklist