feat(ingestion/redshift): collapse lineage to permanent table #9704
Conversation
…com:sid-acryl/datahub-fork into master+ing-473-redshift-temp-tables-lineage
…com:sid-acryl/datahub-fork into master+ing-473-redshift-temp-tables-lineage
…com:sid-acryl/datahub-fork into master+ing-473-redshift-temp-tables-lineage
| start_time=datetime.now(), | ||
| session_id="abc", | ||
| create_command="CREATE TABLE #player_activity_temp", | ||
| parsed_result=SqlParsingResult( |
There was a problem hiding this comment.
I would've liked this test to be more broad. For example, we should be mocking the query results from the query that fetches temp tables and relying on the redshift lineage implementation to parse build TempTableRow instances. That way, we would not have any hardcoded SqlParsingResult objects in our tests and would ensure that we can actually fetch temp table definitions correctly
There was a problem hiding this comment.
I mock the query result for test_collapse_temp_lineage.
There was a problem hiding this comment.
Is there a reason you didn't make similar changes to the rest of the tests?
| @@ -1024,6 +1024,18 @@ def _is_dialect_instance( | |||
| return False | |||
|
|
|||
|
|
|||
| def get_query_type( | |||
There was a problem hiding this comment.
is this method still used anywhere? if not, can we remove it?
|
|
||
| query_vs_cursor_mocker = { | ||
| ( | ||
| "-- DataHub Redshift Source temp table DDL query\n select\n *\n " |
There was a problem hiding this comment.
using """ would've been nicer to read than this, but it's fine
Redshift supports temporary table.
A permanent table can be derived from temporary table, we need to show upstream permanent table and it's table & column lineage.
To enable this feature set flag
resolve_temp_table_in_lineageandinclude_table_rename_lineageto true