Issues with Redshift Lineage for many Queries

[gky249]

Hi, We are observing some issues with Redshift lineage, for many queries, this error is thrown :

Error was An Identifier is expected, got Token[value: (] instead..

Could it be a issue related with dependencies on external tables ?
These problematic tables have dependency on tables from "stage_external" schema e.g.

Error parsing query create table "DATABASE"."SCHEMA"."TABLE" as ( with /* Starting points is .... from "DATABASE"."stage_external"."TABLE_X" ....
Error was An Identifier is expected, got Token[value: (] instead..

According to Redshift admin, the user we got have access to stage_external schema but no datasets are ingested from stage_external schema. When I search for "stage_external" in global search, I can see a couple of containers but these containers are empty.
Is it that tables in stage_external are external tables so they aren't ingested in Datahub ?
Datahub user is accessing SVV_TABLE_INFO and external tables aren't available in this view instead there is another view containing info about external tables i.e. SVV_EXTERNAL_TABLES.

For table lineage (parsing create, insert queries), I see these two types of error messages :

for getting lineage. Error was An Identifier is expected, got Token[value: (] instead..
for getting lineage. Error was An Identifier is expected, got Token[value: #] instead..

For Views lineage (CREATE MATERIALIZED VIEW), errors I see are :

Error was 'NoneType' object has no attribute 'raw_name'.
Error was An Identifier is expected, got Comparison[value: contracts_raw.contract_sk = events.contract_sk] instead..

Earlier I mentioned about stage_external schema, now I can see in the logs that the plugin tries to ingest tables/views from stage_external schema but it doesn't find any. i.e.

Processing schema: lt_phoenix.stage_external
Process tables in schema lt_phoenix.stage_external
No tables in cache for lt_phoenix.stage_external, skipping
Process views in schema lt_phoenix.stage_external
No views in cache for lt_phoenix.stage_external, skipping

Datahub version v10.5
Follow the discussion here - https://datahubspace.slack.com/archives/C029A3M079U/p1687961778614119

[treff7es]

@gky249 we are working on replacing sqlineage SQL parser in Redshift as well (in Snowflake and Bigquery, we already replaced it). -> #8921
This change will also add column-level lineage support, and hopefully, it will resolve your SQL parser issues as well.

[shifanm]

Hi @treff7es this is great to know. Thanks. Is there any ETA around this being completed for Redshift?

[github-actions]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[hsheth2]

#8921 was merged, so I'm going to go ahead and close this issue.