Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ISSUE-19454: Fixes broken looker lineage #19456

Merged
merged 4 commits into from
Jan 22, 2025

Conversation

olof-nn
Copy link
Contributor

@olof-nn olof-nn commented Jan 20, 2025

Describe your changes:

Fixes 19454

Looker lineage is broken when a table reference is enclosed by backticks (common when using BigQuery).
This addresses that issue.

Type of change:

  • Bug fix

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
  • I have added a test that covers the exact scenario we are fixing. For complex issues, comment the issue number in the test for future reference.

Copy link
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@olof-nn
Copy link
Contributor Author

olof-nn commented Jan 20, 2025

I have verified that this solves the issue when running it in our environment.

Copy link
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copy link
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Comment on lines +587 to +588
source_table_name = self._clean_table_name(sql_table_name, dialect)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I f i understand correctly this is so the query is compiled to the correct dialect, right?

Copy link
Contributor Author

@olof-nn olof-nn Jan 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cleaning of the table name already occurs, this is just adding the functionality of making `project_id.dataset_id.table_id` -> project_id.dataset_id.table_id. This is common to enclose table references like this in BigQuery. Our looker views are defined like:
sql_table_name: `project_id.dataset_id.table_id` ;;. The Looker ingestion then tries to find lineage references by searching for:

db: `project_id
schema: dataset_id
table: table_id`

Thus we find no lineage between looker <-> BQ

This addition in the _clean_table_name method will address that

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I.e. it is not compiling the query. That is happening only if the table ref is a derived table in Looker. That scenario is already handled in the elif statement

@ulixius9 ulixius9 merged commit fd2575d into open-metadata:main Jan 22, 2025
17 checks passed
@ulixius9 ulixius9 added the To release Will cherry-pick this PR into the release branch label Jan 22, 2025
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants