Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest/dremio): Dremio Source Ingestion #11598

Merged
merged 48 commits into from
Nov 5, 2024
Merged

feat(ingest/dremio): Dremio Source Ingestion #11598

merged 48 commits into from
Nov 5, 2024

Conversation

sagar-salvi-apptware
Copy link
Contributor

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata docs Issues and Improvements to docs product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment labels Oct 11, 2024
@sagar-salvi-apptware sagar-salvi-apptware changed the title feat: Dremio Source Ingestion feat(ingest/dremio): Dremio Source Ingestion Oct 11, 2024
mayurinehate
mayurinehate reviewed Oct 17, 2024
View reviewed changes
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_datahub_source_mapping.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_datahub_source_mapping.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py Outdated Show resolved Hide resolved
mayurinehate
mayurinehate requested changes Oct 17, 2024
View reviewed changes
Copy link
Collaborator

@mayurinehate mayurinehate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please make use of report.warning and/or report.error for cases where user needs to be aware of error details - such as login failure due to missing PAT or incorrect password ?

@mayurinehate mayurinehate self-requested a review October 28, 2024 14:58
@mayurinehate mayurinehate added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Nov 5, 2024
@mayurinehate mayurinehate merged commit 5d17ecb into datahub-project:master Nov 5, 2024
77 checks passed
dremio-brock
dremio-brock reviewed Nov 6, 2024
View reviewed changes
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_sql_queries.py
SYS.PROJECT.HISTORY.JOBS
WHERE
STATUS = 'COMPLETED'
AND LENGTH(queried_datasets)>0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Length is not valid on an array field, use
AND ARRAY_SIZE(queried_datasets)>0
instead

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#11801

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dremio-brock dremio-brock dremio-brock left review comments

@acrylJonny acrylJonny acrylJonny left review comments

@b-rockx b-rockx b-rockx left review comments

@hsheth2 hsheth2 hsheth2 left review comments

@mayurinehate mayurinehate mayurinehate approved these changes

Assignees
No one assigned
Labels
devops PR or Issue related to DataHub backend & deployment docs Issues and Improvements to docs ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green. product PR or Issue related to the DataHub UI/UX
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@sagar-salvi-apptware @hsheth2 @b-rockx @mayurinehate @acrylJonny @dremio-brock