Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(metadata-ingestion): implement mlflow source #7971

Merged
merged 11 commits into from
Sep 26, 2023

Conversation

hariishaa
Copy link
Contributor

Hi! I would like to make a contribution to DataHub project with a source for MLflow Model Registry. There is another abandoned PR with the same purpose (#2725) but it looks too outdated and it is primarily based on ingesting metadata from MLflow Experiments. In my opinion, a more appropriate solution is to make a source implementation based on Model Registry instead. So I decided to create the ingestion source for the latest version of MLflow from scratch.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added devops PR or Issue related to DataHub backend & deployment ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX labels May 4, 2023
@asikowitz asikowitz self-assigned this Jun 7, 2023
Copy link
Collaborator

@asikowitz asikowitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good to me and thanks so much for your contribution! Sorry for the delay in the review. I just have a few docs / style suggestions and a couple requested changes as we've updated our Source infrastructure.

metadata-ingestion/setup.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mlflow.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mlflow.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mlflow.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mlflow.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mlflow.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/mlflow.py Outdated Show resolved Hide resolved
@anshbansal anshbansal added the community-contribution PR or Issue raised by member(s) of DataHub Community label Jun 23, 2023
@asikowitz
Copy link
Collaborator

Hi @hariishaa, I think this is super close. Are you still looking to get this through?

@hariishaa
Copy link
Contributor Author

hariishaa commented Aug 24, 2023

@asikowitz, yeah... Have some time management problems 🙄 Hope to make all the changes soon

@hariishaa
Copy link
Contributor Author

hariishaa commented Sep 6, 2023

Hi @asikowitz! I've resolved all your comments 🙂 Now waiting for the approval

Copy link
Collaborator

@asikowitz asikowitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks so much for the contribution!

@asikowitz asikowitz added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Sep 22, 2023
@asikowitz
Copy link
Collaborator

Merging through what looks like test flakes

@asikowitz asikowitz merged commit 622816d into datahub-project:master Sep 26, 2023
56 of 59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community-contribution PR or Issue raised by member(s) of DataHub Community devops PR or Issue related to DataHub backend & deployment ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green. product PR or Issue related to the DataHub UI/UX
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants