feat(Dataquality aspect): Added Data Quality Metrics aspect to emit data quality metrics metadata into Datahub

[naresh-angala]

Checklist

• [ X ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[jjoyce0510]

Hi there! What is the goal with this PR? Adding context in the description will be quite useful! Thanks in advanced

[naresh-angala]

Hi there! What is the goal with this PR? Adding context in the description will be quite useful! Thanks in advanced

Hi,

PR is about adding Data Quality Metrics capability, working on changes for dynamic Data Quality metrics addition as per PR review comments.

Thanks.

[rtekal]

@naresh-angala I know that this PR is just the model changes. Can you attach a documentation link that conveys the big picture and where it all fits please.

[jjoyce0510]

Requested changes and clarifications have not been addressed on this PR

[rtekal]

Okay, the team will change the PR to Draft and work on the design changes. Thanks.

[sgm44]

PR is about adding Data Quality Metrics capability, working on changes for dynamic Data Quality metrics addition as per PR review comments.

This is the data model changes to support the full ability to capture and report data quality dimensions. There was a bit of back and forth in slack back in Oct 2023 on this topic which include example usage screens found here

Here was the simple Feature Goal statement:
As a data producer, I want quality metrics for my ingested datasets to display within the dataset view in the catalog so that the metrics are available to data consumers.

[sgm44]

@naresh-angala and @rtekal -- where are the graphQL and UI updates related to this feature? Right now this looks like just PDL updates. Without the rest I don't see how datahub gets any value outside of the ability to ingest and store the data which IMHO is pretty basic.

[naresh-angala]

@naresh-angala and @rtekal -- where are the graphQL and UI updates related to this feature? Right now this looks like just PDL updates. Without the rest I don't see how datahub gets any value outside of the ability to ingest and store the data which IMHO is pretty basic.

@sgm44 -- Intial plan was to get the Data quality model changes be reviewed and accepted.
Would be updating the code with GraphQL and UI related change subsequently after the model.

[naresh-angala]

@jjoyce0510 -- Can you share details on below,

1. Can a feature be contributed with multiple PRs.
2. If feature need to be contributed with single PR, let us know comments on below approach
   --> Marking this PR in draft mode and add the below changes incrementally,
   a) Dynamic list of Dimensions with new aspect
   b) Mapper changes and GraphQL changes
   c) UI changes to display the quality metrics

Thanks.

[naresh-angala]

@jjoyce0510 -- Can you share details on below,

1. Can a feature be contributed with multiple PRs.
2. If feature need to be contributed with single PR, let us know comments on below approach
   --> Marking this PR in draft mode and add the below changes incrementally,
   a) Dynamic list of Dimensions with new aspect
   b) Mapper changes and GraphQL changes
   c) UI changes to display the quality metrics

Thanks.

@jjoyce0510 -- Please provide details on above points.

[Curiosity007]

@naresh-angala Is there any tentative timeline for this feature to be fully integrated into the UI, GraphQL and backend? This is an integral part of DQ, and would like very much to see this in the newest version

[rtekal]

@naresh-angala told me: Targetting last week of Sep to complete

[naresh-angala]

Requested changes and clarifications have not been addressed on this PR

@jjoyce0510 : Updated the PR with dynamic dimension names and UI changes. Please review.

[naresh-angala]

UI Screen shots: Dataset metrics: Chart view:

Schemafield metrics:

[naresh-angala]

UI Screenshots:
Dataset metrics: Table view: