feat(ingestion): extend feast plugin to ingest tags and owners #11784
Conversation
github-actions
bot
added
ingestion
| @@ -45,7 +45,7 @@ arm64_darwin_preflight() { | |||
| pip3 install --no-use-pep517 scipy | |||
| fi | |||
|
|
|||
| brew_install "openssl@1.1" | |||
| brew_install "openssl@3.0.14" | |||
There was a problem hiding this comment.
Quick question - Why this change?
There was a problem hiding this comment.
I needed to upgrade the dependency version because the 1.1 version was deprecated.
There was a problem hiding this comment.
I reverted to the previous version
| @@ -1,4 +1,4 @@ | |||
| #!/bin/bash -e | |||
| #!/bin/bash -e | |||
| if feature_view.tags.get("name"): | ||
| tag = feature_view.tags.get("name") | ||
| tag_association = TagAssociationClass(tag=builder.make_tag_urn(tag)) | ||
| global_tags_aspect = GlobalTagsClass(tags=[tag_association]) |
There was a problem hiding this comment.
Just note: If you've attached tags in DataHub, this will replace them by default :)
There was a problem hiding this comment.
The goal is to have Feast as our source of truth, so the fields must be defined in Feast repo.
Do you agree with this?
| global_tags_aspect = GlobalTagsClass(tags=[tag_association]) | ||
| aspects.append(global_tags_aspect) | ||
|
|
||
| if feature_view.owner: |
There was a problem hiding this comment.
Just note: If you've attached owners in DataHub UI, this will replace them by default :)
There was a problem hiding this comment.
See comment above
| aspects = [StatusClass(removed=False)] | ||
|
|
||
| if entity.tags.get("name"): | ||
| tag: str = entity.tags.get("name") |
There was a problem hiding this comment.
Minor -
Any way to extract the owner and tags logic into a reusable method
There was a problem hiding this comment.
Already created a reusable method for tags and owner logic. Only thing missing is regarding the _owner_mapping (since we can only ingest the owner from Feast and we need to define the owner_type owner_ship_type_class, the solution i'm seeing is having a mapping to these values):
_owner_mapping: Dict[str, Dict[str, Any]] = { "Datahub": { "owner_type": builder.OwnerType.USER, "owner_ship_type_class": OwnershipTypeClass.DATAOWNER, } }
the issue here is having to specify in Datahub repo the specific use cases that each team might have (for example, in our case we will have an owner "MLOps") which might not be applicable to other teams.
Do you see any other possible solution?
margaridafernandes-trip
changed the title
| @@ -360,12 +363,46 @@ def _get_on_demand_feature_view_workunit( | |||
|
|
|||
| return MetadataWorkUnit(id=on_demand_feature_view_name, mce=mce) | |||
|
|
|||
| def _get_tags_and_owners(self, obj: Union[Entity, FeatureView, FeastField]) -> list: | |||
There was a problem hiding this comment.
We written and defensive. Nice work!
| # FIXME: Update to have more owners | ||
| _owner_mapping: Dict[str, Dict[str, Any]] = { | ||
| "Datahub": { | ||
| "owner_type": builder.OwnerType.USER, |
There was a problem hiding this comment.
Does feast support GROUP owners as well? Thanks for calling this out!
There was a problem hiding this comment.
yes, we can pass the owner as a "Person" (user) or "Group"
There was a problem hiding this comment.
the issue is that from Feast side I can only specify the owner: str (so, for example "Margarida") and this should map to a OwnerType.USER and to the appropriate OwnershipTypeClass.
Checklist