feat(ingest/kafka) meta-mapping for Kafka source w/ Avro schemas

[danielcmessias]

Since Avro permits non-spec field/value pairs, a common pattern is to use this to embed business metadata. Being able to map this directly to DataHub's Owners, Tags, and Terms allows metadata to be kept closer to source, avoiding the need for Data Producers to configure transformers.

New Features

Mapping an Avro list-of-strings to DataHub Tags

Consider the following Avro schema:

{
  "name": "sampleRecord",
  "type": "record",
  "tags": ["tag1", "tag2"],
  "fields": [{
    "name": "field_1",
    "type": "string",
    "tags": ["tag3", "tag4"]
  }]
}

The following new config allows this to be ingested as DataHub tags:

config:
  schema_tags_field: tags
  tag_prefix: kafka

These config values are currently the default (so they wouldn't actually be needed).

Maybe it would make more sense for the default value of schema_tags_field to be something like _datahub_tags?

Field-level tags do not automatically become Dataset-level tags.

Meta-mapping implementation

I've re-used the meta-mapping implementation that already exists for dbt (with a small extension to support referencing nested fields).

Consider the following Avro schema:

{
  "name": "sampleRecord",
  "type": "record",
  "owning_team": "@Data-Science",
  "data_tier": "Bronze",
  "fields": [{
    "name": "field_1",
    "type": "string",
    "gdpr": {
      "pii": true
    }
  }]
}

This can now be mapped to DataHub metadata with the following config:

config:
  meta_mapping:
    owning_team:
      match: "^@(.*)"
      operation: "add_owner"
      config:
        owner_type: group
    data_tier:
      match: "Bronze|Silver|Gold"
      operation: "add_term"
      config:
        term: "{{ $match }}"
  field_meta_mapping:
    gdpr.pii:
      match: true
      operation: "add_tag"
      config:
        tag: "pii"

Note the ability to reference nested fields.

Top-level doc in Avro is used as the Dataset's description

A small extra feature that seemed like a no-brainer.

Consider the schema:

{
  "name": "datahub-0",
  "type": "record",
  "doc": "# This is a header.\nAnd this goes below it.",
  "fields": ...
}

Before, this doc field would be ignored. Now, it is used as the Dataset Description:

Implementation

The implementation is mostly just re-using the OperationProcessor created for the dbt source. Otherwise:

• KafkaSchemaRegistryBase interface is changed from get_schema_metadata() to get_aspects_from_schema(), to support returned multiple Aspects. I've kept the original method for backward compatibility in case there are some forks out there with custom schema-registry implementations.
• schema_util.py does involve passing around a lot of values. I was reluctant to do any refactoring, given this class is used in a bunch of places, but I suspect it does need some cleanup.
• Hopefully, there should be no breaking changes (at least that's the idea 😅 )

Obviously, this is just for Avro right now. I'm not familiar with Protobuf, so I don't know if the same thing could be done there.

Testing

I've made a start on getting test coverage for the new changes. If you feel we need to substantially rework any of the testing I'll probably need some support from you guys to get it done 🙂

Docs

Definitely a bit lacking. The trouble is I don't want to rewrite (or copy/paste) all the meta-mapping examples from the dbt page over to Kafka. Open to suggestions here - if there's potential for this 'meta-mapping pattern' to be used in a few different sources, perhaps we should standardize and have a dedicated page?

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[jjoyce0510]

Hi there!

Thanks for the contrib -- we will be looking into this shortly.

Cheers
John

[danielcmessias]

@jjoyce0510 @hsheth2 I'm also looking into whether it makes sense to add meta-mapping to the S3 recipe in the same way. This seems like a common pattern - when you already have a tagging system on the underlying tool and want a way to automatically translate those tags to DataHub metadata. Feel free to give me a ping on Slack if you guys have thoughts on this! 😃

[hsheth2]

@danielcmessias given the merge conflicts on this one, we ended up adapting + making a bunch of tweaks in #8825

I'm going to close this in favor of the new PR