fix(ingest): Upgrade avro version to fix issues with decimal types

[harsha-mandadi-4026]

• We've recently encountered a problem when attempting to include terms in a Decimal-type column of a DBT model(we use Athena as the data platform).
• A range of Avro versions are specified here and since we use poetry it's installing the lower end of the mentioned Avro versions which is 1.10.2 and this version has a bug for Decimal types.
• The bug is other_props is not being passed to the BytesDecimalSchema class. This is the function in Avro 1.10.2

def make_bytes_decimal_schema(other_props):
    """Make a BytesDecimalSchema from just other_props."""
    return BytesDecimalSchema(other_props.get('precision'), other_props.get('scale', 0))

• Basically, the meta of a given DBT model's is held in other_props, and since that's not passed to the superclass of the BytesDecimalSchema class, the meta is lost only for Decimal column types. This issue has been fixed in the latest version of Avro 1.11.2. See here.
• Because of the above-mentioned reason, we bumped the Avro version(poetry now install Avro 1.11.2) and tested it out and we can now see that terms getting assigned to DBT decimal column types as well.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[hsheth2]

Thanks @harsha-mandadi-4026!

We'll need to tread carefully here because because we use avro for serializing when writing to kafka

Also, it looks like CI might be failing with this change e.g.

Traceback (most recent call last):
  File "/home/runner/work/datahub/datahub/metadata-ingestion/scripts/modeldocgen.py", line 712, in <module>
    generate()
  File "/home/runner/work/datahub/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/runner/work/datahub/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/runner/work/datahub/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/runner/work/datahub/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/runner/work/datahub/datahub/metadata-ingestion/scripts/modeldocgen.py", line 608, in generate
    mcps = list(generate_stitched_record(relationship_graph))
  File "/home/runner/work/datahub/datahub/metadata-ingestion/scripts/modeldocgen.py", line 353, in generate_stitched_record
    field = avro.schema.Field(
TypeError: Field.__init__() got an unexpected keyword argument 'type'

[harsha-mandadi-4026]

Thanks @harsha-mandadi-4026!

We'll need to tread carefully here because because we use avro for serializing when writing to kafka

Also, it looks like CI might be failing with this change e.g.

Traceback (most recent call last):
  File "/home/runner/work/datahub/datahub/metadata-ingestion/scripts/modeldocgen.py", line 712, in <module>
    generate()
  File "/home/runner/work/datahub/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/runner/work/datahub/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/runner/work/datahub/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/runner/work/datahub/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/runner/work/datahub/datahub/metadata-ingestion/scripts/modeldocgen.py", line 608, in generate
    mcps = list(generate_stitched_record(relationship_graph))
  File "/home/runner/work/datahub/datahub/metadata-ingestion/scripts/modeldocgen.py", line 353, in generate_stitched_record
    field = avro.schema.Field(
TypeError: Field.__init__() got an unexpected keyword argument 'type'

Hi @hsheth2 - Thanks for taking a look. I can see in the Field class that it accepts a keyword argument called type but it's been slightly renamed to type_.

class Field(CanonicalPropertiesMixin, EqualByJsonMixin):
    _reserved_properties: Sequence[str] = FIELD_RESERVED_PROPS

    def __init__(self, type_, name, has_default, default=None, order=None, names=None, doc=None, other_props=None, validate_names: bool = True):

this part

                field = avro.schema.Field(
                    type=f["type"],
                    name=f["name"],
                    has_default=False,
                )

in metadata_ingestion/scripts/modeldocgen.py seems to be failing.

Looks like there are quite a few changes in the schema.py file of Avro, I'll take a look.

[hsheth2]

Closing because #9042 was merged