Add otel codegen forward serialization

[sfc-gh-jopel]

Description of PR

• Adds code generator as a protoc plugin to generate pure python marshaling logic
  • The marshaler serializes forwards to a bytearray buffer, front to back by pre-calculating sizes
  • The code generator is triggered using ./scripts/proto_codegen.sh, and the logic of the protoc plugin is in ./scripts/plugin.py. The plugin parses the protoc CodeGeneratorRequest which contains descriptors for the custom message definitions. It extracts relevant information for serialization and then uses the Jinja2 template ./scripts/templates/template.py.jinja2 to generate the code
  • The generated code is under src/snowflake/telemetry/_internal/opentelemetry/proto/**
  • The code generator adds a dev and test dependency on black, isort, grpcio-tools, and Jinja2, but does not add any runtime dependencies
  • The generated code size is 3k lines and ~100kB, which is much smaller than protobuf package at 2MB
• A major performance improvement of this PR was inlining all of the primitive type serialization and size calculation utils from src/snowflake/telemetry/_internal/serialize/**
  • Function calls in python are very expensive so this led to 25% performance improvement alone
  • The downside is that it increases complexity of code generator a little and also increases generated code size
• bytearray with resizing was used instead of BytesIO w/ fixed size buffer because BytesIO writing is slower than bytearray even for buffers up to 4MB, this change was about a 10% performance improvement

Follow up PRs:

• Internalize adapter logic to remove opentelemetry-exporter-otlp-proto-common dep and protobuf: Add adapter code to remove opentelemetry-exporter-otlp-proto-common dep sfc-gh-jopel/snowflake-telemetry-python#4
• Add proxy import so that protobuf is used instead of custom logic if it is already present in the runtime

Testing

• Adds a merge gate check-codegen to ensure that the generated code is the latest generated code from
• Adds property-based testing of marshaler message types using hypothesis module. This module randomly generates inputs, and the test ensure the serialized bytestrings of this custom serialization module and protobuf are the exact same
• Adds a simple unit test to execute the code generator and ensure it works as expected

[sfc-gh-sili]

👍

[sfc-gh-sili]

This seems exactly the same as SerializeToString. I am curious what this is for?

[sfc-gh-sili]

Thanks for the work! Looks great

Things verified:

• Generation scripts finishes successfully
• Codes are properly documented and tested
• Checked that in the POC Add adapter code to remove opentelemetry-exporter-otlp-proto-common dep sfc-gh-jopel/snowflake-telemetry-python#4, our custom proto interface is almost identical to the upstream proto interface

Nit:

• inline_size_function could have an example in the docstring
• gen-requirements.txt could go into scripts

[sfc-gh-tmonk]

These changes look good to me. I like the demonstration of interface equivalence in test_proto_serialization.py.

One nit is to remove the from __future__ import annotations lines, because they should not be required in this library. It only supports Python 3.8 and above.

Thanks for all the great work!