Enhance Kafka exporter to respect max message size

[yurishkuro]

This is a copy of the upstream issue open-telemetry/opentelemetry-collector-contrib#36982 that we encountered, creating a ticket here to track and for better discoverability.

Problem is described here: #6437 (comment)

[Riteek712]

@yurishkuro I would like to contribute to this issue can you please assign me this?

[yurishkuro]

consider it assigned

[Riteek712]

Hi @yurishkuro , I am just informing you that I'm working on this issue, I came down with a cold last week, I am sorry for the delay.

[Riteek712]

@ADI-ROXX sure, can you mention your email for correspondence?

[shivanshuraj1333]

hey folks, I'm working on this as it is needed for us, the splitting logic for traces is not straight forward for spans, had to do binary search over spans.

Here's a PR open-telemetry/opentelemetry-collector-contrib#37176, I have added unit tests.
Would open it up for review once I have manually tested it with a kafka cluster that the splitting is happening as expected.

[ADI-ROXX]

Okay @shivanshuraj1333 👍