Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Print the number of time a header is present in a message #4229

Open
2 tasks done
Shawcs opened this issue Sep 13, 2023 · 0 comments
Open
2 tasks done

Print the number of time a header is present in a message #4229

Shawcs opened this issue Sep 13, 2023 · 0 comments
Labels
status/triage Issues pending maintainers triage type/feature A new feature

Comments

@Shawcs
Copy link

Shawcs commented Sep 13, 2023

Issue submitter TODO list

  • I've searched for an already existing issues here
  • I'm running a supported version of the application which is listed here and the feature is not present there

Is your proposal related to a problem?

Hello all,
we recently got an issue with a topic that was very big but the messages looked small. Here what we saw:

  • When we open the message inside the UI nothing looked fishy and we had a message size of 800bytes and header of 200bytes.
  • However the topic had approximately 400 000 msg and it's size was more than 400 gb.
  • I started a Statistics run on this topic (I can't attach a screen here) but in the key size I saw:
    total 13 MB | min 36 bytes | max 36 bytes | avg 36 bytes
    and for values:
    total 21 MB | min 55 bytes | max 59 bytes | avg 55 bytes

can you explain what the "total size" mean in the context ?

In the end I read directly the messages in the partition file inside the broker and saw that a header was repeated thousand times in a single message. The root cause was a miss configuration in the producer but in the kafka UI nothing show duplicate header.

Describe the feature you're interested in

If a message contains multiple time the same header, even with the same value, it would be nice to print the number of time this header is present in the message. This can highlight some producing problems

Describe alternatives you've considered

No response

Version you're running

0.7.1

Additional context

No response

@Shawcs Shawcs added status/triage Issues pending maintainers triage type/feature A new feature labels Sep 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/triage Issues pending maintainers triage type/feature A new feature
Projects
None yet
Development

No branches or pull requests

1 participant