[Logs Overview] Filter unsuitable tokens

[weltenwort]

📓 Summary

Log message commonly contain identifies and ids that are inherently (pseudo)random and contain characters that would be interpreted as punctuation in prose, such as UUIDs or addresses. In many cases the analyzer splits these into several small and diverse tokens that overwhelm the distance metric of the categorization aggregation. To reduce the likelihood of that we can filter tokens that look like strings consisting only of hexadecimal values before categorization and limit the number of tokens compared.

• 🔗 related to: [Logs Overview] Enhanced logs component for solution UIs #190848

✔️ Acceptance criteria

• The categorization_analyzer is configured such that the char_filter ignores hexadecimal tokens.
• The categorization_analyzer is configured such that a limit filter sets a reasonable maximum token count.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)