[Logs Overview] Improve analyzer by filtering unsuitable tokens

[weltenwort]

📝 Summary

This improves the analyzer configuration used by the logs overview categorization feature so it's less susceptible to numeric and hexadecimal values that overwhelm the rest of the content after tokentization.

closes #197255

🎨 Previews

before

after

🕵️‍♀️ Review notes

For testing the synthtrace scenario distributed_unstructured_logs can be used to create test data:

node scripts/synthtrace.js \
  --from=now-12h \
  --to=now+12h \
  --logLevel=debug \
  distributed_unstructured_logs.ts \
  --scenarioOpts.distribution=poisson \
  --scenarioOpts.rate=10000 \
  --scenarioOpts.messageGroup=userAuthentication

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[cauemarcondes]

LGTM

[weltenwort]

@elasticmachine merge upstream

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 3fee0a8
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-197868-3fee0a8cb5b8

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
logsShared	424.1KB	424.3KB	+233.0B

History

• 💛 Build #246037 was flaky 9e705f3

cc @weltenwort