Enabling tracing by usage of opentelemetry

[rem1-dev]

Enabling tracing by usage of opentelemetry

[rem1-dev]

@shesek this is same as PR #99 but applied to fresh branch created out of new-index. Commits were squashed.

[rem1-dev]

I'm working on solving those conflict files

[rem1-dev]

@shesek conflicts resolved, it's ready to be merged.

[philippem]

@rem1-dev when testing locally we ran into some errors

tempo-1           | level=warn ts=2024-11-13T20:09:28.400374334Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185917 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"
tempo-1           | level=warn ts=2024-11-13T20:09:28.426261953Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185912 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"

which may be related to cardinality or size of the spans.

Possible causes:

- Large Trace Data: The trace contains too many spans or overly large data (such as logs or metadata).
- High Cardinality: Excessive number of unique tags, attributes, or data points being attached to spans, leading to large trace size.
- Excessive Span Detail: Individual spans may contain large payloads or detailed information that increase the trace size.
- Improper Sampling Rate: Sampling configuration might be too high, causing too many traces to be captured, some of which may exceed the size limit.
- Large Payloads in Spans: If spans contain large amounts of data (e.g., large JSON bodies, files, or detailed logs), it could push the trace size over the limit.
- Too Many Traces Collected: An overly aggressive tracing configuration that collects an excessive number of traces, increasing the likelihood of exceeding the size limit.
- Long or Complex Transactions: If a trace represents a very large or complex operation (e.g., a multi-step workflow or a distributed operation), it might naturally grow too large.

[rem1-dev]

@philippem how can I reproduce this locally?