[jaeger-v2] Dangerous use of tracer in plugin/storage/grpc/factory.go

[yurishkuro]

The factory is passed a tracer that it then passes to OTEL's ToClientConn() function, which automatically adds tracing instrumentation to gRPC connection. That means we may start generating new traces as we're trying to save spans to storage, thus causing an infinite loop of trace generation. The write path must always be devoid of tracing (or use very low sampling rate) to avoid this.

Related upstream issue open-telemetry/opentelemetry-collector#10663

[mahadzaryab1]

@yurishkuro is this blocked on the upstream issue? or is there anything we can do right now to fix this?

[yurishkuro]

@mahadzaryab1 there is unfinished PR 5979. It's a bit hacky solution.

I would've preferred for this to be fixed upstream, but that ticket did not see much traction. We could just propose a new hook in OTEL to be usable from binaries like Jaeger, which would allow us to customer the tracer somehow, e.g. by using it with Jaeger Remote Sampler SDK extension which would allow to configure undesired endpoints with 0 probability.

Another option we could try to use is to inject a noop span into the context when the execution comes to places that we do not want to be traced - this will at least prevent downstream tracing into storage writes, but will not help with say healthcheck endpoint tracing (tbh that one could be fixed as a configuration option in the extension itself, to say "disable tracing" - assuming it's an issue today).