Visualising Long Running/In Flight/Live Spans

[JavaPerformance]

Hi All,

I've been playing with Jaeger/OpenTelemetry over the past few days, I adore the functionality and the UI however I'm running into an issue with our particular use case.

We run 10's of thousands of long running Java batch jobs, they range from a few minutes to days. These jobs have numerous steps and call a number of infrastructure components (DB etc).

We'd love to visualise and understand the performance of these jobs at runtime and your Span sequence diagram is ideal.

I have written a Javassist based instrumentor to inject OTEL Spans into our code and 3rd party code which works well but unfortunately (understandably) we can't see the Traces until the parent Span has ended, which in our case leaves us blind for up to a day and is only useful in post mortem analysis.

I have written my own SpanProcessor to export the Spans ( via JaegerGrpcSpanExporter) both onStart() and onEnd() and confirmed that all of the Id's for a particular Span are the same on both calls and that at onStart that hasEnded is false however the results are not what I hoped for.

Rather than merging the two "identical" Spans onEnd() the are both persisted and the UI displays both with one having a run time of 213484886 days 17h ;)

I know I'm asking a lot, but do you think it would be possible and of value to render "In Flight" Spans for those of us that have long running processes?

Paul.

[yurishkuro]

We had issues and attempted PRs to support "partial" spans, but none of that work was ever completed.

Is it necessary for you to represent such long-running workflow with a single parent span? What if you just have a short span that kicks off the whole thing, and then another short span to finish the workflow?

Ultimately, Jaeger UI only works with spans that have well-defined duration. This is not the case for in-flight spans, so there's no good way to represent them on the timeline, so perhaps don't try doing it in the first place.

Also, you could combine the workaround above with still having a common in-flight parent span, but that span won't be shown until it finishes, but the shorter start/finish child spans would allow rendering the trace.

[JavaPerformance]

Thanks for the reply, I'll give the workaround a go and let you know tomorrow.

Now, I'm rubbish at Go :) .... But I'm currently building Jaeger, where would I find the point in the code where I could "Merge" an incoming Span with an already persisted Span with an identical Id? (Yes I'm naively confident :)

[yurishkuro]

#979