[APM] Dependencies Operations - trace cannot be found - with OTel

[AlexanderWert]

When using OTel data (e.g. the OTel demo), in the Dependencies --> Operations --> [SELECT a specific operation] view the most bottom view always shows The selected trace cannot be found:

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[AlexanderWert]

Related: https://github.com/elastic/opentelemetry-dev/issues/517

[MiriamAparicio]

After some investigation we did with @jennypavlova, these are our findings:

The trace waterfall for dependecies operations is filled querying to span documents using term query for span.destination.service.resource and span.name fields, where transaction.id exist, the results for this query are empty, after this query another one is made to retrieve the transactions for those spans, querying the transaction ids

We assumed that in otel, span documents don't have the transaction.id field and that's the reason why the trace waterfall is empy

@AlexanderWert is it possible to include the transaction.id in span documents from Otel?

[AlexanderWert]

@AlexanderWert is it possible to include the transaction.id in span documents from Otel?

That's unfortunately not easily possible as it would require to buffer spans (similar to tail-based sampling) which we definitely want to avoid.

[AlexanderWert]

The trace waterfall for dependecies operations is filled querying to span documents using term query for span.destination.service.resource and span.name fields, where transaction.id exist, the results for this query are empty, after this query another one is made to retrieve the transactions for those spans, querying the transaction ids

@MiriamAparicio Instead of going through the transaction.id couldn't we just do the same with trace.id? And in the second step query for those trace.ids but also transaction.id: * (to only limit to spans that are transactions) and service.name being the same as the one of the corresponding exit span. Would that be possible?

[MiriamAparicio]

Instead of going through the transaction.id couldn't we just do the same with trace.id?

Yeah, that seems possible, we can try and see if it works