Tempo Metrics works only sometimes

[golan118a]

Hi
As I can see in previous posts in this forum, a very common and easy way to monitor the traces duration is with the newly added Tempo Metrics and more specifically the functions max/min/avg_over_time.
My issue is that sometimes it works and sometimes it doesn't. Grafana does not specify there is an error running the TraceQL, all I see is "no data".

I checked and the data stream is consistent and available and without the aggregate function I can see the traces duration clearly.

Every help is appreciated, thanks

[joe-elliott]

Here is a setup guide for TraceQL metrics:

https://grafana.com/docs/tempo/latest/operations/traceql-metrics/

Are there any error logs in the queriers, query frontend or metrics generator? The request/response logs in the query frontend may also give some clues. Look for lines containing "query range response" and "query range request".

[golan118a]

Thank you for the reply.

Following the setup guide seem to have fixed my issue.
I would like though to generate an alert if a trace duration is too long and I know it is not possible to create an alert rule with Tempo as a data source. I know the metrics generated are sent to Prometheus but the question is how do I look for them there?

I have tried running the query:
histogram_quantile(0.95, sum(rate(traces_spanmetrics_latency_bucket[$__rate_interval])) by (le, service))
but the results don't match.

I'm not sure if this question is for this forum, if so I apologize and would appreciate a redirect please.
thanks again

[joe-elliott]

One of your metrics is max and the other is p95. You are also hiding information making it easier to understand the issue. Is this the same service?

[golan118a]

Hi
Yes, both pictures are made by querying the same service.
What I am trying to do is to generate an alert in case a business process takes too long (SLA). I already have traces that encapsulate the entire process which is established with different services, MQs and DBs. Therefor if I could monitor a trace duration I can generate an alert.
Because there is not an option to generate an alert on Tempo, I am trying to find the metrics in Prometheus that will tell me about the traces durations with the metrics sent by Tempo's metrics generator.
In the above example I tried comparing 2 queries where I expected to show similar results or pattern. The first picture is of a query made from a metric that is received from Tempo and sent to Prometheus - traces_spanmetrics_latency_bucket.
The second picture is directly from Tempo - {resource.service.name="XXX"} | max_over_time(duration).
I expected to see some form of correlation.
This is a new dashboard I made just to find a metric I can monitor:

I am grateful for any help I can get, and if this topic isn't for this forum please redirect me,
thanks again