All indicators on the Monitor page are empty

[zenvzenv]

What happened?

I want to use jaeger's monitor page to monitor the running status of the application. I configured METRICS_STORAGE_TYPE according to the instructions, configured otelcol's SpanMetrics Connector and --prometheus.query.support-spanmetrics-connector=true, but it didn't work, the monitor page All indicators are empty in

Steps to reproduce

1. export METRICS_STORAGE_TYPE=prometheus
2. export PROMETHEUS_SERVER_URL=http://localhost:9090
3. Set --prometheus.query.support-spanmetrics-connector=true
4. Configure the spanmetrics connector of otelcol,my otelcol config file like this:

   extensions:
     health_check:
     zpages:
       endpoint: 0.0.0.0:55679
   
   receivers:
     otlp:
       protocols:
         grpc:
   
   processors:
     batch:
   
   exporters:
     prometheus:
       endpoint: "0.0.0.0:8889"
       namespace: promexample
   
     jaeger:
       endpoint: 10.20.74.60:14250
       tls:
         insecure: true
   
     prometheusremotewrite:
       endpoint: http://localhost:9090/api/v1/write
       namespace: promremotewriteexample
       target_info:
         enabled: true
       tls:
         insecure: true
       remote_write_queue:
         enabled: false
         num_consumers: 10
   
   connectors:
     spanmetrics:
       namespace: spanmetrics
       histogram:
         explicit:
           buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
       dimensions:
         - name: http.method
           default: GET
         - name: http.status_code
       exemplars:
         enabled: true
       exclude_dimensions: ['status.code']
       dimensions_cache_size: 1000
       aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"    
       metrics_flush_interval: 15s 
   
   service:
     pipelines:
       traces:
         receivers: [otlp]
         exporters: [jaeger, spanmetrics]
   
       metrics:
         receivers: [spanmetrics]
         exporters: [prometheus]
   
     extensions: [health_check, zpages]

Expected behavior

The monitor page displays performance data

Relevant log output

No response

Screenshot

No response

Additional context

No response

Jaeger backend version

1.47.0

SDK

No response

Pipeline

javaagent->otelcol->spanmetrics connector->prometheus

Stogage backend

clickhouse

Operating system

linux

Deployment model

CLI

Deployment configs

No response

[albertteoh]

Can you see your traces in Jaeger?

Have you tried going through the Troubleshooting steps?

• I noticed you've configured otel with namespace: spanmetrics which means you'll need to add the --prometheus.query.namespace=spanmetrics flag. I suggest checking the Prometheus server to see if you can find spanmetrics.calls_total and spanmetrics.duration_bucket metrics there.

[zenvzenv]

1. I am able to see traces data in jaeger.
2. I've checked out Troubleshooting, but it doesn't seem to be working.

• http://all-in-one:14269/metrics is work and have jaeger_requests_total and jaeger_latency_bucket data
• http://jaeger-query:16687/metrics is not work,so /metrics endpoint have no the following indicators

  jaeger_query_requests_total{operation="get_call_rates",result="ok"} 18
  jaeger_query_requests_total{operation="get_error_rates",result="ok"} 18
  jaeger_query_requests_total{operation="get_latencies",result="ok"} 36
  
  jaeger_query_latency_bucket{operation="get_call_rates",result="ok",le="0.005"} 5
  jaeger_query_latency_bucket{operation="get_call_rates",result="ok",le="0.01"} 13
  jaeger_query_latency_bucket{operation="get_call_rates",result="ok",le="0.025"} 18
  
  jaeger_query_latency_bucket{operation="get_error_rates",result="ok",le="0.005"} 7
  jaeger_query_latency_bucket{operation="get_error_rates",result="ok",le="0.01"} 13
  jaeger_query_latency_bucket{operation="get_error_rates",result="ok",le="0.025"} 18
  
  jaeger_query_latency_bucket{operation="get_latencies",result="ok",le="0.005"} 7
  jaeger_query_latency_bucket{operation="get_latencies",result="ok",le="0.01"} 25
  jaeger_query_latency_bucket{operation="get_latencies",result="ok",le="0.025"} 36
  

3. Prometheus have no latency_bucket and calls_total metrics data
4. I add --prometheus.query.namespace=spanmetrics flag to jaeger start cmd,my cmd is : SPAN_STROAGE_TYPE=grpc-plugin METRICS_STORAGE_TYPE=prometheus PROMETHEUS_SERVER_URL=http:localhost:9090 PROMETHEUS_QUERY_SUPPORT_SPANMETRICS_CONNECTOR=true jaeger-all-in-one --grpc.stroage.plugin.binary=/path/to/jaeger-clickhouse-linux-amd64 --grpc.stroage.configuration=/path/to/config.yml --prometheus.query.namespace=spanmetrics,but I can't see spanmetrics.calls_total and spanmetrics.duration_bucket metrics in prometheus

[albertteoh]

Are you running off a docker image?
If you're running the latest tag, please remove the this image tag with docker rmi -f jaegertracing/all-in-one:latest as you may be using an old cached version.

I also suggest running the docker-compose example: https://github.com/jaegertracing/jaeger/tree/main/docker-compose/monitor#quickstart.

I've just checked and is working as expected for me.

That can be your baseline that you can use to build upon for your specific deployment.

[zenvzenv]

I run docker-compose expample success, but my production environment does not have docker,I just can use CLI to deploy,I deployed it according to the configuration of docker, but the monitor page still has no data.The following are my relevant commands and configurations, please help me to see if there is any problem, thank you

1. prometheus
   • start cmd: prometheus --config.file=otel-prometheus.yml --web.listen-address="0.0.0.0:9091"
   • otel-prometheus.yml like follow:

     global:
       scrape interval: 15s
       evaluation_ interval: 15s
     scrape configs:
       - job _name: 'aggregated -trace-metrics'
         scrape_interval: 25 
         static_configs:
           - targets: ['10.20.78.142:8888']
           - targets: ['10.20.78.142:8889']

2. jaeger
   • start cmd like follow:

   export COLLECTOR_OTLP_ENABLE=false
   export METRICS_STORAGE_TYPE=prometheus
   export PROMETHEUS_SERVER_URL=http://10.20.78.142:9091
   export PROMETHEUS_QUERY_SUPPORT_SPANMETRICS_CONNECTOR=true
   nohup jaeger-all-in-one --query.ui-config jaeger-ui.json

   • jaeger-ui.json like:

   {
     "monitor": {
       "menuEnaled": true
     }
     "dependencies": {
       "menuEnaled": true
     }
   }

3. otelcol-contrib
   • config.xml like:

   receivers:
     otlp:
       protocols:
         grpc:
         http:
   
     jaeger:
       protocols:
         thrift_http:
           endpoint: "0.0.0.0:14278"
   
   process:
     batch:
   
   exporters:
     prometheus:
       endpoint: "0.0.0.0:8889"
   
     jaeger:
       endpoint: "10.20.78.142:14250"
       tls:
         insecure: true
   
   connectors:
     spanmetrics:
   
   service:
     pipeline:
       traces:
         receivers: [otlp, jaeger]
         processors: [batch]
         exporters: [spanmetrics, jaeger]
   
       metrics/spanmetrics:
         receivers: [spanmetrics]
         exporters: [prometheus]

4. I also download and build microsim,and I use microsim -j http://localhost:14278/api/traces -d 24h -s 500ms cmd to generate trace data,I can see trace data and monitor data.It seems that the data imported through port 14278 can display monitor data, but the data imported through otlp protocol (port 4317) cannot display data

[albertteoh]

Okay, so I think we've established that your challenges most likely boil down to configuration.

The reason why I suggest running the stack locally is so you have a chance to understand the configs necessary to get things working for SPM in a simple, self-contained environment that can be quickly brought up and down.

For example, when I copy your OTEL config and paste it into the docker-compose example I can see the following error immediately on startup:

monitor-otel_collector-1  | Error: failed to get config: cannot unmarshal the configuration: 2 error(s) decoding:
monitor-otel_collector-1  |
monitor-otel_collector-1  | * '' has invalid keys: process
monitor-otel_collector-1  | * 'service' has invalid keys: pipeline
monitor-otel_collector-1  | 2023/08/18 11:43:30 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 2 error(s) decoding:
monitor-otel_collector-1  |
monitor-otel_collector-1  | * '' has invalid keys: process
monitor-otel_collector-1  | * 'service' has invalid keys: pipeline
monitor-otel_collector-1 exited with code 1

That's because:

process:
  batch:

should be:

processors:
  batch:

and there may be more configuration issues but at least you can test some of your deployment configs quickly within the local docker-compose setup and be confident they work, then you can deal with any problems relating to deployment and network.

In short, we're reducing the problem space down to separate problems to make it easier to troubleshoot:

• Application configs:
  • OTEL config
  • Prometheus config
  • Jaeger config
• Deployment configs:
  • Container image config
  • Networking config
  • K8s config
  • etc.

[zenvzenv]

Sorry, my OTEL configuration is processsors, because I am an intranet developer, I made a typo by mistake

[zenvzenv]

My trace data is generated by opentelemetry-javaagent and transmitted through port 4317 of OTLP, microsim is sent to port 14278 of OTELCOL's jaeger, and I noticed that OTELCOL's jaeger receiver has been marked as deprecated

[zenvzenv]

1. prometheus
   • start cmd : prometheus --config.file=otel-prometheus.yml
   • prometheus.yml like follow:

     global:
       scrape interval: 15s
       evaluation_ interval: 15s
     scrape_configs:
         static_configs:
           - targets: ['localhost:8888']
           - targets: ['localhost:8889']

2. jaeger
   • start cmd like follow:

     export COLLECTOR_OTLP_ENABLE=false
     export METRICS_STORAGE_TYPE=prometheus
     export PROMETHEUS_SERVER_URL=http://localhost:9090
     export SPAN_STROAGE_TYPE=grpc-plugin
     export PROMETHEUS_QUERY_SUPPORT_SPANMETRICS_CONNECTOR=true
     nohup jaeger-all-in-one --query.ui-config jaeger-ui.json --grpc-stroage-plugin.binary=jaeger-clickhouse --grpc-storage-plugin.configuration-file=clickhouse-config.yml

3. otelcol-contrib
   • config.xml like:

     receivers:
       otlp:
         protocols:
           grpc:
           http:
     
     processors:
       batch:
     
     exporters:
       prometheus:
         namespace: otelcol
         endpoint: "0.0.0.0:8889"
     
       jaeger:
         endpoint: "localhost:14250"
         tls:
           insecure: true
     
     connectors:
       spanmetrics:
     
     service:
       pipelines:
         traces:
           receivers: [otlp, jaeger]
           processors: [batch]
           exporters: [spanmetrics, jaeger]
     
         metrics/spanmetrics:
           receivers: [spanmetrics]
           processors: [batch]
           exporters: [prometheus]

4. The trace data of all services can be accepted and displayed normally,jaeger-all-in-one service has monitor data,but after a while, the monitor data is gone.One of my services (I have 3 in total now) only has data for a short period of time (a few minutes) but then it's gone again

[zenvzenv]

I want to known this is why

[albertteoh]

Have you checked your otel collector logs? I'm surprised it's running because you're using a jaeger receiver but haven't configured it under your receivers section.

You've also got a typo in jaeger- all- in-one's env vars, so I suggest checking the log output:

SPAN_STROAGE_TYPE=grpc-plugin

[albertteoh]

As mentioned in the above comment, we can catch these issues quickly if we copy-paste your configs above into the monitor example OTEL config then make run.

I just did it and took me less than a minute to catch this config issue as well (should be pipelines instead of pipeline):

monitor-otel_collector-1  | Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
monitor-otel_collector-1  |
monitor-otel_collector-1  | * 'service' has invalid keys: pipeline
monitor-otel_collector-1  | 2023/08/24 21:56:03 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
monitor-otel_collector-1  |
monitor-otel_collector-1  | * 'service' has invalid keys: pipeline
monitor-otel_collector-1  |
monitor-otel_collector-1 exited with code 1

[zenvzenv]

1. jaeger and clickhouse grpc-plugin works fine,there is no error message in jaeger log.If I don't use grpc-plugin, the monitor data is also intermittent, some services have data, some services have no data
2. otelcol-contrib works fine,there is no error message in otelcol-contrib log.the pipeline in reality pipelines,it was my hand that made a mistake

[albertteoh]

If I don't use grpc-plugin, the monitor data is also intermittent, some services have data, some services have no data

The intermittent services could be caused by a trace ingestion problem, i.e. spans are not sent at all, dropped or sampled out somehow. If you can't see it in Monitor data, I would not expect the trace data to be there as well.

[zenvzenv]

Those service Traces data without Monitor data can be accepted and displayed normally.

The following is traces data but no monitor data:

The following is traces data and monitor data:

The following is traces data but also intermittent monitor data

[albertteoh]

Can you check the traces where the metrics are missing from the Monitor tab?

One possibility is that the spans have a kind "client". Currently, SPM only considers the server kind spans. See: https://www.jaegertracing.io/docs/1.48/spm/#serviceoperation-missing-in-monitor-tab.

[zenvzenv]

Thank you very much, helped me understand the reason.Spans that are indeed server types will be recorded, and spans of other types will not be recorded.

I can't set the type of span by myself because I use opentelemetry-javaagent. I thought that since one service has Monitor data, the rest of the services should also have Monitor data.

[albertteoh]

Yeah, maybe we should make the span kind configurable for users to allow client as well.

[zenvzenv]

I think this is a very good idea and look forward to the subsequent version evolution