Self-monitoring / meta-monitoring with Jaeger

[paulbruzz]

Hello everyone,

I am trying to implement self-monitoring or meta-monitoring within Jaeger, but I've had difficulty finding a comprehensive guide. The only reference I've encountered is a brief mention in the documentation (here).

Could anyone provide further insights or resources on this topic?
Thank you in advance for your help.

[dosubot[bot]]

Hey @paulbruzz! 👋 I'm here to help you with any bugs, questions, or becoming a contributor. Let's tackle this together!

To implement self-monitoring or meta-monitoring within Jaeger, you can utilize the built-in metrics, logging, and tracing capabilities. Here are the steps and resources available:

1. Metrics:

   • Jaeger microservices expose metrics in Prometheus format by default.

   • Use the following command line options to control metrics exposure:

     --admin.http.host-port <port>
     --metrics-backend <prometheus|expvar>
     --metrics-http-route <endpoint>

   • Each Jaeger component exposes the metrics scraping endpoint on the admin port. For example:

     • jaeger-agent: Port 14271
     • jaeger-collector: Port 14269
     • jaeger-query: Port 16687
     • jaeger-ingester: Port 14270
     • all-in-one: Port 14269

   • You can use the Prometheus monitoring mixin for Jaeger, which includes a prebuilt dashboard for Grafana. More information can be found here.

2. Logging:

   • Jaeger components log to standard out using the go.uber.org/zap library.
   • Logs are JSON encoded strings. Example:

     {"level":"info","ts":1615914981.7914007,"caller":"flags/admin.go:111","msg":"Starting admin HTTP server","http-addr":":14269"}
     {"level":"info","ts":1615914981.7914548,"caller":"flags/admin.go:97","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}

   • Adjust the log level using the --log-level command line switch.

3. Traces:

   • Jaeger can trace its own components, such as requests to the Query service.
   • To see these traces, start the all-in-one component and refresh the UI screen. You will see jaeger-query in the Services dropdown.
   • To disable these traces, set the JAEGER_DISABLED=true environment variable:

     docker run -e JAEGER_DISABLED=true -p 16686:16686 jaegertracing/all-in-one:<version>

For a comprehensive guide, refer to the Monitoring Jaeger documentation ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.