Kamon Observability

[contrebande-labs]

Hello,

I'm running 0.7.0 on Ubuntu 20.04 with a JDK11 JVM. I have enabled Kamon and Kanela and I'm looking for the "example metrics" mentioned in the monitoring guide within Kamon's APM app. All I can find is Akka and JVM stuff, no "various metrics about the public graph". Can someone tell me where to find those, please ? Thanks.

[pm47]

Hello, metrics are uploaded automatically but you have to create your own dashboards and charts. Follow this tutorial to create a chart. Set "Data Source" to "kamon-application" and use the "Visualization" field to select your metrics with autocomplete. To find out about all metrics, look for files named Monitoring.scala in the repository, for example routing-table metrics are defined here.

[contrebande-labs]

Hi, thanks! Wow, that's a lot of hidden gems right there! I forked eclair to have a look at different Kamon reporters to send the data to a custom OpenTelemetry receiver. Will report my findings here.

[pm47]

that's a lot of hidden gems right there!

Indeed there are a lot of useful metrics, we rely on them extensively to monitor our production node.

a custom OpenTelemetry receiver. Will report my findings here

Please do! We're interested.

[contrebande-labs]

So I patched 0.7.0 to upgrade Kamon to 2.5.0 and add all the available reporter dependencies in eclair-node's pom.xml. I also disabled all the reporters by default (including Kamon's own APM) in eclair-node's application.conf. Also in eclair-node's application.conf, I removed the whole kanela.modules block because I was getting Malformed configuration for module on path: jdbc. The module will be ignored. in eclair's logs. Doing this made the warning disappear. All the Maven tests pass and I have been running this configuration in production for over 3 days now and it seems to work well. I am using prometheus for the metrics and jaeger for the traces because these exporters are more mature than Kamon's opentelemetry exporter. I couldn't get InfluxDB to work either. Everything is exported to a local opentelemetry-collector instance that then sends it all to elasticsearch for visualization in Kibana. I'm also concurrently running the Kamon APM exporter at the same time to be able to compare between the two dashboards. I'm planning on disabling the Kamon APM once I'm satisfied with the configuration. For now, I'm missing the logs so I will see if I can have that working as well in the next few days. When its all good, I will patch the master branch on my fork and update the monitoring.md document and then initiate a PR for you to review. Finally, adding the exporter jars to the eclair-node package added 10Mb to the file and having all of these new classes loaded might affect performances so I will need you to tell me how you you want the runtime cpu and memory usage done. Please let me know if this is the correct course of action.