Kamon Metric Visualisation with Grafana

[prongs]

I'm not able to fully understand how to go about visualising some of the metrics. Specifically, the metrics that seem like it should have been a gauge but is actually a histogram.

As an example, I want to track the size of the pending message for Akka actors. This seems like a simple-enough gauge-like metric, but it's actually a "Range Sampler".

As per Kamon Docs

---

Range Samplers

Range samplers are an unusual type of instrument, only seen so far in Kamon. When monitoring queues, like we do for actor mailboxes, just having a number going up and down (like a traditional gauge) that is collected once at a reporting tick isn’t enough. Many things can happen to a queue between recordings, such as you could get a million messages in a queue and process them all before the next recording, seeing a value of 0 that does not reflect the fact that the queue had a million elements just a few seconds ago.
When monitoring a queue size it is not just about knowing where it is at a given moment because that number is probably incorrect right after reading it, but knowing where it was is of great help. The range sampler internally tracks three variables: the current value, the minimum and the maximum observed value. These three values are read and stored in a histogram every 200 milliseconds (by default) and produce a value distribution on each reporting interval.
After each observation, the minimum and maximum tracking variables are reset to the current value, meaning that if no changes occur then the three of them will have the same value when the next observation happens.

---

The docs do not mention how to massage the range sampler to view it as a gauge. Maybe that would be a good thing to add to the docs.

The problem I'm facing is how to make sense of the values. The following shows 3 different observations:

root@host:/# curl http://service:9095/ | grep group | grep akka_group_pending_messages | grep 'WebSocketEngine"'  | grep -v bucket
akka_group_pending_messages_count{system="application",group="application/user/WebSocketEngine"} 88596.0
akka_group_pending_messages_sum{system="application",group="application/user/WebSocketEngine"} 6415.0

root@host:/# curl http://service:9095/ | grep group | grep akka_group_pending_messages | grep 'WebSocketEngine"'  | grep -v bucket
akka_group_pending_messages_count{system="application",group="application/user/WebSocketEngine"} 89496.0
akka_group_pending_messages_sum{system="application",group="application/user/WebSocketEngine"} 6415.0

root@host:/# curl http://service:9095/ | grep group | grep akka_group_pending_messages | grep 'WebSocketEngine"'  | grep -v bucket
akka_group_pending_messages_count{system="application",group="application/user/WebSocketEngine"} 93996.0
akka_group_pending_messages_sum{system="application",group="application/user/WebSocketEngine"} 6415.0

What I understand from the above is that the sum metric didn't increase and the count metric did. Which means new observations were recorded but there were no new messages for these actors.

Now, looking at another actor group:

root@host:/# curl http://service:9095/ | grep group | grep akka_group_pending_messages   | grep -v bucket 
akka_group_pending_messages_count{system="application",group="application/user/Actor/EventsWebSocket"} 207333.0
akka_group_pending_messages_sum{system="application",group="application/user/Actor/EventsWebSocket"} 5010.0

root@host:/# curl http://service:9095/ | grep group | grep akka_group_pending_messages 
akka_group_pending_messages_count{system="application",group="application/user/Actor/EventsWebSocket"} 208230.0
akka_group_pending_messages_sum{system="application",group="application/user/Actor/EventsWebSocket"} 5014.0

Here, both sum and counts have increased. Weirdly, count increased by 897 but sum increased only by 4. Not really sure what that means. Does it mean that the actor only got 4 new messages in 897 observations?

Coming to deriving gauge-like meaning from it, I think I should be doing some sort of division of sum/count, but can't figure out the exact combination. Applying it to the paragraph above, if we divide 4 by 897, we'll get a very small fractional value and that doesn't represent the queue size. Tried the following graphs, none of which make sense to me:

akka_group_pending_messages_sum

Shows Linearly increasing value from 5k to 5.1k and beyond. Not sure how to interpret this

akka_group_pending_messages_sum/akka_group_pending_messages_count

Slowly moving fractional value. Again, not sure of the interpretation.

rate(akka_group_pending_messages_sum)/rate(akka_group_pending_messages_count)

Somewhat correlates with how I think the actor is receiving new messages. As in the graph increases when I expect new messages to arrive and then decreases when I think it's not receiving more messages, or at least it's receiving messages at a slower pace.

But again, unable to interpret the values. Values are ranging from 0 to 0.16. Not sure what fractional value means here

rate(akka_group_pending_messages_sum)

This graph seems similar to the above one and also correlates with the pace of the actor receiving the messages. But same problem here, I'm unable to interpret the values, which range from 0 to 2.29.

[prongs]

Same problem persists for the JVM Heap Size metrics. I'm not sure why that is implemented as a Range sampler instead of as a simple Gauge. Makes things really confusing and metrics don't seem to correspond with the exact behaviour.

A few weeks back, I had a reproducible OOM in my service and I thought I should be able to correlate the Heap Size metric with the service crash. As in, the metrics should show increased heap usage just before the crash. This did not happen, as the service was running with Xmx = 1GB but the graph on grafana showed only 150MB usage. There too, I had the same confusion as described above. The metric should have been a gauge, but is actually a Range Sampler. There again, we were looking at rate(sum)/rate(count), but the values on the graph did not correlate with the actual crash.

Similar challenge is mentioned in this comment: #751 (comment).