Prometheus View of runtime.MemStats

[nfisher]

Motivation

I'm not sure if this belongs in this repo but I think there's some common runtime stats that most monitored services would want to expose similar to those available in /debug/pprof. I would see this feature being responsible for creating a measurement and view for each of the fields in the runtime.MemStats struct.

Impact

As this would be a new function this is an API expansion. The performance impact of adding this view is expected to be approximately equivalent to the cost of calling runtime.ReadMemStats and populating all of the measurements.

Proposed Implementation

Add a package go.opencensus.io/stats/view/runtime with the following functions and data structs;

• New(prefix string) RuntimeViews.
• RuntimeViews.Views() []View - returns a collection of all of the runtime views.
• RuntimeViews.Start(freq uint64) - starts the background polling goroutine.
• RuntimeViews.Stop() - stops the background polling goroutine.

param - prefix

• Used to prefix each of the stats.

param - freq

• Not sure if this is desirable or if it should make the ReadMemStats call on each request.
• Idea would be to periodically call ReadMemStats in a background goroutine and cache the values until the timer has triggered.

return - []View

• Each element in the []View would correspond to a field in the MemStats struct.
• Descriptions would be the first line of each of the fields in the MemStats struct documentation.
• The full names would be ${prefix}/runtime/${SNAKE_CASE_FIELD_NAME} (I'll map these out more fully later).

[semistrict]

This is a really neat idea and in my opinion would be a good addition to this repo. However, currently the stats package is oriented around aggregating data whereas this data is from existing counters in the runtime (already aggregated).

We are in the process of designing new interfaces to this library to support already-aggregated metrics and once that's done I think this would be an excellent test case. See: census-instrumentation/opencensus-specs#118
census-instrumentation/opencensus-proto#53

Once this is implemented in Go, we can do something like this suggestion without having a Goroutine poll runtime.ReadMemStats, instead we will probably have some way to have OpenCensus, at export time, pull the already-aggregated values directly from runtime.ReadMemStats and send them to the exporter without going through the stats package at all.

[odeke-em]

Great thoughts @nfisher! I too encountered a need for this and implemented one almost a month ago when I was instrumenting Caddy server with OpenCensus https://github.com/orijtech/caddy/blob/c8498719b7c1c2a3c707355be2395a35f03e434e/caddy/caddymain/exporters.go#L54-L110

@Ramonza the idea of extracting those metrics only on a Prometheus pull, for Prometheus I believe won't allow other exporters to get these metrics but also if the scrape duration from Prometheus is too large e.g 30 seconds, on a server an event could happen within that time which would then give no indication of the event.

@nfisher here is a by the way(random perhaps wakcy idea), a nicer and perhaps more better way might be to get information from /proc/<PID> on Linux or what top does on OSX :)

[semistrict]

@odeke-em I was not suggesting we extract them in Prometheus but the stats package has an internal timer that we could use for this purpose.

[nfisher]

@Ramonza cool not sure where it is suitably located merely referenced stats/view as a package base because the other views are there. I'm likely to pick this up. It seems the prom client provides it but I'm not sure it's compatible with OC views/measurements.

[semistrict]

I think a new dir under /plugins/ seems like the most appropriate place to put this.

[rakyll]

This is blocked by the work needs to be done to support census-instrumentation/opencensus-proto#53.

[wdhnl]

Now that census-instrumentation/opencensus-proto#53 is merged, can this be revisited?

I'd love to see a way to be able to export this on intervals that isn't prometheus specific (in my case, to Stackdriver).

[burdiyan]

+1. Having Go runtime metrics in Stackdriver would be very helpful.

[burdiyan]

Also seems like blocked label can be removed now, as mention PR was merged actually.

[Zenithar]

+1

[marwan-at-work]

Hi, can this be used with an Exporter that is not using the new metric library? The OCHTTP library defines its own view.View(s) and therefore we automatically get metrics by just registering those views...not sure how to do this here. Thanks!