proposed metrics plugin: gRPC

[jacobmarble]

I would like to add a metrics/monitoring plugin for InfluxDB. InfluxDB already has a spanstore plugin, implemented as a remote gRPC service.

However, it looks like a gRPC plugin for metrics does not exist.

• Have there been discussions about this yet?
• Would this contribution be accepted?
• Should such a plugin be implemented separately from, but with similar patterns to, the spanstore gRPC plugin?

[yurishkuro]

A bit of history - spanstore plugin was not originally created as a remote storage API, but as a sidecar process API using hashicorp/plugin framework. We later realized that since the sidecar plugin actually just implements a gRPC service we could turn it into proper remote storage API without changes. This way we indirectly end up with a remote storage API that is specific to Jaeger (since no industry standard existed).

This is different for metrics storage. Even though technically we could go via similar route, the first question is why would we want to create our own bespoke remote storage API for metrics when there is already a Prometheus Remote Storage API that is wildly used in the industry? Since our current implementation already uses that API, I don't think there is a need for another.

You can integrate InfluxDB for metrics by implementing Prometheus Remote Storage API over InfluxDB, if that does not exist already. By doing so you also make InfluxDB compatible with other systems that rely on that same API.

[yurishkuro]

if the queries can still be expressed in Prometheus Remote Read API, then I think it makes sense to switch to that. @albertteoh any thoughts?

[jacobmarble]

I've looked into this over the weekend. For example, this is the PromQL query behind MetricsReader.GetCallRates():

sum(rate(calls_total{service_name =~ "customer", span_kind =~ "SPAN_KIND_SERVER"}[10m])) by (service_name,operation)

To implement this query with the Prometheus Remote Read API, is a rough equivalent:

query := &prompb.Query{
	StartTimestampMs: 17,
	EndTimestampMs:   18,
	Matchers: []*prompb.LabelMatcher{
		{
			Type:  prompb.LabelMatcher_RE,
			Name:  "service_name",
			Value: "customer",
		},
		{
			Type:  prompb.LabelMatcher_RE,
			Name:  "span_kind",
			Value: "SPAN_KIND_SERVER",
		},
	},
	Hints: &prompb.ReadHints{
		StartMs: startTimestamp.Milliseconds(),
		EndMs:   endTimestamp.Milliseconds(),
		StepMs:  (10 * time.Minute).Milliseconds(),
		Func:    "rate",
		Grouping: []string{"service_name", "operation"},
	},
}

Notice that:

• The remote read API affords only one "function" per call
• That function is a "hint" -- the service is not required to do anything with it
• The result will always be larger than the PromQL query results
• The result will always require post-processing by the client (sum and possibly rate; more in the case of GetErrorRates and GetLatencies)

It seems to me that the primary use case of the Prometheus Remote Read API is to enable long-term storage integrations, and it's not meant to provide the same level of expressiveness as PromQL.

---

If this analysis is correct, then I would like to re-open the proposal for a gRPC metrics plugin.

[albertteoh]

Thanks for the detailed investigation and report @jacobmarble!

If the Prometheus Remote Read API was the original design

Nope, it was just designed for the for the Prometheus query API because it's supported by a fairly large number of vendors: https://promlabs.com/blog/2021/10/14/promql-vendor-compatibility-round-three/

if the queries can still be expressed in Prometheus Remote Read API, then I think it makes sense to switch to that.

Yup, I agree.

The result will always require post-processing by the client (sum and possibly rate; more in the case of GetErrorRates and GetLatencies)

It sounds like an increased memory and CPU pressure on jaeger-query to compute, from the example above, the rate and sum aggregations, instead of offloading the work to the Prometheus backend.

It's probably not a huge problem since Jaeger UI wasn't designed to poll jaeger-query, so the remote read API query would only execute on "refresh" of the Monitor tab.

So all in all, I think this should be okay given the benefits of allowing more metrics backend integrations.

Thoughts?

[jacobmarble]

The result will always require post-processing by the client (sum and possibly rate; more in the case of GetErrorRates and GetLatencies)

It sounds like an increased memory and CPU pressure on jaeger-query to compute, from the example above, the rate and sum aggregations, instead of offloading the work to the Prometheus backend.

It's probably not a huge problem since Jaeger UI wasn't designed to poll jaeger-query, so the remote read API query would only execute on "refresh" of the Monitor tab.

So all in all, I think this should be okay given the benefits of allowing more metrics backend integrations.

I'm not confident that the CPU/memory pressure on jaeger-query would be small enough to overlook. Imagine a metric:

1. with high cardinality
2. with high frequency
3. stored in a database with Prometheus Remote Read service
4. but stored in a database with no hint support

In this case, the database would return raw data, with no aggregation, resulting in a large quantity of results, perhaps 1000x more than would have been returned by a PromQL query.

---

A second concern I have with this API is the headline disclaimer:

This is not currently considered part of the stable API and is subject to change even between non-major version releases of Prometheus.

This disclaimer exists despite the fact the Remote Read API was introduced in Prometheus version 2.0.0, released on November 8th, 2017.

[yurishkuro]

The result will always require post-processing by the client (sum and possibly rate; more in the case of GetErrorRates and GetLatencies)

To me this is quite a strong reason not to switch from Query API to Remote Read API. I have no desire to maintain metrics transformation functions in Jaeger backend.

So overall +1 for Jaeger-specific query API to drive the Monitoring tab. We already have the internal API for that in storage/metricsstore/interface.go, so exposing this via grpc remote api seems sensible (but remote only, not sidecar - cf #4409)

[jacobmarble]

Please continue any discussion here: #4413