[router][common] Multiple fixes in Opentelemetry

[m-nagarajan]

Summary

1. Added otel.venice.metrics.export.interval.in.seconds for OpenTelemetry (OTel) metrics export, with a default value of 60 seconds which is the same without this config right now.
2. Updated MetricEntityState to maintain a 1:1 relationship between OTel instruments and Tehuti sensors, rather than a 1:n relationship, to eliminate unnecessary lookups during the hot path.
3. Stopped emitting OTel metrics for the total store in the router. The aggregation will be done on the receiving side. This will be helpful during creation of pre-aggregates in the metrics processing systems by not having to do storeName != total.
4. Modified venice.response.status_code_category to use success/fail instead of healthy/unhealthy/tardy/throttled/bad_request to keep it standard. Tardy/throttled/bad_request can be inferred from the response status.
5. removed 'incoming_call_count' otel metric and reverted it back to tehuti only metric request as it was not covering all the incoming cases either and this was making things more confusing.
6. Renamed the existing OTel metric call_key_count to key_count and converted it into a histogram. This metric will now measure key counts on the response handling side, including success/fail details and response codes, similar to call_time, and will provide a distribution for key counts.
7. Fixed a bug where the exponential histogram view was configured for only one metric.

How was this PR tested?

GH CI and below log via integration tests shows all the new changes

2025-02-10 16:24:23 - [] INFO [VeniceOpenTelemetryMetricsRepository] [PeriodicMetricReader-1] Logging OpenTelemetry metrics for debug purpose: [ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=venice.router, version=null, schemaUrl=null, attributes={}}, name=venice.router.retry_delay, description=Retry delay time, unit=MILLISECOND, type=HISTOGRAM, data=ImmutableHistogramData{aggregationTemporality=DELTA, points=[ImmutableHistogramPointData{getStartEpochNanos=1739233403028360000, getEpochNanos=1739233463032171000, getAttributes={venice.cluster.name="venice-cluster_1ed021f416bf_e2ee76ba", venice.request.method="single_get", venice.store.name="store_1ed46d18c968_6db4c37d"}, getSum=11.0, getCount=1, hasMin=true, getMin=11.0, hasMax=true, getMax=11.0, getBoundaries=[], getCounts=[1], getExemplars=[]}]}}, ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=venice.router, version=null, schemaUrl=null, attributes={}}, name=venice.router.key_count, description=Count of keys during response handling along with response codes, unit=NUMBER, type=EXPONENTIAL_HISTOGRAM, data=ImmutableExponentialHistogramData{aggregationTemporality=DELTA, points=[ImmutableExponentialHistogramPointData{getStartEpochNanos=1739233403028360000, getEpochNanos=1739233463032171000, getAttributes={http.response.status_code="200", http.response.status_code_category="2xx", venice.cluster.name="venice-cluster_1ed021f416bf_e2ee76ba", venice.request.method="multi_get_streaming", venice.response.status_code_category="success", venice.store.name="store_1ed46d18c968_6db4c37d"}, getScale=3, getSum=100.0, getCount=10, getZeroCount=0, hasMin=true, getMin=10.0, hasMax=true, getMax=10.0, getPositiveBuckets=DoubleExponentialHistogramBuckets{scale: 3, offset: 26, counts: {26=10} }, getNegativeBuckets=EmptyExponentialHistogramBuckets{scale=3, offset=0, bucketCounts=[], totalCount=0}, getExemplars=[]}, ImmutableExponentialHistogramPointData{getStartEpochNanos=1739233403028360000, getEpochNanos=1739233463032171000, getAttributes={http.response.status_code="200", http.response.status_code_category="2xx", venice.cluster.name="venice-cluster_1ed021f416bf_e2ee76ba", venice.request.method="single_get", venice.response.status_code_category="success", venice.store.name="store_1ed46d18c968_6db4c37d"}, getScale=3, getSum=100.0, getCount=100, getZeroCount=0, hasMin=true, getMin=1.0, hasMax=true, getMax=1.0, getPositiveBuckets=DoubleExponentialHistogramBuckets{scale: 3, offset: -1, counts: {-1=100} }, getNegativeBuckets=EmptyExponentialHistogramBuckets{scale=3, offset=0, bucketCounts=[], totalCount=0}, getExemplars=[]}]}}, ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=venice.router, version=null, schemaUrl=null, attributes={}}, name=venice.router.aborted_retry_count, description=Count of aborted retry requests, unit=NUMBER, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1739233403028360000, epochNanos=1739233463032171000, attributes={venice.cluster.name="venice-cluster_1ed021f416bf_e2ee76ba", venice.request.method="single_get", venice.request.retry_abort_reason="no_available_replica", venice.store.name="store_1ed46d18c968_6db4c37d"}, value=1, exemplars=[]}], monotonic=true, aggregationTemporality=DELTA}}, ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=venice.router, version=null, schemaUrl=null, attributes={}}, name=venice.router.call_count, description=Count of all requests during response handling along with response codes, unit=NUMBER, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1739233403028360000, epochNanos=1739233463032171000, attributes={http.response.status_code="200", http.response.status_code_category="2xx", venice.cluster.name="venice-cluster_1ed021f416bf_e2ee76ba", venice.request.method="multi_get_streaming", venice.response.status_code_category="success", venice.store.name="store_1ed46d18c968_6db4c37d"}, value=10, exemplars=[]}, ImmutableLongPointData{startEpochNanos=1739233403028360000, epochNanos=1739233463032171000, attributes={http.response.status_code="200", http.response.status_code_category="2xx", venice.cluster.name="venice-cluster_1ed021f416bf_e2ee76ba", venice.request.method="single_get", venice.response.status_code_category="success", venice.store.name="store_1ed46d18c968_6db4c37d"}, value=100, exemplars=[]}], monotonic=true, aggregationTemporality=DELTA}}, ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=venice.router, version=null, schemaUrl=null, attributes={}}, name=venice.router.call_time, description=Latency based on all responses, unit=MILLISECOND, type=EXPONENTIAL_HISTOGRAM, data=ImmutableExponentialHistogramData{aggregationTemporality=DELTA, points=[ImmutableExponentialHistogramPointData{getStartEpochNanos=1739233403028360000, getEpochNanos=1739233463032171000, getAttributes={http.response.status_code="200", http.response.status_code_category="2xx", venice.cluster.name="venice-cluster_1ed021f416bf_e2ee76ba", venice.request.method="multi_get_streaming", venice.response.status_code_category="success", venice.store.name="store_1ed46d18c968_6db4c37d"}, getScale=3, getSum=39.09754, getCount=10, getZeroCount=0, hasMin=true, getMin=1.842334, hasMax=true, getMax=16.385833, getPositiveBuckets=DoubleExponentialHistogramBuckets{scale: 3, offset: 7, counts: {7=3,8=2,9=2,10=1,11=0,12=0,13=0,14=0,15=0,16=0,17=0,18=0,19=0,20=1,21=0,22=0,23=0,24=0,25=0,26=0,27=0,28=0,29=0,30=0,31=0,32=1} }, getNegativeBuckets=EmptyExponentialHistogramBuckets{scale=3, offset=0, bucketCounts=[], totalCount=0}, getExemplars=[]}, ImmutableExponentialHistogramPointData{getStartEpochNanos=1739233403028360000, getEpochNanos=1739233463032171000, getAttributes={http.response.status_code="200", http.response.status_code_category="2xx", venice.cluster.name="venice-cluster_1ed021f416bf_e2ee76ba", venice.request.method="single_get", venice.response.status_code_category="success", venice.store.name="store_1ed46d18c968_6db4c37d"}, getScale=3, getSum=214.82287399999996, getCount=100, getZeroCount=0, hasMin=true, getMin=1.039375, hasMax=true, getMax=56.24875, getPositiveBuckets=DoubleExponentialHistogramBuckets{scale: 3, offset: 0, counts: {0=2,1=9,2=12,3=12,4=15,5=22,6=13,7=8,8=3,9=0,10=0,11=1,12=0,13=0,14=0,15=1,16=0,17=0,18=0,19=0,20=1,21=0,22=0,23=0,24=0,25=0,26=0,27=0,28=0,29=0,30=0,31=0,32=0,33=0,34=0,35=0,36=0,37=0,38=0,39=0,40=0,41=0,42=0,43=0,44=0,45=0,46=1} }, getNegativeBuckets=EmptyExponentialHistogramBuckets{scale=3, offset=0, bucketCounts=[], totalCount=0}, getExemplars=[]}]}}]

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to explain your proposed changes and call out the behavior change.

1. otel.venice.metrics.export.interval.in.seconds config with a default value of 60 seconds.
2. venice.response.status_code_category dimension will emit values success/fail

[ZacAttack]

Can you explain this change actually? What is the semantic that's trying to be conveyed in this name change?

[m-nagarajan]

for counting requests we have

1. incoming_call_count at request handling path
2. call_count at response handling path

This can help us correlate with the incoming vs response handled cases. I was not inclined to be optimistic here to measure these just at response handling path.
This change makes counting keys to be similar (ie incoming_key_count and key_count) to be able to get similar insights from the response handling path like pass/fail, etc.

[ZacAttack]

I see. Let's document that, maybe even with a bit of javadoc. I'm not sure I would understand this when looking at a dashboard, and I'm not sure I'm clever enough to suggest a rename that would capture this nuance.

[m-nagarajan]

MetricEntity has a description field which has this info. Made it more explicit and added java doc as well.

[m-nagarajan]

Thought more about this and looking at further code, it looks like the request side and response side code for metric handling are not isolated fully and having separate metrics for incoming vs processed is only making it more confusing. Reverted some of the changes to remove incoming context and currently having only call_count and key_count and increment it the similar way we used to before.

[ZacAttack]

These changes overall look good, I'm giving a provisional ship it pending some of the things I've asked. I think overall I have some latent concern about the enum tweak and the potential new overhead for single key lookup, but aside from that I don't have any major objections. Thanks!

[m-nagarajan]

Thanks @ZacAttack for the review. Replied to your comments.

[m-nagarajan]

for counting requests we have

1. incoming_call_count at request handling path
2. call_count at response handling path

This can help us correlate with the incoming vs response handled cases. I was not inclined to be optimistic here to measure these just at response handling path.
This change makes counting keys to be similar (ie incoming_key_count and key_count) to be able to get similar insights from the response handling path like pass/fail, etc.

[lluwm]

Looks good and it makes sense to me!