Gateway usage gauge #33370
Gateway usage gauge #33370
Conversation
ogaca-dd
added
the
qa/done
Uncompressed package size comparisonComparison with ancestor Diff per package
Decision |
Test changes on VMUse this command from test-infra-definitions to manually test this PR changes on a VM: inv aws.create-vm --pipeline-id=54727709 --os-family=ubuntuNote: This applies to commit 8f1ba93 |
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 2d763d6 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | +1.67 | [+0.78, +2.57] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.71 | [+0.65, +0.77] | 1 | Logs bounds checks dashboard |
| ➖ | file_tree | memory utilization | +0.07 | [+0.00, +0.13] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | +0.06 | [-0.72, +0.83] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | +0.05 | [-0.42, +0.52] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.03 | [-0.66, +0.73] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | +0.03 | [-0.04, +0.09] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_0ms_latency_http1 | egress throughput | +0.02 | [-0.82, +0.87] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.01 | [-0.82, +0.84] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.00 | [-0.02, +0.01] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | -0.02 | [-0.30, +0.26] | 1 | Logs |
| ➖ | file_to_blackhole_300ms_latency | egress throughput | -0.02 | [-0.66, +0.61] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http2 | egress throughput | -0.03 | [-0.93, +0.87] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | -0.04 | [-0.83, +0.74] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | -0.95 | [-3.97, +2.07] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -2.52 | [-2.60, -2.45] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | links |
|---|---|---|---|---|
| ✅ | file_to_blackhole_0ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency_linear_load | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | |
| ✅ | quality_gate_logs | lost_bytes | 10/10 | |
| ✅ | quality_gate_logs | memory_usage | 10/10 |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
| @@ -77,7 +77,7 @@ func (c component) SetOTelAttributeTranslator(attrstrans *attributes.Translator) | |||
| } | |||
|
|
|||
| func (c component) ReceiveOTLPSpans(ctx context.Context, rspans ptrace.ResourceSpans, httpHeader http.Header) source.Source { | |||
There was a problem hiding this comment.
nit: you can add hostFromAttributesHandler to this func as well (ignore if you plan to do it in a follow-up PR)
There was a problem hiding this comment.
I tried to minimise the changes as I had many issues with repository cross dependencies. I'll do it in a follow-up PR.
|
/merge |
Devflow running:
|
…/gateway-usage-gauge
Gitlab CI Configuration ChangesRemoved Jobs
Changes Summary
ℹ️ Diff available in the job log. |
| when: never | ||
| - when: always | ||
| timeout: 15 minutes | ||
| # datadog_otel_components_ocb_build: |
There was a problem hiding this comment.
Please create a card to restore it so that we don't forget
There was a problem hiding this comment.
Tracked by OTAGENT-270
|
/merge |
Devflow running:
|
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR adds a new gauge metric `datadog.otel.gateway`. This metric is set at `1` when the collector is used as a gateway otherwise, the value is `0`. If `datadog.host.name` was sent with two different values, then the collector is used as a gateway. Note: [The PR on opentelemetry-mapping-go](DataDog/opentelemetry-mapping-go#479) should be merged before this PR and `go.mod` must use a commit on main instead of `0.24.1-0.20250124131402-bcd6fd1617a6` Associated PRs: - DataDog/opentelemetry-mapping-go#479 - DataDog/datadog-agent#33370 <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes <!--Describe what testing was performed and which tests were added.--> #### Testing The following commands generates the metric `datadog.otel.gateway` at `1` ``` telemetrygen metrics --otlp-insecure --otlp-attributes "datadog.host.name"=\"host1\" telemetrygen metrics --otlp-insecure --otlp-attributes "datadog.host.name"=\"host2\" ``` Same tests were performed for `traces` and for `logs` when `isLogsAgentExporterEnabled` is false. <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.--> --------- Co-authored-by: Yang Song <[email protected]>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR adds a new gauge metric `datadog.otel.gateway`. This metric is set at `1` when the collector is used as a gateway otherwise, the value is `0`. If `datadog.host.name` was sent with two different values, then the collector is used as a gateway. Note: [The PR on opentelemetry-mapping-go](DataDog/opentelemetry-mapping-go#479) should be merged before this PR and `go.mod` must use a commit on main instead of `0.24.1-0.20250124131402-bcd6fd1617a6` Associated PRs: - DataDog/opentelemetry-mapping-go#479 - DataDog/datadog-agent#33370 <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes <!--Describe what testing was performed and which tests were added.--> #### Testing The following commands generates the metric `datadog.otel.gateway` at `1` ``` telemetrygen metrics --otlp-insecure --otlp-attributes "datadog.host.name"=\"host1\" telemetrygen metrics --otlp-insecure --otlp-attributes "datadog.host.name"=\"host2\" ``` Same tests were performed for `traces` and for `logs` when `isLogsAgentExporterEnabled` is false. <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.--> --------- Co-authored-by: Yang Song <[email protected]>
What does this PR do?
This PR adds a
nilargument for the changes done by DataDog/opentelemetry-mapping-go#479.For now, the metric
datadog.otel.gatewayis not sent for OTLP ingest for for OTEL agent. This is eventually done in a later PR.See open-telemetry/opentelemetry-collector-contrib#37499 for additional context.
Note:
OTLPReceiver.ReceiveResourceSpanstakesattributes.HostFromAttributesHandlerfor open-telemetry/opentelemetry-collector-contrib#37499.Motivation
Describe how you validated your changes
Unit tests and e2e tests.
Possible Drawbacks / Trade-offs
Additional Notes