Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(compression): Make the compression type configurable, default: gzip #3705

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat(compression): Make the compression type configurable, default: gzip #3705

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
661245d
feat(compression): Make the compression type configurable, default: zstd
rnishtala-sumo May 14, 2024
99b58e2
Regenerating golden files
rnishtala-sumo May 14, 2024
63a0454
Move the compression config to the top-level, change default to gzip
rnishtala-sumo May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .changelog/3705.added.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
feat(compression): Make the compression type configurable, default: zstd
1 change: 1 addition & 0 deletions deploy/helm/sumologic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,6 +432,7 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `otellogswindows.metrics.enabled` | Enable OpenTelemetry Logs Collector for Windows Nodes metrics | `true` |
| `otellogswindows.serviceLabels` | Add custom labels to OpenTelemetry Logs Collector for Windows Nodes Service | `{}` |
| `otellogswindows.additionalDaemonSets` | OpenTelemetry Logs Collector for Windows Nodes Daemonset per node customization options. See [Best Practices](https://help.sumologic.com/docs/send-data/kubernetes/best-practices/#setting-different-resources-on-different-nodes-for-logs-collector). | `{}` |
| `metadata.compression` | Compression for logs, metrics and events | `zstd` |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about putting this at the top-level, just compression? Right now, it affects the log collector and the event collector, so not just metadata. But it doesn't affect the metrics collector.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, that's a good point, will make this change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved compression to the top level. Also added the config option to the otlphttp exporter of the metrics collector.

| `metadata.image.repository` | Image repository for otelcol docker container. | `` |
| `metadata.image.tag` | Image tag for otelcol docker container. | `` |
| `metadata.image.pullPolicy` | Image pullPolicy for otelcol docker container. | `IfNotPresent` |
Expand Down
2 changes: 2 additions & 0 deletions deploy/helm/sumologic/conf/events/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ exporters:
sumologic:
client: {{ include "sumologic.sumo_client" . }}
endpoint: {{ include "sumologic.events.exporter.endpoint" . }}
compression: {{ .Values.metadata.compression }}
log_format: {{ include "sumologic.events.exporter.format" . }}
sending_queue:
enabled: true
Expand All @@ -18,6 +19,7 @@ exporters:
sumologic/sumologic-mock:
client: {{ include "sumologic.sumo_client" . }}
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.metadata.compression }}
log_format: {{ include "sumologic.events.exporter.format" . }}
sending_queue:
enabled: true
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/sumologic/conf/logs/collector/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
exporters:
otlphttp:
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.{{ .Values.sumologic.clusterDNSDomain }}.:4318
compression: {{ .Values.metadata.compression }}
sending_queue:
queue_size: 10
# this improves load balancing at the cost of more network traffic
Expand Down
6 changes: 6 additions & 0 deletions deploy/helm/sumologic/conf/logs/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ exporters:
{{ if .Values.sumologic.logs.container.enabled }}
sumologic/containers:
endpoint: ${SUMO_ENDPOINT_DEFAULT_LOGS_SOURCE}
compression: {{ .Values.metadata.compression }}
log_format: {{ include "logs.otelcol.container.exporter.format" . }}
sending_queue:
enabled: true
Expand All @@ -20,6 +21,7 @@ exporters:
{{- if eq (include "sumologic-mock.forward-logs-metadata" .) "true" }}
sumologic/sumologic-mock-containers:
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.metadata.compression }}
log_format: {{ include "logs.otelcol.container.exporter.format" . }}
sending_queue:
enabled: true
Expand All @@ -35,6 +37,7 @@ exporters:
{{ if .Values.sumologic.logs.systemd.enabled }}
sumologic/systemd:
endpoint: ${SUMO_ENDPOINT_DEFAULT_LOGS_SOURCE}
compression: {{ .Values.metadata.compression }}
log_format: json
sending_queue:
enabled: true
Expand All @@ -47,6 +50,7 @@ exporters:
{{- if eq (include "sumologic-mock.forward-logs-metadata" .) "true" }}
sumologic/sumologic-mock-systemd:
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.metadata.compression }}
log_format: json
sending_queue:
enabled: true
Expand All @@ -63,6 +67,7 @@ exporters:
sumologic:
client: {{ include "sumologic.sumo_client" . }}
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
compression: {{ .Values.metadata.compression }}
log_format: otlp
sending_queue:
enabled: true
Expand All @@ -76,6 +81,7 @@ exporters:
sumologic/sumologic-mock:
client: {{ include "sumologic.sumo_client" . }}
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.metadata.compression }}
log_format: otlp
sending_queue:
enabled: true
Expand Down
10 changes: 10 additions & 0 deletions deploy/helm/sumologic/conf/metrics/otelcol/exporters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ sumologic/sumologic-mock-default:
client: {{ include "sumologic.sumo_client" . }}
metric_format: {{ include "metrics.otelcol.exporter.format" . }}
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.metadata.compression }}
## Sumo doesn't yet natively support OTLP Histograms
decompose_otlp_histograms: true
## Configuration for sending queue
Expand All @@ -29,6 +30,7 @@ sumologic/sumologic-mock-http:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: {{ include "sumologic-mock.receiver-endpoint" . }}
compression: {{ .Values.metadata.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -47,6 +49,7 @@ sumologic/default:
client: {{ include "sumologic.sumo_client" . }}
metric_format: {{ include "metrics.otelcol.exporter.format" . }}
endpoint: {{ include "metrics.otelcol.exporter.endpoint" . }}
compression: {{ .Values.metadata.compression }}
## Sumo doesn't yet natively support OTLP Histograms
decompose_otlp_histograms: true
## Configuration for sending queue
Expand All @@ -68,6 +71,7 @@ sumologic/apiserver:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_APISERVER_METRICS_SOURCE}
compression: {{ .Values.metadata.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -82,6 +86,7 @@ sumologic/control_plane:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_CONTROL_PLANE_METRICS_SOURCE}
compression: {{ .Values.metadata.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -96,6 +101,7 @@ sumologic/controller:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_CONTROLLER_METRICS_SOURCE}
compression: {{ .Values.metadata.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -110,6 +116,7 @@ sumologic/kubelet:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_KUBELET_METRICS_SOURCE}
compression: {{ .Values.metadata.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -124,6 +131,7 @@ sumologic/node:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_NODE_METRICS_SOURCE}
compression: {{ .Values.metadata.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -138,6 +146,7 @@ sumologic/scheduler:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_SCHEDULER_METRICS_SOURCE}
compression: {{ .Values.metadata.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand All @@ -152,6 +161,7 @@ sumologic/state:
client: {{ include "sumologic.sumo_client" . }}
metric_format: prometheus
endpoint: ${SUMO_ENDPOINT_STATE_METRICS_SOURCE}
compression: {{ .Values.metadata.compression }}
sending_queue:
enabled: true
{{- if .Values.metadata.persistence.enabled }}
Expand Down
3 changes: 3 additions & 0 deletions deploy/helm/sumologic/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1556,6 +1556,9 @@ metadata:
## Add custom labels to all otelcol statefulset PVC (logs and metrics)
pvcLabels: {}

## Compression for logs, metrics and events:
compression: zstd

## Configure metrics pipeline.
## This section affects only otelcol provider.
metrics:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/events_otc/basic.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_EVENTS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/events_otc/options.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_EVENTS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
2 changes: 2 additions & 0 deletions tests/helm/testdata/goldenfile/events_otc/sumologic-mock.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,15 @@ data:
verbosity: detailed
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_EVENTS_SOURCE}
log_format: otlp
sending_queue:
enabled: true
storage: file_storage
sumologic/sumologic-mock:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: http://RELEASE-NAME-sumologic-mock.sumologic.svc.cluster.local.:3000/receiver
log_format: otlp
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/logs_otc/basic.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ data:
config.yaml: |
exporters:
otlphttp:
compression: zstd
disable_keep_alives: true
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/logs_otc/debug.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ data:
debug:
verbosity: detailed
otlphttp:
compression: zstd
disable_keep_alives: true
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/logs_otc/options.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ data:
config.yaml: |
exporters:
otlphttp:
compression: zstd
disable_keep_alives: true
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/logs_otc_daemonset/multiple_multiline.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ data:
config.yaml: |
exporters:
otlphttp:
compression: zstd
disable_keep_alives: true
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/debug.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ data:
verbosity: detailed
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
2 changes: 2 additions & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/debug_with_sumologic_mock.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ data:
verbosity: detailed
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
log_format: otlp
sending_queue:
Expand All @@ -26,6 +27,7 @@ data:
storage: file_storage
sumologic/sumologic-mock:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: http://RELEASE-NAME-sumologic-mock.sumologic.svc.cluster.local.:3000/receiver
log_format: otlp
sending_queue:
Expand Down
4 changes: 4 additions & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/debug_with_sumologic_mock_http.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ data:
debug:
verbosity: detailed
sumologic/containers:
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_LOGS_SOURCE}
log_format: json
sending_queue:
Expand All @@ -24,6 +25,7 @@ data:
queue_size: 10000
storage: file_storage
sumologic/sumologic-mock-containers:
compression: zstd
endpoint: http://RELEASE-NAME-sumologic-mock.sumologic.svc.cluster.local.:3000/receiver
log_format: json
sending_queue:
Expand All @@ -32,6 +34,7 @@ data:
queue_size: 10000
storage: file_storage
sumologic/sumologic-mock-systemd:
compression: zstd
endpoint: http://RELEASE-NAME-sumologic-mock.sumologic.svc.cluster.local.:3000/receiver
log_format: json
sending_queue:
Expand All @@ -40,6 +43,7 @@ data:
queue_size: 10000
storage: file_storage
sumologic/systemd:
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_LOGS_SOURCE}
log_format: json
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/otel.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_logs_otc/templates.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_LOGS_SOURCE}
log_format: otlp
sending_queue:
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_metrics_otc/additional_endpoints.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic/default:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
decompose_otlp_histograms: true
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_METRICS_SOURCE}
max_request_body_size: 16777216
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_metrics_otc/basic.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic/default:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
decompose_otlp_histograms: true
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_METRICS_SOURCE}
max_request_body_size: 16777216
Expand Down
8 changes: 8 additions & 0 deletions tests/helm/testdata/goldenfile/metadata_metrics_otc/custom.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ data:
exporters:
sumologic/apiserver:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_APISERVER_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -26,6 +27,7 @@ data:
timeout: 30s
sumologic/control_plane:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_CONTROL_PLANE_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -37,6 +39,7 @@ data:
timeout: 30s
sumologic/controller:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_CONTROLLER_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -48,6 +51,7 @@ data:
timeout: 30s
sumologic/default:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
decompose_otlp_histograms: true
endpoint: ${SUMO_ENDPOINT_DEFAULT_METRICS_SOURCE}
max_request_body_size: 16777216
Expand All @@ -60,6 +64,7 @@ data:
timeout: 30s
sumologic/kubelet:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_KUBELET_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -71,6 +76,7 @@ data:
timeout: 30s
sumologic/node:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_NODE_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -82,6 +88,7 @@ data:
timeout: 30s
sumologic/scheduler:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_SCHEDULER_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand All @@ -93,6 +100,7 @@ data:
timeout: 30s
sumologic/state:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
endpoint: ${SUMO_ENDPOINT_STATE_METRICS_SOURCE}
max_request_body_size: 16777216
metric_format: prometheus
Expand Down
1 change: 1 addition & 0 deletions tests/helm/testdata/goldenfile/metadata_metrics_otc/debug.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ data:
verbosity: detailed
sumologic/default:
client: k8s_%CURRENT_CHART_VERSION%
compression: zstd
decompose_otlp_histograms: true
endpoint: ${SUMO_ENDPOINT_DEFAULT_OTLP_METRICS_SOURCE}
max_request_body_size: 16777216
Expand Down
Loading