Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(operator)!: Add configuration option for dropping OTLP attributes #15857

Merged
merged 19 commits into from
Jan 28, 2025
Merged

feat(operator)!: Add configuration option for dropping OTLP attributes #15857

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f043e37
feat(operator): Add configuration option for dropping OTLP attributes
xperimental Jan 20, 2025
c62b991
Add support in implementation
xperimental Jan 20, 2025
708f32d
Refactor OTLP validation
xperimental Jan 20, 2025
4c7741b
Add validation for global drop configuration
xperimental Jan 20, 2025
6e1fd39
Extend validation to support global configuration inheritance
xperimental Jan 20, 2025
a1b18d4
Fix tenant validation when it does not contain own attribute list
xperimental Jan 20, 2025
4861ff1
Add validation for dropping OpenShift required OTLP attributes
xperimental Jan 20, 2025
61a7d98
Update API docs
xperimental Jan 21, 2025
69fc46f
Fix generated API code
xperimental Jan 21, 2025
6fe2420
Ignore lint errors about deprecated API attribute
xperimental Jan 21, 2025
d0f4117
Remove StructuredMetadata from OTLP configuration options
xperimental Jan 21, 2025
a424841
Remove StructuredMetadata from implementation and tests
xperimental Jan 21, 2025
4784d9d
Remove structured_metadata attributes from defaults
xperimental Jan 21, 2025
07618ae
Update text for validation error
xperimental Jan 23, 2025
ededa50
Move list of OTLP attributes to common package
xperimental Jan 23, 2025
54029d5
Refactor DefaultOTLPAttributes
xperimental Jan 23, 2025
902a0bb
Update docs for DisableRecommendedAttributes
xperimental Jan 28, 2025
b9c8a33
Add paragraph about attribute names to documentation
xperimental Jan 28, 2025
a217b10
Merge remote-tracking branch 'upstream/main' into otlp-support-drop
xperimental Jan 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions operator/api/loki/v1/lokistack_types.go
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DisableRecommendedAttributes docs have to be updated... and I wonder if it's useful at all 🤔 mainly because we now we know that attributes are attached to structured metadata by default. On the other hand, IIRC stream labels are in added to the index, since some of the labels we have in the recommendedAttributes have high cardinality maybe it makes sense for this toggle to exist. WDYT?

Original file line number Diff line number Diff line change
Expand Up @@ -837,12 +837,12 @@ type OTLPSpec struct {
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Stream Labels"
StreamLabels *OTLPStreamLabelSpec `json:"streamLabels,omitempty"`

// StructuredMetadata configures which attributes are saved in structured metadata.
// Drop configures which attributes are dropped from the log entry.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention here the detail on how the attributes should be in their OTEL formatted name and not Loki?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wondered about how to do that. Let me try adding some text to the OTLPSpec, because this is the same for both StreamLabels and Drop.

//
// +optional
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Structured Metadata"
StructuredMetadata *OTLPMetadataSpec `json:"structuredMetadata,omitempty"`
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Dropped Attributes"
Drop *OTLPMetadataSpec `json:"drop,omitempty"`
}

type OTLPStreamLabelSpec struct {
Expand Down
2 changes: 2 additions & 0 deletions operator/api/loki/v1/v1.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ var (
ErrOTLPTenantMissing = errors.New("if no global OTLP configuration is present which defines at least one stream label, every tenant must have an OTLP configuration")
// ErrOTLPTenantNoStreamLabel when a tenant is defined but has no stream labels and there also no global stream labels.
ErrOTLPTenantNoStreamLabel = errors.New("if no global OTLP configuration is present which defines at least one stream label, every tenant must define at least one stream label")
// ErrOTLPInvalidDrop when a OTLP configuration tries to drop an attribute also listed as a stream label
ErrOTLPInvalidDrop = errors.New("can not drop OTLP attribute also listed as a stream label")

// ErrRuleMustMatchNamespace indicates that an expression used in an alerting or recording rule is missing
// matchers for a namespace.
Expand Down
4 changes: 2 additions & 2 deletions operator/api/loki/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

116 changes: 57 additions & 59 deletions operator/bundle/community-openshift/manifests/loki-operator.clusterserviceversion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing
certified: "false"
containerImage: docker.io/grafana/loki-operator:0.7.1
createdAt: "2025-01-16T18:30:58Z"
createdAt: "2025-01-21T12:58:10Z"
description: The Community Loki Operator provides Kubernetes native deployment
and management of Loki and related logging components.
features.operators.openshift.io/disconnected: "true"
Expand Down Expand Up @@ -373,62 +373,61 @@ spec:
path: limits.global.ingestion.perStreamRateLimitBurst
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:number
- description: StreamLabels configures which resource attributes are converted
to Loki stream labels.
displayName: Stream Labels
path: limits.global.otlp.streamLabels
- description: ResourceAttributes lists the names of the resource attributes
that should be converted into Loki stream labels.
displayName: Resource Attributes
path: limits.global.otlp.streamLabels.resourceAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.global.otlp.streamLabels.resourceAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.global.otlp.streamLabels.resourceAttributes[0].regex
- description: StructuredMetadata configures which attributes are saved in structured
metadata.
displayName: Structured Metadata
path: limits.global.otlp.structuredMetadata
- description: Drop configures which attributes are dropped from the log entry.
displayName: Dropped Attributes
path: limits.global.otlp.drop
- description: LogAttributes lists the names of log attributes that should be
included in structured metadata.
displayName: Log Attributes
path: limits.global.otlp.structuredMetadata.logAttributes
path: limits.global.otlp.drop.logAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.global.otlp.structuredMetadata.logAttributes[0].name
path: limits.global.otlp.drop.logAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.global.otlp.structuredMetadata.logAttributes[0].regex
path: limits.global.otlp.drop.logAttributes[0].regex
- description: ResourceAttributes lists the names of resource attributes that
should be included in structured metadata.
displayName: Resource Attributes
path: limits.global.otlp.structuredMetadata.resourceAttributes
path: limits.global.otlp.drop.resourceAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.global.otlp.structuredMetadata.resourceAttributes[0].name
path: limits.global.otlp.drop.resourceAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.global.otlp.structuredMetadata.resourceAttributes[0].regex
path: limits.global.otlp.drop.resourceAttributes[0].regex
- description: ScopeAttributes lists the names of scope attributes that should
be included in structured metadata.
displayName: Scope Attributes
path: limits.global.otlp.structuredMetadata.scopeAttributes
path: limits.global.otlp.drop.scopeAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.global.otlp.structuredMetadata.scopeAttributes[0].name
path: limits.global.otlp.drop.scopeAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.global.otlp.structuredMetadata.scopeAttributes[0].regex
path: limits.global.otlp.drop.scopeAttributes[0].regex
- description: StreamLabels configures which resource attributes are converted
to Loki stream labels.
displayName: Stream Labels
path: limits.global.otlp.streamLabels
- description: ResourceAttributes lists the names of the resource attributes
that should be converted into Loki stream labels.
displayName: Resource Attributes
path: limits.global.otlp.streamLabels.resourceAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.global.otlp.streamLabels.resourceAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.global.otlp.streamLabels.resourceAttributes[0].regex
- description: CardinalityLimit defines the cardinality limit for index queries.
displayName: Cardinality Limit
path: limits.global.queries.cardinalityLimit
Expand Down Expand Up @@ -534,62 +533,61 @@ spec:
path: limits.tenants.ingestion.perStreamRateLimitBurst
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:number
- description: StreamLabels configures which resource attributes are converted
to Loki stream labels.
displayName: Stream Labels
path: limits.tenants.otlp.streamLabels
- description: ResourceAttributes lists the names of the resource attributes
that should be converted into Loki stream labels.
displayName: Resource Attributes
path: limits.tenants.otlp.streamLabels.resourceAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.tenants.otlp.streamLabels.resourceAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.tenants.otlp.streamLabels.resourceAttributes[0].regex
- description: StructuredMetadata configures which attributes are saved in structured
metadata.
displayName: Structured Metadata
path: limits.tenants.otlp.structuredMetadata
- description: Drop configures which attributes are dropped from the log entry.
displayName: Dropped Attributes
path: limits.tenants.otlp.drop
- description: LogAttributes lists the names of log attributes that should be
included in structured metadata.
displayName: Log Attributes
path: limits.tenants.otlp.structuredMetadata.logAttributes
path: limits.tenants.otlp.drop.logAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.tenants.otlp.structuredMetadata.logAttributes[0].name
path: limits.tenants.otlp.drop.logAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.tenants.otlp.structuredMetadata.logAttributes[0].regex
path: limits.tenants.otlp.drop.logAttributes[0].regex
- description: ResourceAttributes lists the names of resource attributes that
should be included in structured metadata.
displayName: Resource Attributes
path: limits.tenants.otlp.structuredMetadata.resourceAttributes
path: limits.tenants.otlp.drop.resourceAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.tenants.otlp.structuredMetadata.resourceAttributes[0].name
path: limits.tenants.otlp.drop.resourceAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.tenants.otlp.structuredMetadata.resourceAttributes[0].regex
path: limits.tenants.otlp.drop.resourceAttributes[0].regex
- description: ScopeAttributes lists the names of scope attributes that should
be included in structured metadata.
displayName: Scope Attributes
path: limits.tenants.otlp.structuredMetadata.scopeAttributes
path: limits.tenants.otlp.drop.scopeAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.tenants.otlp.structuredMetadata.scopeAttributes[0].name
path: limits.tenants.otlp.drop.scopeAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.tenants.otlp.structuredMetadata.scopeAttributes[0].regex
path: limits.tenants.otlp.drop.scopeAttributes[0].regex
- description: StreamLabels configures which resource attributes are converted
to Loki stream labels.
displayName: Stream Labels
path: limits.tenants.otlp.streamLabels
- description: ResourceAttributes lists the names of the resource attributes
that should be converted into Loki stream labels.
displayName: Resource Attributes
path: limits.tenants.otlp.streamLabels.resourceAttributes
- description: Name contains either a verbatim name of an attribute or a regular
expression matching many attributes.
displayName: Name
path: limits.tenants.otlp.streamLabels.resourceAttributes[0].name
- description: If Regex is true, then Name is treated as a regular expression
instead of as a verbatim attribute name.
displayName: Treat name as regular expression
path: limits.tenants.otlp.streamLabels.resourceAttributes[0].regex
- description: Blocked defines the list of rules to block matching queries.
displayName: Blocked
path: limits.tenants.queries.blocked
Expand Down
Loading
Loading