Support otel_logs codec in S3 source (#5028) #5030

danhli · 2024-10-08T05:40:34Z

Description

Add a new codec option, otel_logs, in S3 source. With this option, OTLP JSON files can be ingested and processed natively, similar to ingesting through http endpoint using otel_logs_source.

Issues Resolved

Resolves #5028

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Daniel Li <[email protected]>

dlvenable · 2024-10-08T19:16:30Z

...to-common/src/main/java/org/opensearch/dataprepper/plugins/otel/codec/OTLPJsonLogsCodec.java

+import java.util.function.Consumer;
+
+@DataPrepperPlugin(name = "opentelemetry_logs", pluginType = InputCodec.class)
+public class OTLPJsonLogsCodec extends OTLPJsonLogsDecoder implements InputCodec {


Can you rename this to OTelLogsCodec? What was the reason for Json in the name?

Renamed it to OTelLogsInputCodec.

dlvenable · 2024-10-08T19:20:26Z

...to-common/src/main/java/org/opensearch/dataprepper/plugins/otel/codec/OTLPJsonLogsCodec.java

+import java.io.InputStream;
+import java.util.function.Consumer;
+
+@DataPrepperPlugin(name = "opentelemetry_logs", pluginType = InputCodec.class)


Maybe this should be otel_json_logs or otel_logs_json?

Changed it to otel_logs.

Signed-off-by: Daniel Li <[email protected]>

dlvenable

Thank you @danhli . I have a few comments to help with documentation.

dlvenable · 2024-10-09T19:22:08Z

...on/src/main/java/org/opensearch/dataprepper/plugins/otel/codec/OTelLogsInputCodecConfig.java

+ * Configuration class for {@link OTelLogsInputCodec}.
+ */
+public class OTelLogsInputCodecConfig {
+    static final String JSON_FORMAT = "json"; 


This should be an enum. It will help with our auto-generated schema and documentation.

I've submitted a few PRs to correct these, such as #5025 and #5035.

Please see these:

https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/key-value-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/keyvalue/WhitespaceOption.java

https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/key-value-processor/src/test/java/org/opensearch/dataprepper/plugins/processor/keyvalue/WhitespaceOptionTest.java

This is where it is used:

data-prepper/data-prepper-plugins/key-value-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/keyvalue/KeyValueProcessorConfig.java

Lines 129 to 133 in 6e51f51

@JsonProperty("whitespace")

@JsonPropertyDescription("Specifies whether to be lenient or strict with the acceptance of " +

"unnecessary white space surrounding the configured value-split sequence. " +

"In this case, strict means that whitespace is trimmed and lenient means it is retained in the key name and in the value." +

"Default is <code>lenient</code>.")

Added OTelLogsFormatOption.

dlvenable · 2024-10-09T19:22:58Z

...on/src/main/java/org/opensearch/dataprepper/plugins/otel/codec/OTelLogsInputCodecConfig.java

+/**
+ * Configuration class for {@link OTelLogsInputCodec}.
+ */
+public class OTelLogsInputCodecConfig {


Please add documentation:

See an example:

data-prepper/data-prepper-plugins/flatten-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/flatten/FlattenProcessorConfig.java

Lines 19 to 20 in 6e51f51

@JsonPropertyOrder

@JsonClassDescription("The <code>flatten</code> processor transforms nested objects inside of events into flattened structures.")

dlvenable · 2024-10-09T19:24:07Z

...on/src/main/java/org/opensearch/dataprepper/plugins/otel/codec/OTelLogsInputCodecConfig.java

+    static final String JSON_FORMAT = "json"; 
+
+    @JsonProperty("format")
+    private String format = JSON_FORMAT;


Please add documentation:

Examples:

data-prepper/data-prepper-plugins/key-value-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/keyvalue/KeyValueProcessorConfig.java

Lines 129 to 135 in 6e51f51

@JsonProperty("whitespace")

@JsonPropertyDescription("Specifies whether to be lenient or strict with the acceptance of " +

"unnecessary white space surrounding the configured value-split sequence. " +

"In this case, strict means that whitespace is trimmed and lenient means it is retained in the key name and in the value." +

"Default is <code>lenient</code>.")

@NotNull

private WhitespaceOption whitespace = DEFAULT_WHITESPACE;

…t#5028) Signed-off-by: Daniel Li <[email protected]>

danhli · 2024-10-11T16:55:52Z

@dlvenable I have completed the documentation and enum changes. Please review.

dlvenable

Thank you @danhli for this great contribution!

danhli added 2 commits October 7, 2024 22:03

Support otlp_json_logs codec in S3 source (opensearch-project#5028)

f78f11c

Signed-off-by: Daniel Li <[email protected]>

Add copyright info (opensearch-project#5028)

a67c31a

Signed-off-by: Daniel Li <[email protected]>

danhli requested review from chenqi0805, engechas, graytaylor0, dinujoh, kkondaka, KarstenSchnitter, dlvenable and oeyh as code owners October 8, 2024 05:40

Update class loader (opensearch-project#5028)

e3c5b91

Signed-off-by: Daniel Li <[email protected]>

danhli changed the title ~~Support otlp_json_logs codec in S3 source (#5028)~~ Support opentelemetry_logs codec in S3 source (#5028) Oct 8, 2024

Change the codec's name to opentelemetry_logs (opensearch-project#5028)

ac291d6

Signed-off-by: Daniel Li <[email protected]>

dlvenable requested changes Oct 8, 2024

View reviewed changes

Change the codec's name to otel_logs (opensearch-project#5028)

ae110fd

Signed-off-by: Daniel Li <[email protected]>

danhli changed the title ~~Support opentelemetry_logs codec in S3 source (#5028)~~ Support otel_logs codec in S3 source (#5028) Oct 9, 2024

kkondaka previously approved these changes Oct 9, 2024

View reviewed changes

dlvenable requested changes Oct 9, 2024

View reviewed changes

Add documentation and otel_logs format option enum (opensearch-projec…

dfeeb97

…t#5028) Signed-off-by: Daniel Li <[email protected]>

danhli dismissed kkondaka’s stale review via dfeeb97 October 11, 2024 16:47

dlvenable approved these changes Oct 11, 2024

View reviewed changes

kkondaka approved these changes Oct 11, 2024

View reviewed changes

dlvenable merged commit 8dc66d2 into opensearch-project:main Oct 11, 2024
46 of 47 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support otel_logs codec in S3 source (#5028) #5030

Support otel_logs codec in S3 source (#5028) #5030

danhli commented Oct 8, 2024 •

edited

Loading

dlvenable Oct 8, 2024

danhli Oct 9, 2024 •

edited

Loading

dlvenable Oct 8, 2024

danhli Oct 9, 2024 •

edited

Loading

dlvenable left a comment

dlvenable Oct 9, 2024

danhli Oct 11, 2024

dlvenable Oct 9, 2024

danhli Oct 11, 2024

dlvenable Oct 9, 2024

danhli Oct 11, 2024

danhli commented Oct 11, 2024

dlvenable left a comment

	@JsonProperty("whitespace")
	@JsonPropertyDescription("Specifies whether to be lenient or strict with the acceptance of " +
	"unnecessary white space surrounding the configured value-split sequence. " +
	"In this case, strict means that whitespace is trimmed and lenient means it is retained in the key name and in the value." +
	"Default is <code>lenient</code>.")

	@JsonPropertyOrder
	@JsonClassDescription("The <code>flatten</code> processor transforms nested objects inside of events into flattened structures.")

Support otel_logs codec in S3 source (#5028) #5030

Support otel_logs codec in S3 source (#5028) #5030

Conversation

danhli commented Oct 8, 2024 • edited Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

danhli Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danhli Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

dlvenable left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danhli commented Oct 11, 2024

dlvenable left a comment

Choose a reason for hiding this comment

danhli commented Oct 8, 2024 •

edited

Loading

danhli Oct 9, 2024 •

edited

Loading

danhli Oct 9, 2024 •

edited

Loading