Skip to content

Commit

Permalink
Remove references to deprecated processor names otel_trace_raw and …
Browse files Browse the repository at this point in the history
…`service_map_stateful` (#5017)

Signed-off-by: linghengqian <[email protected]>
  • Loading branch information
linghengqian authored Oct 3, 2024
1 parent aff479b commit 72e2f27
Show file tree
Hide file tree
Showing 14 changed files with 29 additions and 29 deletions.
6 changes: 3 additions & 3 deletions data-prepper-plugins/trace-peer-forwarder-processor/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Trace Peer Forwarder Processor

This processor is used to reduce the number of Events that will be forwarded in a Trace Analytics pipeline by half when using [Peer Forwarder](https://github.com/opensearch-project/data-prepper/blob/main/docs/peer_forwarder.md).
It groups the events based on `trace_id` similar to `service_map_stateful` and `otel_trace_raw ` processors.
It groups the events based on `trace_id` similar to `service_map` and `otel_traces ` processors.

In [Trace Analytics pipeline](https://github.com/opensearch-project/data-prepper/blob/main/docs/trace_analytics.md#trace-analytics-pipeline) each event is duplicated, when it is sent from `otel-trace-pipeline` to `raw-pipeline` and `service-map-pipeline`.
So, the event will be forwarded once in each pipeline. Using this processor event will be forwarded only once in `otel-trace-pipeline` to correct peer.
Expand All @@ -26,7 +26,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
service-map-pipeline:
Expand All @@ -35,7 +35,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
```
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- stdout:
service-map-pipeline:
Expand All @@ -62,7 +62,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- stdout:
```
Expand Down
2 changes: 1 addition & 1 deletion docs/core_apis.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ authentication:
### Peer Forwarder
Peer forwarder can be configured to enable stateful aggregation across multiple Data Prepper nodes. For more information on configuring Peer Forwarder, see [Peer Forwarder Configuration](https://github.com/opensearch-project/data-prepper/blob/main/docs/peer_forwarder.md).
It is supported by `service_map_stateful`, `otel_trace_raw` and `aggregate` processors.
It is supported by `service_map`, `otel_traces` and `aggregate` processors.

### Shutdown Timeouts
When the DataPrepper `shutdown` API is invoked, the sink and processor `ExecutorService`'s are given time to gracefully shutdown and clear any in-flight data. The default graceful shutdown timeout for these `ExecutorService`'s is 10 seconds. This can be configured with the following optional parameters:
Expand Down
4 changes: 2 additions & 2 deletions docs/peer_forwarder.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@

## Peer Forwarder
An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors.
An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map`, `otel_traces` processors.

Peer Forwarder groups events based on the identification keys provided the processors.
For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured.
For `service_map` and `otel_traces` it's `traceId` by default and can not be configured.
It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys).

---
Expand Down
10 changes: 5 additions & 5 deletions docs/trace_analytics.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ The [OpenTelemetry source](../data-prepper-plugins/otel-trace-source/README.md)
### Processor

We have two processor for the Trace Analytics feature,
* *otel_trace_raw* - This is a processor that receives collection of [Span](../data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records sent from [otel-trace-source](../data-prepper-plugins/otel-trace-source/README.md), does stateful processing on extracting and filling-in trace group related fields.
* *otel_traces* - This is a processor that receives collection of [Span](../data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records sent from [otel-trace-source](../data-prepper-plugins/otel-trace-source/README.md), does stateful processing on extracting and filling-in trace group related fields.
* *otel_trace_group* - This is a processor that fills in the missing trace group related fields in the collection of [Span](../data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the opensearch backend.
* *service_map_stateful* - This processor performs the required preprocessing on the trace data and build metadata to display the service-map OpenSearch Dashboards dashboards.
* *service_map* - This processor performs the required preprocessing on the trace data and build metadata to display the service-map OpenSearch Dashboards dashboards.


### OpenSearch sink
Expand Down Expand Up @@ -118,7 +118,7 @@ raw-pipeline:
# The raw processor does bulk request to your OpenSearch sink, so configure the batch_size higher.
batch_size: 3200
processor:
- otel_trace_raw:
- otel_traces:
- otel_trace_group:
hosts: [ "https://localhost:9200" ]
# Change to your credentials
Expand Down Expand Up @@ -152,7 +152,7 @@ service-map-pipeline:
pipeline:
name: "otel-trace-pipeline"
processor:
- service_map_stateful:
- service_map:
# The window duration is the maximum length of time the data prepper stores the most recent trace data to evaluvate service-map relationships.
# The default is 3 minutes, this means we can detect relationships between services from spans reported in last 3 minutes.
# Set higher value if your applications have higher latency.
Expand Down Expand Up @@ -250,7 +250,7 @@ pipeline authors the ability to configure other processors to modify spans or tr
To provide a migration path, Data Prepper 1.4 introduced the following changes.
* The `otel_trace_source` has an optional parameter `record_type` which can be set to `event`. When configured, it will output event objects.
* The `otel_trace_raw` replaces `otel_trace_raw_prepper` for event-based spans.
* The `otel_traces` replaces `otel_trace_raw_prepper` for event-based spans.
* The `otel_trace_group` replaces `otel_trace_group_prepper` for event-based spans.
In Data Prepper 2.0, the `otel_trace_source` will only output Events. Data Prepper 2.0 also removes
Expand Down
4 changes: 2 additions & 2 deletions examples/config/example-pipelines.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://<your-domain>.<region>.es.amazonaws.com" ]
Expand All @@ -26,7 +26,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: [ "https://<your-domain>.<region>.es.amazonaws.com" ]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Resources:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- stdout:
- Name: DATA_PREPPER_CONFIG_YAML
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
# trace_flush_interval: 6
sink:
- stdout:
4 changes: 2 additions & 2 deletions examples/dev/k8s/data-prepper.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ data:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://opensearch:9200" ]
Expand All @@ -40,7 +40,7 @@ data:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://opensearch:9200"]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -30,7 +30,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
4 changes: 2 additions & 2 deletions examples/jaeger-hotrod/pipelines.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -27,7 +27,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
4 changes: 2 additions & 2 deletions examples/trace_analytics.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -29,7 +29,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
4 changes: 2 additions & 2 deletions examples/trace_analytics_no_ssl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -28,7 +28,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down
4 changes: 2 additions & 2 deletions examples/trace_analytics_no_ssl_2x.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ raw-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- otel_trace_raw:
- otel_traces:
sink:
- opensearch:
hosts: [ "https://node-0.example.com:9200" ]
Expand All @@ -27,7 +27,7 @@ service-map-pipeline:
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://node-0.example.com:9200"]
Expand Down

0 comments on commit 72e2f27

Please sign in to comment.