diff --git a/docs/restate/tracing.md b/docs/restate/tracing.md index 8544cb19..f067dfe6 100644 --- a/docs/restate/tracing.md +++ b/docs/restate/tracing.md @@ -11,13 +11,13 @@ Restate supports the following tracing features: * Exporting traces to OTLP-compatible systems (e.g. Jaeger) * Correlating parent traces of incoming gRPC/Connect HTTP requests, using the [W3C TraceContext](https://github.com/w3c/trace-context) specification. -## Setup OTLP exporter +## Setting up OTLP exporter -To set up the OTLP exporter, you need to set the configuration entry `observability.tracing.endpoint` to point to your trace collector. -The exporter sends OTLP trace data via gRPC ([OTLP exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md)). +Set up the [OTLP exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md) by pointing the configuration entry `observability.tracing.endpoint` to your trace collector. +The exporter sends OTLP trace data via gRPC. -### Exporting OTLP traces to Jaeger +## Exporting OTLP traces to Jaeger Jaeger accepts OTLP trace data via gRPC on port `4317`. @@ -33,24 +33,68 @@ Configure the tracing endpoint in Restate as a fully specified URL: `http:// + + +```shell +docker run --name restate_dev --rm -d -e RESTATE_OBSERVABILITY__TRACING__ENDPOINT=http://localhost:4317 --network=host ghcr.io/restatedev/restate-dist:VAR::RESTATE_DIST_VERSION +``` + + + + +```shell +docker run --name restate_dev --rm -d -e RESTATE_OBSERVABILITY__TRACING__ENDPOINT=http://host.docker.internal:4317 -p 8081:8081 -p 9091:9091 -p 9090:9090 ghcr.io/restatedev/restate-dist:VAR::RESTATE_DIST_VERSION +``` + + + + +Go to the Jaeger UI at http://localhost:16686. + +If you now spin up your Restate services and send requests to them, you will see the traces appear in the Jaeger UI. + +An example from the [shopping cart example](https://github.com/restatedev/example-shopping-cart-typescript): +![Observability](/img/observability.jpeg) + +## Setting up Jaeger file exporter + +If you can't configure a Jaeger agent, you can export traces by writing them to files, using the Jaeger JSON format. To do so, set up the configuration entry `observability.tracing.json_file_export_path` pointing towards the path where trace files should be written. You can import the trace files using the Jaeger UI: ![Jaeger UI File import](/img/jaeger-import-file.png) + ## Understanding traces +The traces contain detailed information about the context calls that were done during the invocation (e.g. sleep, one-way calls, interaction with state): + +![Understanding traces](/img/understanding_traces.png) -Similarly to logs, traces export [attributes/tags](#components-and-log-event-context-fields) that correlate the trace with the service and/or invocation. For example, you can filter directly in the Jaeger UI all the traces belonging to the service `org.example.ExampleService` by setting the tag filter `rpc.service=org.example.ExampleService`. +The initial `ingress_service_invocation` spans show when the gRPC/Connect HTTP request was received by Restate. The `invoke` span beneath it shows when Restate invoked the service endpoint to process the request. -Restate traces don't look like traditional HTTP server traces, because of the inner workings of Restate and OpenTelemetry/Jaeger. For each invocation, a span named `service_invocation` is created to mark the beginning of the invocation, and a child span `end_invocation` is created to mark the end of an invocation. You can easily check for every invocation if it ended or not by checking whether the span `service_invocation` has the `end_invocation` child or not. +The tags of the spans contain the metadata of the context calls (e.g. call arguments, invocation id). -The spans `ingress_service_invocation` informs when the gRPC/Connect HTTP request is received, and `invoker_invocation_task` informs when the runtime invokes the service endpoint to process the request. +When a service invokes another service, the child invocation is linked automatically to the parent invocation, as you can see in the image. +Note that the spans of one-way calls are shown as separate traces. The parent invocation only shows that the one-way call was scheduled, not its entire tracing span. +To see this information, search for the trace of the one-way call by filtering on the invocation id tag: +``` +restate.invocation.sid="example.MyExampleService-AzEyMw==-0189b536906b746c8da6f83f0257acda" +``` -![Jaeger trace](/img/jaeger-trace.png) +## Searching traces -When a service invokes another service, the child invocation will be automatically linked to the span `service_invocation` of the parent invocation. +Traces export attributes and tags that correlate the trace with the service and/or invocation. For example, in the Jaeger UI, you can filter the traces belonging to the service `org.example.ExampleService` by setting the tag filter `rpc.service=org.example.ExampleService`. Or you can filter on the method (`rpc.method`) or invocation id (`restate.invocation.sid`) or any other tag: -We recommend to set up either Jaeger or Jaeger File trace exporter with filter `info,restate_worker::partition::effects=debug`. With this filter, the traces will also contain all the steps executed by the Restate internal state machine to drive the invocation to completion. +![Jaeger method search](/img/jaeger_search.png) \ No newline at end of file diff --git a/docs/tour-of-restate.mdx b/docs/tour-of-restate.mdx index d715dabb..8e8c1de9 100644 --- a/docs/tour-of-restate.mdx +++ b/docs/tour-of-restate.mdx @@ -1262,7 +1262,7 @@ Payment call succeeded for idempotency key dc2048f8-62c1-4b3e-bb92-25dc9da14bac ## Observability -Restate exposes OpenTelemetry traces of your invocations +Restate exposes OpenTelemetry traces of your invocations. Run the Jaeger container with: ```shell diff --git a/static/img/jaeger-trace.png b/static/img/jaeger-trace.png deleted file mode 100644 index b214d812..00000000 Binary files a/static/img/jaeger-trace.png and /dev/null differ diff --git a/static/img/jaeger_search.png b/static/img/jaeger_search.png new file mode 100644 index 00000000..b59bf64f Binary files /dev/null and b/static/img/jaeger_search.png differ diff --git a/static/img/observability.jpeg b/static/img/observability.jpeg new file mode 100644 index 00000000..2f61f253 Binary files /dev/null and b/static/img/observability.jpeg differ diff --git a/static/img/understanding_traces.png b/static/img/understanding_traces.png new file mode 100644 index 00000000..a8df6a0b Binary files /dev/null and b/static/img/understanding_traces.png differ