Support instrumentation transition from Span Events to Log Events

[codefromthecrypt]

What happened?

I'm using Jaeger 2.0 and not sure if this is a config bug or something unimplemented, or a bug. In any case, I try to export log events (how genai messages are captured now) and get this:

 Failed to export logs to localhost:4317, error code: StatusCode.UNIMPLEMENTED

Note: span events are no longer possible in genai, as of 1.28.0 it is only log events

Steps to reproduce

openai~=1.54.3

# 1.28.1 is required for Log Events API/SDK
opentelemetry-sdk~=1.28.1
opentelemetry-exporter-otlp-proto-grpc~=1.28.1
opentelemetry-distro~=0.49b1
opentelemetry-instrumentation-openai-v2~=2.0b0

# NOTE: OpenTelemetry Python Log Events APIs is in beta
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._events import EventLoggerProvider
from opentelemetry.trace import set_tracer_provider
from opentelemetry._logs import set_logger_provider
from opentelemetry._events import set_event_logger_provider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk._logs.export import SimpleLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

from openai import OpenAI
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter()))
set_tracer_provider(provider)

provider = LoggerProvider()
provider.add_log_record_processor(SimpleLogRecordProcessor(OTLPLogExporter()))
set_logger_provider(provider)

event_provider = EventLoggerProvider(provider)
set_event_logger_provider(event_provider)

OpenAIInstrumentor().instrument()

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a short poem on OpenTelemetry."}],
)

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true OPENAI_API_KEY=sk-not-tellin python main.py
Overriding of current EventLoggerProvider is not allowed
Failed to export logs to localhost:4317, error code: StatusCode.UNIMPLEMENTED
Failed to export logs to localhost:4317, error code: StatusCode.UNIMPLEMENTED

Expected behavior

I would expect the logs to be exported without an error

Relevant log output

otel-collector-1  | 2024/11/14 07:58:41 application version: git-commit=3f752c2f434fabde20694816c264cfe8b546c575, git-version=v2.0.0, build-date=2024-11-10T18:53:12Z
otel-collector-1  | 2024/11/14 07:58:41 No '--config' flags detected, using default All-in-One configuration with memory storage.
otel-collector-1  | 2024/11/14 07:58:41 To customize All-in-One behavior, pass a proper configuration.
otel-collector-1  | 2024-11-14T07:58:41.897Z	info	[email protected]/service.go:135	Setting up own telemetry...
otel-collector-1  | 2024-11-14T07:58:41.897Z	warn	[email protected]/service.go:190	service::telemetry::metrics::address is being deprecated in favor of service::telemetry::metrics::readers
otel-collector-1  | 2024-11-14T07:58:41.897Z	info	telemetry/metrics.go:70	Serving metrics	{"address": "0.0.0.0:8888", "metrics level": "Detailed"}
otel-collector-1  | 2024-11-14T07:58:41.897Z	info	builders/builders.go:26	Development component. May change in the future.	{"kind": "exporter", "data_type": "traces", "name": "jaeger_storage_exporter"}
otel-collector-1  | 2024-11-14T07:58:41.897Z	warn	[email protected]/factory.go:49	jaeger receiver will deprecate Thrift-gen and replace it with Proto-gen to be compatbible to jaeger 1.42.0 and higher. See https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/18485 for more details.	{"kind": "receiver", "name": "jaeger", "data_type": "traces"}
otel-collector-1  | 2024-11-14T07:58:41.897Z	info	builders/extension.go:50	Development component. May change in the future.	{"kind": "extension", "name": "healthcheckv2"}
otel-collector-1  | 2024-11-14T07:58:41.898Z	info	[email protected]/service.go:207	Starting jaeger...	{"Version": "git-commit=3f752c2f434fabde20694816c264cfe8b546c575, git-version=v2.0.0, build-date=2024-11-10T18:53:12Z", "NumCPU": 4}
otel-collector-1  | 2024-11-14T07:58:41.898Z	info	extensions/extensions.go:39	Starting extensions...
otel-collector-1  | 2024-11-14T07:58:41.898Z	info	extensions/extensions.go:42	Extension is starting...	{"kind": "extension", "name": "expvar"}
otel-collector-1  | 2024-11-14T07:58:41.898Z	info	expvar/extension.go:45	Starting expvar server	{"kind": "extension", "name": "expvar", "addr": ":27777"}
otel-collector-1  | 2024-11-14T07:58:41.899Z	info	extensions/extensions.go:59	Extension started.	{"kind": "extension", "name": "expvar"}
otel-collector-1  | 2024-11-14T07:58:41.899Z	info	extensions/extensions.go:42	Extension is starting...	{"kind": "extension", "name": "healthcheckv2"}
otel-collector-1  | 2024-11-14T07:58:41.899Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "extension", "name": "healthcheckv2", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	extensions/extensions.go:59	Extension started.	{"kind": "extension", "name": "healthcheckv2"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	extensions/extensions.go:42	Extension is starting...	{"kind": "extension", "name": "jaeger_storage"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	jaegerstorage/extension.go:119	Initializing storage 'some_storage'	{"kind": "extension", "name": "jaeger_storage"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	memory/factory.go:76	Memory storage initialized	{"kind": "extension", "name": "jaeger_storage", "configuration": {"MaxTraces":100000}}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	extensions/extensions.go:59	Extension started.	{"kind": "extension", "name": "jaeger_storage"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	extensions/extensions.go:42	Extension is starting...	{"kind": "extension", "name": "remote_sampling"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	remotesampling/extension.go:104	Starting file-based sampling strategy provider	{"kind": "extension", "name": "remote_sampling", "path": "./cmd/jaeger/sampling-strategies.json"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	static/provider.go:149	Loading sampling strategies	{"kind": "extension", "name": "remote_sampling", "filename": "./cmd/jaeger/sampling-strategies.json"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	warn	static/provider.go:71	Default operations level strategies will not be included for Ratelimiting service strategies.This behavior will be changed in future releases. Cf. https://github.com/jaegertracing/jaeger/issues/5270	{"kind": "extension", "name": "remote_sampling"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "extension", "name": "remote_sampling", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-11-14T07:58:41.900Z	info	remotesampling/extension.go:245	Starting remote sampling HTTP server	{"kind": "extension", "name": "remote_sampling", "endpoint": ":5778"}
otel-collector-1  | 2024-11-14T07:58:41.901Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "extension", "name": "remote_sampling", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-11-14T07:58:41.901Z	info	remotesampling/extension.go:279	Starting remote sampling GRPC server	{"kind": "extension", "name": "remote_sampling", "endpoint": ":5779"}
otel-collector-1  | 2024-11-14T07:58:41.901Z	info	extensions/extensions.go:59	Extension started.	{"kind": "extension", "name": "remote_sampling"}
otel-collector-1  | 2024-11-14T07:58:41.901Z	info	extensions/extensions.go:42	Extension is starting...	{"kind": "extension", "name": "jaeger_query"}
otel-collector-1  | 2024-11-14T07:58:41.901Z	info	jaegerquery/server.go:128	Archive storage not configured	{"kind": "extension", "name": "jaeger_query"}
otel-collector-1  | 2024-11-14T07:58:41.901Z	info	jaegerquery/server.go:145	Metric storage not configured	{"kind": "extension", "name": "jaeger_query"}
otel-collector-1  | 2024-11-14T07:58:41.901Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "extension", "name": "jaeger_query", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-11-14T07:58:41.902Z	info	app/static_handler.go:92	Using UI configuration	{"kind": "extension", "name": "jaeger_query", "path": ""}
otel-collector-1  | 2024-11-14T07:58:41.902Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "extension", "name": "jaeger_query", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-11-14T07:58:41.902Z	info	app/server.go:270	Query server started	{"kind": "extension", "name": "jaeger_query", "http_addr": "[::]:16686", "grpc_addr": "[::]:16685"}
otel-collector-1  | 2024-11-14T07:58:41.903Z	info	extensions/extensions.go:59	Extension started.	{"kind": "extension", "name": "jaeger_query"}
otel-collector-1  | 2024-11-14T07:58:41.903Z	info	app/server.go:355	Starting GRPC server	{"kind": "extension", "name": "jaeger_query", "port": 16685, "addr": ":16685"}
otel-collector-1  | 2024-11-14T07:58:41.903Z	info	app/server.go:336	Starting HTTP server	{"kind": "extension", "name": "jaeger_query", "port": 16686, "addr": ":16686"}
otel-collector-1  | 2024-11-14T07:58:41.906Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-11-14T07:58:41.906Z	info	[email protected]/otlp.go:112	Starting GRPC server	{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4317"}
otel-collector-1  | 2024-11-14T07:58:41.907Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-11-14T07:58:41.907Z	info	[email protected]/otlp.go:169	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
otel-collector-1  | 2024-11-14T07:58:41.907Z	info	[email protected]/service.go:230	Everything is ready. Begin running and processing data.

Screenshot

Additional context

See open-telemetry/opentelemetry.io#5575 (review)

Jaeger backend version

v2.0.0

SDK

OpenTelemetry Python 1.28.1

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

No response

Deployment configs

name: genai-instrumentation

services:
  otel-collector:
    image: jaegertracing/jaeger:2.0.0
    command: [
      "--set", "receivers.otlp.protocols.http.endpoint=0.0.0.0:4318",
      "--set", "receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317"
    ]
    ports:
      - "16686:16686"  # UI
      - "4317:4317"  # grpc
      - "4318:4318"  # http

[codefromthecrypt]

maybe I added this to the wrong repo. If it isn't a doc related change, please move it to the right repo. Sorry!

[yurishkuro]

Jaeger only stores traces, no logs receiver is mounted. Even if you configure a receiver, there is no exporter of logs in the jaeger binary, ie a pipeline for logs cannot be defined.

What's the motivation here? Are events being logged with tracing context but not as span events? Would there be events without trace context?

For the former it would still not be possible to send to Jaeger today, because span events are encapsulated within a span, Jaeger cannot store them separately (could work if we had partial span support).

[codefromthecrypt]

Thanks for asking the motivation. Totally understand historically logs were out of scope. As I was AWOL, I wasn't sure if there already were discussions on how to handle span events tossed for log events. As you can imagine from a user POV, an instrumentation library upgrade requiring different infrastructure for what they would perceive as "the same data" would be annoying.

One thought would be a "demo mode" processor which could re-attach log events as span event, so that the experience ends up the same, so you only need jaeger still to do a demo.

For example, any log that isn't a log event dumped, and any log event that has no trace ID, span ID. For portability, it could patch the span so that they end up as span events. easier said than done maybe, and this assumes you can update spans after the fact, which I don't know if is the case.