[bug] Span/Activity cannot be exported to Tempo if Activity.StatusDescription has the length of 128 characters or greater

[JediAlexLis]

Package

OpenTelemetry.Exporter.OpenTelemetryProtocol

Package Version

Package Name	Version
OpenTelemetry.Exporter.OpenTelemetryProtocol	>= 1.11.0
OpenTelemetry.Exporter.Console	>= 1.11.0

Runtime Version

net8.0

Description

A span with Status=Error and StatusDescription with the length of 128 characters and greater cannot be exported to tempo instance if OpenTelemetry.Exporter.OpenTelemetryProtocol v1.11.0 and greater is used.

The self-diagnostics logs says the following:

2025-01-30T20:37:18.0840959Z:HTTP request to {0} failed. Exception: {1}{http://localhost:4318/v1/traces}{System.Net.Http.HttpRequestException: Response status code does not indicate success: 400 (Bad Request).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.OtlpHttpExportClient.SendExportRequest(Byte[] buffer, Int32 contentLength, DateTime deadlineUtc, CancellationToken cancellationToken)}

This bug isn't reproducable in case where the length of the status description is 127 characters and less. Also, it's not reproducable in the previous versions of the package.

Steps to Reproduce

1. Create docker-compose.yaml and paste the following snippet to it:

services:

  tempo:
    image: grafana/tempo:latest
    command: [ "-config.file=/etc/tempo.yaml" ]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - ./tempo-data:/var/tempo
    ports:
      - "14268:14268"  # jaeger ingest
      - "3200:3200"   # tempo
      - "9095:9095" # tempo grpc
      - "4317:4317"  # otlp grpc
      - "4318:4318"  # otlp http
      - "9411:9411"   # zipkin

2. Create tempo.yaml in the same place and paste the following snippet to it:

stream_over_http_enabled: true
server:
  http_listen_port: 3200
  log_level: info


cache:
  background:
    writeback_goroutines: 5
  caches:
  - roles:
    - frontend-search  
    memcached: 
      host: memcached:11211

query_frontend:
  search:
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09
    metadata_slo:
        duration_slo: 5s
        throughput_bytes_slo: 1.073741824e+09
  trace_by_id:
    duration_slo: 100ms
  metrics:
    max_duration: 120h                # maximum duration of a metrics query, increase for local setups
    query_backend_after: 5m
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09

distributor:
  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can
      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
        thrift_http:                   #
          endpoint: "tempo:14268"      # for a production deployment you should only enable the receivers you need!
        grpc:
          endpoint: "tempo:14250"
        thrift_binary:
          endpoint: "tempo:6832"
        thrift_compact:
          endpoint: "tempo:6831"
    zipkin:
      endpoint: "tempo:9411"
    otlp:
      protocols:
        grpc:
          endpoint: "tempo:4317"
        http:
          endpoint: "tempo:4318"
    opencensus:
      endpoint: "tempo:55678"

ingester:
  max_block_duration: 5m               # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

compactor:
  compaction:
    block_retention: 24h                # overall Tempo trace retention. set for demo purposes

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces
  processor:
    local_blocks:
      filter_server_spans: false
      flush_to_storage: true

storage:
  trace:
    backend: local                     # backend configuration to use
    wal:
      path: /var/tempo/wal             # where to store the wal locally
    local:
      path: /var/tempo/blocks

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks] # enables metrics generator
      generate_native_histograms: both
      

3. Create tempo-data folder in the same place.
4. Run docker-compose up -d to start tempo locally.
5. In a browser (or by using curl, Postman, whatever), send GET-request http://localhost:3200/ready until you get ready response from tempo.
6. Create a .NET 8 console app, add OpenTelemetry.Exporter.Console and OpenTelemetry.Exporter.OpenTelemetryProtocol of version 1.11.0 or 1.11.1, and put the following code into Program.cs:

using System.Diagnostics;
using System.Text;

using OpenTelemetry;
using OpenTelemetry.Trace;

namespace OTel
{
    internal class Program
    {
        static void Main(string[] args)
        {
            using var activitySource = new ActivitySource("MyTest");

            using var tracerProvider = Sdk.CreateTracerProviderBuilder()
                .AddSource("MyTest")
                .AddConsoleExporter()
                .AddOtlpExporter(options =>
                {
                    options.Endpoint = new Uri("http://localhost:4318/v1/traces");
                    options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
                })
                .Build();

            var length = 128; // 127 characters seem to be the maximum length of StatusDescription supported by the package
            var longMessage = Enumerable.Repeat('*', length)
                .Aggregate(new StringBuilder(), (builder, ch) => builder.Append(ch))
                .ToString();

            using var activity = activitySource.StartActivity("NewActivity");
            activity!.SetTag("count", 10);
            activity.SetStatus(ActivityStatusCode.Error, longMessage);
        }
    }
}

7. Add OTEL_DIAGNOSTICS.json with the following content:

{
  "LogDirectory": ".",
  "FileSize": 32768,
  "LogLevel": "Warning"
}

8. Add the following section into your csproj file:

<ItemGroup>
  <None Update="OTEL_DIAGNOSTICS.json">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </None>
</ItemGroup>

9. Start the console app.
10. When it's finished, copy Activity.TraceId value from the console, and send http://localhost:3200/api/traces/<your trace id>. You will get 404 response.
11. Go to /bin/Debug/net8.0, find the self-diagnostincs logs, and see the message about 400 response.

Expected Result

1. The span should have been successfully exported to tempo and queried.
2. No exceptions should have been logged by the self-diagnostics.

Actual Result

1. The span hasn't been exported to tempo, and the query returns 404.
2. The exception about 400 response has been logged by the self-diagnostics.

Additional Context

No response

[rajkumar-rangaraj]

Does this work with earlier versions of OpenTelemetry.Exporter.OpenTelemetryProtocol?

[JediAlexLis]

Yes, it works with v1.9.0 and v1.10.0