[bug] GrpcExportClient does not work for secure endpoints on net framework apps 462 or higher

[cosminlwpw]

Package

OpenTelemetry.Exporter.OpenTelemetryProtocol

Package Version

Package Name	Version
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.11.1

Runtime Version

net48

Description

Setting exporter endpoint to a secure gRPC endpoint of otel-collector results in an exception.
otel-collector instance is running grpc secure on 4319.

`
OpenTelemetry.Exporter.OpenTelemetryProtocol.Tests.IntegrationTests.TraceExportResultIsSuccess(protocol: Grpc, endpoint: ":4319", exportProcessorType: Simple or Batch, forceFlush: True, scheme: "https")
 Source: IntegrationTests.cs line 40
 Duration: 2 sec

Message: 
Assert.Equal() Failure: Values differ
Expected: Success
Actual: Failure

Stack Trace: 
IntegrationTests.TraceExportResultIsSuccess(OtlpExportProtocol protocol, String endpoint, ExportProcessorType exportProcessorType, Boolean forceFlush, String scheme) line 103

Standard Output: 
TracerProviderSdk event: 'Building TracerProvider.'
TracerProviderSdk event: 'Sampler added = "OpenTelemetry.Trace.ParentBasedSampler".'
TracerProviderSdk event: 'Processors added = "OpenTelemetry.SimpleActivityExportProcessor".'
TracerProviderSdk event: 'TracerProvider built successfully.'
Activity started. Name = 'Grpc Test Activity', Id = '00-ff2826f11110ecea734e717a40f01455-a463c7f4f1743663-01'.
Activity stopped. Name = 'Grpc Test Activity', Id = '00-ff2826f11110ecea734e717a40f01455-a463c7f4f1743663-01'.
Exporter failed send data to collector to https://localhost:4319/opentelemetry.proto.collector.trace.v1.TraceService/Export endpoint. Data will not be sent. Exception: Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="failed to connect to all addresses", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1738165998.323000000","description":"Failed to pick subchannel","file":"......\src\core\ext\filters\client_channel\client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1738165998.323000000","description":"failed to connect to all addresses","file":"......\src\core\lib\transport\error_utils.cc","file_line":163,"grpc_status":14}]}") ---> Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1738165998.323000000","description":"Failed to pick subchannel","file":"......\src\core\ext\filters\client_channel\client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1738165998.323000000","description":"failed to connect to all addresses","file":"......\src\core\lib\transport\error_utils.cc","file_line":163,"grpc_status":14}]}
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Core.Internal.AsyncCall2.UnaryCall(TRequest msg)
at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method2 method, String host, CallOptions options, TRequest request)
at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.GrpcExportClient.SendExportRequest(Byte[] buffer, Int32 contentLength, DateTime deadlineUtc, CancellationToken cancellationToken) in C:\git\opentelemetry-dotnet\src\OpenTelemetry.Exporter.OpenTelemetryProtocol\Implementation\ExportClient\ GrpcExportClient .cs:line 59
ForceFlush invoked for processor type 'SimpleActivityExportProcessor' returned result 'True'.
'TracerProvider' Disposed.
`

Steps to Reproduce

Make sure to set net48 OpenTelemetry.Exporter.OpenTelemetryProtocol.Tests.csproj
Run OpenTelemetry.Exporter.OpenTelemetryProtocol.Tests.IntegrationTests.TraceExportResultIsSuccess
You can just leave one test case [InlineData(OtlpExportProtocol.Grpc, ":4319", ExportProcessorType.Simple, true, "https")] and change [Trait] to just [Theory].

Expected Result

no exceptions, telemetry data ends up in otel-collector -> exporters -> .etc

Actual Result

exception

Additional Context

No response

[rajkumar-rangaraj]

@cosminlwpw Did you try connecting to a service which does not use self-signed certificate?

[cosminlwpw]

@cosminlwpw Did you try connecting to a service which does not use self-signed certificate?

@rajkumar-rangaraj Uhm, no...and I don't have access to a resource that does not use self-signed certificates. But certificate chain is correct.
This is usually the case in development stage (at least in my cases).
Also, same test but with targetframework net core is working correctly (but that is probably to be expected)

[rajkumar-rangaraj]

I tested this scenario using a secure gRPC endpoint over HTTPS targeting .NET Framework 4.6.2 and higher, and it worked without any issues.

[cosminlwpw]

I tested this scenario using a secure gRPC endpoint over HTTPS targeting .NET Framework 4.6.2 and higher, and it worked without any issues.

very peculiar; since for me this looks like an integration issue, can you share OS (+version where client app/scenario is run) and otel-collector service version for which this scenario worked for you?
Thanks

[cosminlupw]

ok, I made it work by setting GRPC_DEFAULT_SSL_ROOTS_FILE_PATH env. var. point to where my crt is.
I guess issue can be closed though, it would be great to have this documented, in changelog at least? I first figured it out captain obviously and then I found these chats from the past #1795

[Kielek]

@cosminlupw, maybe you could review our documentation and create PR with improvements?

Ref: https://github.com/open-telemetry/opentelemetry-dotnet/blob/core-1.11.1/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/README.md

We can consider adjusting also CHANGELOG if needed.

[cosminlupw]

tempting, but I have doubts on how to go about this without adding confusion; if this needs just a doc change and/or impl. change i.e. GRPC_DEFAULT_SSL_ROOTS_FILE_PATH is actually a Grpc.Core env. variable, its implementation detail, and not really otel related.

The only way(s) I see it part of otel-net docs is adding a new Environment Variable OTEL_EXPORTER_OTLP_CERTIFICATE and:
a. actually set GRPC_DEFAULT_SSL_ROOTS_FILE_PATH := OTEL_EXPORTER_OTLP_CERTIFICATE if set
b. change GrpcExportClient and find maybe a straightforward way to load the certificate from stores

Maybe @rajkumar-rangaraj can jump-in and share his opinion, since on his setup, the secure connection worked without any additional changes.

[rajkumar-rangaraj]

@cosminlwpw I did not use a local collector; instead, I tested with a service which supports gRPC endpoint over HTTPS.

[rajkumar-rangaraj]

There is a work in progress to add support for OTEL_EXPORTER_OTLP_CERTIFICATE - #5918. Until that work is complete, workaround is to use GRPC_DEFAULT_SSL_ROOTS_FILE_PATH.