System.Diagnostics.Activity not started when handling delayed messages with open telemetry enabled

[jonesr-out]

Describe the bug

Description

System.Diagnostics.Activity.Current is null when trying to access this property within a command handler for commands sent with a delay. The behavior is only seen in versions later than NServiceBus 9.0.4. It works as expected in version 9.0.4.

Expected behavior

System.Diagnostics.Activity.Current should not be null since a new span should have been created.

Actual behavior

System.Diagnostics.Activity.Current is null.

Versions

NServiceBus >= 9.0.5

Steps to reproduce

The issue is reproducible running this sample application, https://github.com/jonesr-out/nservicebus-otel-bug. Note that MessageGenerator is setting a 100ms delay using DelayDeliveryWith.

Relevant log output

No response

Additional Information

Workarounds

• Downgrade to version 1.0.4

[TravisNickels]

Thanks for reporting this. We are going to take a look at this and try to reproduce and triage it. We will let you know what the results are.

[boblangley]

We've validated the bug and added it to our priority bug backlog and will work on it as soon as we can.

[ydie22]

I think the activity is being discarded by sampling.
activity = ActivitySources.Main.StartActivity(name: ActivityNames.IncomingMessageActivityName, ActivityKind.Consumer, CreateNewRootActivityContext(), tags: null, links: links); returns null. Maybe the trace flags should include Recorded ?
Also, further in the code, an attempt to start the resulting activity is made, which is an error given the StartActivity call (also setting the ID format is prohibited on already started activities).

[mauroservienti]

Quick update: #7264 has been raised to address this issue.