OpenTelemetry failing when more than 1 test

[driverpt]

Expected Behavior

Tests should boot properly

Actual Behaviour

GlobalOpenTelemetry is being initialized twice in between Tests. (see GlobalOpenTelemetry#get).

Gradle Output:

    io.micronaut.http.server.exceptions.ServerStartupException at AwsApiProxyTestServer.java:107
        Caused by: com.amazonaws.serverless.exceptions.ContainerInitializationException at MicronautLambdaContainerHandler.java:260
            Caused by: io.micronaut.context.exceptions.BeanInstantiationException at DefaultBeanContext.java:1921
                Caused by: io.micronaut.context.exceptions.BeanInstantiationException at DefaultBeanContext.java:2367
                    Caused by: java.lang.IllegalStateException at GlobalOpenTelemetry.java:99
                        Caused by: java.lang.Throwable at GlobalOpenTelemetry.java:107

Code where it fails:

  public static void set(OpenTelemetry openTelemetry) {
    synchronized (mutex) {
      if (globalOpenTelemetry != null) {
        throw new IllegalStateException( // <------ Code Fails Here
            "GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be "
                + "called only once before any calls to GlobalOpenTelemetry.get. If you are using "
                + "the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal "
                + "instead. Previous invocation set to cause of this exception.",
            setGlobalCaller);
      }
      globalOpenTelemetry = new ObfuscatedOpenTelemetry(openTelemetry);
      setGlobalCaller = new Throwable();
    }
  }

Fix suggestion: On JUnit Extension call GlobalOpenTelemetry.resetForTest() on @BeforeAll if test is PER_CLASS, else on @BeforeEach.

Steps To Reproduce

1. Create new project in Micronaut Launch
2. Add Open Telemetry (including exporter)
3. Set otel.exporter=none for Test Environment
4. Run the tests in CLI with ./gradlew build

Environment Information

• OS: MacOS
• Corretto Java 11

Example Application

No response

Version

3.7.0

[driverpt]

UPDATE

By forcing GlobalOpenTelemetry.resetForTest() on JUnit Before Lifecycle, i'm now getting the following error:

Message: Unable to start GRPC server: Failed to bind to address 0.0.0.0/0.0.0.0:50051
Path Taken: ManagedChannel.serverChannel(GrpcEmbeddedServer server,ExecutorService executorService,List clientInterceptors)
io.micronaut.http.server.exceptions.ServerStartupException: Error starting Micronaut container: Bean definition [io.grpc.ManagedChannel] could not be loaded: Error instantiating bean of type  [io.grpc.ManagedChannel]

Message: Unable to start GRPC server: Failed to bind to address 0.0.0.0/0.0.0.0:50051
Path Taken: ManagedChannel.serverChannel(GrpcEmbeddedServer server,ExecutorService executorService,List clientInterceptors)
	at app//io.micronaut.function.aws.proxy.test.AwsApiProxyTestServer.start(AwsApiProxyTestServer.java:107)
	at app//io.micronaut.function.aws.proxy.test.AwsApiProxyTestServer.start(AwsApiProxyTestServer.java:55)
	at app//io.micronaut.test.extensions.AbstractMicronautExtension.beforeClass(AbstractMicronautExtension.java:318)
	at app//io.micronaut.test.extensions.junit5.MicronautJunit5Extension.beforeAll(MicronautJunit5Extension.java:84)
	at app//org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$12(ClassBasedTestDescriptor.java:395)
	at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)

[graemerocher]

@n0tl3ss can you take a look?

[n0tl3ss]

@driverpt I tried to recreate your issue without success. Can you check does DefaultOpenTelemetryFactory.java class has resetForTest method in your classpath.

I created app using launcher. I selected tracing-opentelemetry-zipkin feature and I chose Junit for testing. I created simple controller that returns string "test" and two test classes with same test code to initialise the Opentelemetry. In debugger I saw that resetForTest is properly called twice at the end of every test class.

Here is a test code:

    @Inject
    @Client("/")
    HttpClient httpClient;

    @Test
    void testItWorks() {
        Assertions.assertTrue(application.isRunning());

        String retrieve = httpClient.toBlocking().retrieve(
                HttpRequest.GET("/test"),
                Argument.of(String.class));


        Assertions.assertEquals("test", retrieve);
    }

[driverpt]

Try adding these dependencies:

    implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-grpc")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp")
    implementation("io.opentelemetry:opentelemetry-extension-aws")
    implementation("io.opentelemetry:opentelemetry-sdk-extension-aws")
    implementation("io.opentelemetry.instrumentation:opentelemetry-aws-sdk-2.2")

    runtimeOnly("ch.qos.logback:logback-classic")

    runtimeOnly("net.logstash.logback:logstash-logback-encoder:7.2")
    runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-logback-mdc-1.0")

[n0tl3ss]

All test passes, Can you try to recreate it and publish the app on the github?

[driverpt]

@n0tl3ss , let me create a Project. In the meanwhile, can you try adding a simple JMS Producer?

[driverpt]

Sample Project: https://github.com/driverpt/micronaut-otel-issue

[n0tl3ss]

Hi @driverpt,

This might be workaround:

handler = new MicronautLambdaHandler(new LambdaApplicationContextBuilder()
                    .environments(Environment.FUNCTION, MicronautLambdaContext.ENVIRONMENT_LAMBDA, Environment.TEST));

Problem is that MicronautLambdaHandler doesn't thinks that it is in test Environment. I think Micronaut AWS module should extend that case.