CircleCI/Testcontainers (neo4j) JVM connection timeout error

[ao508]

Hi there,

I'm having some difficulty running Testcontainers in CircleCI if anyone can point me in the right direction.

circleci config file here
full circleci output here

Executing the circleci job locally (macos) does not reproduce the error, it's only happening in the circleci env.

cmd I use locally

circleci local execute --job run_checkstyle

All we need is to provide our java app with a temporary neo4j database to access when running unit tests.

I would like to avoid running a temporary neo4j outside of the jvm since Testcontainers can provide an embedded db but I'm not having much luck with connecting the jvm to the embedded db while in circleci.

However it does look like the container is starting up, at least initially

18:18:53.574 [main] INFO 🐳 [neo4j:3.5.0] - Starting container with ID: b7f4e26f4175d2cd04a5ce1e9961d1fa37a0d71eb1da258a4a3ba56004fc19c2

but then we reach this

18:20:54.381 [main] DEBUG 🐳 [neo4j:3.5.0] - Wait strategy threw an exception
org.rnorth.ducttape.TimeoutException: java.util.concurrent.TimeoutException
	at org.rnorth.ducttape.timeouts.Timeouts.callFuture(Timeouts.java:70)
	at org.rnorth.ducttape.timeouts.Timeouts.doWithTimeout(Timeouts.java:60)
	at org.testcontainers.containers.wait.strategy.WaitAllStrategy.waitUntilReady(WaitAllStrategy.java:53)
	at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:923)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:466)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:329)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:327)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:315)
	at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:242)
	at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$200(TestcontainersExtension.java:229)
	at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$null$1(TestcontainersExtension.java:59)
	at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$getOrComputeIfAbsent$0(ExtensionValuesStore.java:80)
	at org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.get(ExtensionValuesStore.java:185)
	at org.junit.jupiter.engine.execution.ExtensionValuesStore.getOrComputeIfAbsent(ExtensionValuesStore.java:87)
	at org.junit.jupiter.engine.execution.NamespaceAwareStore.getOrComputeIfAbsent(NamespaceAwareStore.java:53)
	at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$beforeAll$2(TestcontainersExtension.java:59)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:59)
	at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.lambda$invokeBeforeAllCallbacks$7(ClassTestDescriptor.java:358)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.invokeBeforeAllCallbacks(ClassTestDescriptor.java:358)
	at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.before(ClassTestDescriptor.java:197)
	at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.before(ClassTestDescriptor.java:74)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:132)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.util.concurrent.TimeoutException: null
	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at org.rnorth.ducttape.timeouts.Timeouts.callFuture(Timeouts.java:65)
	... 55 common frames omitted

and

18:20:55.358 [main] DEBUG org.testcontainers.utility.ResourceReaper - Removed container and associated volume(s): neo4j:3.5.0
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 133.079 s <<< FAILURE! - in org.mskcc.cmo.metadb.service.MetadbServiceTest
[ERROR] org.mskcc.cmo.metadb.service.MetadbServiceTest  Time elapsed: 133.076 s  <<< ERROR!
org.testcontainers.containers.ContainerLaunchException: Container startup failed
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
Caused by: org.rnorth.ducttape.TimeoutException: java.util.concurrent.TimeoutException
Caused by: java.util.concurrent.TimeoutException

Any thoughts or guidance is greatly appreciated 🙏

Other info:

• docker version: 20.10.2
• java version: 1.8.0_261
• maven 3.6.1
• circleci (local) 0.1.15932+3471602 (homebrew)

[kiview]

Just for sake of documenting the current state of the investigations:

The CircleCI config uses

    steps:
      - setup_remote_docker:
          version: 20.10.2

Since it also uses the machine executor, this will might be the cause of the problems. Note this section from the CircleCI docs:

The use of the setup_remote_docker key is reserved for configs in which your primary executor is a docker container. If your executor is machine (and you want to use docker commands in your config) you do not need to use the setup_remote_docker key.

[ao508]

Unfortunately that’s not it. I switched to installing docker as well with no luck.

I’ll be online soon and will share some more info.

[ao508]

Resolved.

version: 2.1

jobs:
  run_checkstyle:
    working_directory: /tmp/repos/cmo-metadb
    machine: true
    steps:
      - checkout
      - run:
          < OTHER STEPS >

workflows:
  run_mvn_checkstyle:
    jobs:
      - run_checkstyle

I had tried this variation before but without removing the use of orbs. Maybe that was part of the issue as well? Anyway, I appreciate that the solution was basically just removing a few lines from the config file.

[kiview]

So to get it working, you removed the circleci/[email protected] orb and the usage of

    docker:
        - image: cimg/openjdk:8.0.275

which, was added by inlining the defaults?

I haven't used CircleCI actively for some time, but using the orb might have implicitly switched over to using the Docker executor instead of the machine one (at least the orb configures the Docker executor).

Either way, happy that you found a working config 👍