Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(s3stream): fix deadlock issue #1808

Merged
merged 1 commit into from
Aug 15, 2024
Merged

fix(s3stream): fix deadlock issue #1808

merged 1 commit into from
Aug 15, 2024

Conversation

ShadowySpirits
Copy link
Contributor

The deadlock occurs between:

  • at com.automq.stream.s3.S3Storage.lambda$append0$16(S3Storage.java:488)
  • at com.automq.stream.s3.wal.impl.object.RecordAccumulator.lambda$unsafeUpload$16(RecordAccumulator.java:531)
"0-9-s3-read-cb-executor-1" #247 [4207] daemon prio=5 os_prio=0 cpu=1546.08ms elapsed=16773.60s tid=0x00007f1a5a4a9000 nid=4207 waiting on condition  [0x00007f1a38290000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x000004006c325990> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:221)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:754)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:990)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock([email protected]/ReentrantReadWriteLock.java:959)
        at com.automq.stream.s3.S3Stream.close(S3Stream.java:339)
        at com.automq.stream.s3.S3StreamClient$StreamWrapper.lambda$close$2(S3StreamClient.java:326)
        at com.automq.stream.s3.S3StreamClient$StreamWrapper$$Lambda/0x00007f1a77c88a00.get(Unknown Source)
        at com.automq.stream.s3.S3StreamClient.runInLock(S3StreamClient.java:257)
        at com.automq.stream.s3.S3StreamClient$StreamWrapper.close(S3StreamClient.java:322)
        at com.automq.stream.s3.S3StreamClient.lambda$closeStreams$13(S3StreamClient.java:214)
        at com.automq.stream.s3.S3StreamClient$$Lambda/0x00007f1a77d3c450.accept(Unknown Source)
        at java.util.concurrent.ConcurrentHashMap.forEach([email protected]/ConcurrentHashMap.java:1605)
        at com.automq.stream.s3.S3StreamClient.closeStreams(S3StreamClient.java:212)
        at com.automq.stream.s3.S3StreamClient.forceClose(S3StreamClient.java:230)
        at com.automq.stream.s3.failover.ForceCloseStorageFailureHandler.handle(ForceCloseStorageFailureHandler.java:32)
        at com.automq.stream.s3.failover.StorageFailureHandlerChain.handle(StorageFailureHandlerChain.java:27)
        at com.automq.stream.s3.S3Storage.lambda$append0$16(S3Storage.java:488)
        at com.automq.stream.s3.S3Storage$$Lambda/0x00007f1a77b9ce28.accept(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/CompletableFuture.java:863)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire([email protected]/CompletableFuture.java:841)
        at java.util.concurrent.CompletableFuture.postComplete([email protected]/CompletableFuture.java:510)
        at java.util.concurrent.CompletableFuture.completeExceptionally([email protected]/CompletableFuture.java:2194)
        at com.automq.stream.s3.wal.impl.object.RecordAccumulator.lambda$unsafeUpload$16(RecordAccumulator.java:531)
        at com.automq.stream.s3.wal.impl.object.RecordAccumulator$$Lambda/0x00007f1a77d3c000.accept(Unknown Source)
        at java.lang.Iterable.forEach([email protected]/Iterable.java:75)
        at com.automq.stream.s3.wal.impl.object.RecordAccumulator.lambda$unsafeUpload$17(RecordAccumulator.java:531)
        at com.automq.stream.s3.wal.impl.object.RecordAccumulator$$Lambda/0x00007f1a77b9e650.accept(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/CompletableFuture.java:863)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire([email protected]/CompletableFuture.java:841)
        at java.util.concurrent.CompletableFuture.postComplete([email protected]/CompletableFuture.java:510)
        at java.util.concurrent.CompletableFuture.complete([email protected]/CompletableFuture.java:2179)
        at com.automq.stream.s3.operator.AbstractObjectStorage$MergedReadTask.handleReadCompleted(AbstractObjectStorage.java:830)
        at com.automq.stream.s3.operator.AbstractObjectStorage.lambda$tryMergeRead0$31(AbstractObjectStorage.java:596)
        at com.automq.stream.s3.operator.AbstractObjectStorage$$Lambda/0x00007f1a7740a918.run(Unknown Source)
        at com.automq.stream.utils.FutureUtil.suppress(FutureUtil.java:31)
        at com.automq.stream.s3.operator.AbstractObjectStorage.lambda$tryMergeRead0$32(AbstractObjectStorage.java:596)
        at com.automq.stream.s3.operator.AbstractObjectStorage$$Lambda/0x00007f1a773b6970.accept(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/CompletableFuture.java:863)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire([email protected]/CompletableFuture.java:841)
        at java.util.concurrent.CompletableFuture.postComplete([email protected]/CompletableFuture.java:510)
        at java.util.concurrent.CompletableFuture.complete([email protected]/CompletableFuture.java:2179)
        at com.automq.stream.s3.operator.AbstractObjectStorage.lambda$acquireReadPermit$37(AbstractObjectStorage.java:679)
        at com.automq.stream.s3.operator.AbstractObjectStorage$$Lambda/0x00007f1a77409530.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
        at java.lang.Thread.runWith([email protected]/Thread.java:1583)
        at java.lang.Thread.run([email protected]/Thread.java:1570)

@ShadowySpirits ShadowySpirits merged commit f26ea25 into main Aug 15, 2024
6 checks passed
@ShadowySpirits ShadowySpirits deleted the fix_dead_lock branch August 15, 2024 08:03
@ShadowySpirits ShadowySpirits restored the fix_dead_lock branch August 15, 2024 08:03
@ShadowySpirits ShadowySpirits deleted the fix_dead_lock branch August 15, 2024 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants