Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka, MutinyEmitter.sendAndAwait context propagation issue #2551

Open
mapuci opened this issue Mar 22, 2024 · 0 comments
Open

Kafka, MutinyEmitter.sendAndAwait context propagation issue #2551

mapuci opened this issue Mar 22, 2024 · 0 comments
Labels

Comments

@mapuci
Copy link

mapuci commented Mar 22, 2024

I have created issue on quarkus git repo, but I found no one brave enough to face it :)
quarkusio/quarkus#39098 (reproducer link is at the bottom)

The issue is: when using any kind of emitter to send message to kafka in a blocking manner (using regular blocking JTA @ Transactional) the transaction will sometimes still be bound to 2 threads kafka-producer-network-thread and regular executor thread in time of commit (on executor thread ofc.). This causes TX to fail, in a dramatic way. It happens randomly, chance is like 1:1000.

2024-03-01 08:32:55,162 WARN  [com.arj.ats.arjuna] (executor-thread-2) ARJUNA012094: Commit of action id 0:ffff0a010a48:e441:65e18477:d1d0 invoked while multiple threads active within it.
2024-03-01 08:32:55,163 WARN  [com.arj.ats.arjuna] (executor-thread-2) ARJUNA012107: CheckedAction::check - atomic action 0:ffff0a010a48:e441:65e18477:d1d0 commiting with 2 threads active!
2024-03-01 08:32:55,163 WARN  [com.arj.ats.jta] (executor-thread-2) ARJUNA016039: onePhaseCommit on < formatId=131077, gtrid_length=35, bqual_length=36, tx_uid=0:ffff0a010a48:e441:65e18477:d1d0, node_name=quarkus, branch_uid=0:ffff0a010a48:e441:65e18477:d1d3, subordinatenodename=null, eis_name=0 > (io.agroal.narayana.LocalXAResource@2ea67dc) failed with exception XAException.XA_RBROLLBACK: javax.transaction.xa.XAException: Error trying to transactionCommit local transaction: Enlisted connection used without active transaction

image

The issue is that executor thread is blocked until message gets ACKed. But this is done BEFORE contextPropagation clears propagated contexts on kafka-producer-network-thread. So it is kind of race condition I guess.
Problem can be bypassed using different thread when sending to kafka, thread without TX context propagation. e.g. @ManagedExecutorConfig(cleared = ThreadContext.TRANSACTION)

Quarkus reproducer:

  • https://github.com/MatejPucihar/kafka-db-tx
  • reproducer stress tests the app, since the chance of EX is low, I have no other idea, how to reproduce it.
  • EX is reproduced for both MutinyEmitter and regular Emitter.

That are all the insights I can provide :) I'm for sure not an expert with reactive or context propagation, so take my insights with a grain of salt. The reproducer is real.
If this is not a bug, it would at least deserve some honorable mention in quarkus documentation I guess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants