[Bug] java.lang.IllegalArgumentException: The log sequence number must be non-decreasing. The current log sequence number is 27783, but the new log sequence number is 27783

[luoyuxia]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main

Minimal reproduce step

Create a pk table with 3 replica factor, and then write to the table. The exception may well happen..

What doesn't meet your expectations?

Shouldn't throw such exception

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[luoyuxia]

When kv records come into KvTablet, we'll increase the log sequence number(which is log offset) in PreWriteBuffer. But if the kv batch is duplicated in idempotent writing, it won't append the change log to logTablet.. So it brings the inconsistent of the log offset between PreWriteBuffer and LogTablet..
It happens that the lsn (log sequnce number)PreWriteBuffer is 10, but the end log offset of LogTablet is 8 since the last two log entry is not appended to LogTablet since the the kvBatch is considered as duplicated..

When another batch is written into KvTablet, it'll get the lsn assigned to the record from the end offset logTablet, which is 8.. But the current lsn of PreWriteBuffer is 10. When put the log into PreWriteBuffer, it will throw the exception The log sequence number must be non-decreasing..

Althogh I'm not sure that's the root cause of my case, but it does be a problem that should be resolved..
I'm trying to find whether is it expected forduplicate kv batch..