Enable overlapping RW+RO transactions

[libor-peltan-cznic]

Hi,
I tried porting a very complex piece of code from LMDB to MDBX.

The show-stopper appeared to be the usage of overlapping trasactions, when a single thread opens both read-write and read-only trasactions in various scenarios (any of them opened first, any of them "long-lived", ...). Some of the scenarios can't be easily removed, as the RO transaction is intentionally kept as a consistent snapshot, while the database is being modified in the other txn.

Is there a strong reason why MDBX doesn't support this, opposing to LMDB?

I found a debug option to allow this. Does it have any drawbacks? Will it be removed in the future?

Thanks much!

[erthink]

In general, you intentionally breaks the atomicity, consistency and isolation of ACID when use MDBX/LMDB's read- and write-transaction simultaneously. I.e. doing this you read and mix both an old and a new versions of the data.

Furthermore, it is no warranty (without specific check & retry) that you see thru read-transaction exactly the previous snapshot immediatelly before the write-transaction was started. And if the order of starting read and write transactions is not fixed/determined, then you will get a rather strange system with random behavior (or sometimes random, which is even worse).

Practice shows that such mixing leads to heisenbugs and is usually the result of a design error(s), misunderstandings and/or omissions. Therefore, libmdbx prohibits transaction overlap by default, but allows to disable checking by debugging flags (including because such bad behavior is used in old tests).

In the future these and other checks still be disableable, but via more general approach, including #200, #223, etc.

[libor-peltan-cznic]

So far I'm using LMDB with the env's MDB_NOTLS flag. From what I know, int this setup the transaction data are not bound to threads, and simultaneous transactions work the same way regardless if different threads opened them, or the same thread. I may be lacking the internal view on LMDB (or MDBX) implementation, but at least from the outer view (on API), it is not clear why this shouldn't work the consistent way - i.e. that in the RO txn, the data are seen consistently in the state when the RO txn was opened, an the RW txn can manipulate the data independently. Do you think there is a difference between MDBX and LMDB operation (with MDBX_NOTLS and MDB_NOTLS flag on, respectively), or are they working the same way in this case?

[erthink]

The five points:

1. (just in case) No any object (i.e. a transaction or a cursor) could be used by multiple thread simultaneously.
2. MDBX_NOTLS works exactly the same as MDB_NOTLS, so you can use transaction from any thread.
   However, a write transaction always bought to single thread (since holds the shared mutex for ACID serialization).
3. If a write transaction starts after a read transaction, then there is no guarantee that one or more other write transactions have not been committed between the launches of these transactions.
4. It is strange enough if some code takes data from a read transaction and does something based on it in a write transaction - briefly, this the corner case which I told above.
5. Nonetheless, if you are sure to mix RW and RO txns then you can avoid "transaction overlapping checks" by debug flags (for now) or by other approach in the future.