Allow more explicit control over transaction lifetimes

[inexorabletash]

Imported from https://www.w3.org/Bugs/Public/show_bug.cgi?id=11528

Based on lots of developer feedback, in addition to the current automatic commit model it would be nice to support:

• explicitly committing, to avoid extra round-trips (known performance bottleneck)
• ways to defer committing to allow other async processing to occur within transaction scope

Both make the transaction model more complex, so we should tread carefully.

[inexorabletash]

My thoughts on both of these are captured in these gists:

• https://gist.github.com/inexorabletash/d55a6669a040e92e47c6
• https://gist.github.com/inexorabletash/8c122c84a65de65c35b3

[inexorabletash]

In addition, for many puts, the overhead of all the individual success events can jank the main thread. We may want to design this in such a way that the transaction just opts-out of the per-request events entirely - possibly put() and friends just return null in those cases!

[inexorabletash]

TPAC 2019 Web Apps breakout:

Looked at two proposals:

• On IDBTransaction add waitUntil(promise). Can be called multiple times. Try redefine autocommit behavior in terms of these promises resolving (i.e. unify requests and promises).
• On IDBDatabase.transaction() add an option manualCommit: true; commit() must be called explicitly.

Notes:

• Expect most usage to be by wrappers/libraries.
• Attendees were pretty split - explicit commit is more like traditional database APIs, promises build on SW/WebLocks patterns and reduce risk of coding deadlocks
• Using either option puts the transaction into a new mode where new requests can be made at any time.
• At first glance, seems like either option can be polyfilled on the other

Next steps: make concrete proposals; reach out to library authors to get feedback on preferences

[Jack-Works]

I'd like to give my real-world example of how pain is it to use Indexed DB v2.

I have a database called crypto-keys than contains WebCrypto CryptoKeys object, let's assume it is in version 1.

In version 2, I need to change from CryptoKeys to JsonWebKey(a plain object) but I can't do it in the upgrade transaction, because in the WebCrypto spec, export a CryptoKey to JsonWebKey is an async operation.

Even worse, in Indexed DB v2, I can't query the database version (it's a v3 API and only Chrome support it now) so I can't open the current version (1) based on the query result and prepare all async data I need, then open the database in version 2 to use previously prepared data to move from 'CryptoKeytoJsonWebKey`.

In the end, I have to open my database like this: (and I think this kind of solution is very, very ugly)

0. Set UpgradeData to undefined
1. Open the database of version firstVersionThatRequiresAsyncUpgrade (might fail cause DB version might be greater than that) with UpgradeData, if failed, goto step 4
2. If firstVersionThatRequiresAsyncUpgrade === latestVersion, return the db, it is upgraded asynchronously to the latest version
3. Run the asyncUpgradePrepare for database, set result to UpgradeData
4. firstVersionThatRequiresAsyncUpgrade++
5. Go to step 1

[Jamesernator]

The web locks API shares a lot in common with this, I wonder if the concepts actually map one-to-one, for example we have "read" vs "readwrite" transactions, which map nicely to "shared" vs "exclusive" locks.

Then there's also requests for timeouts on indexxedDB transactions, this could naturally be built on top stealing locks. In a lot of ways this would actually be safer for indexxeddb as the whole transaction can be rolled back instead of committed.

I wonder if would actually be possible to simply share the architecture such that in request a lock, instead of simply taking a name, it could take an indexxeddb transaction record instead. This way indexeddb wouldn't need to build it's own concept of locking.

[janl]

Putting my hat in the ring here for more control over transaction lifetimes and/or better composability with promises/async/await.

My use case is the implementation of an abstract data structure, think linked list, or tree, with persistence and encryption-at-rest. In practice that means I need to be able to read a node in my structure from IndexedDB, decrypt it (which is an async operation) and then use the decrypted content to identify which next node to load from IndexedDB to traverse my data structure.

Say I want to give transaction semantics to a list reorder, or tree insert operation, I will have to read and write a number of entries from IndexedDB and further entries based on encrypting those entries. All of which I can’t do at the moment because of the auto commit behaviour.

I can live with either an explicit txn.commit() API or a “predicate pushdown” variant like the txn.waitUntil(Promise) proposal in https://github.com/inexorabletash/indexeddb-promises#advanced-usage

The goal here is to provide special-purpose data management APIs that support transparent encryption-at-rest in scenarios where a simple (and today ubiquitous) device-level encryption is not satisfactory (e.g. when devices are shared among multiple users with the same system “login”).

Aside from the consistency guarantees that I can’t implement today, this works quite well already. I’d love to be able to offer this as a proper safe solution with official Web API support.

I’m happy to elaborate or answer any further questions on this.

Thanks! <3

[evanstade]

We discussed this at TPAC 2024 again and I believe there was more or less consensus on this option:

IDBDatabase.transaction() add an option manualCommit: true; commit() must be called explicitly.

The advantages are that it's simple to understand and implement.

The main drawback is that it's a footgun to make it possible to wedge an entire database by forgetting to call commit(). Offsetting this is that (a) this is a non-default, i.e. advanced, usage pattern and (b) most IDB usage is via wrapper libraries that will hopefully make it harder to misuse. However I filed #425 to see if we can create a secondary safety mechanism to protect against misuse.