storage: optimize multiple objects write #1059
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously multiple objects write required 2 round trips
to DB per object: one to fetch object state from DB and second
to update object. Each query is indexed and fast, but
with latency to DB comparable to query execution time it adds
significant overhead.
This change optimizes multiple objects write in 2 steps:
Instead of reading DB state for each object and then deciding
on possible write operation, perform write operation unconditionally
with correct predicates (version and permissions) defined where
applicable. That said write operation might not succeed if row doesn't
match predicate. Write query is structured in such way, that
final state of the row in the database is returned, regardless
whether writeop successed or not. By inspecting returned row we can
infer whether it was success, version conflict or permission error.
Now that each object is written to DB in a single query, there
is no dependencies between queries and all of them can be blasted
to DB in a batch without waiting for result of each. Whole batch
continues to be executed in a single transaction, so outcome is
the same, but batching negates latency penalty.
To support batching access to native pgx connection is required, for that
ExecuteInTxPgx
variant was added and all call-site of StorageWrite converted to it.