perf: Persistent executor

[dima74]

Context

Meta issue: optimizing single peer tps #4727.
It was identified that executor related things takes noticeable amount of time (#3716 (comment)).
Fixes #3716

Solution

Single executor WASM instance will be used for validating all transactions of a block. It gives approximately 10-15% improvement of single peer tps (from 2900 to 3300). However there is a problem with lifetimes and I have to use a hack with std::mem::transmute to bypass borrow checker. Would be glad to hear opinions/suggestions about it.

Review notes (optional)

Primary change is that data stored in wasmtime::Store was changed from CommonState<...> to Option<CommonState<...>>

Checklist

• I've read CONTRIBUTING.md.
• (optional) I've written unit tests for the code changes.
• All review comments have been resolved.
• All CI checks pass.

[Erigara]

Not really comfortable with using unsafe.

What is going under the hood is that WasmCache contain inside state transaction with lifetime <'world, 'block, 'state>, but than we actually pass inside StateTransaction with a lifetimes shorter than that with a promise that we will clear state after transaction is validated, right?

[dima74]

Currently WasmCache stores &'world mut StateTransaction<'block, 'state> inside. We create WasmCache before obtaining any StateTransaction, so the first problem is that there is no 'world lifetime when we create WasmCache.

I tried to eliminate 'world lifetime by storing Arc<RefCell<StateTransaction<'block, 'state>>> inside WasmCache. In this case problem will be in categorize_transactions function. Basically:

fn categorize_transactions<'block, 'state>(
    transactions: Vec<AcceptedTransaction>,
    state_block: &'block mut StateBlock<'state>,
) -> Vec<CommittedTransaction> {
    let wasm_cache: WasmCache<'block, 'state> = WasmCache::new();
    validate(tx1, state_block, &mut wasm_cache);
    validate(tx2, state_block, &mut wasm_cache);  // here borrow check error
    ...
}

fn validate(
    tx: AcceptedTransaction,
    state_block: &'block mut StateBlock<'state>,
    wasm_cache: &mut WasmCache<'block, 'state>,
) {
    ...
}

&mut WasmCache<'block, 'state> is invariant over 'block thus calling validate multiple times will fail borrow check

[DCNick3]

The way you have to repeat impls for the Option<CommonState<...>> is unfortunate... Would it make sense to make all uses of CommonState wrapped into an Option? Or would this be too disruptive?

[dima74]

I also don't like it, but using Option<CommonState> everywhere will make code more complex I think

[dima74]

Rebased after #5113