STR-949 chainstate manager v2 and minor chainstate restructures

[delbonis]

Description

This is a second attempt at doing the chainstate manager now with some better idea of the context, plus some additional types to support work in the other PR. I expect some more things will be copied over that are related and minimally invasive.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature/Enhancement (non-breaking change which adds functionality or enhances an existing one)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactor
• New or updated tests
• Dependency Update

Notes to Reviewers

Checklist

• I have performed a self-review of my code.
• I have commented my code where necessary.
• I have updated the documentation if needed.
• My changes do not introduce new warnings.
• I have added tests that prove my changes are effective or that my feature works.
• New and existing tests pass with my changes.

Related Issues

[delbonis]

Really happy with the general refactoring I did to the CSM logic, this will hopefully make it simpler to make work when rebasing the changes in the other branch.

[delbonis]

Still have to simplify the actual db impls to remove the now-unnecessary chainstate tables/fields.

[bewakes]

Great work, looks much cleaner.

[storopoli]

ACK 164930d

[delbonis]

Will make some minor changes in response to these comments and rebase.

[delbonis]

Rebased and applied fixes.

[github-actions]

Commit: 83ef9ef

SP1 Performance Test Results

program	cycles	success
BTC_BLOCKSPACE	30,357,713	✅
EL_BLOCK	101,655	✅
CL_BLOCK	58,268	✅
L1_BATCH	30,387,702	✅
L2_BATCH	5,473	✅
CHECKPOINT	15,255	✅

[codecov]

Codecov Report

Attention: Patch coverage is 24.63343% with 514 lines in your changes missing coverage. Please review.

Project coverage is 54.85%. Comparing base (67a711d) to head (1ff4722).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/storage/src/managers/chainstate.rs	0.00%	74 Missing ⚠️
crates/state/src/state_op.rs	40.17%	70 Missing ⚠️
bin/strata-client/src/rpc_server.rs	0.00%	53 Missing ⚠️
bin/strata-client/src/main.rs	0.00%	52 Missing ⚠️
...rates/consensus-logic/src/csm/client_transition.rs	25.00%	42 Missing ⚠️
crates/primitives/src/epoch.rs	0.00%	28 Missing ⚠️
...tes/sequencer/src/block_template/block_assembly.rs	0.00%	24 Missing ⚠️
crates/sequencer/src/duty/worker.rs	0.00%	23 Missing ⚠️
crates/sequencer/src/checkpoint/worker.rs	0.00%	20 Missing ⚠️
crates/consensus-logic/src/fork_choice_manager.rs	0.00%	19 Missing ⚠️
... and 14 more

@@            Coverage Diff             @@
##             main     #633      +/-   ##
==========================================
- Coverage   55.15%   54.85%   -0.31%     
==========================================
  Files         311      313       +2     
  Lines       33321    33449     +128     
==========================================
- Hits        18379    18347      -32     
- Misses      14942    15102     +160     

Files with missing lines	Coverage Δ
crates/btcio/src/reader/query.rs	47.33% <100.00%> (ø)
crates/chaintsn/src/transition.rs	82.86% <100.00%> (ø)
crates/db/src/traits.rs	0.00% <ø> (ø)
crates/proof-impl/cl-stf/src/lib.rs	94.87% <100.00%> (ø)
crates/rocksdb-store/src/chain_state/db.rs	98.54% <100.00%> (-0.17%)	⬇️
crates/rocksdb-store/src/lib.rs	0.00% <ø> (ø)
crates/sequencer/src/block_template/handle.rs	0.00% <ø> (ø)
crates/status/src/status_manager.rs	47.12% <100.00%> (+2.54%)	⬆️
crates/test-utils/src/lib.rs	87.50% <ø> (ø)
crates/state/src/chain_state.rs	91.07% <90.90%> (-0.31%)	⬇️
... and 23 more

... and 5 files with indirect coverage changes

[bewakes]

Writing WriteBatch to db seems a bit weird in that it seems to not allow for updating an idx. The rest is good.

[storopoli]

ACK f10c7ca

[delbonis]

Okay this should be good to go now. I don't want to force merge this.

[sapinb]

unused file likely retained during conflict fix.
This was moved inside crates/sequencer

[delbonis]

Force merging because we have a few approvals already and we want to keep velocity up.