Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable parallel root during persistence #12392

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Rjected
Copy link
Member

@Rjected Rjected commented Nov 8, 2024

An attempt at letting us enable parallel state root while persistence is running. Unfortunately this does cause some large state root calculation times after pruning:

2024-11-08T01:16:13.274042Z DEBUG engine::tree: Calculating state root in parallel, parent found in memory parent_hash=0x62b7207e38f8ee326718cd84122e65a0c529c95c7c972da3031e16be1a521c84 historical=0xce0b5402a4891a163a6a7bfc7082b64479be4f869f658f768a0fb9cbb3b0ad54
2024-11-08T01:16:13.276804Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=41
2024-11-08T01:16:13.291558Z DEBUG provider::storage_writer: Appended block data range=2634289..=2634292
// ... more logs
2024-11-08T01:16:13.668259Z DEBUG engine::persistence: Running pruner block_num=2634292
2024-11-08T01:16:13.668301Z DEBUG pruner: Pruner started tip_block_number=2634292
// ... more logs
2024-11-08T01:16:13.669352Z DEBUG pruner: Pruner finished tip_block_number=2634292 elapsed=1.049125ms deleted_entries=136 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 10000, deleted: 136 }), time_limit: Some(PruneTimeLimit { limit: 100ms, start: Instant { tv_sec: 1135547, tv_nsec: 839583333 } }) } output=PrunerOutput { progress: Finished, segments: [(Headers, SegmentOutput { progress: Finished, pruned: 21, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(2634292), tx_number: None }) }), (Transactions, SegmentOutput { progress: Finished, pruned: 115, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(2634292), tx_number: Some(97718838) }) }), (Receipts, SegmentOutput { progress: Finished, pruned: 0, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(2634292), tx_number: Some(97718838) }) })] } stats=[(Headers, 21, Finished), (Transactions, 115, Finished)]
// ... many commits
2024-11-08T01:16:17.394935Z DEBUG engine::tree: Calculated state root root_elapsed=4.120853875s block=NumHash { number: 2634303, hash: 0x89638859120c12b17c853c18862b59e9a8f0e05e8d726f2b07d40e58c615e098 }

Still debugging why this occurs, it seems to happen after (and is somehow blocked by) many many rw-transaction commits, which is unexpected, since we should only be committing once in persistence.

@Rjected Rjected added C-perf A change motivated by improving speed, memory usage or disk footprint A-blockchain-tree Related to sidechains, reorgs and pending blocks C-benchmark A change that impacts how or what we benchmark labels Nov 8, 2024
@Rjected Rjected force-pushed the dan/enable-parallel-root-persistence branch from 153df6f to 13d7dd1 Compare November 19, 2024 20:29
@Rjected Rjected force-pushed the dan/enable-parallel-root-persistence branch from 13d7dd1 to 46efc21 Compare November 19, 2024 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-blockchain-tree Related to sidechains, reorgs and pending blocks C-benchmark A change that impacts how or what we benchmark C-perf A change motivated by improving speed, memory usage or disk footprint
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant