[storage] fix the replay-verify stuck

[areshand]

Description

The problem is the 2nd retry was the same process with old cache or lock on DB. Now, do the retry outside and can still leverage the resumable replay without redundant work。

using old versioned node cache will cause all non-exe thread panic and stuck forever.
also the lock on ledger db cannot be dropped even when the AptosDb goes out of scope in the 2nd iteration.

Test Plan

https://github.com/aptos-labs/aptos-core/actions/runs/6937303262

[msmouse]

first print will be "2th"? 😂

And I mean.. res doesn't need to be kept across iterations if we add the return under the code == 1 branch. (updated my previous comment to use while True:, which makes more sense.

[areshand]

there is some poetry code inspection in CI/CD and enforce there is a return from this function, since it doesn't know if code == 1 would eventually return. updated i = 0

[msmouse]

okay.. then

1. print("try {i}") should be before func(...)
2. looks nicer to do:

(partition_number, code, msg) = (None, None, None)

for i in range(1, NUM_TRIES+1):
    print("try {i}")
    (partition_number, code, msg) = func(*args, **kwargs)
    if code != 1:
        break

return (partition_number, code, msg)

[areshand]

looks beautify

[msmouse]

Accidental approval, sorry.

[github-actions]

✅ Forge suite compat success on aptos-node-v1.7.3 ==> 90bc3c610554c4dad55b9b81a09f717c1d89580c

Compatibility test results for aptos-node-v1.7.3 ==> 90bc3c610554c4dad55b9b81a09f717c1d89580c (PR)
1. Check liveness of validators at old version: aptos-node-v1.7.3
compatibility::simple-validator-upgrade::liveness-check : committed: 4994 txn/s, latency: 6474 ms, (p50: 6300 ms, p90: 9800 ms, p99: 13900 ms), latency samples: 189780
2. Upgrading first Validator to new version: 90bc3c610554c4dad55b9b81a09f717c1d89580c
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1786 txn/s, latency: 15808 ms, (p50: 19300 ms, p90: 21800 ms, p99: 22500 ms), latency samples: 92880
3. Upgrading rest of first batch to new version: 90bc3c610554c4dad55b9b81a09f717c1d89580c
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1837 txn/s, latency: 15899 ms, (p50: 19200 ms, p90: 22000 ms, p99: 22300 ms), latency samples: 91880
4. upgrading second batch to new version: 90bc3c610554c4dad55b9b81a09f717c1d89580c
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3605 txn/s, latency: 8704 ms, (p50: 9900 ms, p90: 11400 ms, p99: 12100 ms), latency samples: 144200
5. check swarm health
Compatibility test for aptos-node-v1.7.3 ==> 90bc3c610554c4dad55b9b81a09f717c1d89580c passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite realistic_env_max_load success on 90bc3c610554c4dad55b9b81a09f717c1d89580c

two traffics test: inner traffic : committed: 7500 txn/s, latency: 5209 ms, (p50: 5100 ms, p90: 6300 ms, p99: 11400 ms), latency samples: 3255200
two traffics test : committed: 100 txn/s, latency: 2403 ms, (p50: 2300 ms, p90: 2800 ms, p99: 4000 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.204, avg: 0.196", "QsPosToProposal: max: 0.208, avg: 0.177", "ConsensusProposalToOrdered: max: 0.691, avg: 0.649", "ConsensusOrderedToCommit: max: 0.577, avg: 0.545", "ConsensusProposalToCommit: max: 1.266, avg: 1.194"]
Max round gap was 1 [limit 4] at version 1562403. Max no progress secs was 3.987601 [limit 10] at version 1562403.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on aptos-node-v1.7.3 ==> 90bc3c610554c4dad55b9b81a09f717c1d89580c

Compatibility test results for aptos-node-v1.7.3 ==> 90bc3c610554c4dad55b9b81a09f717c1d89580c (PR)
Upgrade the nodes to version: 90bc3c610554c4dad55b9b81a09f717c1d89580c
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 4215 txn/s, latency: 4818 ms, (p50: 4800 ms, p90: 7800 ms, p99: 8400 ms), latency samples: 244520
5. check swarm health
Compatibility test for aptos-node-v1.7.3 ==> 90bc3c610554c4dad55b9b81a09f717c1d89580c passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking