[execution window] further optimizations #15053

bchocho · 2024-10-23T05:26:28Z

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-10-23T05:26:31Z

⏱️ 2h 3m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
test-target-determinator	28m	🟩 🟩 🟩 🟩 🟩 (+1 more)
rust-images / rust-all	14m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
rust-cargo-deny	9m	🟩 🟩 🟩 🟩 🟩
check-dynamic-deps	5m	🟩 🟩 🟩 🟩 🟩
semgrep/ci	3m	🟩 🟩 🟩 🟩 🟩 (+1 more)
general-lints	2m	🟩 🟩 🟩 🟩 🟩
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+1 more)
file_change_determinator	57s	🟩 🟩 🟩 🟩 🟩
rust-images / rust-all	52s	🟥

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

par iter for filter add a max txns slighly bump up pipeline pending latency Remove QS backpressure reinstate max_txns_to_execute fix index out of bounds fix applying max_user_txns_to_execute also constrain windowed blocks to max_block_txns filter expired txns execution backpressure with block gas limit fix waiting for window prepare vs execute use average instead of median Don't use block if gas used is less than the min block gas limit spread more re-instate quorum store backpressure naive? attempt at removing txns from proof manager on commit ordered / unordered caches Revert "naive? attempt at removing txns from proof manager on commit" This reverts commit df2743b. reduce backlog reduce further add a constant 10ms ignore proof backpressure for now Revert "ignore proof backpressure for now" This reverts commit cf8952f. clean up proof backpressure use subtracting sub make the proof backpressure smaller Add more logging hack up a way for vote backpressure not to panic Add sequential workload test run test on PR fix realistic_env_sequential_fairness_workload_sweep

github-actions · 2025-01-27T23:33:33Z

❌ Forge suite `realistic_env_max_load` failure on `771e3aaca6a1413ae5d092dc01424ece5e1c4897`

Workload                                      | submitted/s  | committed/s  | expired/s    | rejected/s   | chain txn/s  | latency      | p50 lat      | p90 lat      | p99 lat      | mempool->block | prop->order  | order->commit | actual dur   | idx_fn       | idx_cache    | idx_data    
0: ResourceGroupsGlobalWriteTag1KB[B:5.0k]    | 1979.92      | 1979.92      | 0.00         | 0.00         | 2983         | 2.069        | 2.000        | 2.400        | 2.800        | 0.627          | 0.303        | 0.246         | 51           | 0            | 0.000        | 0.000       
1: TokenV1FTMintAndTransfer[B:4.0k]           | 1273.18      | 1273.18      | 0.00         | 0.00         | 2309         | 2.507        | 2.400        | 2.800        | 3.100        | 0.550          | 0.315        | 0.971         | 51           | 0            | 0.000        | 0.000       
2: LiquidityPoolSwap[B:2.0k]                  | 857.08       | 857.08       | 0.00         | 0.00         | 1863         | 1.904        | 1.900        | 2.100        | 2.300        | 0.454          | 0.279        | 0.405         | 51           | 0            | 0.000        | 0.000       
3: NoOpFeePayer[B:5.0k]                       | 1942.41      | 1942.41      | 0.00         | 0.00         | 2970         | 2.113        | 2.000        | 2.500        | 2.900        | 0.577          | 0.322        | 0.500         | 51           | 0            | 0.000        | 0.000       
4: ModifyGlobalResource[B:4.0k]               | 2055.95      | 2055.95      | 0.00         | 0.00         | 3077         | 1.751        | 1.700        | 2.000        | 2.200        | 0.375          | 0.282        | 0.414         | 51           | 0            | 0.000        | 0.000       
5: SmartTablePicture1MWith256Change[B:0.0k]   | 7.00         | 7.00         | 0.00         | 0.00         | 987          | 3.478        | 3.300        | 4.000        | 4.100        | 0.705          | 0.394        | 1.945         | 51           | 0            | 0.000        | 0.000       
background traffic             | submitted/s  | committed/s  | expired/s    | rejected/s   | latency      | p50 lat      | p90 lat      | p99 lat     
background with traffic 0      | 995.98       | 995.98       | 0.00         | 0.00         | 1.730        | 1.800        | 1.800        | 2.400       
background with traffic 1      | 991.37       | 991.37       | 0.00         | 0.00         | 1.955        | 1.800        | 2.700        | 2.700       
background with traffic 2      | 1011.20      | 1011.20      | 0.00         | 0.00         | 1.603        | 1.500        | 1.800        | 2.100       
background with traffic 3      | 995.51       | 995.51       | 0.00         | 0.00         | 1.719        | 1.800        | 2.100        | 2.400       
background with traffic 4      | 995.67       | 995.67       | 0.00         | 0.00         | 1.631        | 1.500        | 1.800        | 2.100       
background with traffic 5      | 993.07       | 993.07       | 0.00         | 0.00         | 3.696        | 3.600        | 6.000        | 6.900       
Test Failed: Failed latency check, for ["P50 latency for background with traffic 5 is 3.6s and exceeds limit of 3s"]

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_latency
             at ./testsuite/forge/src/success_criteria.rs:564:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:267:9
   3: <aptos_testcases::load_vs_perf_benchmark::LoadVsPerfBenchmark as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/load_vs_perf_benchmark.rs:447:21
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   5: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:639:37
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
   8: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
   9: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  10: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  11: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  12: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  13: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  14: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  15: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  16: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  17: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  18: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  19: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  20: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  21: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  22: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  23: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  24: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  25: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  26: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  27: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  28: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  29: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  30: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  31: main
  32: __libc_start_main
  33: _start
Trailing Log Lines:
  28: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  29: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  30: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  31: main
  32: __libc_start_main
  33: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="e8131a4c-3190-4ae3-af8c-66010981be4f">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="CompositeNetworkTest(CpuChaosWrapper(network:multi-region-network-emulation(workload vs perf test))) with ">
            <failure message="Failed latency check, for [&quot;P50 latency for background with traffic 5 is 3.6s and exceeds limit of 3s&quot;]

Stack backtrace:
   0: anyhow::error::&lt;impl anyhow::Error&gt;::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_latency
             at ./testsuite/forge/src/success_criteria.rs:564:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:267:9
   3: &lt;aptos_testcases::load_vs_perf_benchmark::LoadVsPerfBenchmark as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/load_vs_perf_benchmark.rs:447:21
   4: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   5: &lt;aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:639:37
   6: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
   8: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
   9: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  10: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  11: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  12: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  13: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  14: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  15: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  16: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  17: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  18: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  19: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  20: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  21: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  22: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  23: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  24: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  25: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  26: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  27: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  28: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  29: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  30: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  31: main
  32: __libc_start_main
  33: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:381"},"thread_name":"main","hostname":"forge-e2e-pr-15053-1738019963-771e3aaca6a1413ae5d092dc01424ece5","timestamp":"2025-01-27T23:33:29.216231Z","message":"Deleting namespace forge-e2e-pr-15053: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:398"},"thread_name":"main","hostname":"forge-e2e-pr-15053-1738019963-771e3aaca6a1413ae5d092dc01424ece5","timestamp":"2025-01-27T23:33:29.216254Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-15053"}

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge177-0       1/1     Running     0          13m
aptos-node-0-validator-0                1/1     Running     0          13m
aptos-node-1-fullnode-eforge177-0       1/1     Running     0          13m
aptos-node-1-validator-0                1/1     Running     0          13m
aptos-node-2-fullnode-eforge177-0       1/1     Running     0          13m
aptos-node-2-validator-0                1/1     Running     0          13m
aptos-node-3-validator-0                1/1     Running     0          13m
aptos-node-4-validator-0                1/1     Running     0          13m
aptos-node-5-validator-0                1/1     Running     0          13m
aptos-node-6-validator-0                1/1     Running     0          13m
forge-testnet-deployer-vqdhj            0/1     Completed   0          13m
genesis-aptos-genesis-eforge177-qsjz6   0/1     Completed   0          13m

bchocho added the CICD:run-forge-e2e-perf Run the e2e perf forge only label Oct 23, 2024

bchocho changed the base branch from main to brian/exec-window-exec-optimize-2 October 23, 2024 06:01