Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: fix failing sim tests #6389

Merged
merged 12 commits into from
Feb 6, 2024
Merged

chore: fix failing sim tests #6389

merged 12 commits into from
Feb 6, 2024

Conversation

nazarhussain
Copy link
Contributor

Motivation

Make tests suit more stable.

Description

Fix failing sim tests.

Steps to test or reproduce

  • Run all tests

@nazarhussain nazarhussain requested a review from a team as a code owner February 2, 2024 15:03
@nazarhussain nazarhussain self-assigned this Feb 2, 2024
nazarhussain
nazarhussain commented Feb 2, 2024
View reviewed changes
packages/cli/test/utils/simulation/validator_clients/index.ts
const clId = `${options.id}-${client}`;

const opts: ValidatorGeneratorOptions = {
...options,
id: clId,
keys: options.keys ?? {type: "no-keys"},
genesisTime: options.genesisTime + forkConfig.GENESIS_DELAY,
genesisTime: options.genesisTime,
Copy link
Contributor Author

@nazarhussain nazarhussain Feb 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix the log time between validator and beacon.

@nazarhussain
Copy link
Contributor Author

nazarhussain commented Feb 2, 2024 •
edited
Loading

There is one more commit in this PR f11e452fbace8f70420f14861b15745d5ac4f4f0 but it's changes are not reflected in the Github UI.

@nazarhussain nazarhussain changed the base branch from unstable to stable February 2, 2024 15:09
@nazarhussain nazarhussain changed the base branch from stable to unstable February 2, 2024 15:09
nazarhussain
nazarhussain commented Feb 2, 2024
View reviewed changes
packages/cli/test/utils/simulation/validator_clients/lighthouse.ts
@@ -39,7 +39,6 @@ export const generateLighthouseValidatorNode: ValidatorNodeGenerator<ValidatorCl
"beacon-nodes": beaconUrls[0],
"debug-level": "debug",
"init-slashing-protection": null,
"allow-unsynced": null,
Copy link
Contributor Author

@nazarhussain nazarhussain Feb 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This param is not supported with new release of lighthouse. This is one clear example that we often don't even look into logs of sim tests to see why it's failing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the latest run on unstable branch, I don't see any error that would indicate that Lighthouse failed to start. Where would I find this, just in the log files? I feel like a service failing to start should be printed out in the console.

This param is not supported with new release of lighthouse

There was some discussion on this in #5942, I'd agree with Lion here that we use fixed versions everywhere, our CI needs to be deterministic.

And then we need to set up a CI run that tests against latest version

To ensure that Lodestar works with new releases there should be a new workflow that runs on a cron job (say, daily) against un-pinned version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the latest run on unstable branch, I don't see any error that would indicate that Lighthouse failed to start. Where would I find this, just in the log files? I feel like a service failing to start should be printed out in the console.

Lighthouse Validator could not start in the mixed clients tests. In the logs file saw the error.

@codecov Codecov
Copy link

codecov bot commented Feb 2, 2024 •
edited
Loading

Codecov Report

Merging #6389 (038e3e2) into unstable (00dfa63) will decrease coverage by 0.01%.
Report is 2 commits behind head on unstable.
The diff coverage is n/a.

❗ Current head 038e3e2 differs from pull request most recent head ebd884a. Consider uploading reports for the commit ebd884a to get more accurate results

Additional details and impacted files 
@@             Coverage Diff              @@
##           unstable    #6389      +/-   ##
============================================
- Coverage     60.15%   60.14%   -0.01%     
============================================
  Files           407      407              
  Lines         46512    46512              
  Branches       1551     1551              
============================================
- Hits          27978    27975       -3     
- Misses        18502    18505       +3     
  Partials         32       32

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 2, 2024 •
edited
Loading

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: 4182f4d Previous: b511d83 Ratio
pickEth1Vote - max votes 28.794 ms/op 7.9996 ms/op 3.60
forkChoice updateHead vc 600000 bc 64 eq 300000 73.587 ms/op 19.292 ms/op 3.81
BeaconState.hashTreeRoot - No change 862.00 ns/op 285.00 ns/op 3.02
Full benchmark results
Benchmark suite Current: 4182f4d Previous: b511d83 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 721.58 us/op 581.13 us/op 1.24
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 113.15 us/op 101.94 us/op 1.11
BLS verify - blst-native 1.3691 ms/op 1.3212 ms/op 1.04
BLS verifyMultipleSignatures 3 - blst-native 2.8893 ms/op 2.7855 ms/op 1.04
BLS verifyMultipleSignatures 8 - blst-native 6.3831 ms/op 6.1743 ms/op 1.03
BLS verifyMultipleSignatures 32 - blst-native 22.981 ms/op 22.341 ms/op 1.03
BLS verifyMultipleSignatures 64 - blst-native 45.225 ms/op 44.525 ms/op 1.02
BLS verifyMultipleSignatures 128 - blst-native 91.327 ms/op 94.350 ms/op 0.97
BLS deserializing 10000 signatures 1.0418 s/op 932.26 ms/op 1.12
BLS deserializing 100000 signatures 10.392 s/op 9.6014 s/op 1.08
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.6388 ms/op 1.3639 ms/op 1.20
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.8443 ms/op 1.5074 ms/op 1.22
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.7710 ms/op 3.0484 ms/op 0.91
BLS verifyMultipleSignatures - same message - 64 - blst-native 4.2774 ms/op 3.5817 ms/op 1.19
BLS verifyMultipleSignatures - same message - 128 - blst-native 6.8786 ms/op 8.1083 ms/op 0.85
BLS aggregatePubkeys 32 - blst-native 30.867 us/op 28.682 us/op 1.08
BLS aggregatePubkeys 128 - blst-native 117.44 us/op 101.86 us/op 1.15
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 83.238 ms/op 56.169 ms/op 1.48
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 98.030 ms/op 50.948 ms/op 1.92
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 76.608 ms/op 28.995 ms/op 2.64
getSlashingsAndExits - default max 447.86 us/op 185.94 us/op 2.41
getSlashingsAndExits - 2k 976.49 us/op 361.14 us/op 2.70
proposeBlockBody type=full, size=empty 8.6805 ms/op 5.1395 ms/op 1.69
isKnown best case - 1 super set check 757.00 ns/op 314.00 ns/op 2.41
isKnown normal case - 2 super set checks 711.00 ns/op 309.00 ns/op 2.30
isKnown worse case - 16 super set checks 780.00 ns/op 303.00 ns/op 2.57
CheckpointStateCache - add get delete 8.1890 us/op 5.2240 us/op 1.57
validate api signedAggregateAndProof - struct 4.0815 ms/op 2.7545 ms/op 1.48
validate gossip signedAggregateAndProof - struct 3.5324 ms/op 2.7523 ms/op 1.28
validate gossip attestation - vc 640000 1.6503 ms/op 1.3513 ms/op 1.22
batch validate gossip attestation - vc 640000 - chunk 32 213.78 us/op 162.20 us/op 1.32
batch validate gossip attestation - vc 640000 - chunk 64 203.46 us/op 142.37 us/op 1.43
batch validate gossip attestation - vc 640000 - chunk 128 179.85 us/op 135.53 us/op 1.33
batch validate gossip attestation - vc 640000 - chunk 256 194.11 us/op 125.89 us/op 1.54
pickEth1Vote - no votes 1.8504 ms/op 1.2242 ms/op 1.51
pickEth1Vote - max votes 28.794 ms/op 7.9996 ms/op 3.60
pickEth1Vote - Eth1Data hashTreeRoot value x2048 28.389 ms/op 16.100 ms/op 1.76
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 39.910 ms/op 21.790 ms/op 1.83
pickEth1Vote - Eth1Data fastSerialize value x2048 828.71 us/op 670.30 us/op 1.24
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.9606 ms/op 5.4942 ms/op 1.27
bytes32 toHexString 861.00 ns/op 536.00 ns/op 1.61
bytes32 Buffer.toString(hex) 361.00 ns/op 286.00 ns/op 1.26
bytes32 Buffer.toString(hex) from Uint8Array 711.00 ns/op 415.00 ns/op 1.71
bytes32 Buffer.toString(hex) + 0x 400.00 ns/op 286.00 ns/op 1.40
Object access 1 prop 0.30000 ns/op 0.15700 ns/op 1.91
Map access 1 prop 0.20600 ns/op 0.14000 ns/op 1.47
Object get x1000 10.987 ns/op 7.4290 ns/op 1.48
Map get x1000 1.0800 ns/op 0.72900 ns/op 1.48
Object set x1000 80.301 ns/op 49.571 ns/op 1.62
Map set x1000 66.881 ns/op 39.382 ns/op 1.70
Return object 10000 times 0.34640 ns/op 0.23440 ns/op 1.48
Throw Error 10000 times 5.0865 us/op 3.7928 us/op 1.34
fastMsgIdFn sha256 / 200 bytes 4.2700 us/op 3.2470 us/op 1.32
fastMsgIdFn h32 xxhash / 200 bytes 420.00 ns/op 281.00 ns/op 1.49
fastMsgIdFn h64 xxhash / 200 bytes 513.00 ns/op 340.00 ns/op 1.51
fastMsgIdFn sha256 / 1000 bytes 14.003 us/op 11.295 us/op 1.24
fastMsgIdFn h32 xxhash / 1000 bytes 561.00 ns/op 415.00 ns/op 1.35
fastMsgIdFn h64 xxhash / 1000 bytes 567.00 ns/op 449.00 ns/op 1.26
fastMsgIdFn sha256 / 10000 bytes 128.27 us/op 104.39 us/op 1.23
fastMsgIdFn h32 xxhash / 10000 bytes 2.3830 us/op 1.9630 us/op 1.21
fastMsgIdFn h64 xxhash / 10000 bytes 1.6500 us/op 1.3790 us/op 1.20
send data - 1000 256B messages 26.177 ms/op 24.084 ms/op 1.09
send data - 1000 512B messages 35.873 ms/op 27.263 ms/op 1.32
send data - 1000 1024B messages 51.999 ms/op 39.641 ms/op 1.31
send data - 1000 1200B messages 53.099 ms/op 39.747 ms/op 1.34
send data - 1000 2048B messages 61.828 ms/op 48.795 ms/op 1.27
send data - 1000 4096B messages 62.259 ms/op 42.573 ms/op 1.46
send data - 1000 16384B messages 138.60 ms/op 116.00 ms/op 1.19
send data - 1000 65536B messages 619.41 ms/op 459.48 ms/op 1.35
enrSubnets - fastDeserialize 64 bits 2.5430 us/op 1.3200 us/op 1.93
enrSubnets - ssz BitVector 64 bits 708.00 ns/op 497.00 ns/op 1.42
enrSubnets - fastDeserialize 4 bits 290.00 ns/op 187.00 ns/op 1.55
enrSubnets - ssz BitVector 4 bits 716.00 ns/op 468.00 ns/op 1.53
prioritizePeers score -10:0 att 32-0.1 sync 2-0 166.16 us/op 111.88 us/op 1.49
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 193.20 us/op 144.88 us/op 1.33
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 294.10 us/op 166.52 us/op 1.77
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 483.03 us/op 304.73 us/op 1.59
prioritizePeers score 0:0 att 64-1 sync 4-1 480.48 us/op 362.78 us/op 1.32
array of 16000 items push then shift 2.0896 us/op 1.6719 us/op 1.25
LinkedList of 16000 items push then shift 12.831 ns/op 9.1550 ns/op 1.40
array of 16000 items push then pop 135.71 ns/op 106.43 ns/op 1.28
LinkedList of 16000 items push then pop 11.374 ns/op 8.8160 ns/op 1.29
array of 24000 items push then shift 3.2542 us/op 2.6281 us/op 1.24
LinkedList of 24000 items push then shift 13.104 ns/op 9.2110 ns/op 1.42
array of 24000 items push then pop 196.95 ns/op 118.84 ns/op 1.66
LinkedList of 24000 items push then pop 12.820 ns/op 8.8380 ns/op 1.45
intersect bitArray bitLen 8 8.0610 ns/op 5.8810 ns/op 1.37
intersect array and set length 8 141.39 ns/op 64.743 ns/op 2.18
intersect bitArray bitLen 128 54.796 ns/op 35.933 ns/op 1.52
intersect array and set length 128 1.6902 us/op 898.52 ns/op 1.88
bitArray.getTrueBitIndexes() bitLen 128 2.9800 us/op 1.6640 us/op 1.79
bitArray.getTrueBitIndexes() bitLen 248 5.0880 us/op 2.9160 us/op 1.74
bitArray.getTrueBitIndexes() bitLen 512 9.4670 us/op 5.4930 us/op 1.72
Buffer.concat 32 items 1.8070 us/op 1.0310 us/op 1.75
Uint8Array.set 32 items 3.6100 us/op 1.9940 us/op 1.81
Set add up to 64 items then delete first 6.9654 us/op 4.5213 us/op 1.54
OrderedSet add up to 64 items then delete first 10.350 us/op 5.6444 us/op 1.83
Set add up to 64 items then delete last 7.0395 us/op 5.2442 us/op 1.34
OrderedSet add up to 64 items then delete last 10.727 us/op 6.3262 us/op 1.70
Set add up to 64 items then delete middle 7.7638 us/op 4.9606 us/op 1.57
OrderedSet add up to 64 items then delete middle 12.221 us/op 8.1332 us/op 1.50
Set add up to 128 items then delete first 15.144 us/op 10.843 us/op 1.40
OrderedSet add up to 128 items then delete first 20.516 us/op 14.156 us/op 1.45
Set add up to 128 items then delete last 15.329 us/op 10.474 us/op 1.46
OrderedSet add up to 128 items then delete last 23.393 us/op 12.194 us/op 1.92
Set add up to 128 items then delete middle 15.481 us/op 9.6065 us/op 1.61
OrderedSet add up to 128 items then delete middle 30.656 us/op 19.961 us/op 1.54
Set add up to 256 items then delete first 30.214 us/op 22.159 us/op 1.36
OrderedSet add up to 256 items then delete first 39.726 us/op 30.206 us/op 1.32
Set add up to 256 items then delete last 28.274 us/op 19.724 us/op 1.43
OrderedSet add up to 256 items then delete last 45.649 us/op 25.125 us/op 1.82
Set add up to 256 items then delete middle 28.095 us/op 18.769 us/op 1.50
OrderedSet add up to 256 items then delete middle 68.075 us/op 46.919 us/op 1.45
transfer serialized Status (84 B) 3.1160 us/op 1.7430 us/op 1.79
copy serialized Status (84 B) 2.3920 us/op 1.2830 us/op 1.86
transfer serialized SignedVoluntaryExit (112 B) 3.2450 us/op 1.8580 us/op 1.75
copy serialized SignedVoluntaryExit (112 B) 2.8080 us/op 1.4270 us/op 1.97
transfer serialized ProposerSlashing (416 B) 4.6000 us/op 2.3520 us/op 1.96
copy serialized ProposerSlashing (416 B) 3.5550 us/op 2.3320 us/op 1.52
transfer serialized Attestation (485 B) 4.0640 us/op 2.7190 us/op 1.49
copy serialized Attestation (485 B) 4.2610 us/op 2.1400 us/op 1.99
transfer serialized AttesterSlashing (33232 B) 6.3560 us/op 2.5840 us/op 2.46
copy serialized AttesterSlashing (33232 B) 16.102 us/op 7.4190 us/op 2.17
transfer serialized Small SignedBeaconBlock (128000 B) 5.9880 us/op 3.2540 us/op 1.84
copy serialized Small SignedBeaconBlock (128000 B) 67.293 us/op 24.000 us/op 2.80
transfer serialized Avg SignedBeaconBlock (200000 B) 8.8390 us/op 3.5130 us/op 2.52
copy serialized Avg SignedBeaconBlock (200000 B) 76.533 us/op 25.068 us/op 3.05
transfer serialized BlobsSidecar (524380 B) 9.9640 us/op 3.4260 us/op 2.91
copy serialized BlobsSidecar (524380 B) 227.51 us/op 97.154 us/op 2.34
transfer serialized Big SignedBeaconBlock (1000000 B) 10.474 us/op 3.6350 us/op 2.88
copy serialized Big SignedBeaconBlock (1000000 B) 378.08 us/op 170.74 us/op 2.21
pass gossip attestations to forkchoice per slot 8.3962 ms/op 4.3615 ms/op 1.93
forkChoice updateHead vc 100000 bc 64 eq 0 1.0758 ms/op 699.55 us/op 1.54
forkChoice updateHead vc 600000 bc 64 eq 0 11.261 ms/op 5.2638 ms/op 2.14
forkChoice updateHead vc 1000000 bc 64 eq 0 11.639 ms/op 7.5918 ms/op 1.53
forkChoice updateHead vc 600000 bc 320 eq 0 7.0919 ms/op 4.3343 ms/op 1.64
forkChoice updateHead vc 600000 bc 1200 eq 0 8.3350 ms/op 4.4933 ms/op 1.85
forkChoice updateHead vc 600000 bc 7200 eq 0 12.375 ms/op 6.0450 ms/op 2.05
forkChoice updateHead vc 600000 bc 64 eq 1000 24.329 ms/op 11.952 ms/op 2.04
forkChoice updateHead vc 600000 bc 64 eq 10000 16.631 ms/op 12.587 ms/op 1.32
forkChoice updateHead vc 600000 bc 64 eq 300000 73.587 ms/op 19.292 ms/op 3.81
computeDeltas 500000 validators 300 proto nodes 9.9875 ms/op 6.7752 ms/op 1.47
computeDeltas 500000 validators 1200 proto nodes 8.7178 ms/op 7.1986 ms/op 1.21
computeDeltas 500000 validators 7200 proto nodes 8.0750 ms/op 11.587 ms/op 0.70
computeDeltas 750000 validators 300 proto nodes 12.682 ms/op 14.016 ms/op 0.90
computeDeltas 750000 validators 1200 proto nodes 13.567 ms/op 11.250 ms/op 1.21
computeDeltas 750000 validators 7200 proto nodes 12.703 ms/op 10.671 ms/op 1.19
computeDeltas 1400000 validators 300 proto nodes 28.281 ms/op 21.698 ms/op 1.30
computeDeltas 1400000 validators 1200 proto nodes 27.035 ms/op 22.145 ms/op 1.22
computeDeltas 1400000 validators 7200 proto nodes 23.317 ms/op 21.340 ms/op 1.09
computeDeltas 2100000 validators 300 proto nodes 38.095 ms/op 31.638 ms/op 1.20
computeDeltas 2100000 validators 1200 proto nodes 35.139 ms/op 35.360 ms/op 0.99
computeDeltas 2100000 validators 7200 proto nodes 36.717 ms/op 33.301 ms/op 1.10
altair processAttestation - 250000 vs - 7PWei normalcase 3.9551 ms/op 3.5124 ms/op 1.13
altair processAttestation - 250000 vs - 7PWei worstcase 4.6192 ms/op 5.0127 ms/op 0.92
altair processAttestation - setStatus - 1/6 committees join 249.71 us/op 177.75 us/op 1.40
altair processAttestation - setStatus - 1/3 committees join 419.85 us/op 362.42 us/op 1.16
altair processAttestation - setStatus - 1/2 committees join 596.23 us/op 432.31 us/op 1.38
altair processAttestation - setStatus - 2/3 committees join 780.86 us/op 592.90 us/op 1.32
altair processAttestation - setStatus - 4/5 committees join 973.24 us/op 850.01 us/op 1.14
altair processAttestation - setStatus - 100% committees join 1.1754 ms/op 1.0752 ms/op 1.09
altair processBlock - 250000 vs - 7PWei normalcase 10.511 ms/op 14.841 ms/op 0.71
altair processBlock - 250000 vs - 7PWei normalcase hashState 44.131 ms/op 53.462 ms/op 0.83
altair processBlock - 250000 vs - 7PWei worstcase 49.872 ms/op 54.456 ms/op 0.92
altair processBlock - 250000 vs - 7PWei worstcase hashState 125.53 ms/op 127.60 ms/op 0.98
phase0 processBlock - 250000 vs - 7PWei normalcase 3.5616 ms/op 4.8616 ms/op 0.73
phase0 processBlock - 250000 vs - 7PWei worstcase 43.115 ms/op 43.676 ms/op 0.99
altair processEth1Data - 250000 vs - 7PWei normalcase 974.45 us/op 791.69 us/op 1.23
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 21.556 us/op 25.243 us/op 0.85
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 96.324 us/op 73.653 us/op 1.31
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 35.436 us/op 29.311 us/op 1.21
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 21.320 us/op 17.955 us/op 1.19
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 182.09 us/op 234.30 us/op 0.78
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.6301 ms/op 1.6380 ms/op 1.00
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 2.8535 ms/op 2.2153 ms/op 1.29
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 2.1197 ms/op 1.9838 ms/op 1.07
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 6.3495 ms/op 4.8918 ms/op 1.30
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 3.4970 ms/op 2.9178 ms/op 1.20
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 7.7502 ms/op 8.0022 ms/op 0.97
Tree 40 250000 create 918.68 ms/op 692.95 ms/op 1.33
Tree 40 250000 get(125000) 255.22 ns/op 216.86 ns/op 1.18
Tree 40 250000 set(125000) 2.8464 us/op 1.7186 us/op 1.66
Tree 40 250000 toArray() 33.881 ms/op 26.382 ms/op 1.28
Tree 40 250000 iterate all - toArray() + loop 31.817 ms/op 26.077 ms/op 1.22
Tree 40 250000 iterate all - get(i) 102.20 ms/op 79.970 ms/op 1.28
MutableVector 250000 create 20.323 ms/op 13.841 ms/op 1.47
MutableVector 250000 get(125000) 7.9340 ns/op 6.7130 ns/op 1.18
MutableVector 250000 set(125000) 776.33 ns/op 307.75 ns/op 2.52
MutableVector 250000 toArray() 5.6435 ms/op 4.1528 ms/op 1.36
MutableVector 250000 iterate all - toArray() + loop 5.6638 ms/op 4.2532 ms/op 1.33
MutableVector 250000 iterate all - get(i) 1.8678 ms/op 1.5663 ms/op 1.19
Array 250000 create 5.3680 ms/op 3.8613 ms/op 1.39
Array 250000 clone - spread 4.6921 ms/op 1.6772 ms/op 2.80
Array 250000 get(125000) 2.9110 ns/op 1.3670 ns/op 2.13
Array 250000 set(125000) 6.6460 ns/op 5.9540 ns/op 1.12
Array 250000 iterate all - loop 229.22 us/op 174.70 us/op 1.31
effectiveBalanceIncrements clone Uint8Array 300000 96.390 us/op 73.295 us/op 1.32
effectiveBalanceIncrements clone MutableVector 300000 621.00 ns/op 596.00 ns/op 1.04
effectiveBalanceIncrements rw all Uint8Array 300000 305.10 us/op 202.75 us/op 1.50
effectiveBalanceIncrements rw all MutableVector 300000 244.56 ms/op 149.13 ms/op 1.64
phase0 afterProcessEpoch - 250000 vs - 7PWei 145.65 ms/op 131.21 ms/op 1.11
phase0 beforeProcessEpoch - 250000 vs - 7PWei 80.610 ms/op 52.957 ms/op 1.52
altair processEpoch - mainnet_e81889 638.69 ms/op 545.85 ms/op 1.17
mainnet_e81889 - altair beforeProcessEpoch 116.28 ms/op 86.707 ms/op 1.34
mainnet_e81889 - altair processJustificationAndFinalization 22.389 us/op 14.994 us/op 1.49
mainnet_e81889 - altair processInactivityUpdates 8.6903 ms/op 6.2196 ms/op 1.40
mainnet_e81889 - altair processRewardsAndPenalties 84.167 ms/op 64.320 ms/op 1.31
mainnet_e81889 - altair processRegistryUpdates 6.6380 us/op 2.8370 us/op 2.34
mainnet_e81889 - altair processSlashings 1.1870 us/op 591.00 ns/op 2.01
mainnet_e81889 - altair processEth1DataReset 1.2630 us/op 850.00 ns/op 1.49
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.1135 ms/op 1.4618 ms/op 1.45
mainnet_e81889 - altair processSlashingsReset 8.5720 us/op 4.2760 us/op 2.00
mainnet_e81889 - altair processRandaoMixesReset 11.927 us/op 5.1900 us/op 2.30
mainnet_e81889 - altair processHistoricalRootsUpdate 1.4950 us/op 758.00 ns/op 1.97
mainnet_e81889 - altair processParticipationFlagUpdates 2.9710 us/op 1.8180 us/op 1.63
mainnet_e81889 - altair processSyncCommitteeUpdates 895.00 ns/op 785.00 ns/op 1.14
mainnet_e81889 - altair afterProcessEpoch 154.17 ms/op 124.43 ms/op 1.24
capella processEpoch - mainnet_e217614 2.2839 s/op 3.3101 s/op 0.69
mainnet_e217614 - capella beforeProcessEpoch 531.58 ms/op 759.48 ms/op 0.70
mainnet_e217614 - capella processJustificationAndFinalization 39.274 us/op 30.704 us/op 1.28
mainnet_e217614 - capella processInactivityUpdates 50.963 ms/op 31.331 ms/op 1.63
mainnet_e217614 - capella processRewardsAndPenalties 526.58 ms/op 520.08 ms/op 1.01
mainnet_e217614 - capella processRegistryUpdates 27.217 us/op 42.476 us/op 0.64
mainnet_e217614 - capella processSlashings 702.00 ns/op 1.2270 us/op 0.57
mainnet_e217614 - capella processEth1DataReset 697.00 ns/op 892.00 ns/op 0.78
mainnet_e217614 - capella processEffectiveBalanceUpdates 8.0220 ms/op 8.7749 ms/op 0.91
mainnet_e217614 - capella processSlashingsReset 5.7980 us/op 6.5340 us/op 0.89
mainnet_e217614 - capella processRandaoMixesReset 8.9360 us/op 9.6140 us/op 0.93
mainnet_e217614 - capella processHistoricalRootsUpdate 887.00 ns/op 923.00 ns/op 0.96
mainnet_e217614 - capella processParticipationFlagUpdates 2.3610 us/op 2.0530 us/op 1.15
mainnet_e217614 - capella afterProcessEpoch 350.39 ms/op 318.60 ms/op 1.10
phase0 processEpoch - mainnet_e58758 578.59 ms/op 483.24 ms/op 1.20
mainnet_e58758 - phase0 beforeProcessEpoch 203.79 ms/op 145.16 ms/op 1.40
mainnet_e58758 - phase0 processJustificationAndFinalization 21.150 us/op 20.542 us/op 1.03
mainnet_e58758 - phase0 processRewardsAndPenalties 66.153 ms/op 59.715 ms/op 1.11
mainnet_e58758 - phase0 processRegistryUpdates 16.106 us/op 16.259 us/op 0.99
mainnet_e58758 - phase0 processSlashings 906.00 ns/op 682.00 ns/op 1.33
mainnet_e58758 - phase0 processEth1DataReset 1.1990 us/op 437.00 ns/op 2.74
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.4125 ms/op 1.1795 ms/op 1.20
mainnet_e58758 - phase0 processSlashingsReset 6.8480 us/op 3.5160 us/op 1.95
mainnet_e58758 - phase0 processRandaoMixesReset 11.793 us/op 4.2090 us/op 2.80
mainnet_e58758 - phase0 processHistoricalRootsUpdate 980.00 ns/op 498.00 ns/op 1.97
mainnet_e58758 - phase0 processParticipationRecordUpdates 8.1430 us/op 3.9010 us/op 2.09
mainnet_e58758 - phase0 afterProcessEpoch 123.44 ms/op 97.762 ms/op 1.26
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.7854 ms/op 1.4372 ms/op 1.24
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.9688 ms/op 1.6939 ms/op 1.16
altair processInactivityUpdates - 250000 normalcase 39.977 ms/op 27.089 ms/op 1.48
altair processInactivityUpdates - 250000 worstcase 43.183 ms/op 26.569 ms/op 1.63
phase0 processRegistryUpdates - 250000 normalcase 24.226 us/op 16.258 us/op 1.49
phase0 processRegistryUpdates - 250000 badcase_full_deposits 814.94 us/op 538.47 us/op 1.51
phase0 processRegistryUpdates - 250000 worstcase 0.5 206.32 ms/op 144.71 ms/op 1.43
altair processRewardsAndPenalties - 250000 normalcase 75.653 ms/op 64.732 ms/op 1.17
altair processRewardsAndPenalties - 250000 worstcase 76.820 ms/op 60.652 ms/op 1.27
phase0 getAttestationDeltas - 250000 normalcase 16.606 ms/op 11.435 ms/op 1.45
phase0 getAttestationDeltas - 250000 worstcase 16.773 ms/op 13.457 ms/op 1.25
phase0 processSlashings - 250000 worstcase 121.26 us/op 120.38 us/op 1.01
altair processSyncCommitteeUpdates - 250000 234.73 ms/op 172.64 ms/op 1.36
BeaconState.hashTreeRoot - No change 862.00 ns/op 285.00 ns/op 3.02
BeaconState.hashTreeRoot - 1 full validator 203.93 us/op 182.90 us/op 1.12
BeaconState.hashTreeRoot - 32 full validator 2.0420 ms/op 1.9075 ms/op 1.07
BeaconState.hashTreeRoot - 512 full validator 27.926 ms/op 18.427 ms/op 1.52
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 268.80 us/op 173.67 us/op 1.55
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 3.2138 ms/op 2.2348 ms/op 1.44
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 43.806 ms/op 28.607 ms/op 1.53
BeaconState.hashTreeRoot - 1 balances 172.69 us/op 124.25 us/op 1.39
BeaconState.hashTreeRoot - 32 balances 1.6848 ms/op 1.2617 ms/op 1.34
BeaconState.hashTreeRoot - 512 balances 16.629 ms/op 13.669 ms/op 1.22
BeaconState.hashTreeRoot - 250000 balances 260.27 ms/op 196.39 ms/op 1.33
aggregationBits - 2048 els - zipIndexesInBitList 85.389 us/op 21.077 us/op 4.05
byteArrayEquals 32 108.67 ns/op 79.178 ns/op 1.37
Buffer.compare 32 66.898 ns/op 57.085 ns/op 1.17
byteArrayEquals 1024 2.5282 us/op 2.1502 us/op 1.18
Buffer.compare 1024 83.588 ns/op 73.402 ns/op 1.14
byteArrayEquals 16384 40.864 us/op 39.383 us/op 1.04
Buffer.compare 16384 357.06 ns/op 311.48 ns/op 1.15
byteArrayEquals 123687377 279.26 ms/op 311.36 ms/op 0.90
Buffer.compare 123687377 7.3101 ms/op 14.199 ms/op 0.51
byteArrayEquals 32 - diff last byte 83.928 ns/op 110.39 ns/op 0.76
Buffer.compare 32 - diff last byte 63.391 ns/op 64.277 ns/op 0.99
byteArrayEquals 1024 - diff last byte 2.2675 us/op 2.2030 us/op 1.03
Buffer.compare 1024 - diff last byte 78.708 ns/op 73.458 ns/op 1.07
byteArrayEquals 16384 - diff last byte 34.136 us/op 33.602 us/op 1.02
Buffer.compare 16384 - diff last byte 281.74 ns/op 277.46 ns/op 1.02
byteArrayEquals 123687377 - diff last byte 278.19 ms/op 293.66 ms/op 0.95
Buffer.compare 123687377 - diff last byte 9.3343 ms/op 10.625 ms/op 0.88
byteArrayEquals 32 - random bytes 7.9220 ns/op 8.2430 ns/op 0.96
Buffer.compare 32 - random bytes 85.413 ns/op 71.115 ns/op 1.20
byteArrayEquals 1024 - random bytes 7.5790 ns/op 7.5650 ns/op 1.00
Buffer.compare 1024 - random bytes 73.532 ns/op 68.977 ns/op 1.07
byteArrayEquals 16384 - random bytes 7.2220 ns/op 6.7450 ns/op 1.07
Buffer.compare 16384 - random bytes 75.156 ns/op 66.609 ns/op 1.13
byteArrayEquals 123687377 - random bytes 12.930 ns/op 13.340 ns/op 0.97
Buffer.compare 123687377 - random bytes 87.140 ns/op 78.170 ns/op 1.11
regular array get 100000 times 56.724 us/op 49.224 us/op 1.15
wrappedArray get 100000 times 52.520 us/op 48.673 us/op 1.08
arrayWithProxy get 100000 times 18.322 ms/op 16.520 ms/op 1.11
ssz.Root.equals 61.512 ns/op 59.515 ns/op 1.03
byteArrayEquals 59.441 ns/op 58.271 ns/op 1.02
Buffer.compare 12.564 ns/op 13.603 ns/op 0.92
shuffle list - 16384 els 7.8416 ms/op 7.8434 ms/op 1.00
shuffle list - 250000 els 110.95 ms/op 109.51 ms/op 1.01
processSlot - 1 slots 21.872 us/op 20.052 us/op 1.09
processSlot - 32 slots 4.1910 ms/op 4.9405 ms/op 0.85
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 66.515 ms/op 77.404 ms/op 0.86
getCommitteeAssignments - req 1 vs - 250000 vc 3.1032 ms/op 3.2194 ms/op 0.96
getCommitteeAssignments - req 100 vs - 250000 vc 3.9624 ms/op 4.4885 ms/op 0.88
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4857 ms/op 4.8954 ms/op 0.92
findModifiedValidators - 10000 modified validators 575.27 ms/op 879.23 ms/op 0.65
findModifiedValidators - 1000 modified validators 465.12 ms/op 689.40 ms/op 0.67
findModifiedValidators - 100 modified validators 475.71 ms/op 739.03 ms/op 0.64
findModifiedValidators - 10 modified validators 463.07 ms/op 670.39 ms/op 0.69
findModifiedValidators - 1 modified validators 490.12 ms/op 525.57 ms/op 0.93
findModifiedValidators - no difference 535.27 ms/op 588.36 ms/op 0.91
compare ViewDUs 6.3279 s/op 5.1235 s/op 1.24
compare each validator Uint8Array 2.2900 s/op 1.8032 s/op 1.27
compare ViewDU to Uint8Array 1.6873 s/op 1.4426 s/op 1.17
migrate state 1000000 validators, 24 modified, 0 new 1.1852 s/op 850.09 ms/op 1.39
migrate state 1000000 validators, 1700 modified, 1000 new 1.5683 s/op 1.2777 s/op 1.23
migrate state 1000000 validators, 3400 modified, 2000 new 1.9227 s/op 1.6582 s/op 1.16
migrate state 1500000 validators, 24 modified, 0 new 1.1795 s/op 986.37 ms/op 1.20
migrate state 1500000 validators, 1700 modified, 1000 new 1.2915 s/op 1.3907 s/op 0.93
migrate state 1500000 validators, 3400 modified, 2000 new 1.5887 s/op 1.5246 s/op 1.04
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.4800 ns/op 5.2400 ns/op 1.05
state getBlockRootAtSlot - 250000 vs - 7PWei 994.82 ns/op 650.51 ns/op 1.53
computeProposers - vc 250000 12.372 ms/op 9.7218 ms/op 1.27
computeEpochShuffling - vc 250000 124.05 ms/op 105.90 ms/op 1.17
getNextSyncCommittee - vc 250000 220.59 ms/op 163.39 ms/op 1.35
computeSigningRoot for AttestationData 43.611 us/op 30.320 us/op 1.44
hash AttestationData serialized data then Buffer.toString(base64) 2.9785 us/op 2.4508 us/op 1.22
toHexString serialized data 1.4109 us/op 1.2597 us/op 1.12
Buffer.toString(base64) 286.67 ns/op 252.66 ns/op 1.13

by benchmarkbot/action

@nflaig
Copy link
Member

nflaig commented Feb 2, 2024

There is one more commit in this PR f11e452fbace8f70420f14861b15745d5ac4f4f0 but it's changes are not reflected in the Github UI.

It was just a transient change on your branch as b766546 removed the line and f11e452 added it again.

see unstable branch

lodestar/packages/cli/test/utils/simulation/SimulationEnvironment.ts

Line 249 in 00dfa63

genesisState: this.genesisState,

wemeetagain
wemeetagain reviewed Feb 5, 2024
View reviewed changes
packages/beacon-node/src/network/libp2p/index.ts
@@ -1,7 +1,8 @@
import {PeerId} from "@libp2p/interface";
import {Registry} from "prom-client";
import {ENR} from "@chainsafe/enr";
import {identify} from "@libp2p/identify";
// TODO: We should use this fork until https://github.com/libp2p/js-libp2p/pull/2387
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We needed a way to disable identify-push, since on fork boundaries we run into this issue.

This fork just deletes the identify-push code.

@nazarhussain
Copy link
Contributor Author

https://github.com/ChainSafe/lodestar/actions/runs/7788167067?pr=6389

image

@nazarhussain nazarhussain merged commit c4bf385 into unstable Feb 6, 2024
13 of 15 checks passed
@nazarhussain nazarhussain deleted the nh/sim-test-check branch February 6, 2024 08:37
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.16.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig left review comments

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees

@nazarhussain nazarhussain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@nazarhussain @nflaig @wemeetagain