Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix] Fix exploration in losses #1898

Merged
merged 1 commit into from
Feb 11, 2024
Merged

[BugFix] Fix exploration in losses #1898

merged 1 commit into from
Feb 11, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 11, 2024

No description provided.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 11, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1898

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (3 Unrelated Failures)

As of commit 107e875 with merge base 2cfd9b6 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 11, 2024
@vmoens vmoens added the bug Something isn't working label Feb 11, 2024
@github-actions GitHub Actions
Copy link

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}10$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1413s 67.9473ms 14.7173 Ops/s 15.3519 Ops/s $\color{#d91a1a}-4.13\%$
test_sync 52.6561ms 37.5277ms 26.6470 Ops/s 28.2904 Ops/s $\textbf{\color{#d91a1a}-5.81\%}$
test_async 65.0110ms 33.4904ms 29.8593 Ops/s 29.0470 Ops/s $\color{#35bf28}+2.80\%$
test_simple 0.5111s 0.4427s 2.2590 Ops/s 2.2018 Ops/s $\color{#35bf28}+2.60\%$
test_transformed 0.6824s 0.6095s 1.6407 Ops/s 1.5946 Ops/s $\color{#35bf28}+2.89\%$
test_serial 1.4835s 1.4154s 0.7065 Ops/s 0.6734 Ops/s $\color{#35bf28}+4.92\%$
test_parallel 1.4701s 1.3929s 0.7179 Ops/s 0.6990 Ops/s $\color{#35bf28}+2.70\%$
test_step_mdp_speed[True-True-True-True-True] 0.2031ms 21.3703μs 46.7939 KOps/s 47.0020 KOps/s $\color{#d91a1a}-0.44\%$
test_step_mdp_speed[True-True-True-True-False] 40.9860μs 13.0284μs 76.7553 KOps/s 76.1730 KOps/s $\color{#35bf28}+0.76\%$
test_step_mdp_speed[True-True-True-False-True] 65.0910μs 12.5876μs 79.4432 KOps/s 80.4584 KOps/s $\color{#d91a1a}-1.26\%$
test_step_mdp_speed[True-True-True-False-False] 30.9280μs 7.7114μs 129.6775 KOps/s 129.5513 KOps/s $\color{#35bf28}+0.10\%$
test_step_mdp_speed[True-True-False-True-True] 81.2120μs 22.8887μs 43.6897 KOps/s 44.3705 KOps/s $\color{#d91a1a}-1.53\%$
test_step_mdp_speed[True-True-False-True-False] 37.0400μs 14.4031μs 69.4297 KOps/s 69.9679 KOps/s $\color{#d91a1a}-0.77\%$
test_step_mdp_speed[True-True-False-False-True] 66.5240μs 13.8223μs 72.3471 KOps/s 74.1988 KOps/s $\color{#d91a1a}-2.50\%$
test_step_mdp_speed[True-True-False-False-False] 40.1850μs 8.9120μs 112.2080 KOps/s 112.4166 KOps/s $\color{#d91a1a}-0.19\%$
test_step_mdp_speed[True-False-True-True-True] 68.2270μs 24.2081μs 41.3086 KOps/s 41.9621 KOps/s $\color{#d91a1a}-1.56\%$
test_step_mdp_speed[True-False-True-True-False] 50.2030μs 15.7105μs 63.6518 KOps/s 63.3226 KOps/s $\color{#35bf28}+0.52\%$
test_step_mdp_speed[True-False-True-False-True] 56.0640μs 13.8817μs 72.0371 KOps/s 74.1026 KOps/s $\color{#d91a1a}-2.79\%$
test_step_mdp_speed[True-False-True-False-False] 34.9260μs 9.0208μs 110.8548 KOps/s 111.6259 KOps/s $\color{#d91a1a}-0.69\%$
test_step_mdp_speed[True-False-False-True-True] 0.1109ms 25.3481μs 39.4507 KOps/s 39.9162 KOps/s $\color{#d91a1a}-1.17\%$
test_step_mdp_speed[True-False-False-True-False] 62.3160μs 17.0403μs 58.6845 KOps/s 58.8203 KOps/s $\color{#d91a1a}-0.23\%$
test_step_mdp_speed[True-False-False-False-True] 36.9090μs 15.1220μs 66.1287 KOps/s 67.9927 KOps/s $\color{#d91a1a}-2.74\%$
test_step_mdp_speed[True-False-False-False-False] 0.1021ms 10.3957μs 96.1933 KOps/s 99.2145 KOps/s $\color{#d91a1a}-3.05\%$
test_step_mdp_speed[False-True-True-True-True] 61.1150μs 24.1778μs 41.3602 KOps/s 41.4250 KOps/s $\color{#d91a1a}-0.16\%$
test_step_mdp_speed[False-True-True-True-False] 77.6550μs 15.5884μs 64.1504 KOps/s 63.5477 KOps/s $\color{#35bf28}+0.95\%$
test_step_mdp_speed[False-True-True-False-True] 71.7540μs 15.9348μs 62.7555 KOps/s 62.6597 KOps/s $\color{#35bf28}+0.15\%$
test_step_mdp_speed[False-True-True-False-False] 60.7140μs 9.9716μs 100.2848 KOps/s 99.1684 KOps/s $\color{#35bf28}+1.13\%$
test_step_mdp_speed[False-True-False-True-True] 37.6100μs 25.4613μs 39.2753 KOps/s 39.5820 KOps/s $\color{#d91a1a}-0.77\%$
test_step_mdp_speed[False-True-False-True-False] 74.6700μs 16.8224μs 59.4444 KOps/s 59.0523 KOps/s $\color{#35bf28}+0.66\%$
test_step_mdp_speed[False-True-False-False-True] 68.9590μs 17.3399μs 57.6705 KOps/s 58.9308 KOps/s $\color{#d91a1a}-2.14\%$
test_step_mdp_speed[False-True-False-False-False] 73.4570μs 11.1805μs 89.4412 KOps/s 86.3863 KOps/s $\color{#35bf28}+3.54\%$
test_step_mdp_speed[False-False-True-True-True] 70.5010μs 26.5804μs 37.6216 KOps/s 37.9666 KOps/s $\color{#d91a1a}-0.91\%$
test_step_mdp_speed[False-False-True-True-False] 76.4820μs 18.1420μs 55.1206 KOps/s 54.5332 KOps/s $\color{#35bf28}+1.08\%$
test_step_mdp_speed[False-False-True-False-True] 42.5390μs 17.3189μs 57.7405 KOps/s 57.8689 KOps/s $\color{#d91a1a}-0.22\%$
test_step_mdp_speed[False-False-True-False-False] 0.1101ms 11.2185μs 89.1384 KOps/s 87.5479 KOps/s $\color{#35bf28}+1.82\%$
test_step_mdp_speed[False-False-False-True-True] 75.6080μs 27.5128μs 36.3468 KOps/s 36.4625 KOps/s $\color{#d91a1a}-0.32\%$
test_step_mdp_speed[False-False-False-True-False] 71.5340μs 18.9668μs 52.7236 KOps/s 52.2393 KOps/s $\color{#35bf28}+0.93\%$
test_step_mdp_speed[False-False-False-False-True] 45.2150μs 18.1873μs 54.9834 KOps/s 55.2599 KOps/s $\color{#d91a1a}-0.50\%$
test_step_mdp_speed[False-False-False-False-False] 35.7270μs 12.3199μs 81.1693 KOps/s 80.9134 KOps/s $\color{#35bf28}+0.32\%$
test_values[generalized_advantage_estimate-True-True] 12.1366ms 9.4525ms 105.7921 Ops/s 105.0923 Ops/s $\color{#35bf28}+0.67\%$
test_values[vec_generalized_advantage_estimate-True-True] 36.5673ms 33.4585ms 29.8878 Ops/s 28.0404 Ops/s $\textbf{\color{#35bf28}+6.59\%}$
test_values[td0_return_estimate-False-False] 0.2524ms 0.1851ms 5.4029 KOps/s 5.5880 KOps/s $\color{#d91a1a}-3.31\%$
test_values[td1_return_estimate-False-False] 23.9410ms 23.2186ms 43.0690 Ops/s 42.4699 Ops/s $\color{#35bf28}+1.41\%$
test_values[vec_td1_return_estimate-False-False] 35.5925ms 33.5225ms 29.8307 Ops/s 28.1193 Ops/s $\textbf{\color{#35bf28}+6.09\%}$
test_values[td_lambda_return_estimate-True-False] 37.0139ms 33.6246ms 29.7401 Ops/s 30.1105 Ops/s $\color{#d91a1a}-1.23\%$
test_values[vec_td_lambda_return_estimate-True-False] 35.3377ms 33.5888ms 29.7718 Ops/s 28.1375 Ops/s $\textbf{\color{#35bf28}+5.81\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.1497ms 7.9795ms 125.3212 Ops/s 120.0107 Ops/s $\color{#35bf28}+4.43\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.3804ms 2.0080ms 498.0115 Ops/s 529.5910 Ops/s $\textbf{\color{#d91a1a}-5.96\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.5742ms 0.3464ms 2.8869 KOps/s 2.8102 KOps/s $\color{#35bf28}+2.73\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 42.1962ms 41.5399ms 24.0732 Ops/s 21.1898 Ops/s $\textbf{\color{#35bf28}+13.61\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.5934ms 3.0320ms 329.8145 Ops/s 325.6808 Ops/s $\color{#35bf28}+1.27\%$
test_dqn_speed 1.8605ms 1.3643ms 732.9833 Ops/s 718.6996 Ops/s $\color{#35bf28}+1.99\%$
test_ddpg_speed 3.9942ms 2.8586ms 349.8195 Ops/s 358.4081 Ops/s $\color{#d91a1a}-2.40\%$
test_sac_speed 11.9210ms 8.9596ms 111.6116 Ops/s 112.1395 Ops/s $\color{#d91a1a}-0.47\%$
test_redq_speed 17.4682ms 13.8697ms 72.0998 Ops/s 71.0920 Ops/s $\color{#35bf28}+1.42\%$
test_redq_deprec_speed 15.0641ms 14.4769ms 69.0754 Ops/s 67.3391 Ops/s $\color{#35bf28}+2.58\%$
test_td3_speed 9.8681ms 8.8267ms 113.2920 Ops/s 108.4744 Ops/s $\color{#35bf28}+4.44\%$
test_cql_speed 39.8579ms 37.4360ms 26.7122 Ops/s 26.5060 Ops/s $\color{#35bf28}+0.78\%$
test_a2c_speed 8.1414ms 7.6289ms 131.0807 Ops/s 127.6891 Ops/s $\color{#35bf28}+2.66\%$
test_ppo_speed 8.9556ms 7.8324ms 127.6752 Ops/s 125.3305 Ops/s $\color{#35bf28}+1.87\%$
test_reinforce_speed 8.0522ms 6.8175ms 146.6812 Ops/s 145.4443 Ops/s $\color{#35bf28}+0.85\%$
test_iql_speed 36.8346ms 33.4901ms 29.8596 Ops/s 29.5914 Ops/s $\color{#35bf28}+0.91\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.1562ms 2.8779ms 347.4773 Ops/s 330.7379 Ops/s $\textbf{\color{#35bf28}+5.06\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8278ms 0.5271ms 1.8971 KOps/s 1.8861 KOps/s $\color{#35bf28}+0.58\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7467ms 0.4991ms 2.0037 KOps/s 1.9958 KOps/s $\color{#35bf28}+0.40\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.3230ms 2.9422ms 339.8841 Ops/s 333.2569 Ops/s $\color{#35bf28}+1.99\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6297ms 0.5137ms 1.9466 KOps/s 1.9054 KOps/s $\color{#35bf28}+2.16\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6522ms 0.4899ms 2.0413 KOps/s 1.9913 KOps/s $\color{#35bf28}+2.51\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.1465ms 2.9925ms 334.1673 Ops/s 317.4279 Ops/s $\textbf{\color{#35bf28}+5.27\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8438ms 0.6477ms 1.5440 KOps/s 1.5179 KOps/s $\color{#35bf28}+1.72\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8774ms 0.6163ms 1.6226 KOps/s 1.5856 KOps/s $\color{#35bf28}+2.33\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.1879ms 2.8763ms 347.6724 Ops/s 320.5599 Ops/s $\textbf{\color{#35bf28}+8.46\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8539ms 0.5271ms 1.8971 KOps/s 1.8442 KOps/s $\color{#35bf28}+2.87\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7760ms 0.4987ms 2.0053 KOps/s 1.9528 KOps/s $\color{#35bf28}+2.68\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.2463ms 2.9387ms 340.2828 Ops/s 316.4096 Ops/s $\textbf{\color{#35bf28}+7.55\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6362ms 0.5127ms 1.9506 KOps/s 1.8691 KOps/s $\color{#35bf28}+4.36\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7117ms 0.4922ms 2.0319 KOps/s 1.9627 KOps/s $\color{#35bf28}+3.52\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.2567ms 2.9990ms 333.4428 Ops/s 307.5377 Ops/s $\textbf{\color{#35bf28}+8.42\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.7727ms 0.6491ms 1.5406 KOps/s 1.5206 KOps/s $\color{#35bf28}+1.31\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8272ms 0.6255ms 1.5988 KOps/s 1.5960 KOps/s $\color{#35bf28}+0.18\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1307s 9.1170ms 109.6858 Ops/s 112.7122 Ops/s $\color{#d91a1a}-2.69\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 17.4715ms 13.7189ms 72.8924 Ops/s 71.1260 Ops/s $\color{#35bf28}+2.48\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 6.2990ms 2.6589ms 376.0999 Ops/s 385.4751 Ops/s $\color{#d91a1a}-2.43\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1146s 10.6870ms 93.5712 Ops/s 114.4955 Ops/s $\textbf{\color{#d91a1a}-18.28\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 18.0012ms 13.8494ms 72.2054 Ops/s 61.3713 Ops/s $\textbf{\color{#35bf28}+17.65\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 3.7133ms 2.5348ms 394.5083 Ops/s 378.9099 Ops/s $\color{#35bf28}+4.12\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1151s 8.9918ms 111.2120 Ops/s 112.2667 Ops/s $\color{#d91a1a}-0.94\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 16.9962ms 13.9754ms 71.5541 Ops/s 69.7735 Ops/s $\color{#35bf28}+2.55\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 4.2217ms 2.8563ms 350.1075 Ops/s 355.5665 Ops/s $\color{#d91a1a}-1.54\%$

@github-actions GitHub Actions
Copy link

$\color{#35bf28}\textsf{\Large✔\kern{0.2cm}\normalsize OK}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}0$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1220s 0.1191s 8.3988 Ops/s 8.4871 Ops/s $\color{#d91a1a}-1.04\%$
test_sync 0.1716s 0.1028s 9.7315 Ops/s 9.7118 Ops/s $\color{#35bf28}+0.20\%$
test_async 0.2547s 92.0749ms 10.8607 Ops/s 10.9221 Ops/s $\color{#d91a1a}-0.56\%$
test_single_pixels 0.1434s 0.1429s 6.9994 Ops/s 7.3240 Ops/s $\color{#d91a1a}-4.43\%$
test_sync_pixels 83.3938ms 80.9820ms 12.3484 Ops/s 12.3094 Ops/s $\color{#35bf28}+0.32\%$
test_async_pixels 0.2024s 75.8110ms 13.1907 Ops/s 12.8617 Ops/s $\color{#35bf28}+2.56\%$
test_simple 0.9248s 0.8518s 1.1740 Ops/s 1.2032 Ops/s $\color{#d91a1a}-2.43\%$
test_transformed 1.1671s 1.0987s 0.9102 Ops/s 0.9299 Ops/s $\color{#d91a1a}-2.12\%$
test_serial 2.5579s 2.4822s 0.4029 Ops/s 0.4164 Ops/s $\color{#d91a1a}-3.26\%$
test_parallel 2.3045s 2.1311s 0.4692 Ops/s 0.4800 Ops/s $\color{#d91a1a}-2.23\%$
test_step_mdp_speed[True-True-True-True-True] 0.3445ms 32.7022μs 30.5790 KOps/s 30.9501 KOps/s $\color{#d91a1a}-1.20\%$
test_step_mdp_speed[True-True-True-True-False] 0.2035ms 19.9219μs 50.1961 KOps/s 51.6945 KOps/s $\color{#d91a1a}-2.90\%$
test_step_mdp_speed[True-True-True-False-True] 40.8800μs 18.9939μs 52.6485 KOps/s 54.0855 KOps/s $\color{#d91a1a}-2.66\%$
test_step_mdp_speed[True-True-True-False-False] 0.2206ms 11.4126μs 87.6228 KOps/s 89.7045 KOps/s $\color{#d91a1a}-2.32\%$
test_step_mdp_speed[True-True-False-True-True] 0.2402ms 35.0614μs 28.5214 KOps/s 29.0250 KOps/s $\color{#d91a1a}-1.74\%$
test_step_mdp_speed[True-True-False-True-False] 0.2180ms 22.0960μs 45.2571 KOps/s 46.9955 KOps/s $\color{#d91a1a}-3.70\%$
test_step_mdp_speed[True-True-False-False-True] 40.5500μs 20.9133μs 47.8164 KOps/s 49.3142 KOps/s $\color{#d91a1a}-3.04\%$
test_step_mdp_speed[True-True-False-False-False] 38.2910μs 13.2934μs 75.2255 KOps/s 76.7469 KOps/s $\color{#d91a1a}-1.98\%$
test_step_mdp_speed[True-False-True-True-True] 0.2461ms 37.3609μs 26.7659 KOps/s 27.4808 KOps/s $\color{#d91a1a}-2.60\%$
test_step_mdp_speed[True-False-True-True-False] 44.6300μs 23.9031μs 41.8355 KOps/s 43.3863 KOps/s $\color{#d91a1a}-3.57\%$
test_step_mdp_speed[True-False-True-False-True] 0.2206ms 20.8126μs 48.0478 KOps/s 49.6612 KOps/s $\color{#d91a1a}-3.25\%$
test_step_mdp_speed[True-False-True-False-False] 0.1999ms 13.2051μs 75.7286 KOps/s 76.4928 KOps/s $\color{#d91a1a}-1.00\%$
test_step_mdp_speed[True-False-False-True-True] 0.2363ms 38.9366μs 25.6828 KOps/s 26.4882 KOps/s $\color{#d91a1a}-3.04\%$
test_step_mdp_speed[True-False-False-True-False] 44.4910μs 25.6701μs 38.9558 KOps/s 39.4336 KOps/s $\color{#d91a1a}-1.21\%$
test_step_mdp_speed[True-False-False-False-True] 0.2097ms 22.6637μs 44.1234 KOps/s 45.8811 KOps/s $\color{#d91a1a}-3.83\%$
test_step_mdp_speed[True-False-False-False-False] 0.1983ms 15.0006μs 66.6640 KOps/s 67.1000 KOps/s $\color{#d91a1a}-0.65\%$
test_step_mdp_speed[False-True-True-True-True] 0.2278ms 36.9768μs 27.0440 KOps/s 27.8128 KOps/s $\color{#d91a1a}-2.76\%$
test_step_mdp_speed[False-True-True-True-False] 46.3810μs 23.6905μs 42.2110 KOps/s 43.1153 KOps/s $\color{#d91a1a}-2.10\%$
test_step_mdp_speed[False-True-True-False-True] 0.2408ms 24.9421μs 40.0929 KOps/s 41.9149 KOps/s $\color{#d91a1a}-4.35\%$
test_step_mdp_speed[False-True-True-False-False] 0.2092ms 15.1846μs 65.8563 KOps/s 67.9268 KOps/s $\color{#d91a1a}-3.05\%$
test_step_mdp_speed[False-True-False-True-True] 70.1910μs 38.8621μs 25.7320 KOps/s 26.2549 KOps/s $\color{#d91a1a}-1.99\%$
test_step_mdp_speed[False-True-False-True-False] 0.2373ms 26.0036μs 38.4563 KOps/s 39.5994 KOps/s $\color{#d91a1a}-2.89\%$
test_step_mdp_speed[False-True-False-False-True] 0.2407ms 26.3653μs 37.9286 KOps/s 39.1062 KOps/s $\color{#d91a1a}-3.01\%$
test_step_mdp_speed[False-True-False-False-False] 47.1500μs 17.0322μs 58.7125 KOps/s 60.2764 KOps/s $\color{#d91a1a}-2.59\%$
test_step_mdp_speed[False-False-True-True-True] 0.2540ms 40.8470μs 24.4816 KOps/s 25.2693 KOps/s $\color{#d91a1a}-3.12\%$
test_step_mdp_speed[False-False-True-True-False] 0.2229ms 27.7949μs 35.9779 KOps/s 37.2718 KOps/s $\color{#d91a1a}-3.47\%$
test_step_mdp_speed[False-False-True-False-True] 0.2170ms 26.3697μs 37.9223 KOps/s 38.7011 KOps/s $\color{#d91a1a}-2.01\%$
test_step_mdp_speed[False-False-True-False-False] 40.4600μs 16.9887μs 58.8627 KOps/s 60.9117 KOps/s $\color{#d91a1a}-3.36\%$
test_step_mdp_speed[False-False-False-True-True] 0.2410ms 42.2175μs 23.6868 KOps/s 24.5001 KOps/s $\color{#d91a1a}-3.32\%$
test_step_mdp_speed[False-False-False-True-False] 0.2540ms 29.5045μs 33.8931 KOps/s 34.8497 KOps/s $\color{#d91a1a}-2.74\%$
test_step_mdp_speed[False-False-False-False-True] 0.2301ms 28.0387μs 35.6649 KOps/s 36.6964 KOps/s $\color{#d91a1a}-2.81\%$
test_step_mdp_speed[False-False-False-False-False] 0.2072ms 18.5972μs 53.7714 KOps/s 54.4902 KOps/s $\color{#d91a1a}-1.32\%$
test_values[generalized_advantage_estimate-True-True] 25.1558ms 24.4674ms 40.8707 Ops/s 42.1100 Ops/s $\color{#d91a1a}-2.94\%$
test_values[vec_generalized_advantage_estimate-True-True] 88.9054ms 3.3438ms 299.0566 Ops/s 308.3832 Ops/s $\color{#d91a1a}-3.02\%$
test_values[td0_return_estimate-False-False] 0.1004ms 62.4579μs 16.0108 KOps/s 16.5428 KOps/s $\color{#d91a1a}-3.22\%$
test_values[td1_return_estimate-False-False] 56.9652ms 52.2007ms 19.1568 Ops/s 19.6225 Ops/s $\color{#d91a1a}-2.37\%$
test_values[vec_td1_return_estimate-False-False] 2.0859ms 1.7650ms 566.5786 Ops/s 570.5089 Ops/s $\color{#d91a1a}-0.69\%$
test_values[td_lambda_return_estimate-True-False] 83.7220ms 83.2368ms 12.0139 Ops/s 12.2381 Ops/s $\color{#d91a1a}-1.83\%$
test_values[vec_td_lambda_return_estimate-True-False] 3.9963ms 1.7989ms 555.9050 Ops/s 558.5684 Ops/s $\color{#d91a1a}-0.48\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 23.7822ms 23.3319ms 42.8597 Ops/s 43.8335 Ops/s $\color{#d91a1a}-2.22\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.8984ms 0.7044ms 1.4197 KOps/s 1.4213 KOps/s $\color{#d91a1a}-0.12\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7209ms 0.6546ms 1.5277 KOps/s 1.5459 KOps/s $\color{#d91a1a}-1.17\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.6012ms 1.4558ms 686.9130 Ops/s 691.6232 Ops/s $\color{#d91a1a}-0.68\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9486ms 0.6752ms 1.4809 KOps/s 1.5080 KOps/s $\color{#d91a1a}-1.80\%$
test_dqn_speed 8.0398ms 1.4647ms 682.7569 Ops/s 688.2164 Ops/s $\color{#d91a1a}-0.79\%$
test_ddpg_speed 4.6386ms 2.8967ms 345.2192 Ops/s 357.4223 Ops/s $\color{#d91a1a}-3.41\%$
test_sac_speed 9.3397ms 8.8188ms 113.3947 Ops/s 116.6904 Ops/s $\color{#d91a1a}-2.82\%$
test_redq_speed 11.6263ms 10.8295ms 92.3403 Ops/s 91.9025 Ops/s $\color{#35bf28}+0.48\%$
test_redq_deprec_speed 13.2042ms 12.3157ms 81.1970 Ops/s 84.7741 Ops/s $\color{#d91a1a}-4.22\%$
test_td3_speed 17.6260ms 8.9343ms 111.9288 Ops/s 113.8952 Ops/s $\color{#d91a1a}-1.73\%$
test_cql_speed 27.0829ms 25.9469ms 38.5402 Ops/s 37.8449 Ops/s $\color{#35bf28}+1.84\%$
test_a2c_speed 5.9837ms 5.2294ms 191.2273 Ops/s 181.6829 Ops/s $\textbf{\color{#35bf28}+5.25\%}$
test_ppo_speed 6.3894ms 5.4922ms 182.0774 Ops/s 174.7492 Ops/s $\color{#35bf28}+4.19\%$
test_reinforce_speed 6.1430ms 4.2650ms 234.4653 Ops/s 224.7197 Ops/s $\color{#35bf28}+4.34\%$
test_iql_speed 21.1694ms 19.5621ms 51.1193 Ops/s 49.9260 Ops/s $\color{#35bf28}+2.39\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.7855ms 3.5559ms 281.2201 Ops/s 283.4109 Ops/s $\color{#d91a1a}-0.77\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7895ms 0.5906ms 1.6932 KOps/s 1.7091 KOps/s $\color{#d91a1a}-0.93\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8275ms 0.5689ms 1.7578 KOps/s 1.7557 KOps/s $\color{#35bf28}+0.12\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.8745ms 3.5918ms 278.4098 Ops/s 283.6886 Ops/s $\color{#d91a1a}-1.86\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8050ms 0.5672ms 1.7631 KOps/s 1.7346 KOps/s $\color{#35bf28}+1.65\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6822ms 0.5424ms 1.8435 KOps/s 1.7886 KOps/s $\color{#35bf28}+3.07\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.9309ms 3.7131ms 269.3203 Ops/s 271.6676 Ops/s $\color{#d91a1a}-0.86\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.0061ms 0.6951ms 1.4387 KOps/s 1.4017 KOps/s $\color{#35bf28}+2.64\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8110ms 0.6674ms 1.4983 KOps/s 1.4890 KOps/s $\color{#35bf28}+0.63\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.7234ms 3.5571ms 281.1313 Ops/s 282.1640 Ops/s $\color{#d91a1a}-0.37\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.9016ms 0.5905ms 1.6936 KOps/s 1.7687 KOps/s $\color{#d91a1a}-4.25\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8059ms 0.5651ms 1.7697 KOps/s 1.8308 KOps/s $\color{#d91a1a}-3.34\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.9221ms 3.6007ms 277.7237 Ops/s 281.5873 Ops/s $\color{#d91a1a}-1.37\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.9117ms 0.5831ms 1.7148 KOps/s 1.7247 KOps/s $\color{#d91a1a}-0.57\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7210ms 0.5605ms 1.7840 KOps/s 1.7745 KOps/s $\color{#35bf28}+0.54\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.9453ms 3.7395ms 267.4130 Ops/s 272.2990 Ops/s $\color{#d91a1a}-1.79\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9682ms 0.7112ms 1.4061 KOps/s 1.3934 KOps/s $\color{#35bf28}+0.92\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9434ms 0.6949ms 1.4390 KOps/s 1.4396 KOps/s $\color{#d91a1a}-0.04\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1236s 10.2655ms 97.4141 Ops/s 97.9701 Ops/s $\color{#d91a1a}-0.57\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 18.8967ms 16.4628ms 60.7428 Ops/s 61.5145 Ops/s $\color{#d91a1a}-1.25\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 7.2047ms 3.2320ms 309.4105 Ops/s 320.0474 Ops/s $\color{#d91a1a}-3.32\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1162s 9.9776ms 100.2243 Ops/s 100.6688 Ops/s $\color{#d91a1a}-0.44\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 18.7660ms 16.3905ms 61.0108 Ops/s 54.5046 Ops/s $\textbf{\color{#35bf28}+11.94\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 7.3007ms 3.1594ms 316.5199 Ops/s 326.0892 Ops/s $\color{#d91a1a}-2.93\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1196s 10.3549ms 96.5730 Ops/s 98.1502 Ops/s $\color{#d91a1a}-1.61\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 19.0120ms 16.6018ms 60.2345 Ops/s 60.3848 Ops/s $\color{#d91a1a}-0.25\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 8.3218ms 3.4513ms 289.7424 Ops/s 296.0346 Ops/s $\color{#d91a1a}-2.13\%$

@vmoens vmoens merged commit 1bd5ec6 into main Feb 11, 2024
64 of 67 checks passed
@vmoens vmoens deleted the fix-loss-exploration branch February 27, 2024 00:47
vmoens added a commit that referenced this pull request Feb 28, 2024
@vmoens
[BugFix] Fix exploration in losses (#1898)
c75a7b7
@vmoens vmoens mentioned this pull request Feb 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vmoens @facebook-github-bot