[RLlib] Cleanup examples folder vol. 23: Add example script for custom metrics on `EnvRunners` (using `MetricsLogger` API). #47969

sven1977 · 2024-10-10T12:04:14Z

Cleanup examples folder vol. 23: Add example script for custom metrics on EnvRunners (using MetricsLogger API).

Activated in CI
Example creates 2D heatmap for pacman per episode, logs a custom max. and mean metric (per episode over a sliding window), and the number of lives as EMA-smoothed.

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_23_custom_metrics_and_callbacks

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_23_custom_metrics_and_callbacks

Signed-off-by: sven1977 <[email protected]>

simonsays1980

LGTM. Awesome example. Documenting the MetricsLogger´ and showing users how to use it correctly is so important. I think still that this is one of the complexest parts of RLlib` and gave some suggestions for further cases.

simonsays1980 · 2024-10-10T12:16:03Z

rllib/examples/metrics/custom_metrics_in_env_runners.py

+    - the mean distance travelled by MsPacman per episode (over an infinite window).
+    - the number of lifes of MsPacman EMA-smoothed over time.
+
+    This callback can be setup to only log stats on certain EnvRunner indices through


What happens if an EnvRunner crashes?

Good question. The index first goes out of commission (other EnvRunners will NOT change their indices b/c of another EnvRunner's crash), but only until the actor is automatically restarted. The latter only happens if config.recreate_failed_env_runners=True, of course.

simonsays1980 · 2024-10-10T12:22:23Z

rllib/examples/metrics/custom_metrics_in_env_runners.py

+custom `Algorithm.training_step()` methods, custom loss functions, custom callbacks,
+and custom EnvRunners.
+
+This example:


Awesome example!! This shows a lot how to use the MetricsLogger.

To increase complexity, we could:

Run this only in evaluation and run evaluation each nth training step.

Reset some metrics in between.

Log two group of metrics (for two groups of env-runners).

Good point! There are two more PRs in flight: MetricsLogger on algorithm.training_step and MetricsLogger inside loss function. A third one could be: MetricsLogger only on eval env runners 🙌

simonsays1980 · 2024-10-10T12:23:01Z

rllib/examples/metrics/custom_metrics_in_env_runners.py

+How to run this script
+----------------------
+`python [script file name].py --enable-new-api-stack --wandb-key [your WandB key]
+--wandb-projecy [some project name]`


"--wandb-projecy" -> "--wandb-project"

The same options occur below in "For logging ..."

Yeah, that's ok. The statement below is the generic enable-logging statement, the one above are the recommended args for this particular script.

simonsays1980 · 2024-10-10T12:25:31Z

rllib/examples/metrics/custom_metrics_in_env_runners.py

+            episode.get_infos(-1)["lives"],
+            reduce="mean",  # <- default (must be "mean" for EMA smothing)
+            ema_coeff=0.01,  # <- default EMA coefficient (`window` must be None)
+        )


Maybe add a comment in regard to metrics_logger.reduce when to use it and why not here.

simonsays1980 · 2024-10-10T12:26:43Z

rllib/utils/metrics/metrics_logger.py

+                self.stats = tree.map_structure_with_path(_reduce, stats_to_return)
+        # Provide proper error message if reduction fails due to bad data.
+        except Exception as e:
+            raise ValueError(


Here I wonder what happens when a user resets in between two training steps a key.

Great question. Users should - ideally - never use this API and let RLlib determine, when to call .reduce() on a MetricsLogger object. They should only call .peek() to see individual reduced stats at the moment w/o actually altering/reducing the underlying stats object.

Maybe I need to make this more clear in the docstring ...

Signed-off-by: sven1977 <[email protected]>

…and_callbacks Signed-off-by: Sven Mika <[email protected]>

…and_callbacks

…nup_examples_folder_23_custom_metrics_and_callbacks Signed-off-by: sven1977 <[email protected]> # Conflicts: # rllib/BUILD

…m_metrics_and_callbacks' into cleanup_examples_folder_23_custom_metrics_and_callbacks

sven1977 added 4 commits October 9, 2024 16:23

wip

06a01d6

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

4401c06

…nup_examples_folder_23_custom_metrics_and_callbacks

wip

ff9391b

Signed-off-by: sven1977 <[email protected]>

wip

d0be670

Signed-off-by: sven1977 <[email protected]>

sven1977 requested a review from simonsays1980 as a code owner October 10, 2024 12:04

sven1977 assigned simonsays1980 Oct 10, 2024

sven1977 added 2 commits October 10, 2024 14:05

Merge branch 'master' of https://github.com/ray-project/ray into clea…

1fca4d9

…nup_examples_folder_23_custom_metrics_and_callbacks

wip

70802ef

Signed-off-by: sven1977 <[email protected]>

simonsays1980 approved these changes Oct 10, 2024

View reviewed changes

fix

af6753e

Signed-off-by: sven1977 <[email protected]>

sven1977 enabled auto-merge (squash) October 10, 2024 13:02

github-actions bot disabled auto-merge October 10, 2024 13:02

github-actions bot added the go add ONLY when ready to merge, run all tests label Oct 10, 2024

sven1977 enabled auto-merge (squash) October 10, 2024 13:29

fix

25021fe

Signed-off-by: sven1977 <[email protected]>

github-actions bot disabled auto-merge October 10, 2024 14:38

sven1977 enabled auto-merge (squash) October 10, 2024 14:51

sven1977 added rllib RLlib related issues rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples rllib-newstack rllib-envrunners Issues around the sampling backend of RLlib labels Oct 10, 2024

Merge branch 'master' into cleanup_examples_folder_23_custom_metrics_…

0b9d7c5

…and_callbacks Signed-off-by: Sven Mika <[email protected]>

github-actions bot disabled auto-merge October 10, 2024 16:19

sven1977 enabled auto-merge (squash) October 10, 2024 16:20

Merge branch 'master' into cleanup_examples_folder_23_custom_metrics_…

b6bd489

…and_callbacks

github-actions bot disabled auto-merge October 11, 2024 10:28

sven1977 enabled auto-merge (squash) October 11, 2024 10:36

sven1977 added 2 commits October 11, 2024 14:59

Merge branch 'master' of https://github.com/ray-project/ray into clea…

5560732

…nup_examples_folder_23_custom_metrics_and_callbacks Signed-off-by: sven1977 <[email protected]> # Conflicts: # rllib/BUILD

Merge remote-tracking branch 'origin/cleanup_examples_folder_23_custo…

6380b77

…m_metrics_and_callbacks' into cleanup_examples_folder_23_custom_metrics_and_callbacks

github-actions bot disabled auto-merge October 11, 2024 13:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] Cleanup examples folder vol. 23: Add example script for custom metrics on `EnvRunners` (using `MetricsLogger` API). #47969

[RLlib] Cleanup examples folder vol. 23: Add example script for custom metrics on `EnvRunners` (using `MetricsLogger` API). #47969

sven1977 commented Oct 10, 2024 •

edited

Loading

simonsays1980 left a comment

simonsays1980 Oct 10, 2024

sven1977 Oct 10, 2024

simonsays1980 Oct 10, 2024

sven1977 Oct 10, 2024

simonsays1980 Oct 10, 2024

simonsays1980 Oct 10, 2024

sven1977 Oct 10, 2024

simonsays1980 Oct 10, 2024

simonsays1980 Oct 10, 2024

sven1977 Oct 10, 2024

sven1977 Oct 10, 2024

sven1977 Oct 10, 2024

[RLlib] Cleanup examples folder vol. 23: Add example script for custom metrics on EnvRunners (using MetricsLogger API). #47969

Are you sure you want to change the base?

[RLlib] Cleanup examples folder vol. 23: Add example script for custom metrics on EnvRunners (using MetricsLogger API). #47969

Conversation

sven1977 commented Oct 10, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

simonsays1980 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[RLlib] Cleanup examples folder vol. 23: Add example script for custom metrics on `EnvRunners` (using `MetricsLogger` API). #47969

[RLlib] Cleanup examples folder vol. 23: Add example script for custom metrics on `EnvRunners` (using `MetricsLogger` API). #47969

sven1977 commented Oct 10, 2024 •

edited

Loading