Observability of validators for relayers

[tkporter]

Description

Goal of this was to have insight into validators of important sets being "up"

Introduces a new metric used by relayers: hyperlane_observed_validator_latest_index, e.g.:

hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test1",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 664
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test1",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 641
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test2",hyperlane_baselib_version="0.1.0",origin="test1",validator="0x15d34aaf54267db7d7c367839aaf71a00a2c6a65"} 670
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test2",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 665
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test3",hyperlane_baselib_version="0.1.0",origin="test1",validator="0x15d34aaf54267db7d7c367839aaf71a00a2c6a65"} 652
hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test3",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 664
hyperlane_observed_validator_latest_index{agent="relayer",app_context="testapp",destination="test1",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 658
hyperlane_observed_validator_latest_index{agent="relayer",app_context="testapp",destination="test1",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 641

Tapping into metadata building for multisig ISMs, the relayer will update the metric with the latest indices for the validators in a set. In order to prevent the cardinality being ridiculously high, only certain validator sets are tracked. This is done by introducing an app_context label (I'm very open to other names here, for some reason whenever idk how to name some kind of identifier I end up calling it a context 😆)

The app context can either be:

• if a new setting, --metricAppContexts, is specified, a message will be classified based off the first matching list it matches. E.g. --metricAppContexts '[{"name": "testapp", "matchingList": [{"recipient_address": "0xd84379ceae14aa33c123af12424a37803f885889", "destination_domain": 13371 }] }]'. This is nice for e.g. warp route deployments, where the ISM is maybe not a default ISM, and can be changed
• if a message doesn't get classified this way, it can also be classified with the "default_ism" app context, which is just for any message that happens to use the default ISM as its "root" ISM

This way we have insight in to the default ISM and any application-specific ISMs.

Some things to note:

• it's possible for a message to actually have more than one validator set, e.g. if it's using an aggregation ISM. In this case, we'll have metrics on the union of all validator sets for that app context
• Some effort is required to make sure that metrics don't stick around for a validator that has actually been removed from the set. To handle this, we cache the validator set for an app context and clear out the entire set each time we set the metrics

Drive-by changes

• Zod's nonempty function for strings is deprecated, moves to .min(1) instead

Related issues

• Fixes Observe 3rd Party validators are signing at the tip in the relayer #1762

Backward compatibility

yes

Testing

Ran locally - I think i'll probably add something in e2e tests, but opening now

[changeset-bot]

⚠️ No Changeset found

Latest commit: 4bb3afd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR