feat: introduce node health as a metric

[danisharora099]

This is a feature request

Problem

Based on the Reliability RFC,

Node health is a metric meant to determine the connectivity state of a light node and its present ability to reliably send and receive messages from the network. We consider this reliability to be dependent on amount of simultaneous connections to responsive service nodes. Unfortunately the more connections light node establishes - the more bandwidth is consumed. To address this we RECOMMEND following states:

unhealthy - no connections to service nodes are available regardless of protocol;
minimally healthy:
Filter has one service node connection;
LightPush protocol has one service node connection;
sufficiently healthy:
Filter has at least 2 connections available to service nodes;
LightPush has at least 2 connections available to service nodes;

Proposed Solutions

Introduce this as an API

Notes

[weboko]

Some notes:

• we should expose state from WakuNode directly;
• we should expose event based API for consumers to be up to date with changes;
• documentation would be important to provide;

Follow up to this feature would be to develop strategies for recovery from mentioned health states: #2076

[vpavlin]

I feel like 2 nodes might not be enough for calling the state sufficiently healthy - we are working with decentrlaized peer-to-peer potentially unreliable connections:)

[weboko]

thread: https://discord.com/channels/1110799176264056863/1263470028540346391/1263470032172613776

as we discussed it on PM - this can be approached differently, node can be considered minimally health or unhealthy given the example from the description.

it's the matter how we correlate individual healthiness of protocols by & or |

for js-waku in particular it should be an exceptional case when lightPush is unhealthy and filter is fine, reason for it is:

• we maintain up to date peers in the background;
• if there are drops or failures - we rotate peers;

given that js-waku will transition into fully healthy or fully unhealthy state eventually

for implementation it seems only reasonable to expose health state per protocol and then combine
from my side, giving it another thought and read for RFC - I think we should use & so that if even one protocol unhealthy - whole node is.