Dynamically switch between local and remote cluster

[mihirnimgade]

The idea is to be able to run a local and remote cluster at the same time and have the telemetry link fallback to making requests to the local cluster if remote cluster requests start to fail.

The reason we want to support this is to cover the case where you have intermittent/unreliable internet access on the telemetry link host. When internet access goes down, the telemetry link will automatically switch over to using the local cluster which means access to telemetry data (at least the data coming from the radio) is uninterrupted during race time.

Here's what potential implementations might look like:

1. Modify the futures done callback to try making a request to the local cluster if the remote cluster request fails. This is probably the simplest implementation. One advantage with this approach is that loss of data is very unlikely.

2. Once a certain number of requests to the remote cluster fail, switch the request URL to the local cluster. The subsequent requests will be made to the local cluster. This does mean that some messages get lost during the remote to local switch. After switching to the local cluster, spin up a background thread that periodically pings the remote cluster. Once the background thread indicates that the remote cluster is accessible again, switch the request URL back to point to the remote cluster. Requests will then be made to the remote cluster.

TODO

• Add command-line argument to link_telemetry.py that enables/disables dynamic switching.
• Add new fields to the telemetry.toml for the two sets (local and remote) of URLs and secret keys.
• Update relevant documentation.
• Update telemetry.toml template.