Interpolate environment variables in the config

[aakoshh]

Does an extra round of environment variable interpolation on config values.

Original

Another very nice thing to have:
Right now fendermint doesn't seem to interpolate env variables inside env variables.

2024-02-28T12:56:52.320848Z ERROR fendermint/app/src/main.rs:75: failed to execute Options { home_dir: "/local", log_dir: None, log_file_prefix: None, mode: "dev", log_level: Info, global: GlobalArgs { network: Testnet }, command: Run(RunArgs) }: error parsing settings

Caused by:
    invalid type: string "${NOMAD_PORT_fendermint_abci}", expected an integer for key `abci.listen.port` in the environment

A bit of context on NOMAD_PORT_* thing:
this is an env variable set by container orchestrator inside container so I can easily bind to a random port assigned by Nomad. The ports are defined in nomad deployment file.

# env | grep NOMAD_PORT
NOMAD_PORT_cometbft_health=30896
NOMAD_PORT_connect_proxy_cometbft_test=31873
NOMAD_PORT_cometbft_rpc=27657
NOMAD_PORT_cometbft_metrics=27660
NOMAD_PORT_fendermint_p2p=27659
NOMAD_PORT_fendermint_abci=27658
NOMAD_PORT_cometbft_p2p=27656

Very useful for avoiding port collisions.
For example:
If I want to run another subnet on the same pool of hosts. To avoid port collision with another IPC deployment I want to change ports. When I hardcode ports like this: FM_ABCI__LISTEN__PORT = 27658 I have to not forget to change it in multiple places (like cometbft configuration). When I use templating like this FM_ABCI__LISTEN__PORT = NOMAD_PORT_fendermint_abci it is automatically changed when I change it in port declaration in Nomad:

      port "cometbft-rpc" {
        to = 27657
      }

      port "cometbft-p2p" {
        to     = 27656
        static = 27656
      }

      port "cometbft-metrics" {
        to     = 27660
        static = 27660
      }

      port "cometbft-health" {}

      port "fendermint-abci" {
        to     = 27658
        static = 27658
      }

      port "fendermint-p2p" {
        to     = 27659
        static = 27659
      }

In this situation ports are hardcoded in declaration too so I cannot run two IPC nodes on the same host.
I can make Nomad assign a random port

      port "fendermint-abci" {}

And it can be useful in situation where I have only 4 physical servers and 4 IPC nodes. Affinity would make one IPC run on one server, but in case I do some upgrades to a server and it comes offline for an hour 2 IPC nodes can run on one physical server. And if 3 servers is suddenly offline instead of having whole network down 4 IPC nodes can temporarily run on one physical server. Automatically, so no downtime. And when everything is up again they will migrate gracefully.

CMT_P2P_EXTERNAL_ADDRESS = "${NOMAD_HOST_ADDRESS_cometbft_p2p}"

This will interpolate to whatever ip:port cometbft runs now so other nodes can find it.

[fridrik01]

Ah this is nice to have! This looks good to me and nice tests and PR summary!