Setting Overhaul

[josecelano]

@da2ce7 is working on a setting Overhaul.

@josecelano has proposed moving the initial setting setup to its binary. The reason behind that is that the configuration file makes things harder when using docker to run the application. @josecelano has been working on a PR to add docker support.

@Power2All has also dockerized the application on a fork. That solution uses a shell script to generate the config.toml file before running the application. It takes the values from env vars. This solution does not need any change in the source code.

I've created this discussion to collect all the information about settings.

@da2ce7 @josecelano and @WarmBeer were discussing a solution today. It seems we need a more friendly solution to handle the configuration.

Requirements

• It has to work for both the development and production environment.
• It has to be docker or cloud friendly.

Notes

• It seems a lot of plan environments it's still very common. Digital Ocean App Platform allows you to inject only env vars.
• Azure Container Instances do not allow you to inject a single file. It allows you only to mount a volume.
• With Kubernetes you can inject both env vars and config maps. Kubernetes have two types of configuration variables: the standard ones (ConfigMaps) and the secrets.
• The current config.toml file is not a plan list of env vars. It has groups and it also has arrays. Notice that you can have more than one UDP tracker and HTTP tracker, and you need a group of configuration values for each.

Proposed solutions

We can reply with the proposed solution. Just give it a name so we can easily refer to it.

Links

• Full docker support #123
• https://github.com/Power2All/torrust-axum
• [proposal] Create an installer with a flag (argument) #122
• Setting Overhaul #105

cc @WarmBeer @cgbosse

[josecelano]

hi, I just want to add a useful link https://12factor.net/

"The twelve-factor app is a methodology for building software-as-a-service apps".

I think that's a good collection of "good practices" for operations.

It has a specific section for configuration: https://12factor.net/config

And also for logs, which it's something we have also discussed in the past: https://12factor.net/logs

And another one for Dev/prod parity: https://12factor.net/dev-prod-parity

In my PR for docker support, I'm trying to use the same image for development and production. Or at least, you can use ARGs to modify it. I think ideally everyone should be able to use the official image without needing to build a new one.

[josecelano]

Solution 1: generate config data from .config.toml file or from TORRUST_TRACKER_CONFIG env var.

Abstract

Running the application with docker has some problems:

• You need to execute it twice. The first time it generates the config.toml file
• If you want to run it with docker you can mount a volume/file or generate the file with a shell script before actually running the application. You can see here an example of the script solution.

Problems

• Some docker orchestrators do not allow to mount files, only directories.
• We would like to run the application in a stateless mode. Right now you can run it without any database. You could build a docker image with the default config file and the application would work without mounting any volume. The problem is we need to change the configuration at least when we launch it. @da2ce7 has proposed to allow to change of some settings even at runtime. But it's out of the scope of this solution.

Solution

It seems the minimum viable solution could be to use env var to keep the same information we have in the config.toml file. That's what @Power2All did here. We could change the code to check for both the file and the env vars.

This solution would:

• Use a single env var TORRUST_TRACKER_CONFIG containing a json file with the same information we have in the toml file. The first iteration could use also a toml format. So basically, the app can take the same info from a file or an env var.

Pros:

• It's compatible with the current version. It only checks for the env var if the file does not exist. @da2ce7 has proposed removing the file, but for development, without docker, we need an easy way to inject the configuration.
• If you are using an orchestrator to launch the container, I think it's not a problem to have a long json value.

Cons:

• If you want to run manually the application with docker it could be hard to pass the value:

Mounting the configuration file in the storage/config dir would be like:

docker run -it \
    --user="$TORRUST_TRACKER_USER_UID" \
    --publish 6969:6969/udp \
    --publish 7070:7070/tcp \
    --publish 1212:1212/tcp \
    --volume "$(pwd)/storage":"/app/storage" \
    torrust-tracker

but if you want use an env var with json:

docker run -it \
    --user="$TORRUST_TRACKER_USER_UID" \
    --publish 6969:6969/udp \
    --publish 7070:7070/tcp \
    --publish 1212:1212/tcp \
    --volume "$(pwd)/storage":"/app/storage" \
    --env TORRUST_TRACKER_CONFIG="BIG .... JSON VALUE"
    torrust-tracker

You can use --env-file but it does not support all the features the config.toml file has (groups and arrays).

Anyway, I suppose you can create a file, store it whenever you want to store it and read the contents and pass it to that command.

docker run -it \
    --user="$TORRUST_TRACKER_USER_UID" \
    --publish 6969:6969/udp \
    --publish 7070:7070/tcp \
    --publish 1212:1212/tcp \
    --volume "$(pwd)/storage":"/app/storage" \
    --env TORRUST_TRACKER_CONFIG=`cat your file`
    torrust-tracker

If you want to test it:

$ echo "hello world" > greetings.txt
TEST_ENV=`cat greetings.txt`
docker run -it --env TEST_ENV="$TEST_ENV" alpine env | grep "hello world"
TEST_ENV=hello world

I prefer to keep the config.toml file or even better a new .config.toml file that is gitignored. If we want to keep the compatibility we can use the same name.

[josecelano]

Proposal: Simplify configuration

hi @da2ce7 @WarmBeer, I've been thinking about the configuration file. I think regardless of the final format we decide to use (.env, json, yaml, toml, XML, ...), we should try to make it as much simple as possible. There are a couple of things that make the configuration more complex:

1. Groups: [[udp_trackers]], [[http_trackers]], [http_api], [http_api.access_tokens]. They make the file more human-friendly, but we cannot port it easily to the .env format.
2. Arrays: [[udp_trackers]], [[http_trackers]]. Allowing more than one UDP or HTTP tracker makes the configurations harder.

If we keep a simple data structure (a list of pair key-value), we could generate any format (json, toml, XML...), even the simplest one, .env. Many hosting providers only allow injecting env vars. The application would be more portable.

On the other hand, do we need to launch more than one UDP or HTTP tracker @WarmBeer? Why do not you launch a new app instance? Why would you need more than one port? Is there any reason for that?

[Power2All]

On the other hand, do we need to launch more than one UDP or HTTP tracker @WarmBeer? Why do not you launch a new app instance? Why would you need more than one port? Is there any reason for that?

Yes, in case you want to bind on certain IP's. Some could run multiple instances. IPv6 and such. This was my option as a solution.

[da2ce7]

Torrust currently uses the config-rs crate for the management a configuration file.

The configuration code is centralized within the config.rs file, and loaded from the main.rs file.

Previously I worked on a pull request to #105 that tried to rework the entire configuration concept, it used rust's built-in json serialization and deserialization, it auto generated the configuration and supported upgrading from the old configuration. However when writing this pull request, I instead got lost in a side-project, fixing the clippy warning for torrust: #110 #115 , and since then there has been significant changes that makes the re-base difficult.

One of the key features in my design was to make all the configuration options optional, and sharing configuration properties where possible.

When working on the docker support for torrust, @josecelano opened this detailed discussion on the various possibilities for configuration.

I have also considered these options, and since have found a interesting crate called Figment, that has many interesting properties.

[josecelano]

Many changes were implemented in this issue: #401