Best practices for running on Kubernetes?

[juzna]

We'd like to run sftpgo on Kubernetes / GKE, mainly because we run most our apps there, and also to add some redundancy and increase throughput by running multiple replicas. Does anyone has experience with that?

Running multiple replicas doesn't seem to be explicitly supported.

The main problem we're hitting is the required temp storage, which would be different on different hosts. It seems like this would cause trouble and data corruption. Or would it be safe?

We'd be using Google Cloud storage for all users, so theoretically no local storage should be necessary. We can have only read-only config file (e.g. keys of the server) and use Postgres as data provider. However, I'm not sure if something could break if we made /var/lib/sftpgo/ read-only. Any thoughts?

Thanks

[drakkan]

Hi,

I have no direct experience with this setup, however I think other users run sftpgo this way. If you use GCS as backend you shoud set "prefer_database_credentials": true, inside sftpgo.json so the GCS credentials will not be stored on the host storage.

Please test using this configuration and let me know if you have troubles

P.S. if you use SFTPGo at your company please consider a sponsorship this will allow me to spend more time working on the project, thank you!

[juzna]

Thanks, all seems to work fine.

Couple of gotchas we've encountered (I'll later see if I can send PRs for docs, but if not, at least it's here):

• without persistent disk storage, sftpgo would generate new host keys on every start, which doesn't make clients happy. So we generated them once and stored them as Kubernetes secrets, so that they're shared across all replicas.
• obviously, we can't use sqlite data provider with no shared disk, because each replica would have it's own independent config 🤦

We've also decided to use Kubernetes for the whole configuration, including user management. Basically, we have a static config (from dumpdata) committed in git, which is passed to the server via Kubernetes ConfigMap. We then use memory data provider which loads the config at startup. 💡 Idea: It would be nice to make the HTTP UI read-only, to prevent people from accidentally doing changes only stored in memory.

The static config has however one problem, because for GCS, each configured user needs credentials to access GCS. We don't want to store these credentials in git with the remaining config. So we wrote a short Python script :sorry: which merges non-secret config with secrets, and sftpgo can then load the full config file. 💡 Idea: Do you think this would be beneficial feature to have? Either native support for loading secrets from a separate file, or the script to merge config file?

Thx

[drakkan]

Thanks, all seems to work fine.

good!

Couple of gotchas we've encountered (I'll later see if I can send PRs for docs, but if not, at least it's here):

• without persistent disk storage, sftpgo would generate new host keys on every start, which doesn't make clients happy. So we generated them once and stored them as Kubernetes secrets, so that they're shared across all replicas.

yes, Kubernetes/Docker secrets is the way to go. I forgot about this sorry

• obviously, we can't use sqlite data provider with no shared disk, because each replica would have it's own independent config facepalm

consider a shared database, like PostgreSQL / MySQL, with your configuration it seems very difficult to change / add a user

We've also decided to use Kubernetes for the whole configuration, including user management. Basically, we have a static config (from dumpdata) committed in git, which is passed to the server via Kubernetes ConfigMap. We then use memory data provider which loads the config at startup. Idea: It would be nice to make the HTTP UI read-only, to prevent people from accidentally doing changes only stored in memory.

you can now set permissions for SFTPGo admins and so it is easy to make an admin read only

The static config has however one problem, because for GCS, each configured user needs credentials to access GCS. We don't want to store these credentials in git with the remaining config. So we wrote a short Python script :sorry: which merges non-secret config with secrets, and sftpgo can then load the full config file. Idea: Do you think this would be beneficial feature to have? Either native support for loading secrets from a separate file, or the script to merge config file?

if you set "prefer_database_credentials": false the GCS credentials are stored inside the credentials dir and so loaded from files.

However if you try git master, GCS credentials are encrypted and can be optionally stored inside an external KMS

Thx

[MegalodonBite]

Thank you for this discussion.

We are currently testing a similar setup in a Kubernetes Cluster.

We use a configMap for:

• host keys
• user settings
• application settings

We would like to avoid using the database as a dependency to be able to exclude a source of error.

@juzna
Is the setup as above still in use?
Did you gain any new insights that occurred while running the setup?

[saralhimanshu]

Thank you for this discussion.

We are currently testing a similar setup in a Kubernetes Cluster.

We use a configMap for:

• host keys
• user settings
• application settings

We would like to avoid using the database as a dependency to be able to exclude a source of error.

@juzna Is the setup as above still in use? Did you gain any new insights that occurred while running the setup?

[saralhimanshu]

Can you please share kubernetes configmap / secret template for storing host keys and consuming via variable or other method ?

Thank you in advance for your support !!

[saralhimanshu]

Can anyone please share kubernetes configmap / secret template for storing host keys and consuming via variable or other method ?

Thank you in advance for your support !!

[MegalodonBite]

@saralhimanshu
using this chart https://github.com/sagikazarmark/helm-charts/tree/master/charts/sftpgo

in values.yaml

config:
      sftpd:
          host_keys:
            - /etc/secretvolume/id_ed25519
            - /etc/secretvolume/id_rsa
 volumes:
  - name: secretvolume
    secret:
      secretName: sftpgohostkeys
volumeMounts:
  - name: secretvolume
    mountPath: /etc/secretvolume
    readOnly: true

configmap:

apiVersion: v1
data:
    id_ed25519: YOUR_KEY
    id_rsa: YOUR_KEY
kind: Secret
metadata:
    name: sftpgohostkeys

Add more keys, if you like with id_something

Please let me know, if you have suggestions for improvement