libssh overrides configured keys

[UweSauter]

OS: CentOS 8.1
libssh: 0.9.0-4.el8
tmate-ssh-server: e1a5e08

libssh has its own configuration in /etc/libssh . I cannot say in which version this was introduced.
The configuration directives in /etc/libssh/libssh_server.config are

# Parse system-wide crypto configuration file
Include /etc/crypto-policies/back-ends/libssh.config
# Parse OpenSSH configuration file for consistency
Include /etc/ssh/sshd_config

This causes all keys that are loaded previously to be exchanged with the system keys.

In my case I run ./tmate-ssh-server -k /etc/tmate/ -p 22022 to use a different set of SSH keys than the system.

To diagnose this issue I did run strace -ff ./tmate-ssh-server -k /etc/tmate/ -p 22022 2>&1 | tee /tmp/tmate-ssh-server.strace.log. As you can see in the following except of /tmp/tmate-ssh-server.strace.log, once a child gets forked from the parent process, libssh reads the system SSH keys.

# grep open /tmp/tmate-ssh-server.strace.log | grep etc
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/tmate//ssh_host_rsa_key", O_RDONLY) = 5
openat(AT_FDCWD, "/etc/tmate//ssh_host_ed25519_key", O_RDONLY) = 5
[pid 13324] openat(AT_FDCWD, "/etc/libssh/libssh_server.config", O_RDONLY) = 7
[pid 13324] openat(AT_FDCWD, "/etc/crypto-policies/back-ends/libssh.config", O_RDONLY) = 8
[pid 13324] openat(AT_FDCWD, "/etc/ssh/sshd_config", O_RDONLY) = 8
[pid 13324] openat(AT_FDCWD, "/etc/ssh/ssh_host_rsa_key", O_RDONLY) = 9
[pid 13324] openat(AT_FDCWD, "/etc/ssh/ssh_host_ed25519_key", O_RDONLY) = 9
[pid 13324] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 0

This effectively means that you cannot run tmate-ssh-server with SSH keys independent of the system keys. A workaround would be to edit /etc/libssh/libssh_server.config but I don't know how practical this would be.

[UweSauter]

I think this change came with libssh-0.9.0. See their announcement at https://www.libssh.org/2019/06/28/libssh-0-9-0/ :
We also added support for server side configuration parsing. This is mostly useful for defining ciphers, mac modes and hashes. We also improved the performance and reduced the copying of data for internal data structures.
Which in this case breaks the server side…

Edit: typo

[Jakuje]

There is a bind option SSH_BIND_OPTIONS_PROCESS_CONFIG which prevents this default behavior:

https://gitlab.com/libssh/libssh-mirror/-/blob/master/src/options.c#L1630

[UweSauter]

There's also an issue open on the client side which addresses this issue: tmate-io/tmate#186

[ansasaki]

Hello @UweSauter,

This was treated as a bug in Fedora and the inclusion of the OpenSSH server configuration was removed from the libssh_server.config file.

I'll check the tmate server code to see if it is possible to call the ssh_bind_options_parse_config() explicitly before setting the options to avoid them to be overridden. Once the configuration file is parsed, the automatic parsing is not executed.

[ansasaki]

@UweSauter Could you please check if applying the PR #74 solves the issue for you?

[UweSauter]

#74 looks good as far as I can tell. Thanks.