artifact: permission error to read certificates

[ahjohannessen]

I got this on Flatcar Linux this morning:

failed to download artifact "https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.35/grpc_health_probe-linux-amd64": getter subprocess failed: exit status 1: failed to download artifact: Get "https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.35/grpc_health_probe-linux-amd64": tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs/ca-certificates.crt: permission denied

Seems something changed with regards to artifact permissions to read certificates:

tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs/ca-certificates.crt: permission denied

After upgrading to 1.9.1 -> 1.9.3. Temporarily solved it by setting disable_filesystem_isolation = true, which probably is not a permanent fix or good idea?

On Fedora CoreOS machines I do not have this issue (yet).

Nomad version

1.9.3

Operating system and Environment details

Flatcar Container Linux

Flatcar Container Linux by Kinvolk stable 4081.2.0 for VMware
core@app03 ~ $ uname -a
Linux app03 6.6.60-flatcar #1 SMP PREEMPT_DYNAMIC Tue Nov 12 16:20:46 -00 2024 x86_64 Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz GenuineIntel GNU/Linux

Fedora CoreOS:

Fedora CoreOS 41.20241027.3.0
core@app04:~$ uname -a
Linux app04 6.11.5-300.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 22 20:11:15 UTC 2024 x86_64 GNU/Linux

[tgross]

On Fedora CoreOS machines I do not have this issue (yet).

@ahjohannessen when you say you don't have this issue on CoreOS but you do on Flatcar, are you talking about the exact same version of Nomad? Also, don't both those distros run all the software as containers?

[ahjohannessen]

On Fedora CoreOS machines I do not have this issue (yet).

@ahjohannessen when you say you don't have this issue on CoreOS but you do on Flatcar, are you talking about the exact same version of Nomad? Also, don't both those distros run all the software as containers?

@tgross

Same version of Nomad. I install the binaries with ansible-nomad, no container install.

For things like consul, consul-template, nomad and vault I prefer setting it up running outside containers. Everything else goes into containers that Nomad controls :)