Unable to build Oak Containers compatible C++ binary from Nix devenv

[bmclarnon]

#4619 attempts to define a C++ binary that will run in a Oak Container, but it runs into two issues:

1. the binary expects the dynamic linker to be in the nix store (i.e., /nix/store/.../lib/, not /lib64/)
2. the binary requires a newer glibc version than is supported by any of the distroless images (2.38 vs 2.36).

More generally, these issues can be summarized as "binaries built with the Nix C++ toolchain aren't generally suitable for running on other Linux distributions". Example: https://discourse.nixos.org/t/how-do-i-build-a-binary-on-nixos-that-i-can-run-on-other-distros/11230

As an alternative, I tried using a hermetic GCC toolchain, which produces binaries with a standard path and targets a less-cutting-edge glibc version. That solves (1) and (2), but introduces the problem that the compiler itself can't run on NixOS because dynamic linker path is wrong. This is in theory something that'd be fixed by https://github.com/Mic92/nix-ld, but I wasn't able to get it working.

Steps to repro (using gcc):

$ ./scripts/docker_run nix develop .#ci
$ wget https://toolchains.bootlin.com/downloads/releases/toolchains/x86-64/tarballs/x86-64--glibc--stable-2021.11-5.tar.bz2
$ tar -xjf x86-64--glibc--stable-2021.11-5.tar.bz2
$ ./x86-64--glibc--stable-2021.11-5/bin/x86_64-buildroot-linux-gnu-gcc --version                                                                                                   

If successful, this will print the gcc version. Otherwise, it'll print ./x86-64--glibc--stable-2021.11-5/bin/x86_64-buildroot-linux-gnu-gcc: cannot execute: required file not found.

One viable alternative would be finding a way to use the Nix C++ toolchain to produce binaries that can run on other Linux platforms. Another would be finding (or creating) a NixOS-based OCI base image. Help appreciated!

[bmclarnon]

If it's not feasible to get nix-ld working, it should be possible to define a new bazel toolchain that uses the nix-provided gcc and cross-compiles (via a sysroot) for other Linux distributions. But defining a C++ toolchain is pretty verbose, so it isn't my preferred solution.

[tiziano88]

@bmclarnon would you like to try out if #4622 solves your issue, before I merge it?

[bmclarnon]

@bmclarnon would you like to try out if #4622 solves your issue, before I merge it?

The tests in #4619 passed when I used the new CI base image. Thanks for the really quick fix!