Envoy 1.19 / Centos 7 / GLIBC_2_18 compatibility issue

[WeirdBob]

Describe the bug
When trying to use func-e on centos 7, envoy 1.19 fails to start with the message: versions/1.19.0/bin/envoy: /lib64/libc.so.6: version `GLIBC_2.18' not found

To Reproduce
Steps to reproduce the behavior:

1. Install func-e on centos-7
2. Try to run envoy (--version is enough)
3. See error

Expected behavior
Envoy is compiled with a compatible glibc version (as it was the case with previous getenvoy builds)

Environment (please complete the relevant information):

• OS: Centos 7
• Envoy Version: 1.19.0

Additional context
Thanks for you work even if it doesn't work (yet) in my case !

[codefromthecrypt]

hi @WeirdBob getenvoy and all things associated with that are gone now. specific to this topic, we aren't using a different build pipeline: the envoy binary here was extracted from the official docker image during archive-envoy

That said, this is an issue.. we don't have a CI dimension for centos. We can take a look and try to help figure out what can be done about it. Thanks for reporting!

[codefromthecrypt]

@mathetake maybe you can help elaborate options on this one as you are more familiar with the parts inside. Here are some bread crumbs, though some may be irrelevant.

This isn't an attempt to find a reason to not support centos, rather show what might be in front of us given the envoy project's decisions. Custom builds are extremely expensive so, upstream adjustments are preferred vs going back into the labor pit of perpetually broken custom builds.

• the centos build image is marked as experimental, but appears to be latest centos 7
• the official build wants glibc >= 2.27 since 1.15
• the build has an if-statement around centos 7, so this might imply we can raise support issues...

[codefromthecrypt]

verified this is the case even on envoy's build image. this hints at a custom build being needed, and there are various things on the internet about glibc locking what can be used on centos 7 (and corresponding rhel).

$ docker run -ti --rm envoyproxy/envoy-build-centos:2144d692c47e4fc5f4d4e2dab27f08a084c5b346
[root@c4f6c72ca205 /]# curl https://func-e.io/install.sh | bash -s -- -b /usr/local/bin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9791  100  9791    0     0   4914      0  0:00:01  0:00:01 --:--:--  4915
tetratelabs/func-e info checking GitHub for latest tag
tetratelabs/func-e info found version: 0.6.0 for v0.6.0/linux/amd64
tetratelabs/func-e info installed /usr/local/bin/func-e
[root@c4f6c72ca205 /]# func-e run --version
looking up latest version
downloading https://archive.tetratelabs.io/envoy/download/v1.19.0/envoy-v1.19.0-linux-amd64.tar.xz
starting: /root/.func-e/versions/1.19.0/bin/envoy --version --admin-address-path /root/.func-e/runs/1628720114081400543/admin-address.txt
/root/.func-e/versions/1.19.0/bin/envoy: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /root/.func-e/versions/1.19.0/bin/envoy)
error: envoy exited with status: 1

Meanwhile, if someone has another source of the envoy binary (built specifically for centos 7), that could be used by overriding ENVOY_VERSIONS_URL to something compliant with the same schema

Anyway over to @mathetake for some more intelligent feedback :D Meanwhile I suppose at least we can add a centos 8 dimension to the CI tests to ensure that works.

[codefromthecrypt]

added some comments to issues about pending official tarballs and packages

[mathetake]

OK the problem summarizes as: Envoy's official built binary uses too new libc++ - which uses unsupported ABI in CentOS's too old system glibc. Therefore, we have to build Envoy on CentOS/RHEL 7, but getenvoy project no longer exists and our own CentOS 7 build would never be provided here.

Here's two work arounds;

• build Envoy by yourself on CentOS/RHEL7.
• install the newer glibc into the CentOS, and use patchelf to modify the Envoy's binary so it uses the installed new glibc instead of old system glibc.
  • Please refer to this thread in Envoy slack(https://envoyproxy.slack.com/archives/C78HA81DH/p1545069327234900)

But both of them means you cannot use via func-e. I will think about the solution without building by ourselves, and support CentOS7/RHEL via func-e. But I don't think of any idea for now.

[codefromthecrypt]

I'm going to move this to the archive-envoy repository as it is similar to any other non-standard build request (such as nightlies). func-e already works on centos 7 (ex list run work), it is a matter of the source of binaries at this point. Thanks for the elaboration @mathetake!

[codefromthecrypt]

In another issue @lizan mentioned upstream won't support CentOS 7 packages. Note that this is different than supporting people with custom builds. For example, fixing custom builds of CentOS 7 is in progress.

For people who keep running into this, my personal advise...

• don't use getenvoy packages as they aren't maintained or supported (no new packages == no new versions or patches)
• if you can do your own build, do it.
• if you can't do your own build ask upstream who have the most power to support centos in the build, and keep it working, to package RPMs for CentOS 7

Of course another option is moving off CentOS 7. Regardless, please let's not add more issues here, as this project only follows what formal packages support. We've had a burden of having the brunt of Q&A that the envoy project team themselves should be answering, and if they had to answer these questions all the time, maybe they would relent in supporting CentOS 7 again. Adding a bunch of issues here burdens a project with no empowerment or influence to change the mind of the envoy proxy team