LD_PRELOAD Causes Error on Deployment

[scottwater]

I am investing using this buildpack.

Durning my deployment to heroku, the build is rejected do to an error (cannot open shared object file).

Some of the output (same error over and over) is below.

Any suggestions for a next step? Any other info I could provide?

Thanks,
Scott

kickoff:jemal g push staging jemal:master
Counting objects: 12, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (12/12), 931 bytes | 0 bytes/s, done.
Total 12 (delta 9), reused 2 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Deleting 1 files matching .slugignore patterns.
remote: -----> jemalloc app detected
remote: -----> Vendoring binaries
remote: Fetching https://github.com/mojodna/heroku-buildpack-jemalloc/releases/download/4.2.1/jemalloc-4.2.1-1.tar.gz
remote: -----> Configuring build environment
remote: -----> Building runtime environment
remote: -----> Ruby app detected
remote: -----> Compiling Ruby/Rails
remote: -----> Using Ruby version: ruby-2.3.3
remote: -----> Installing dependencies using bundler 1.13.6
remote: Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
remote: ERROR: ld.so: object '/app/vendor/jemalloc/lib/libjemalloc.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

[mojodna]

Good catch.

Are you setting LD_PRELOAD via heroku config?

If so, change /app/vendor/jemalloc/lib/libjemalloc.so.1 to /app/vendor/jemalloc/lib/libjemalloc.so (the former doesn't exist, despite what the docs and jemalloc.sh say).

If not, set that instead of using jemalloc.sh.

[mojodna]

Actually, I'm wrong. /app/vendor/jemalloc/lib/libjemalloc.so.1 does exist in the tarball. I don't know what's going on here.

Which stack are you using?

[scottwater]

@mojodna I am using the cedar-14 stack with Ruby 2.3.3. I had set the LD_PREPOLOAD via Heroku config.

I had originally thought it was a typo as well, but I connected via bash and was able to verify the libjemalloc.so.1 does exist.

Exact commands executed:
heroku config:set LD_PRELOAD=/app/vendor/jemalloc/lib/libjemalloc.so.1

heroku config -s | grep LD_PRELOAD
LD_PRELOAD='/app/vendor/jemalloc/lib/libjemalloc.so.1'

Please let me know if I can provide any other details/info.

Thanks,
Scott

[mojodna]

I was able to reproduce the error by running heroku run bash against a newly-created app with LD_PRELOAD set (prior to pushing any commits). That was because the app hadn't been built yet, and the dyno only contained the cedar-14 runtime. After pushing a commit containing an empty index.html, I re-ran heroku run bash and things worked swimmingly.

Try reconnecting to the dyno via bash and run (and report back):

ldd /app/vendor/jemalloc/lib/libjemalloc.so.1

Output should be (and there should be no error when connecting):

~ $ ldd /app/vendor/jemalloc/lib/libjemalloc.so.1
	linux-vdso.so.1 =>  (0x00007ffc5dffb000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f53da66d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f53da2a8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f53daacb000)

[scottwater]

I get the following:

~ $ ldd /app/vendor/jemalloc/lib/libjemalloc.so.1
linux-vdso.so.1 => (0x00007ffdb438c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f652d6b6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f652d2f1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f652db14000)

I did a bit more testing. On a brand new/empty rails project (4.2.7.1), I get the error ERROR: ld.so: object '/app/vendor/jemalloc/lib/libjemalloc.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file, but the build is still deployed.

In my app, looking through more of the output I also see a JSON:ParseError (see http://d.pr/n/2bZd)

I tried to play around with some of the gem versions of sprockets (3.7.0) and sprocket-rails (3.2.0), but I have not been able to narrow down the issue.

Thanks,
Scott

[mojodna]

Can you share access with me so I can do some poking around? mojodna / seth at mojodna dot net

[czj]

Hi @mojodna I'm having the same issue with a Heroku app :

ERROR: ld.so: object '/app/vendor/jemalloc/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

I'm using the same buildpack/configuration on Scalingo without any specific issues. If you want, I could add you on the open-source project we run with this lib on Heroku.

[9mm]

Curious about this if it still exists, I am wanting to use this in production on Heroku, rails 5, ruby 2.4.1, I'm hesitant to pull the trigger though.

[nateberkopec]

Let's try and get it fixed if it is an issue.

Can everyone using LD_PRELOAD as an env variable (like in your "config settings" in Heroku or whatever) in this thread please try removing that and just use jemalloc.sh in your Procfile?

i.e.:

web: jemalloc.sh bundle exec puma -C config/puma.rb

[9mm]

I havent tried using it yet, but I could try now. I did have one other question, do you know how I might do this if I'm using pgbouncer? It's bin/start-pgbouncer-stunnel below

web: bin/start-pgbouncer-stunnel bundle exec puma -C config/puma.rb
clock: bundle exec clockwork config/clock.rb
worker: bundle exec que ./config/environment.rb

[nateberkopec]

@9mm I think jemalloc.sh bin/start-pgbouncer-stunnel bundle exec puma -C config/puma.rb should work but I am no bash master.

[havenwood]

I ran into this issue a while back setting LD_PRELOAD but just prefixing jemalloc.sh is now working for me.

[knightq]

It works with the script, but it makes the Procfile strongly environment-dependent.
This "solution" should be considered only a workaround, not a fix to the original issue problem, IMHO.

[brian-kephart]

I too was unable to get LD_PRELOAD to work. Using jemalloc.sh in the Procfile did work, and reduced my overall memory usage significantly, but increased my swap memory usage. Some days, after nearly a full day's uptime, I'd see ~17MB of swap memory when my total memory usage was under 70%. With the stock allocator, those numbers are more like 2-6MB at 80-85% usage.

I bring it up here in case this sort of misbehavior is the result of having the script prepended only to processes in the Procfile, rather than all commands run on the dyno, making the Procfile option less viable. Has anyone else seen this?

This behavior was consistent on both the Cedar-14 and Heroku-16 stacks, running Rails 5.0 with Ruby 2.4.1 on a Hobby web dyno.

[nateberkopec]

@brian-kephart 17mb of swap is nothing to worry about. If anything, it indicates the allocator is being more efficient because it's grouping memory pages in such a way that the operating system can swap larger portions out of the resident set.

[brian-kephart]

I managed to fix this problem by setting LD_PRELOAD in the script in /.profile.d.

I think that the problem we're having is a result of having the environment variable present before the buildpack scripts pull in the binaries. Setting the variable within the buildpack seems to solve it.

Before I submit a pull request, is there any reason not to set LD_PRELOAD within the buildpack?

[ohaddahan]

@brian-kephart I believe the issue with LD_PRELOAD is that Heroku is taking the user environment while doing the build , which is wrong and can even be considered a security hazard.
I opened a support case on that a long time ago but they didn't provide a solution.
It seems that using LD_PRELOAD now doesn't cause a build error , they seem to ignore the stderr stream containing the error message.
So other than looking ugly in the logs it should be safe.
But I like your solution better , one question , how do you confirm it's actually being used?
heroku run bash / rails console and check LD_PRELOAD value?

[ohaddahan]

@brian-kephart another option , ugly as it may is to wrap your and run:

1. heroku heroku config:unset LD_PRELOAD
2. git push
3. heroku heroku config:set LD_PRELOAD /app/vendor/jemalloc/lib/libjemalloc.so

This should remove all build log errors and apply it to the environment in full.
It's just ugly and require 3 restarts instead of 1 , I expected Heroku to do this when they do the build on their side , unset/set special Linux variables like LD_PRELOAD.

[scottwater]

I can confirm @ohaddahan's experience with it no longer causing a build error.

I had read @nateberkopec's recent blog post and tried again yesterday. I still see many errors during the deploy, but it is successful.

If there is an easy way to test that it is properly and working/active that would be really helpful.

[ohaddahan]

@scottwater you can log into your machine and check either:
heroku rails console -> File.exists?( ENV['LD_PRELOAD'] )
or
heroku bash -> ls $LD_PRELOAD

If it exists and set, all Linux process must load the shared library before they start running.
https://blog.fpmurphy.com/2012/09/all-about-ld_preload.html

[brian-kephart]

@ohaddahan Yes, you can check that the LD_PRELOAD value is correct as you described, but follow mojodna's instructions in this thread from 12/27/16 to make sure the specified file is actually there. If the LD_PRELOAD value is correct, the file is present, and you see no runtime errors, all should be well. There might be better checks in the jemalloc documentation, though.

[ohaddahan]

@brian-kephart basically if ls $LD_PRELOAD run and you don't see any LD_PRELOAD failed ... error (since ls also loaded it) , it's fine.
If you really want to be on the safe side , you can write a small executable that will try loading some symbols from jemalloc and add it to your application slug and than run it from inside Heroku shell. But that's a little over the top :)

[nateberkopec]

Xavier Noria asked me a similar question once - is it possible to be completely sure jemalloc was running in the Ruby process. As you mentioned @ohaddahan, I think trying to use a jemalloc specific feature or symbol would be the only way to be totally sure.

[ohaddahan]

@nateberkopec actually I think it can be simpler , attaching to the ruby process with gdb or running collect / strace / ltrace on the process.
You can also use LD_DEBUG=all to see move info , but all of these seem over the top.
LD_PRELOAD is really vocal about succeeding or not :)

[gaffneyc]

I believe this would be fixed by #18 when using JEMALLOC_ENABLED as LD_PRELOAD is being set in .profile.d/jemalloc so builds shouldn't be using it. Also, in my testing of bin/compile, none of the config variables were set with the expectation that buildpack developers would read any they need from env-dir/VARIABLE (which #18 does for JEMALLOC_VERSION).

[wongy91]

how do i verify if jemalloc is running on heroku/dokku (im running RoR)

[citizen428]

@wongy91 Here's one way:

$ ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS']"
-lpthread -ljemalloc -lgmp -ldl -lobjc

[k5o]

@citizen428 I'm not OP, but running $ ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS']" in heroku run bash yields:

-lpthread -lgmp -ldl -lcrypt -lm

But I can confirm it shows -ljemalloc when running that command in my local environment. For prod, I've prefixed my dynos with jemalloc.sh and also set JEMALLOC_ENABLED=true in my Heroku config and my app runs fine. Additionally, running $ ldd /app/vendor/jemalloc/lib/libjemalloc.so.1 in heroku bash will yield

	linux-vdso.so.1 =>  (0x00007ffeabfeb000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4adb7e9000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4adb41f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4adbc46000)

This is all to ask, am I right in believing that jemalloc is likely running despite not seeing it built when running the command you've provided? I just installed jemalloc onto my server so I'm not entirely sure if I've done everything right.

[ohaddahan]

@k5o you can try this CheckJEMalloc , this actually tries using a symbol from JEMalloc.

[bjeanes]

@ohaddahan that script is a great idea.

However, like @k5o, my RbConfig::CONFIG['LIBS'] also doesn't contain -ljemalloc so I used that module as a test, and it returned:

irb(main):001:0> CheckJEMalloc.je_malloc_exists?
FFI::NotFoundError : Function 'je_malloc' not found in [/app/vendor/jemalloc/lib/libjemalloc.so.2]
=> false
irb(main):002:0> ENV['LD_PRELOAD']
=> "/app/vendor/jemalloc/lib/libjemalloc.so.2"
irb(main):003:0> puts `file ENV['LD_PRELOAD']`
/app/vendor/jemalloc/lib/libjemalloc.so.2: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=78d9b314f3040553f25655f32de403fa5f472f97, with debug_info, not stripped
=> nil

Now, I wonder if that script only works with older jemalloc (I have libjemalloc.so.2 but this discussion seems to only mention libjemalloc.so.1).

[k5o]

I am seeing the same result as @bjeanes, @ohaddahan

[ohaddahan]

RbConfig::CONFIG['LIBS'] will only work if Ruby was compiled with JEMalloc.

You can try and use a different symbol from je_malloc.so which is JEMalloc specific.
You can also verify it by running nm -a libjemalloc.so.2 to see that symbol exists.

Also, run heroku run bash , run env LD_PRELOAD=.... rails console and retry, also check that you're not getting any LD_PRELOAD warnings about not being able to load the library

[bjeanes]

Unfortunately, on Heroku:

bash: nm: command not found

I tried linking directly to malloc from that object, which did work, but this is beyond my expertise so I'm not sure if that would work have otherwise worked anyway.

RbConfig::CONFIG['LIBS'] will only work if Ruby was compiled with JEMalloc.

Yeah I realise that. That was just context for why I was using the Ruby script you posted instead.

check that you're not getting any LD_PRELOAD warnings about not being able to load the library

No warnings

[ohaddahan]

I'm able to run nm in heroku run bash , I'm seeing that je_malloc symbol was stripped.
I need to play around with it and find some other symbol that can be dynamically loaded.

[bjeanes]

I'm able to run nm in heroku run bash

Ah perhaps we are on different stacks then or nm is provided by one of your buildpacks.