Add section to docs Addressing considerations for using shared libraries

[mabruzzo]

While helping somebody on slack with some linking issues, I realized we should probably add a short section to the documentation to explain how to use Grackle as a shared library.

We already make a bunch of scattered references to the need for defining DYLD_LIBRARY_PATH or LD_LIBRARY_PATH but we could use some improvements:

• it would be nice to have this explanation be centralized in a section on the Installation page that other sections can point to (current the lack of centralization got worse when I introduced the CMake build-system)
• I think we should probably be a little more specific about the exact changes that need to be made and phrase the change as instruction (currently, we phrase it this information like a reminder and assume the reader knows what we are talking about).
  • For example, we should tell people to add something like export LD_LIBRARY_PATH=/path/to/grackle-install/lib:$LD_LIBRARY_PATH to their shell startup script (e.g. .bashrc or .zshrc).
  • Then they also need to source the startup script with export ...
  • The variable needs to be defined whenever they run the application
• I also think we should make it a little more clear what this change does/why it needs to be made:
  • it is really hard to learn about this on your own because lots of documentation/websites use the term ambiguously use the term "liker" without much clarification. I originally found this EXTREMELY confusing because we usually use the -L flag to specify Grackle's location to a "static-linker" (like ld) during a build process and I couldn't figure out why we also needed the LD_LIBRARY_PATH. It was hard to understand that LD_LIBRARY_PATH provided information to distinct software -- the component of the operating system called a dynamic loader (a.k.a a "dynamic linker").
  • lots of online discussion warn against using LD_LIBRARY_PATH outside of software development.
    • While this is good advice, use of LD_LIBRARY_PATH is a necessary evil for using Grackle.
    • I was originally concerned by this and hesitant to make these changes because I didn't extent the full extent of the risks. So it may make sense to address them.
    • If LD_LIBRARY_PATH only specifies a directory holding a small subset of custom shared libraries (i.e. no core libraries like glibc, zlib, Glib), then the risks are extremely minimal. The worst thing that happens is if you want to use 2 different versions grackle (or you are trying to develop grackle).
  • Maybe we hide some of these details behind a toggle button (so the details aren't overwhelming)
• technically, the instructions are different if someone installs to a system-install location (on linux you need to invoke sudo ldconfig -v).¹ We can also make it clear when exactly this becomes unnecessary

Footnotes

1. The libtool provides some breadcrumbs about the fact that this needs to happen, but the CMake build-system doesn't make any such suggestion. We should be clear that rules may differ on different operating systems (I vaguely recall reading that some ldconfig commands that work on linux do very different and very destructive things on certain BSD systems -- but I may be conflating things) ↩

[mabruzzo]

For posterity, I have attached a copy of a post I made on the slack explaining things (we will need to rework some of this so that the documentation reads coherently).

Linking an application against a shared library involves 2 parts:

First, is “static-linking.” This is the part of the build-process of your application where a “static linker” like ld (the “static linker” may be wrapped by your compiler), combines data from all of your compiled object files. It encodes meta information about the shared libraries (like ABI version & transitive dependencies) that the application requires.

• You might instruct the dynamic linker that the application uses grackle with the -lgrackle flag.
• If grackle isn’t used installed to a system location, you commonly need to tell the dynamic linker where it can find gracke with the -L flag.
• Be aware that the dynamic linker does NOT usually¹ encode the grackle’s location inside the application’s metadata (even if you specified it with the -L flag).

The second is “dynamic loading” (sometimes called “dynamic linking”), which happens behind the scenes when you launch your executable.

• when you launch your executable, the operating system loads all of the required shared libraries needed by the executable (these requirements are encoded in the executable’s metadata). The component of the operating system that does this is called a “dynamic loader”.
• in the common case where the metadata doesn’t specify the shared library’s location, the operating system needs to go searching for it.
  • If the shared library isn’t inside of a system location, you can provide a hint with the LD_LIBRARY_PATH location
  • Alternatively, the dynamic loader will check for the library in system. For a couple reasons (including performance) the dynamic loader in most (all?) linux distributions does not directly look for libraries in the system directories. Instead, it checks a pre-built cache that lists all known libraries in the system directories.
  • there are other rules too for finding shared libraries (e.g. if you encode the location within the executable’s metadata), but that is beyond the

To summarize, after you install grackle (on linux) you need to provide the dynamic loader with hints about where it can find it.

• if it is installed to an atypical location, you need to use LD_LIBRARY_PATH
• if it is installed to a system location, you need to rebuilt the cache of libraries in the system locations. On linux distributions this can usually be achieved with sudo ldconfig -v (this may not be the same on all flavors of unix —you should usually read man-pages first).

The extra levels of indirection exists to help with dropping in newer ABI-compatible versions of shared libraries and for shipping precompiled versions of software.²

Footnotes

1. There may be some platforms (I think maybe macOS?) where the linker does use information from the -L flag to encode grackle’s location. There are ways to force the dynamic linker to encode where the grackle library is found in the executable (usually with the -rpath flag). The CMake build-system does this while building Grackle’s tests. This generally isn’t considered a best practice (if your application has a lot of complex dependencies this can subtly mess things up!) ↩

2. Grackle can’t currently make use of either of these features of capabilities. The current API makes pursuit of ABI-stability untenable (maybe we can fix that in a 4.0 release). Given our reliance on hdf5 (at least the way we currently link to it), shipping precompiled versions of Grackle would also be a tremendous amount of work. ↩

[mabruzzo]

When writing this documentation, it might be useful to also refer to Issue #198 (since it also provides a description of static linking and dynamic loading)