Why don’t Unix/Linux systems traverse through directories until they find the required version of a linked library

dynamic-linkingldlibrariesshared library

I have a binary executable named "alpha" that requires a linked library (libz.so.1.2.7) which is placed at /home/username/myproduct/lib/libz.so.1.2.7

I export the same to my terminal instance before spawning my binary executable by executing the following command.

export LD_LIBRARY_PATH=/home/username/myproduct/lib/:$LD_LIBRARY_PATH

Now, when I spawn another application "bravo" that requires the same library but of different version, i.e (libz.so.1.2.8) which is available in
/lib/x86_64-linux-gnu/libz.so.1.2.8, system throws the following error.

version `ZLIB_1.2.3.3' not found (required by /usr/lib/x86_64-linux-gnu/libxml2.so.2)

If I unset the LD_LIBRARY_PATH, "bravo" starts up fine. I understand that the above behaviour is because LD_LIBRARY_PATH takes precedence over the directory paths defined in /etc/ld.so.conf while looking for linked libraries and consequently the above error occurred. I am just curious about why have not the developers of UNIX/LINUX designed the OS to search for linked libraries in other directories according to the hierarchy if the first instance of library is of different version.

Simply put, UNIX/LINUX systems traverse through a set of directories until it finds the required library. But why does it not do the same until it finds the expected version rather than accepting the first instance of library irrespective of its version?

Best Answer

But why does it not do the same until it finds the expected version rather than accepting the first instance of library irrespective of its version?

It does, as far as it’s aware. zlib.so.1.2.7 and zlib.so.1.2.8 both have an soname of zlib.so.1, so your alpha and bravo binaries say they need zlib.so.1. The dynamic loader loads the first matching library it finds; it doesn’t know that version 1.2.8 provides additional symbols which bravo needs. (This is why distributions take pains to specify additional dependency information, such as zlib1g (>= 1.2.8) for bravo.)

You might think this should be easy to fix, but it isn’t, not least because binaries and libraries list the symbols they need separately from the libraries they need, so the loader can’t check that a given library provides all the symbols that are needed from it. Symbols can be provided in a variety of ways, and introducing a link between symbols and the libraries providing them could break existing binaries. There’s also the added fun of symbol interposition to complicate things (and make security-sensitive developers tear their hair out).

Some libraries provide version information which ends up being stored in .gnu.version_r, with a link to the providing library, which would help here, but libz isn’t one of them.

(Given the sonames, I’d expect your alpha binary to work fine with zlib.so.1.2.8.)

LD

You can add a file to this directory with the path to your newly installed library like so:

$ cat /etc/ld.so.conf.d/myboost.conf
/usr/local/lib/boost1.55

Then run this command:

$ ldconfig -v

This will process all the libraries and rebuild a "cache", /etc/ld.so.cache. This cache is what's used to locate libraries when they're specified like so: -lboost_thread-mgw46-mt-sd-1_54.

Example output

$ ldconfig -v
/usr/lib64/atlas:
        libclapack.so.3 -> libclapack.so.3.0
        libptcblas.so.3 -> libptcblas.so.3.0
        libf77blas.so.3 -> libf77blas.so.3.0
        libcblas.so.3 -> libcblas.so.3.0
        liblapack.so.3 -> liblapack.so.3.0
        libptf77blas.so.3 -> libptf77blas.so.3.0
        libatlas.so.3 -> libatlas.so.3.0
/usr/lib64/wxSmithContribItems:
        libwxflatnotebook.so.0 -> libwxflatnotebook.so.0.0.1
...

When adding paths to the setup, I like to confirm by going through this output to make sure things are getting picked up the way I expect them to.

LD's cache

You can always print the contents of the .cache file as well using this command:

$ ldconfig -p | head -10
2957 libs found in cache `/etc/ld.so.cache'
    lib3ds-1.so.3 (libc6,x86-64) => /usr/lib64/lib3ds-1.so.3
    libzvbi.so.0 (libc6,x86-64) => /usr/lib64/libzvbi.so.0
    libzvbi-chains.so.0 (libc6,x86-64) => /usr/lib64/libzvbi-chains.so.0
    libzrtpcpp-1.4.so.0 (libc6,x86-64) => /usr/lib64/libzrtpcpp-1.4.so.0
    libzmq.so.1 (libc6,x86-64) => /usr/lib64/libzmq.so.1
    libzmq.so (libc6,x86-64) => /usr/lib64/libzmq.so
    libzipios.so.0 (libc6,x86-64) => /usr/lib64/libzipios.so.0
    libzipios.so (libc6,x86-64) => /usr/lib64/libzipios.so
    libzip.so.1 (libc6,x86-64) => /usr/lib64/libzip.so.1

Why's my ldd output still using 1.53?

This is because your binary is using dynamic libraries. So when the binary was compiled, it was against the 1.55 versions of the libraries. However when you interrogate the binary using ldd, it's within the environment that's using the contents of the .cache file. So the library within the cache that is associated with the symbols used by this binary match those for 1.53, hence you're seeing those libraries.

Your environment knows nothing of the 1.55 libraries, only your build environment, i.e. your Makefile, is aware of this.

Dynamic libraries

Think of the functions within these as symbols. The symbols are a name, and so these names often don't change from one version of a library to another. So if you were to look at a library such as boost, you can use the tool readelf to get a list of the symbols within one of these .so files.

Example

$ readelf -Ws /usr/lib64/libboost_date_time-mt.so | head -10

Symbol table '.dynsym' contains 261 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 000000335aa096a8     0 SECTION LOCAL  DEFAULT    9 
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt8bad_castD2Ev@GLIBCXX_3.4 (2)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt6locale5_ImplD1Ev@GLIBCXX_3.4 (2)
     4: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  UND _ZTINSt6locale5facetE@GLIBCXX_3.4 (2)
     5: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND wcslen@GLIBC_2.2.5 (3)
     6: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  UND _ZTISt11logic_error@GLIBCXX_3.4 (2)

In the above output you can see some of the FUNC definitions, these are the names that are used to "link" a function in an executable with a function from some .so library.

I'm over simplifying this, and probably explaining things slightly off, but I'm only trying to give you a general idea of how the mechanics of how the machinery under the hood works.

References

How do I list the symbols in a .so file

Debian – or should not shared libraries be executable (e.g. Red Hat vs Debian)

Debian changed this back in 1997, in Policy version 2.2 (see #7129, although that doesn’t give much detail at all and I haven’t found any other discussion); the reason is given thus in current versions of Policy:

Shared libraries should not be installed executable, since the dynamic linker does not require this and trying to execute a shared library usually results in a core dump.

So it’s really about managing expectations, and the principle of least surprise: don’t mark something as executable if executing it isn’t useful. On Debian and its derivatives, the only libraries which are supposed to be executable are those which do something sensible when executed (such as the C library, libpthread, and of course the dynamic linker).

On FreeBSD and OpenBSD, at least, shared libraries from the operating system are not executable. On OpenBSD this is also generally true of shared libraries in /usr/local. On FreeBSD, shared libraries in /usr/local from ports and packages are often executable. (Thanks to JdeBP for the BSD-related information.)

On Linux, I don’t think there are any particular security considerations (at least, not on a basic Linux system), because running a library doesn’t require its executable bit to be set (you can run it using the dynamic linker).