Linux – Locally-installing glibc-2.23 causes all programs to segfault

glibclinux

I'm trying to upgrade glibc on a system on which I do not have root access. Therefore, I'm installing to a local prefix. I would like some help understanding best practices for setting this up, as well as help resolving a particular issue. (The quick summary of my issue: when I include the newly-installed glibc lib path in my LD_LIBRARY_PATH, every program I tried to run, including ls, vim, pwd, etc. segfault.)

Background information:

$ uname -a
Linux 3.13.0-68-generic #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Compiler/toolchain: I'm running a locally compiled and installed from source version of gcc 5.3.0. It seems to work fine. This is installed at ~/toolchains/gcc_5.3.0

$ ls ~/toolchains/gcc_5.3.0
bin  include  lib  lib32  lib64  libexec  share

Attempting to install: glibc-2.23 from source with --prefix=~/local/

I don't have sudo on this machine (it's a shared cluster; the policy is to install your own toolchains if you need customizability, as I do).

$ echo $LD_LIBRARY_PATH
~/toolchains/gcc_5.3.0/lib:~/toolchains/gcc_5.3.0/lib64

The system-installed version of glibc is 2.19:

$ ldd --version
ldd (Ubuntu EGLIBC 2.19-0ubuntu6.7) 2.19

(Above and below, I'm substituting ~ for absolute paths above for clarity)

Problem:

I'm able to compile and install glibc-2.23 with gcc 5.3.0 as well as the system-installed gcc 4.8.4. Compilation and installation to ~/local/ works fine when LD_LIBRARY_PATH is set as above. However, in order to leverage the new glibc libraries (installed in ~/local/lib), I added ~/local/lib to the end of my current LD_LIBRARY_PATH:

$ echo $LD_LIBRARY_PATH
~/toolchains/gcc_5.3.0/lib:~/toolchains/gcc_5.3.0/lib64:~/local/lib

As soon as I do this, everything I try to run segfaults. I can't even ls or run vim. I just see bash print "Segmentation fault". I have to change my LD_LIBRARY_PATH, and then everything works fine again.

I'm not able to run gdb or strace or anything to try to figure out what's going on (those segfault, too).

Questions:

  1. Any ideas about what is happening here?

  2. I have a feeling that my approach for installation and/or setting of LD_LIBRARY_PATH is not right. What is the best practice for locally-installed gcc and locally-installed glibc? Do I need to more carefully match up versions? I just grabbed the latest stable sources of each.

  3. For my own knowledge for the future, given that gdb doesn't work, are there other ways to debug this sort of thing so I can precisely locate where the segfault is occurring?

Thank you for any thoughts you have.

Edit:
I'm generally trying to get an updated set of tools on my system, and
get required dev headers and libraries, etc. For example, to use some
advanced features of perf_events, I need a bunch of other things, such
as libaudit. That, of course, needs ldap, berkeley db, etc., etc.
Ultimately I'm left with needing some headers that seem to only be
provided by more modern versions of glibc. For example,
here's an error that I get when I'm trying to compile berkeley db;
this type seems to be defined in dirent.h, which is a header in glibc,
although the system is not finding it on the packages installed on
my system:

 -fPIC -DPIC -o .libs/os_dir.o

../src/os/os_dir.c: In function '__os_dirlist':

../src/os/os_dir.c:45:2: error: unknown type name 'DIR'

  DIR *dirp;

  ^

I'd be interested to hear if there are alternative approaches here to getting access to development headers and libs that my system is not finding on its own. The error above with DIR is perhaps a good example.

Best Answer

Because of a mismatch of ld-linux-x86-64.so.2 (man ld.so) and libc.so.

If you want to run gdb under the LD_LIBRARY_PATH setting, run as follows:

export LD_LIBRARY_PATH=~/local/lib
/lib64/ld-linux-x86-64.so.2 --library-path /lib64 /usr/bin/gdb /bin/ls

This runs /usr/bin/gdb in the old library environment and /bin/ls in the new library environment. Similarly, you can run only one command in the new library environment as follows:

export LD_LIBRARY_PATH=~/local/lib
~/local/lib/ld-linux-x86-64.so.2 /bin/echo
Related Question