Backtrace of “git clone” running inside qemu-user-emulation based arm-chroot

chrootdpkggdbmultiarchqemu

I'm running a wheezy:armhf chroot using qemu user emulation on my jessie:x86_64 system. Somehow, a git clone on a particular private repository will hang inside the chroot, while succeed natively. This might be a bug, who knows? To improve my karma, I want to find out what's going on!

As a side-note: the hang I'm experiencing is occurring with git-2.0 inside jessie:armel chroot as well… The hang does not occur inside a full-system-emulation. So I went on digging in the wheezy:armhf rabbithole, just because I had to choose one… I cannot test on a native machine…

So. There is no git-dbg packet, I roll my own. Inside the wheezy:armhf chroot:

sudo apt-get install build-essential fakeroot
sudo apt-get build-dep git
apt-get source git && cd git-1.7.10.4
DEB_CFLAGS_APPEND="-fno-stack-protector" DEB_CXXFLAGS_APPEND="-fno-stack-protector" DEB_BUILD_MAINT_OPTIONS=hardening=-stackprotector,-fortify DEB_BUILD_OPTIONS="noopt nostrip nocheck" fakeroot dpkg-buildpackage -j´getconf _NPROCESSORS_ONLN`
sudo dpkg -i ../git_1.7.10.4-1+wheezy1_armhf.deb

As far as I read the gcc-documentation, setting DEB_CFLAGS_APPEND and DEB_CXXFLAGS_APPEND additionally with -fno-stack-protector is not needed, but anyhow, want to be sure)

Then, using qemu's builtin gdb_stub inside the chroot I'm doing:

QEMU_GDB=1234 git clone /path/to/breaking/repo /tmp/bla

Debugging inside the qemu throws an unsupported syscal 26 error.

Firing up gdb-multiarch outside the chroot, to connect:

gdb-multiarch -q
(gdb) set architecture arm                    # prevents "warning: Architecture rejected target-supplied description"
(gdb) target remote localhost:1234
(gdb) set sysroot /opt/chroots/wheezy:armhf
(gdb) file /opt/chroots/wheezy:armhf/usr/bin/git
Reading symbols from /opt/chroots/wheezy:armhf/usr/bin/git...done. # good! has debug symbols!
(gdb) list                                    # works! code is not stripped
(gdb) step
Cannot find bounds of current function        # meh...
(gdb) backtracke
#0  0xf67e0c90 in ?? ()
#1  0x00000000 in ?? ()                       # wtf?

Giving a continue to let the clone happen will result in a hang, sending a ctrl-c is ignored.

Generating a core-file and loading it into gdb (inside the chroot) will give me a corrupt stack:

gdb -q /usr/bin/git qemu_git_20140514-160951_22373.core
Reading symbols from /usr/bin/git...done.
[New LWP 22373]
Cannot access memory at address 0xf67fe948
Cannot access memory at address 0xf67fe944
(gdb) bt
#0  0xf678b3e4 in ?? ()
#1  0xf678b3d4 in ?? ()
#2  0xf678b3d4 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Now I'm lost.

Where is the problem? Did I miss some detail in the qemu-user-emulation? Do I have to use a completely emulated arm-machine? A misunderstanding in cross-debugging? gdb-multiarch limitations? The creation of debug-packages?

Thanks for any suggestions, pointers, hints, tips, comments and what-not.

My best guess in the moment is based on the fact that git does a clone (I can see two processes/threads), but the QEMU_GDB environment variable is unset by qemu after using. Hence only the initial process is going to gdb. See here for example.

But still: I should be able to properly debug the parent process? I can easily cross-debug a hello-world MWE.

Best Answer

Turns out that this particular hang of "git clone" is a qemu-related problem... Other problems in the qemu-user-emulation prevail, so I have to fall back to full-system emulation... ;-(

Using a qemu-user-static, compiled from their git (the qemu-2.0.0+dfsg-4+b1 currently in jessie has less fixes and won't work for my case...):

git clone git://git.qemu-project.org/qemu.git $HOME/qemu.git && cd $HOME/qemu.git
./configure --static --disable-system --target-list=arm-linux-user --prefix=$HOME/qemu.install --disable-libssh2
make && make install
sudo cp $HOME/qemu.install/bin/qemu-arm /opt/chroots/wheezy:armhf/usr/bin/qemu-arm-static

so...

However, I'm still not able to get backtrace of complex programs...

Related Question