Linux – Does kernel need to be compiled in same dev environment as userspace

I made it :-)

I basically followed Gilles's advice and decided to do it properly: i.e. manage a complete cross-compilation of GLIBC. I started from crosstool-ng, and was initially disappointed - seeing that it didn't support my old kernel. I kept at it, though - manually editing the configuration file saved by crosstool-ng to do changes like these on the default arm-gnueabi build configuration:

$ ct-ng arm-unknown-linux-gnueabi
$ ct-ng menuconfig
...
$ vi .config
$ cat .config
...
CT_KERNEL_VERSION="2.6.17"
CT_KERNEL_V_2_6_17=y
CT_LIBC_VERSION="2.13"
CT_LIBC_GLIBC_V_2_13=y
CT_LIBC_GLIBC_MIN_KERNEL_VERSION="2.6.9"
CT_LIBC_GLIBC_MIN_KERNEL="2.6.9
...
$ ct-ng +libc

After numerous tests and failed attempts, the above changes did it - I got a compiled version of GLIBC that would work with my kernel, and copied the resulting files to my Debian Lenny ARM machine:

$ cd .build/arm-unknown-linux-gnueabi/build/build-libc-final/
$ tar zcpf newlibc.tgz $(find . -type f -iname \*.so)
$ scp newlibc.tgz root@mybook:.

I went all the way and moved past squeeze: I debootstrapped a /wheezy and then - very carefully - overwrote the GLIBC versions of the armel-debootstrapped /wheezy with my own:

# # In the ARM machine
# cd /wheezy/lib/arm-linux-gnueabi/
# mv /var/tmp/ohMyGod/libc.so libc-2.13.so
# mv /var/tmp/ohMyGod/rt/librt.so librt-2.13.so
...

...etc, making sure I didn't miss any shared libraries.

Finally, I copied over the ldd and ldconfig binaries (which were also part of GLIBC), and chrooted inside my /wheezy.

It worked.

I can only assume that the compilation of GLIBC from a chroot-ed 'qemu-arm' emulation inside a x86, somehow messed things up - maybe the configure process detects some stuff from the running environment - whereas the cross-compilation can't be misled.

So naturally I moved to the next step, and used a busybox-static shell to replace the {/bin,/sbin,...} folders of my old lenny with the wheezy ones - and rebooted into my brand new Wheezy :-)

I hereby claim that my WD MyBook World Edition is the only one on the planet running Debian Wheezy :-) If anyone else is interested, I can upload a tarball of the libc files someplace.

Very roughly:

/dev contains device nodes, which in earlier Unix systems was the only way to interact with the kernel. There are two types of these, block devices and character devices. The corresponding API is geared to something that will allow block-based I/O (some kind of disk) or character based I/O (e.g. a serial port).

/sys (and /proc) were added later, possibly inspired by the Plan 9 OS. They provide complete directory subtrees, and the file entries in these subtrees contain text that describes the internal state of the kernel module when read, or, when written, set the internal state.

So a typical application would be:

You want to write a kernel driver for some kind of storage device? Use a /dev node to access the device itself, and /sys (or /proc) entries to fine-tune how the storage gets accessed.