How to use netconsole to help solve a kernel panic

gentookernelkernel-panicnetworking

When I try to use a raid as root, my system panics.

I want to read the kernel output right before the panic, but can't scroll up.
After Googling a bit, I wanted to try netconsole.
but I get no output at all on the 2nd machine

relevant parts of my current grub.conf

title Gentoo Linux 3.0.6
root (hd0,0)
kernel /boot/linux-3.0.6-gentoo root=/dev/md3

title Gentoo Linux 3.0.6 (debug netconsole)                                                                                                    
root (hd0,0)                                                                                                                                   
kernel /boot/linux-3.0.6-gentoo-debug root=/dev/md3 netconsole=@/,6666@192.168.0.27/00:18:f3:a8:09:61                                          
#kernel /boot/linux-3.0.6-gentoo-debug root=/dev/md3 netconsole=6665@192.168.0.26/eth0,6666@192.168.0.27/00:18:f3:a8:09:61

title Gentoo Linux 3.0.6 (no root/auto root)
root (hd0,0)
kernel /boot/linux-3.0.6-gentoo
  • the 1st fails with a kernel panic
  • the 2nd is the one i try to use netconsole
  • the 3th starts but use the wrong/old root

The other computer is running a Kubuntu LiveCD where I started nc -u -l 6666.
And if I start the (first) computer with 3th grub option, I can run nc -u 192.168.0.27 6666 to communicate with the 2nd computer.

The difference between /boot/linux-3.0.6-gentoo and /boot/linux-3.0.6-gentoo-debug is that I enabled:

  • Kernel hacking -> Kernel debugging
  • Kernel hacking -> Delay each boot printk message by N milliseconds
  • Device Drivers -> Network device support -> Network console logging support

Questions

  1. Is my (2nd) grub config correctly defined?
  2. Are there more kernel config options I should be enabling?
  3. Is there more information about the specific machine hardware I should provide detail for?
  4. Is there some way to just enable me to scroll back up after a panic?

Best Answer

I just had the same problem on gentoo with a 3.5.0 kernel. No matter what online example I tried no output got to the client until I ran the following command:

 dmesg -n 7

I originally tried

dmesg -n 8

which was suggested in the kernel documentation but that returned

localhost ~ # dmesg -n 8
dmesg: unknown level '8'

where dmesg -n 7 worked

localhost ~ # dmesg -n 7
localhost ~ # echo check > /sys/block/md0/md/sync_action

Here is what I got on the client

datastore1 ~ # nc -l -p 30000 -u
[ 9698.913098] md: data-check of RAID array md0
[ 9698.913112] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 9698.913124] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[ 9698.913150] md: using 128k window, over a total of 1048512k.
[ 9718.197555] md: md0: data-check done.

BTW, Here is how I setup the netconsole: 

modprobe netconsole netconsole="@/eth0,30000@192.168.1.31/"
Related Question