objdump
+ gdb
minimal runnable example
TLDR:
Now for a the full educational test setup:
main.c
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int myfunc(int i) {
*(int*)(NULL) = i; /* line 7 */
return i - 1;
}
int main(int argc, char **argv) {
/* Setup some memory. */
char data_ptr[] = "string in data segment";
char *mmap_ptr;
char *text_ptr = "string in text segment";
(void)argv;
mmap_ptr = (char *)malloc(sizeof(data_ptr) + 1);
strcpy(mmap_ptr, data_ptr);
mmap_ptr[10] = 'm';
mmap_ptr[11] = 'm';
mmap_ptr[12] = 'a';
mmap_ptr[13] = 'p';
printf("text addr: %p\n", text_ptr);
printf("data addr: %p\n", data_ptr);
printf("mmap addr: %p\n", mmap_ptr);
/* Call a function to prepare a stack trace. */
return myfunc(argc);
}
Compile, and run to generate core:
gcc -ggdb3 -std=c99 -Wall -Wextra -pedantic -o main.out main.c
ulimit -c unlimited
rm -f core
./main.out
Output:
text addr: 0x4007d4
data addr: 0x7ffec6739220
mmap addr: 0x1612010
Segmentation fault (core dumped)
GDB points us to the exact line where the segfault happened, which is what most users want while debugging:
gdb -q -nh main.out core
then:
Reading symbols from main.out...done.
[New LWP 27479]
Core was generated by `./main.out'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000400635 in myfunc (i=1) at main.c:7
7 *(int*)(NULL) = i;
(gdb) bt
#0 0x0000000000400635 in myfunc (i=1) at main.c:7
#1 0x000000000040072b in main (argc=1, argv=0x7ffec6739328) at main.c:28
which points us directly to the buggy line 7.
Binutils analysis
First:
file core
tells us that the core
file is actually an ELF file:
core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from './main.out'
which is why we are able to inspect it more directly with usual binutils tools.
A quick look at the ELF standard shows that there is actually an ELF type dedicated to it:
Elf32_Ehd.e_type == ET_CORE
Further format information can be found at:
man 5 core
Then:
readelf -Wa core
gives some hints about the file structure. Memory appears to be contained in regular program headers:
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
NOTE 0x000468 0x0000000000000000 0x0000000000000000 0x000b9c 0x000000 0
LOAD 0x002000 0x0000000000400000 0x0000000000000000 0x001000 0x001000 R E 0x1000
LOAD 0x003000 0x0000000000600000 0x0000000000000000 0x001000 0x001000 R 0x1000
LOAD 0x004000 0x0000000000601000 0x0000000000000000 0x001000 0x001000 RW 0x1000
and there is some more metadata present in a notes area, notably prstatus
contains the PC:
Displaying notes found at file offset 0x00000468 with length 0x00000b9c:
Owner Data size Description
CORE 0x00000150 NT_PRSTATUS (prstatus structure)
CORE 0x00000088 NT_PRPSINFO (prpsinfo structure)
CORE 0x00000080 NT_SIGINFO (siginfo_t data)
CORE 0x00000130 NT_AUXV (auxiliary vector)
CORE 0x00000246 NT_FILE (mapped files)
Page size: 4096
Start End Page Offset
0x0000000000400000 0x0000000000401000 0x0000000000000000
/home/ciro/test/main.out
0x0000000000600000 0x0000000000601000 0x0000000000000000
/home/ciro/test/main.out
0x0000000000601000 0x0000000000602000 0x0000000000000001
/home/ciro/test/main.out
0x00007f8d939ee000 0x00007f8d93bae000 0x0000000000000000
/lib/x86_64-linux-gnu/libc-2.23.so
0x00007f8d93bae000 0x00007f8d93dae000 0x00000000000001c0
/lib/x86_64-linux-gnu/libc-2.23.so
0x00007f8d93dae000 0x00007f8d93db2000 0x00000000000001c0
/lib/x86_64-linux-gnu/libc-2.23.so
0x00007f8d93db2000 0x00007f8d93db4000 0x00000000000001c4
/lib/x86_64-linux-gnu/libc-2.23.so
0x00007f8d93db8000 0x00007f8d93dde000 0x0000000000000000
/lib/x86_64-linux-gnu/ld-2.23.so
0x00007f8d93fdd000 0x00007f8d93fde000 0x0000000000000025
/lib/x86_64-linux-gnu/ld-2.23.so
0x00007f8d93fde000 0x00007f8d93fdf000 0x0000000000000026
/lib/x86_64-linux-gnu/ld-2.23.so
CORE 0x00000200 NT_FPREGSET (floating point registers)
LINUX 0x00000340 NT_X86_XSTATE (x86 XSAVE extended state)
objdump
can easily dump all memory with:
objdump -s core
which contains:
Contents of section load1:
4007d0 01000200 73747269 6e672069 6e207465 ....string in te
4007e0 78742073 65676d65 6e740074 65787420 xt segment.text
Contents of section load15:
7ffec6739220 73747269 6e672069 6e206461 74612073 string in data s
7ffec6739230 65676d65 6e740000 00a8677b 9c6778cd egment....g{.gx.
Contents of section load4:
1612010 73747269 6e672069 6e206d6d 61702073 string in mmap s
1612020 65676d65 6e740000 11040000 00000000 egment..........
which matches exactly with the stdout value in our run.
Tested in Ubuntu 16.04 amd64, GCC 6.4.0, binutils 2.26.1.
If other people clean up ...
... you usually don't find nothing. But luckily Linux has a handler for this which you can specify at runtime. In /usr/src/linux/Documentation/sysctl/kernel.txt you will find:
[/proc/sys/kernel/]core_pattern is used to specify a core dumpfile pattern name.
- If the first character of the pattern is a '|', the kernel will treat
the rest of the pattern as a command to run. The core dump will be
written to the standard input of that program instead of to a file.
(thanks)
According to the source this is handled by the abrt
program (that's Automatic Bug Reporting Tool, not abort), but on my Arch Linux it is handled by systemd. You may want to write your own handler or use the current directory.
But what's in there?
Now what it contains is system specific, but according to the all knowing encyclopedia:
[A core dump] consists of the recorded state of the working memory of a computer
program at a specific time[...]. In practice, other key pieces of
program state are usually dumped at the same time, including the
processor registers, which may include the program counter and stack
pointer, memory management information, and other processor and
operating system flags and information.
... so it basically contains everything gdb
ever wanted, and more.
Yeah, but I'd like me to be happy instead of gdb
You can both be happy since gdb
will load any core dump as long as you have a exact copy of your executable: gdb path/to/binary my/core.dump
. You should then be able to continue business as usual and be annoyed by trying and failing to fix bugs instead of trying and failing to reproduce bugs.
Best Answer
The
core(5)
manpage describes the parameters affecting core dumps in detail, including their naming etc.To answer your stated question, there is no generalisable way to find a core dump. By default, core is dumped in the process's current working directory, if the process is allowed to write there, if there's enough room on the containing filesystem, if there's no existing core dump (under some circumstances), and if the file size and core file size limits (as set by
ulimit
or similar mechanisms) allow it. But/proc/sys/kernel/core_pattern
provides many different ways of processing core dumps, so you really need to look at that too and figure out what's going on.In your case, I don't know why the core couldn't be found initially, but I do know why you stopped getting cores after setting the redirection up: when using a pipe in
core_pattern
, the processing program must be specified using an absolute pathname.tee
on its own won't be used; you need to specify/usr/bin/tee
. Note that you should take particular care with this type of setup on multi-user systems, because the program run to process the core dump is run asroot
.On Debian derivatives I install
corekeeper
, which writes core dumps in an easily-usable manner to per-user directories under/var/crash
.