Linux – the difference between /proc/self/stack and output from pstack

debuggingkernellinux-kernelmonitoringproc

I have been looking through the documentation for /proc and the "stack" object being a new'ish object in proc, I have also looked through the kernel commit to create it — however the documentation does not detail exactly what is in the /proc/self/stack file — and since I intuitively expected it to be the actual stack of the process — however the old pstack tool gives a different (and more believable) output.

So as an example of the stack for bash

$ cat /proc/self/stack 
[<ffffffff8106f955>] do_wait+0x1c5/0x250
[<ffffffff8106fa83>] sys_wait4+0xa3/0x100
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

and, using pstack

$ pstack $$
#0  0x00000038cfaa664e in waitpid () from /lib64/libc.so.6
#1  0x000000000043ed42 in ?? ()
#2  0x000000000043ffbf in wait_for ()
#3  0x0000000000430bc9 in execute_command_internal ()
#4  0x0000000000430dbe in execute_command ()
#5  0x000000000041d526 in reader_loop ()
#6  0x000000000041ccde in main ()

The addresses are different, and obviously the symbols are not at all the same….

Does anybody have an explanation for the difference and/or a document which describes what is actually shown in /proc-stack?

Best Answer

The file /proc/$pid/stacks shows kernel stacks. On your system, memory addresses of the form ffffffff8xxxxxxx are in the space that's reserved for the kernel. There's not much documentation, you can check the source code. In contracst, the pstack program shows user-space stacks (using its knowledge of executable formats).

Related Solutions

Difference Between ‘all’, ‘default’, and ‘eth*’ in /proc/sys/net/ipv[46]/conf/

I've found the answer while still writing the question. I've decided to post it anyway because others may find this insightful, and then answer it myself; I hope this is not frowned upon :)

The user Philipp Matthias Hahn on the linux-kernel mailing list has figured it out at least partially:

As far as I researched for IPv4 some time ago, the "default" value gets
copied to newly created interfaces only once.
"all" on the other hand allways gets applied in addition to the current
setting, but it depends on the exact setting, if its ORed, ANDed, or
whatevered:
    log_martians         OR
    accept_redirects     AND
    forwarding           ?
    mc_forwarding        AND
    medium_id
    proxy_arp            OR
    shared_media         OR
    secure_redirects     OR
    send_redirects       OR
    bootp_relay          AND
    accept_source_route  AND
    rp_filter            AND
    arp_filter           OR
    arp_announce         MAX
    arp_ignore           MAX
    arp_accept
    app_solicit
    disable_policy
    disable_xfrm
    tag
(see include/linux/inetdevice.h:83 for IN_DEV_{AND,OR,MAX}CONF)

Putting a new value in "all" doesn't change the value you read from
"$interface", but it only gets computed and used internally.

He doesn't cover accept_ra but at least it's clear now how all and default work, or rather, how they do not work as I would have expected.

Linux Kernel – Difference Between sock->sk_wmem_alloc and sock->sk_wmem_queued

I emailed Eric Dumazet, who contributes to the Linux network stack, and here is the answer:

sk_wmem_alloc tracks the number of bytes for skb queued after transport stack : qdisc layer and NIC TX ring buffers.

If you have 1 MB of data sitting in TCP write queue, not yet sent (cwnd limit), sk_wmem_queue will be about 1MB, but sk_wmem_alloc will be about 0

A very good document for understanding what these three types of queues (socket buffer, qdisc queue and device queue) are is this article (rather long) article. In a nutshell, the socket starts by pushing the packets directly onto the qdisc queue, which forwards them to the device queue. When the qdisc queue is full, the socket starts buffering the data in its own write queue.

the network stack places packets directly into the queueing discipline or else pushes back on the upper layers (eg socket buffer) if the queue is full

So basically: sk_wmem_queues is the memory used by the socket buffer (sock.sk_write_queue) while sk_wmem_alloc is the memory used by the packets in the qdisc and device queues.

Related Question

Kernel Logs – Difference Between /proc/kmsg and /dev/kmsg