Utilities – Unbuffer vs Stdbuf for Removing Stdout Buffering

bufferiostdoutUtilities

Is there a difference between unbuffer(1) and stdbuf(1)? From what I gather, unbuffer makes more than the "best effort" of calling the libc function set(X)buf at the beginning, and then letting things be?

Best Answer

They work in completely different ways.

The program unbuffer uses expect to run the named command. Because expect creates a pseudo-tty to attach to the stdout of the child process, the child might be fooled into thinking that it should use line-buffering instead of block-buffering. Some programs will change their behaviour when isatty(stdout) is true, others won't and it is very hard to know which will and which won't.

The program stdbuf attempts to put libstdbuf in front of libc for dynamically loaded binaries. Where libstdbuf redefines the default buffering strategy of the libc stdio calls.

I found this out by

 apt-get source expect coreutils

and reading the relevant source for each program.

Related Solutions

Bash – Avoid output buffering for a command group (curly braces) in bash script

(Note to future readers: the tone of exasperation here is not for the question, but for the mistakes I made trying to answer it and the multiple edits they entailed.)

Oh, for pity's sake. The problem is in tail -f. This works just fine:

#!/bin/bash
printf 'hi\n'
{
    for i in 1 2 3 4; do
        sleep 0.5
        /bin/echo $i
    done;
} | cat
printf 'bye\n'

It's not the pipe, it's not the group. It's tail. As in, chasing our own tails!

So, tail -f failed because it doesn't output right away for some reason. Not sure why python -u is failing, but I don't think it's anything in the script. Maybe try unbuffer with it. Try your script with cat, at least, and verify that it's unbuffered in that case.

Earlier failed attempt intentionally left here so future readers can make sense of the comments.

This script exhibits the same kind of buffering problem you're getting:

#!/bin/bash
printf 'hi\n'
{
    for i in 1 2 3 4; do
    sleep 0.5
    printf '%s\n' $i
    done;
} | tail -f
printf 'bye\n'

This one does not. Output inside the group is redirected to stderr, then stderr from the whole group is piped to the command. Since it's stderr, it's unbuffered.

#!/bin/bash
printf 'hi\n'
{
    for i in 1 2 3 4; do
    sleep 0.5
    printf '%s\n' $i 1>&2
    done;
} |& tail -f
printf 'bye\n'

Adapted from Wang HongQin's answer in this question. The difficulty was in finding a way to unbuffer the pipe with braces rather than an explicit command. Had to fiddle around a while to get the redirection working properly.

stdout – Expected Behavior of stdbuf for Subprocesses

straceing the execve (with environ) and write system calls can help see what's going on:

Here with the stdbuf of GNU coreutils 8.25. I beleive FreeBSD's stdbuf works similarly:

exec and no fork:

$ env -i strace -s200 -vfe execve,write /usr/bin/stdbuf -o0 /usr/bin/env /usr/bin/env > /dev/null
execve("/usr/bin/stdbuf", ["/usr/bin/stdbuf", "-o0", "/usr/bin/env", "/usr/bin/env"], []) = 0
execve("/usr/bin/env", ["/usr/bin/env", "/usr/bin/env"], ["_STDBUF_O=0", "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so"]) = 0
execve("/usr/bin/env", ["/usr/bin/env"], ["_STDBUF_O=0", "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so"]) = 0
write(1, "_STDBUF_O=0\n", 12)           = 12
write(1, "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so\n", 60) = 60
+++ exited with 0 +++

LD_PRELOAD and the config in _STDBUF_O is passed to both env commands. The two write() system calls even though the output doesn't go to a terminal confirms the output is not buffered.

fork and exec:

$ env -i strace -s200 -vfe execve,write /usr/bin/stdbuf -o0 /bin/sh -c '/usr/bin/env; :' > /dev/null
execve("/usr/bin/stdbuf", ["/usr/bin/stdbuf", "-o0", "/bin/sh", "-c", "/usr/bin/env; :"], []) = 0
execve("/bin/sh", ["/bin/sh", "-c", "/usr/bin/env; :"], ["_STDBUF_O=0", "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so"]) = 0
Process 16809 attached
[pid 16809] execve("/usr/bin/env", ["/usr/bin/env"], ["_STDBUF_O=0", "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so", "PWD=/home/stephane"]) = 0
[pid 16809] write(1, "_STDBUF_O=0\n", 12) = 12
[pid 16809] write(1, "LD_PRELOAD=/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so\n", 60) = 60
[pid 16809] write(1, "PWD=/home/stephane\n", 19) = 19
[pid 16809] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,

Same situation.

So yes stdbuf applies to the command it runs and all of its descendants (provided they don't clean their environment like the dynamic linker or libc do with LD_PRELOAD for setuid/setgid... applications).