How to Turn Off Stdout Buffering in a Pipe

bufferpipeshell

I have a script which calls two commands:

long_running_command | print_progress

The long_running_command prints progress but I'm unhappy with it. I'm using print_progress to make it nicer (namely, I print the progress in a single line).

The problem: Connection a pipe to stdout also activates a 4K buffer, so the nice print program gets nothing … nothing … nothing … a whole lot … 🙂

How can I disable the 4K buffer for the long_running_command (no, I do not have the source)?

Best Answer

You can use the unbuffer command (which comes as part of the expect package), e.g.

unbuffer long_running_command | print_progress

unbuffer connects to long_running_command via a pseudoterminal (pty), which makes the system treat it as an interactive process, therefore not using the 4-kiB buffering in the pipeline that is the likely cause of the delay.

For longer pipelines, you may have to unbuffer each command (except the final one), e.g.

unbuffer x | unbuffer -p y | z

Related Solutions

Python – Realtime print statements with tee in interactive script

Specify a zero-sized buffer for Python's standard output stream. You can do this by invoking Python with the -u flag, or with the following statement.

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Bash Pipe – Why Data Loss Occurs in This Construction

Because the nc command inside <(...) will also read from stdin.

Simpler example:

$ nc -l 9999 >/tmp/foo &
[1] 5659

$ echo text | cat <(nc -N localhost 9999) -
[1]+  Done                    nc -l 9999 > /tmp/foo

Where did the text go? Through the netcat.

$ cat /tmp/foo
text

Your program and nc compete for the same stdin, and nc gets some of it.

Best Answer

Related Solutions

Python – Realtime print statements with tee in interactive script

Bash Pipe – Why Data Loss Occurs in This Construction

Related Question