Python – Realtime print statements with tee in interactive script

pipepythonrhelsudotee

I have a program long_interactive_script.py which has thousands of print statements. I want to pipe the program through tee (or an alternative) so that I can save the output.

If I do

long_interactive_script.py | tee logfile.txt

Python puts its print statements in a 4K buffer, causing me to get:

nothing, nothing, nothing, nothing, a whole lot of text!, nothing,
nothing, a sudo prompt in the middle of a word, nothing, nothing, a
whole lot of text!

In an attempt to avoid the buffer I tried:

unbuffer long_interactive_script.py | tee logfile.txt

But this causes my script to stop being interactive. So when the script breaks into a sudo prompt, it halts.

Note: I cannot simple sudo BEFORE running the script. The interactive script only requires sudo on some runs, and I don't want to ask for sudo when it isn't necessary.

More…

stdbuf -oL long_interactive_script.py | tee -a logfile.txt

works to some extent. I get all the desired data, but I also get this error:

ERROR: ld.so: object '/usr/lib64/coreutils/libstdbuf.so' from LD_PRELOAD cannot be preloaded: ignored.

Best Answer

Specify a zero-sized buffer for Python's standard output stream. You can do this by invoking Python with the -u flag, or with the following statement.

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Related Solutions

How to Turn Off Stdout Buffering in a Pipe

You can use the unbuffer command (which comes as part of the expect package), e.g.

unbuffer long_running_command | print_progress

unbuffer connects to long_running_command via a pseudoterminal (pty), which makes the system treat it as an interactive process, therefore not using the 4-kiB buffering in the pipeline that is the likely cause of the delay.

For longer pipelines, you may have to unbuffer each command (except the final one), e.g.

unbuffer x | unbuffer -p y | z

Bash – Explanation of Unpredictable Behavior of Tee Command

This,

paste out sum | awk ... | tee sum

has a race condition. paste opens sum to read it, and tee opens it for writing, truncating it. The shell starts both at approximately the same time, so it's up to chance which one gets to open the file first.

Of course in practice, the shell has to start the utilities one at a time, in some particular order. It probably does that from left to right, so paste might have a better chance of going first, but that's an implementation detail, and in any case the OS scheduler decides what runs when.

If paste gets to go first, it opens the file with the data still intact, and probably has enough time to read the data too. If tee gets to open the file before paste has read it, then paste sees an empty file instead.

Here,

paste out sum | awk ... > sum

The shell opens sum for writing, truncating it. It might do that in parallel to starting paste, but since truncating sum doesn't involve starting another utility, it probably happens first. (I'm not exactly sure if there's a rule about the order of processing redirections and starting the commands in a pipeline like this, but I wouldn't count on it.)

There's a tool called sponge to fix this issue (and a dozen questions about it). It collects the input it gets and only writes it after the input is closed. This should have sum updated correctly, always:

paste out sum | awk ... | sponge sum

Best Answer

Related Solutions

How to Turn Off Stdout Buffering in a Pipe

Bash – Explanation of Unpredictable Behavior of Tee Command

Related Question