Is it safe to redirect stdout and stderr to the same file without file descriptor copies

io-redirectionstderrstdout

I start off in empty directory.

$ touch aFile
$ ls
aFile

Then I ls two arguments, one of which isn't in this directory. I redirect both output streams to a file named output. I use >> in order to avoid writing simultaneously.

$ ls aFile not_exist >>output 2>>output
$ cat output
ls: cannot access 'not_exist': No such file or directory
aFile

Which seems to work. Are there any dangers to this approach?

Best Answer

No, it's not just as safe as the standard >>bar 2>&1.

When you're writing

foo >>bar 2>>bar

you're opening the bar file twice with O_APPEND, creating two completely independent file objects[1], each with its own state (pointer, open modes, etc).

This is very much unlike 2>&1 which is just calling the dup(2) system call, and makes the stderr and stdout interchangeable aliases for the same file object.

Now, there's a problem with that:

O_APPEND may lead to corrupted files on NFS filesystems if more than one process appends data to a file at once. This is because NFS does not support appending to a file, so the client kernel has to simulate it, which can't be done without a race condition.

You usually can count on the probability of the file like bar in foo >>bar 2>&1 being written to at the same time from two separate places being quite low. But by your >>bar 2>>bar you just increased it by a dozen orders of magnitude, without any reason.

[1] "Open File Descriptions" in POSIX lingo.

Related Solutions

Shell – Can redirecting stdout and stderr to the same file mangle lines

Yes, exactly that can happen if lines to stdout are long enough.

#!/usr/bin/perl
use strict;
use warnings;

for (1..10) {
    print "START"; print "-" x 100000; print "END\n";
    warn "!\n";
}

Running:

./writer.pl > out 2>&1

Checking: Open file out in an editor and find stderr ! in between -, not always between END and START

This will vary between OS:s, languages and systems in various ways, but your basic assumption is correct. Experiment by varying length of stdout-lines: print "-" x 100, print "-" x 10000 etc.

Bash – How to Redirect Bash stdout+stderr to One File and stderr to Another File

With zsh (and zsh only) and its multios feature:

your-cmd 2> stdout+stderr.log >&2 2> stderr.log

As fd 2 is redirected twice, zsh implements an internal tee to have it sent to both files.

With bash (or any Bourne-like shell (other than zsh where you'd need to disable multios for it to work here)), you can do the teeing by hand with:

{ your-cmd 2>&1 >&3 3>&- | tee stderr.log 3>&-; } > stderr+stdout.log 3>&1

(though you lose the exit status of your-cmd. zsh has it in $pipestatus[1], bash in "${PIPESTATUS[0]}" though (provided the redirection into stderr+stdout.log didn't fail)).

To record the pid of your-cmd, you could do:

{ sh -ec 'echo "$$" > /var/run/pidfile; exec your-cmd' 2>&1 >&3 3>&- |
   tee stderr.log 3>&-; } > stderr+stdout.log 3>&1

With yash and it's process redirection feature:

your-cmd > stdout+stderr.log 2>(tee stderr.log)

(but note that yash will not wait for the termination of that tee command, so the log files may not be complete by the time you run the next command after that).

Something similar (and with the same caveat) can be done with process substitution in bash, zsh and ksh93:

{ your-cmd 2> >(tee stderr.log); } > stderr+stdout.log

To run in background and get the pid:

(exec your-cmd 2> >(tee stderr.log)) > stderr+stdout.log & pid=$!

With rc:

{your-cmd |[2=0] tee stderr.log} > stdout+stderr.log

rc's pipes allow specifying which file descriptors are connected to the pipe. With other shells, it's always fd 1 of the left command and fd 0 of the right one (hence the little dance with fd 3 above to shift file descriptors around). rc will report failure if either your-cmd or tee fails, though the exact number may be lost.

Best Answer

Related Solutions

Shell – Can redirecting stdout and stderr to the same file mangle lines

Bash – How to Redirect Bash stdout+stderr to One File and stderr to Another File

Related Question