Bash – Appending works without >> when redirecting standard output and error

bashdebianio-redirectionls

I have a file test.txt and have no file named test. When I tried

ls test test.txt > new 2>new

I was expecting new to be overwritten since >> is not used. But in the output file I got both contents appended. Why is it so?

Best Answer

TL;DR bash opens and truncates all involved files before anything is written to them. stdout and stderr are both sent to new because bash has already truncated the file (twice) when ls starts printing.

This is how bash prepares/handles I/O redirection. When you ask for a command to be redirected (>) to a file, bash basically opens that file, creating it if necessary. If the file already exists, it is truncated. This is done through the open system call and a few flags, in your case:

open("new", O_WRONLY|O_CREAT|O_TRUNC, 0666)

O_CREAT creates the file if it does not exist, while O_TRUNC truncates it when it does. This open system call is part of bash's initialisation for redirection, meaning that when you use several redirection operations, such as in...

$ ls test test.txt > new 2>new

... bash begins by opening all concerned files. Therefore, before running ls, it opens new twice, with the same flags:

open("new", O_WRONLY|O_CREAT|O_TRUNC, 0666)
open("new", O_WRONLY|O_CREAT|O_TRUNC, 0666)

This means that basically, when running your command, bash does the following (in that order) :

  1. Open new as standard output, create/truncate the file when necessary.
  2. Open new as standard error, create/truncate the file when necessary.
  3. Run ls: this will write contents to new.

As you can see, bash truncates all involved files before starting ls. This means that when running something with ... >new 2>new, new is basically truncated "twice", and then, outputs are redirected to it. The behaviour you are expecting would require bash to capture ls's stdout and stderr independently, and to open them one after the other, just before writing. Basically:

  1. Start ls.
  2. When something comes on stdout, open new, truncate it and write to it.
  3. When something comes on stderr, open new again, truncate it, and write to it.

However, messages may come out interweaven: the redirected program might very well write something to stdout, then something else to stderr, and then back on stdout... It would be horrible to manage all of that (and it might lead to undesirable (undefined?) behaviours...).

Related Question