Bash – Redirect all bash script output (from inside the script) to two files: one append, one rewrite; but discard output to the console

bashexecstdouttee

I can easily redirect both standard and error output of a bash script to the file, while discarding an output to console with simple exec statement:

exec &>>/var/log/backup.log

But how do I use exec to write to two files, appending to first and rewriting the second? Probably some tee magic should be used. And also some way to mute the console should be found.

The reason for this is a backup script of mine. I want to append to the main log the events of latest backup and just write only these events to the current backup log (clearing its previous content), which should be rewritten each backup session.

Best Answer

It can be done, you need process substitution. Redirect the streams into a subprocess that calls tee and redirects the rest into the overwritten file.

exec &> >(tee -a backup.log > overwritten.log)

Note that this will overwrite the file only once in entire script, because the stream remains open until the script exits or another redirect is established.

Related Solutions

Bash – redirect and log script output

function startLogging {
    exec > >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' | tee -a $logfile)
    [ ! -z "$DEBUG" ] && exec 2>&1 || exec 2> >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' >>$logfile)
    echo "=== Log started for $$ at $(date +%F-%T) ==="
}

You need to have $logfile set to something

Bash IO Redirection – Redirect stdout and stderr to File and Display stderr to Console

2>&1 >>outputfile | tee --append outputfile

For easy testing:

echo -n >outputfile; bash -c "echo stdout >&1; echo stderr >&2" 2>&1 >>outputfile |
  tee --append outputfile; echo "outputfile:"; cat outputfile

Edit 1:

This works by writing stdout (only) to the file, making sterr stdout so that it goes through the pipe, and having tee write its output to the same file.

Both writes must be done in append mode (>> instead of >) otherwise both would overwrite each others output.

As the pipe is a buffer there is no guarantee that the output appears in the file in the right order. This would not even change if an application was connected to both file descriptors (two pipes). For guaranteed order both outputs would have to go through the same channel and be marked respectively. Or you would need some really fancy stuff:

If both stdout and stderr were redirected to a file (not the same file!) and both files were on a FUSE volume then the FUSE module could mark each single write with a timestamp so that a second application could sort the data correctly and combine it for the real output file. Or you don't mark the data but have the module create the combined output file. Most probably there is no FUSE module yet which does this...
Both stdout and stderr could be directed to /dev/null. The outputs of the application would be separated by running it through strace -f -s 32000 -e trace=write. You would have to reverse the escaping in that case. Needless to say that the application does not run faster by being traced.
Maybe the same could be reached by using an existing, simple FUSE module and tracing the module instead of the application. This may be faster than tracing the application because (or rather: if) the module probably has a lot less syscalls than the application.
If the application itself can be modified: The app could be stopped after each output (but I think this is possible from the inside only) and continue only after receiving s signal (SIGUSR1 or SIGCONT). The application reading from the pipe would have to check both the pipe and the file for new data and to send the signal after each new data. Depending on the kind of application this may be faster or even slower than the strace method. FUSE would be the maximum speed solution.

Best Answer

Related Solutions

Bash – redirect and log script output

Bash IO Redirection – Redirect stdout and stderr to File and Display stderr to Console

Related Question