Linux – Obtain both merged and separated stdout and stderr

bashlinuxredirection

I am looking for a way to run a program Prog and end up with 3 files:

The stdout of Prog
The stderr of Prog
Both stdout and stderr of Prog combined just as they would be on screen if no redirecting took place.

Is there a combination of redirecting, pipes etc. which could achieve this?

Note: I normally use bash.

Best Answer

As far as I know, there is no elegant way of doing this (for an inelegant way that works but ain't pretty, scroll down to the end of my answer). I doubt you can do any better than:

$ command >stdout.txt 2>stderr.txt && cat stdout.txt stderr.txt > both.txt

There are various cool tricks you can use but none of them seems to succeed in producing the 3 files any better than the above. The main problem is that the file both.txt will not show the messages (STDERR and STDOUT) in the correct order. This is because (as explained here):

When you redirect both standard output and standard error to the same file, you may get some unexpected results. This is due to the fact that STDOUT is a buffered stream while STDERR is always unbuffered. This means that every character of STDERR is written as soon as it is available while STDOUT writes stuff in batches. When both STDOUT and STDERR are going to the same file you may see error messages appear sooner than you would have expected them in relation to the actual output of your program or script. It isn’t anything to be alarmed about but is simply a side-effect of buffered vs. unbuffered streams, you just need to keep it in mind.

The best alternative I could find was using bash subshells, is kind of complex and still does not display the output in the correct order. I made a simple Perl script, test.pl that prints "OUT" to STDOUT and "ERR" to STDERR, repeating the process 3 times:

#/usr/bin/perl 
for($i=0; $i<=2; $i++){
    print STDOUT "OUT\n"; 
    print STDERR "ERR\n"
}

Its normal, un-redirected output is :

$ ./test.pl
OUT
ERR
OUT
ERR
OUT
ERR

To redirect output(s) I ran:

(./test.pl 2> >(tee error.txt) > >(tee out.txt)) > both.txt

This uses tee, a program that will print its input to screen and to a file name. So, I am redirecting STDERR and passing it as input to tee, telling it to write it to the file error.txt. Similarly with STDOUT and the file out.txt. I am placing the whole thing in a subshell ((...)) so I can then capture all of its output and redirect to both.txt.

Now, this works inasmuch as it creates 3 files, one with STDERR, one with STDOUT and one with both. However, as explained above, this results in the messages appearing in the incorrect order in both.txt:

$ cat both.txt 
ERR
ERR
ERR
OUT
OUT
OUT

The only way around this I could find was to append the time it was printed to each line of output and then sorting, but it is getting seriously convoluted and, in your place, I would ask myself if it is really worth it:

 $(./test.pl \
   2> >(while read n; do echo `date +%N`" $n"; echo "$n" >>error.txt; done) \
    > >(while read n; do echo `date +%N`" $n"; echo "$n" >> out.txt; done )) \
 | gawk '{print $2}'> both.txt

Named pipes

What I did was write two programs for .NET 4. One sends output to a named pipe, the other reads from this pipe and displays to the console. Usage is quite simple:

asdf.exe | NamedPipeServer.exe "APipeName"

In another console window:

NamedPipeClient.exe "APipeName"

Unfortunately, this can only redirect stdout (or stdin, or combined), not stderr by itself, due to limitations in the pipe operator (|) in the Windows Command Prompt. If you figure out how to send stderr through that pipe operator, it should work. Alternatively, the server could be modified to launch your program and specifically redirect stderr. If that is necessary, let me know in a comment (or do it yourself); it's not too difficult if you have some C# and .NET "Process" library knowledge.

You can download the server and the client.

If you close the server after connection, the client will close immediately. If you close the client after connection, the server will close as soon as you try to send something through it. It's not possible to reconnect a broken pipe, mostly because I can't be bothered doing something so complicated right now. It's also limited to one client per server.

Source code

These are written in C#. There's not much point trying to explain it. They use the .NET NamedPipeServerStream and NamedPipeClientStream.

The server:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO.Pipes;
using System.IO;

namespace NamedPipeServer
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args == null || args.Length == 0)
            {
                Console.Error.WriteLine("[NamedPipeServer]: Need pipe name.");
                return;
            }
            
            NamedPipeServerStream PipeServer = new NamedPipeServerStream(args[0], System.IO.Pipes.PipeDirection.Out);
            PipeServer.WaitForConnection();
            StreamWriter PipeWriter = new StreamWriter(PipeServer);
            PipeWriter.AutoFlush = true;

            string tempWrite;

            while ((tempWrite = Console.ReadLine()) != null)
            {
                try
                {
                    PipeWriter.WriteLine(tempWrite);
                }
                catch (IOException ex)
                {
                    if (ex.Message == "Pipe is broken.")
                    {
                        Console.Error.WriteLine("[NamedPipeServer]: NamedPipeClient was closed, exiting");
                        return;
                    }
                }
            }

            PipeWriter.Close();
            PipeServer.Close();
        }
    }
}

The client:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO.Pipes;
using System.IO;

namespace NamedPipeClient
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args == null || args.Length == 0)
            {
                Console.Error.WriteLine("[NamedPipeClient]: Need pipe name.");
                return;
            }

            NamedPipeClientStream PipeClient = new NamedPipeClientStream(".", args[0], System.IO.Pipes.PipeDirection.In);
            PipeClient.Connect();
            StreamReader PipeReader = new StreamReader(PipeClient);

            string tempRead;

            while ((tempRead = PipeReader.ReadLine()) != null)
            {
                Console.WriteLine(tempRead);
            }

            PipeReader.Close();
            PipeClient.Close();
        }
    }
}

Redirecting to a file

type NUL>StdErr.temp
start powershell -c Get-Content StdErr.temp -Wait
MyExecutable.exe 2>StdErr.temp

Create an empty file
Start a new console window that watches the file
Run executable and redirect stderr output to that file

This provides the desired effect of one console window to watch stdout (and provide stdin), and another to watch stderr.

Anything that mimics tail would work. The PowerShell method works natively in Windows, but may be a bit slow (i.e. there's some latency between the writing to the file and the displaying to the screen). See this StackOverflow question for other tail alternatives.

The only problem is the temporary file may grow quite large. A possible workaround is to run a loop that only prints if the file has content and clear the file immediately afterwards, but that would cause a race condition.

Best Answer

Related Solutions

Immediately tell which output was sent to stderr

Redirect Stdout to Named Pipe in Windows Command Line

Named pipes

Source code

Redirecting to a file

Related Question