Shell – Send stdin to console and compressed file

linuxshellterminal

I'm running a propietary program which generates text output. Kind of like a logger.

logger

I'd like to both view this output on the terminal to watch the server breathe, and send it to a text file. So normally I'd do:

logger | tee /tmp/log.txt

But these log files can become large enough to overwhelm my VMs disk space. Instead of sending the text to an uncompressed text file, I'd like it to be compressed immediately. So I might do:

logger

in one terminal and 

logger | gzip > /tmp/log.gz

in another terminal. But this doesn't feel very *nixy.

Is there a way I can accomplish this in one command, similar to using tee? Here is what I'm going for, which obviously won't work, but maybe you'll get the idea:

logger | tee gzip > /tmp/log.txt

Best Answer

With ksh93, zsh or bash:

logger | tee >(gzip > /tmp/log.txt.gz)

That uses that ksh feature called process substitution. >(gzip > /tmp/log.txt.gz) is substituted with the path of a file (typically something like /dev/fd/something, but that could also be a temporary named pipe) that refers to the writing end of a pipe. At the other (reading) end of that pipe, the shell connects the standard input of a new process (run in background) that executes the gzip command.

So, when tee writes to that file, it's actually feeding the data to gzip.

On systems with /dev/fd/n, you can do it manually (with any Bourne-like shell but ksh93) like:

{ logger | tee -a /dev/fd/3 | gzip > /tmp/log.txt.gz; } 3>&1

(though in that case /dev/fd/3 refers to the original stdin which we've made available on the file descriptor 3 with 3>&1, not to the pipe to gzip which here is just connected to tee's stdout)

With zsh:

logger >&1 > >(gzip > /tmp/log.txt.gz)

That uses zsh multios feature whereby zsh implements a sort of tee internally to redirect the same output to more than one file (here the original stdout (>&1) and the pipe to gzip using process substitution again.

logger's stdout will actually be the writing end of a pipe. At the other end of the pipe is a shell process that reads it and distributes it both outputs like tee would.

Related Question