Store output of a command into a ring-buffer

logstext processingUtilities

I've got a long-running command that generates a lot of output on stdout. I'd like to be able to retain, for instance, only the last three days or the last gibibyte (avoiding cutting lines in the middle), and, if possible, in file chunks not larger than 20 MiB. Each file chunk is being named with a numeric suffix or a timestamp.

Something like:

my-cmd | magic-command --output-file-template=my-cmd-%t \
                       --keep-bytes=1G \
                       --keep-time=3d \
                       --max-chunk-size=20M \
                       --compress=xz

Would write:

my-cmd-2014-09-05T10:04:23Z

When it's reaching 20M, it would compress it and open a new one, and so on, and after a while would it starts deleting the oldest files.

Does such a command exist?

I'm aware of logrotate and its ability to manage files written by other applications, but I'm looking for something simpler that doesn't involve having to set up a cron job, specify rules, suspend the process, etc.

Best Answer

You can get some of what you want via pipelog, which "allows for rotating or clearing the log of a running process by piping it through an intermediate which responds to external signals", e.g.:

spewstuff | pipelog spew.log -p /tmp/spewpipe.pid -x "gzip spew.log.1"

You can then get the pid from /tmp/spewpipe.pid, and:

kill -s USR1 $(</tmp/spewpipe.pid)

But that you would have to set up with cron or something. There's one catch to this, however. Notice I gzip spew.log.1 -- this is because the -x command is executed after the log is rotated. So you have the further problem of overwriting spew.log.1.gz each time unless you write a short script to do the gzip and move the file afterward, and use that as the -x command.

Full disclosure: I wrote this, so it of course works perfectly. ;) I will keep a compress option in mind, or something that better facilitates it, for version 0.2 (the intended purpose of -x is somewhat different, but it will work as above). Also automated rollover is a good idea...the first version is intentionally minimal as I resisted the temptation to add features that weren't necessary (it is not so hard to set up a cron job for this, after all).

Note that it's intended for text output; if there are potential null bytes, you should use -z -- which replaces the zero with something else. This was a tradeoff to simplify the implementation.

Related Question