I have two scripts running, both of which output a log file. I'd like to make a third script that can sort these logs by time stamp and merge them into one file as they are created. What is a good way to do this, ideally without overwriting the file constantly?
A good way to sort two log files as they are created
logssort
Best Answer
If you use
tail -f
to tail 2 or more files, then the command shows the data line-by-line, and outputs the filename each time the source of data changes. Using this you can write a script to merge the interleaved output from tail according to timestamp by holding onto each line until you see a line from the other file with a later timestamp.For example, using two standard logfiles (
/var/log/messages
and/var/log/cron
) which on my system have the same format for the timestamp at the start of the line (egJun 9 02:55:01
), you can do the following:The awk flips between the 2 files when it sees the
==>
file heading from tail. It keeps data, in 4 arrays, separately for each file, arbitrarily calledaa
andbb
and numbered 0 and 1.dates
holds the timestamp (in seconds from the epoch),lines
holds the input log line,num
holds the count of lines, andmaxes
the highest date for a file. The first 2 arrays are 2-dimensional indexed by file (0 or 1) and count of held lines.As each log line is read, the timestamp is converted to seconds, and saved in a new entry at the end of
dates
, and the line is saved too. The minimum of the current two dates is set inminmax
. The whole held data is scanned and printed according to timestamp order upto this minimum. Printed entries are cleared, and at the end of the while loop, the arrays are squashed to remove these cleared entries.