You can use sed
or awk
to make it with one command. However you'll loose at speed, cause sed
and awk
will need to run through the whole file anyway.
From a speed point of view it's much better to make a function or every time to combination of tail
+ head
. This does have the downside of not working if the input is a pipe, however you can use proccess substitution, in case your shell supports it (look at example below).
first_last () {
head -n 10 -- "$1"
tail -n 10 -- "$1"
}
and just launch it as
first_last "/path/to/file_to_process"
to proceed with process substitution (bash, zsh, ksh like shells only):
first_last <( command )
ps. you can even add a grep
to check if your "global conditions" exist.
The General Way
$ cat input.log | sed -e "s/^/$(date -R) /" >> output.log
How it works:
cat
reads file called input.log
and just prints it to its standard output stream.
Normally the standard output is connected to a terminal, but this little script contains |
so shell redirects the standard output of cat
to standard input of sed
.
sed
reads data (as cat
produces it), processes it (according to the script provided with -e
option) and then prints it to its standard output. The script "s/^/$(date -R) /"
means replace every start of line to a text generated by date -R
command (the general construction for replace command is: s/pattern/replace/
).
Then according to >>
bash
redirects the output of sed
to a file called output.log
(>
means replace file contents and >>
means append to the end).
The problem is the $(date -R)
evaluated once when you run the script so it will insert current timestamp to the beginning of each line. The current timestamp may be far from a moment when a message was generated. To avoid it you have to process messages as they are written to the file, not with a cron job.
FIFO
The standard stream redirection described above called pipe. You can redirect it not just with |
between commands in the script, but through a FIFO file (aka named pipe). One program will write to the file and another will read data and receive it as the first sends.
Pick an example:
$ mkfifo foo.log.fifo
$ while true; do cat foo.log.fifo | sed -e "s/^/$(date -R) /" >> foo.log; done;
# have to open a second terminal at this point
$ echo "foo" > foo.log.fifo
$ echo "bar" > foo.log.fifo
$ echo "baz" > foo.log.fifo
$ cat foo.log
Tue, 20 Nov 2012 15:32:56 +0400 foo
Tue, 20 Nov 2012 15:33:27 +0400 bar
Tue, 20 Nov 2012 15:33:30 +0400 baz
How it works:
mkfifo
creates a named pipe
while true; do sed ... ; done
runs an infinite loop and at every iteration it runs sed
with redirecting foo.log.fifo
to its standard input; sed
blocks in waiting for input data and then processes a received message and prints it to standard output redirected to foo.log
.
At this point you have to open a new terminal window because the loop occupies the current terminal.
echo ... > foo.log.fifo
prints a message to its standard output redirected to the fifo file and sed
receives it and processes and writes to a regular file.
The important note is the fifo just as any other pipe has no sense if one of its sides is not connected to any process. If you try to write to a pipe the current process will block until someone would read data on the other side of the pipe. If you want to read from a pipe the process will block until someone will write data to the pipe. The sed
loop in the example above does nothing (sleeps) until you do echo
.
For your particular situation you just configure your application to write log messages to the fifo file. If you can't configure it - simply delete the original log file and create a fifo file. But note again that if the sed
loop will die for some reason - your program will be blocked upon attempting to write
to the file until someone will read
from the fifo.
The benefit is the current timestamp evaluated and attached to a message as the program writes it to the file.
Asynchronous Processing With tailf
To make writing to the log and processing more independent you can use two regular files with tailf
. An application will write message to a raw file and other process read new lines (follow to writes asynchronously) and process data with writing to the second file.
Let's take an example:
# will occupy current shell
$ tailf -n0 bar.raw.log | while read line; do echo "$(date -R) $line" >> bar.log; done;
$ echo "foo" >> bar.raw.log
$ echo "bar" >> bar.raw.log
$ echo "baz" >> bar.raw.log
$ cat bar.log
Wed, 21 Nov 2012 16:15:33 +0400 foo
Wed, 21 Nov 2012 16:15:36 +0400 bar
Wed, 21 Nov 2012 16:15:39 +0400 baz
How it works:
Run tailf
process that will follow writes to bar.raw.log
and print them to standard output redirected to the infinite while read ... echo
loop. This loop performs two actions: read data from standard input to a buffer variable called line
and then write generated timestamp with the following buffered data to the bar.log
.
Write some messages to the bar.raw.log
. You have to do this in a separate terminal window because the first one will be occupied by tailf
which will follow the writes and do its job. Quite simple.
The pros is your application would not block if you kill tailf
. The cons is less accurate timestamps and duplicating log files.
Best Answer
It might suffice to use watch: