Grep only for lines of process that caused the error

greptext processing

There is a service running as multiple processes and the logs of all come to one huge log (don't ask me why). It looks like this:

process1: bla bla bla
process2: ha ha ha
process3: tarara **error_has_happened**
process3: details of the error here
process1: bla bla bla
process3: more details of the error here
process2: ha ha ha

The error can happen in any of the processes so I want my greps to find this for all processes:

process3: tarara **error_has_happened**
process3: details of the error here
process3: more details of the error here

so basically detect the error_has_happened and the process in which it happened and print the next N lines only from that process. This should happen for all the processes that had this error, assuming the logs of details of the error from different processes are not interleaved

Is it possible to do this using grep alone? how?

Best Answer

You can do it with:

grep "error_has_happened" -A3 logfile.log

Where 3, is the number of lines after error_has_happened appearence that will be shown. However, this will also show other process outputs not only the outputs of the process which sends the error.

A more elaborated command that worked for me in a quick test is:

grep "error_has_happened" logfile.log | cut -d : -f1 | sort -u |
  while IFS= read -r process; do
    grep "^$process:" logfile.log |
      grep -A3 "error_has_happened"
  done

Related Solutions

GNU parallel excessively slow

Try parallel -X. As written in the comments the overhead of starting a new shell and opening files for buffering for each argument is probably the cause.

Be aware that GNU Parallel will never be as fast as xargs because of that. Expect an overhead of 10 ms per job. With -X this overhead is less significant as you process more arguments in one job.

Grep -v: How to exclude only the first (or last) N lines that match

sed provides a simpler way:

... |  sed '/some stuff/ {N; s/^.*\n//; :p; N; $q; bp}' | ...

This way you delete first occurrence.

If you want more:

sed '1 {h; s/.*/iiii/; x}; /some stuff/ {x; s/^i//; x; td; b; :d; d}'

, where count of i is count of occurrences (one or more, not zero).

Multi-line Explanation

sed '1 {
    # Save first line in hold buffer, put `i`s to main buffer, swap buffers
    h
    s/^.*$/iiii/
    x
}

# For regexp what we finding
/some stuff/ {
    # Remove one `i` from hold buffer
    x
    s/i//
    x
    # If successful, there was `i`. Jump to `:d`, delete line
    td
    # If not, process next line (print others).
    b
    :d
    d
}'

In addition

Probably, this variant will work faster, 'cos it reads all rest lines and print them in one time

sed '1 {h; s/.*/ii/; x}; /a/ {x; s/i//; x; td; :print_all; N; $q; bprint_all; :d; d}'

As result

You can put this code into your .bashrc (or config of your shell, if it is other):

dtrash() {
    if [ $# -eq 0 ]
    then
        cat
    elif [ $# -eq 1 ]
    then
        sed "/$1/ {N; s/^.*\n//; :p; N; \$q; bp}"
    else
        count=""
        for i in $(seq $1)
        do
            count="${count}i"
        done
        sed "1 {h; s/.*/$count/; x}; /$2/ {x; s/i//; x; td; :print_all; N; \$q; bprint_all; :d; d}"

    fi
}

And use it this way:

# Remove first occurrence
cat file | dtrash 'stuff' 
# Remove four occurrences
cat file | dtrash 4 'stuff'
# Don't modify
cat file | dtrash