Read file line by line and remember the last position in file

filesread

I want to grep some line from a log file with an input from another file. I'm using this small command to do that:

while read line; do 
    grep "$line" service.log; 
done < input_strings.txt > result.txt

input_strings.txt has like 50 000 strings (one per line). For every of this string I'm currently searching the huge service.log file (with around 2 000 000 lines).

So lets say the 1st string of input_strings.txt is found in service.log at line 10 000, this line gets written to my result.txt. After that, the 2nd string of input_strings.txt will be searched in service.log, BUT starting at line 1 of service.log.

How can I remember the last line I found the 1st entry in service.log? So that I can start the 2nd search-run there?

Best Answer

If you want to get the matches then you don't need to be using a loop at all. It would be much faster to just use a single grep command:

grep -Ff input_strings service.log > results.txt

That said, if you want to do literally what you stated in your question, then you could use a variable to keep track of the line on which the last match was found:

LINE_NUMBER=0
while read LINE; do

    # Search for the next match starting at the line number of the previous match
    MATCH="$(tail -n+${LINE_NUMBER} "service.log" | grep -n "${LINE}" | head -n1)";

    # Extract the line number from the match result
    LINE_NUMBER="${MATCH/:*/}";

    # Extract the matching string from the match result
    STRING="${x#*:}";

    # Output the matching string
    echo "${STRING}";

done < input_strings.txt > result.txt
Related Question