Bash – How to do a grep on remote machine and print out the line which contains those words

bashgreplinuxshellssh

I have few logs files in my machineB under this directory /opt/ptd/Logs/ as shown below – My logs files are pretty big.

david@machineB:/opt/ptd/Logs$ ls -lt
-rw-r--r-- 1 david david  49651720 Oct 11 16:23 ptd.log
-rw-r--r-- 1 david david 104857728 Oct 10 07:55 ptd.log.1
-rw-r--r-- 1 david david 104857726 Oct 10 07:50 ptd.log.2

I am trying to write a generic shell script which should try to parse all my log file in machineB for a particular pattern and print the line which has those patterns. I will be running my below shell script from machineA which has all the ssh keys setup everything meaning I need to remotely grep on the logs files on machineB from machineA.

#!/bin/bash

wordsToInclude="hello,animal,atttribute,metadata"
wordsToExclude="timeout,runner"

# now grep on the various log file for above words and print out the lines accordingly

Meaning, I will have words separated by comma in wordsToInclude variable – If my logs contain hello word then print out that line, also print out the line which contains animal word. Similarly with attribute and metadata words.

And also I will have words separated by comma in wordsToExclude variable – If any of the lines contains those words then don't print out those line.

I am going with the above format for now for storing the words but any better format is fine to me. I can have long list of words in wordsToInclude and wordsToExclude variable so that's why I am going with storing them in those variables.

I know how to do a grep on small set of variables. If I need to do grep from the command line directly on machineB, then I will do it like this –

grep -E 'hello|animal|atttribute|metadata' ptd.log | grep -v 'timeout'

But I am not sure how do I combine this in my shell script so that I can do a remote ssh grep on machineB from machineA.

Best Answer

If you are open to other formats, consider:

inc="hello|animal|atttribute|metadata"
exc="timeout|runner" 
ssh machineB "grep -E '$inc' path/ptd.log | grep -vE '$exc'"

Faster Alternative

If your log files are large and you are grepping for fixed words, as opposed to fancy regular expressions, you may want to consider this approach:

inc='hello
animal
atttribute
metadata'

exc='timeout
runner'

ssh office "grep -F '$inc' ptd.log | grep -vF '$exc'"

By putting each word on a separate line, we can use grep's -F feature for fixed strings. This turns off regex processing, making the process faster.

Related Solutions

Bash – Modifying a shell variable with regex (bash)

sed

Assuming you're using GNU sed:

$ sed -E 's/(\w+) (\w+) (\w+)/\1 \U\2\E \3/' <<< 'big little man'
big LITTLE man

This command makes use of the GNU specific sequences \U and \E which turn the subsequent characters into upper case and cancel case conversion respectively.

awk

While not operating on regular expressions, awk provides another convenient way to capitalize a single word:

$ awk '{print($1, toupper($2), $3)}' <<< 'big little man'
big LITTLE man

bash

Although Bash on its own does not have regular expression based conversions, you can still achieve partial capitalization by treating your string as an array, e.g.

$ (read -a words; echo "${words[0]} ${words[1]^^} ${words[2]}") <<< 'big little man'
big LITTLE man

Here ^^ converts the second element of our array (i.e. the second word) to upper case. The feature was introduced in Bash 4.

Grep Command – How to Search Strings with Square Brackets

You can use simply this:

grep -oP '\[.*?\]' /root/Documents/a.txt

\[ matches a literal [

.*? matches any number of chars, in a non-greddy (lazy) way.

> Laziness Instead of Greediness

\] matches a literal ]

Now, I'll try to explain what went wrong with your regexes.

this one:

grep -oP 'Consumer Number \K.{26}|UCID \K.{10} |Sending Time \K.{09}|Receive Time \K.{09}|Total Time \\[ \K.{8}|STAN is \K.{8}' /root/Documents/a.txt

is broken: grep: missing terminating ] for character class you probably paste it badly here... check that out.

And this one:

grep -oP 'Consumer Number \K.{26}|UCID \K.{10} |Sending Time \K.{09}|Receive Time \K.{09}|Total Time \K.{8}|STAN is \K.{8}' /root/Documents/a.txt

Could be fixed:

grep -oP 'Consumer Number \K.{29}|UCID \K.{10} |Sending Time \K.{09}|Receive Time \K.{09}|Total Time \K[^/ ]{9}|STAN is \K.{8}' /root/Documents/a.txt

but bare in mind that this in not the best way to do it... the problem with your original regex was that you were too permissive; so, you were matching things you don't intended; if you know that some match should only contain, say, [,numbers, space, and ], don't use .{29}... this is SO permissive with no point: be more precise, for example, like this: [][0-9 ]{29}

Best Answer

Faster Alternative

Related Solutions

Bash – Modifying a shell variable with regex (bash)

Grep Command – How to Search Strings with Square Brackets

Related Question