Count lines containing word

text processing

I have a file with multiple lines. I want to know, for each word that appears in the total file, how many lines contain that word, for example:

0 hello world the man is world
1 this is the world
2 a different man is the possible one

The result I'm expecting is:

0:1
1:1
2:1
a:1
different:1
hello:1
is:3
man:2
one:1
possible:1
the:3
this:1
world:2

Note that the count for "world" is 2, not 3, since the word appears on 2 lines. Because of this, translating blanks to newline chars wouldn't be the exact solution.

Best Answer

Another Perl variant, using List::Util

$ perl -MList::Util=uniq -alne '
  map { $h{$_}++ } uniq @F }{ for $k (sort keys %h) {print "$k: $h{$k}"}
' file
0: 1
1: 1
2: 1
a: 1
different: 1
hello: 1
is: 3
man: 2
one: 1
possible: 1
the: 3
this: 1
world: 2

Related Solutions

Using sed, how to format one word per line, removing white space

You can use tr:

tr -s "[[:blank:]]" "\n" < file | grep .

The [:blank:] character class includes all horizontal whitespace. The -s squeezes or reduces multiple character occurrences to one.

The grep removes a blank line (if present).

Text Processing – How to Count Lines Containing One of Two Words but Not Both

With grep:

cat poem.txt \
  | grep -Evi -e '\<an\>.*\<the\>' -e '\<the\>.*\<an\>' \
  | grep -Eci -e '\<(an|the)\>'

This counts the matched lines. You can find an alternative syntax which counts the total number of matches down below.

Breakdown:

The frist grep command filters out all lines containing both 'an' and 'the'. The second grep command counts those lines, containing either 'an' or 'the'.

If you remove the c from the second grep's -Eci, you will see all matches highlighted.

Details:

The -E option enables extended expression syntax (ERE) for grep.
The -i option tells grep to match case-insensitive
The -v option tells grep to invert the result (i.e. match lines not containing the pattern)
The -c option tells grep to output the number of matched lines instead of the lines themselves
The patterns:
1. \< matches the beginning of a word (thanks @glenn-jackman)
2. \> matches the end of a word (thanks @glenn-jackman)
--> That way we can make sure to not match words containing 'the' or 'an' (like 'pan')
1. grep -Evi -e '\<an\>.*\<the\>' thus matches all lines not containing 'an ... the'
2. Similarly, grep -Evi -e '\<the\>.*\<an\>' matches all lines not containing 'the ... an'
3. grep -Evi -e '\<an\>.*\<the\>' -e '\<the.*an\>' is the combination of the 3. and 4.
4. grep -Eci -e '\<(an|the)\>' matches all lines containing either 'an' or 'the' (surrounded by whitespace or start/end of line) and prints the number of matched lines

EDIT 1: Use \< and \> instead of ( |^) and ( |$), as suggested by @glenn-jackman

EDIT 2: In order to count the number of matches instead of the number of matched lines, use the following expression:

cat poem.txt \
  | grep -Evi -e '\<an\>.*\<the\>' -e '\<the\>.*\<an\>' \
  | grep -Eio -e '\<(an|the)\>' \
  | wc -l

This uses the -o option of grep, which prints every match in a separate line (and nothing else) and then wc -l to count the lines.

Best Answer

Related Solutions

Using sed, how to format one word per line, removing white space

Text Processing – How to Count Lines Containing One of Two Words but Not Both

Related Question