Command Line Efficiency – Are Linux Utilities Smart When Running Piped Commands?

This question is hard to answer, as there is no formal definitions of those terms and different people will use them differently. I here only give my use of them, others will have different points if view.

For me tool and utility are synonyms. I use the words for small programs which just do one small job. I'd call e.g. all applications implemented as applets in busybox tools or utility.

Any application is a program for me. I.e. 'ls' is a tool, a utility and a program. Firefox is a program but I wouldn't call it neither tool nor utility.

You can do that with a while loop:

while read l; do
  [ ${#l} -gt 65 ] && \
    echo "$l" | langid --line | grep -q "is" && \
    echo "$l"
done <file

• read l read the input line by line and store the current line in the variable $l.
• [ ${#l} -gt 65 ] if the line contains more than 65 characters.
  • echo "$l" | langid --line | grep -q "is" process the line and grep for the language, notice with -q, grep will be silent. We just want to check if the string is there, no output.
  • echo "$l" If the string is there, print the original line.
• <file use the contents of file as input.

Edit: The above runs the langid command on each line, this is very slow. If you want it to run in one transit (faster) use this:

awk 'FNR==NR{a[NR]=$0}
  FNR!=NR&&$1~"is"{print a[FNR]}' \
<(sed -n '/^.\{65\}/p' file) \
<(sed -n '/^.\{65\}/p' file | langid --line)

• awk processes two "files":
  • The output of sed -n '/^.\{65\}/p' file: All sentences with 65 or more characters.
  • The output of sed -n '/^.\{65\}/p' file | langid --line which processes all lines with 65 or more characters in one transit.
• Inside awk:
  • FNR==NR applies in the first "file"
  • a[NR]=$0 Fill an array with the line number as index
  • FNR!=NR&&$1~"is" applies to the second "file" and checks if the line contains the string is
  • print a[FNR] if thats true, print the corresponding line in the prevously created array a which contains the original sentence.