Ubuntu – How to remove particular words from lines of a text file

grep approach

To create a copy of the file without lines matching "cat" or "rat", one can use grep in reverse (-v) and with the whole-word option (-w).

grep -vwE "(cat|rat)" sourcefile > destinationfile

The whole-word option makes sure it won't match cats or grateful for example. Output redirection of your shell is used (>) to write it to a new file. We need the -E option to enable the extended regular expressions for the (one|other) syntax.

sed approach

Alternatively, to remove the lines in-place one can use sed -i:

sed -i "/\b\(cat\|rat\)\b/d" filename

The \b sets word boundaries and the d operation deletes the line matching the expression between the forward slashes. cat and rat are both being matched by the (one|other) syntax we apparently need to escape with backslashes.

Tip: use sed without the -i operator to test the output of the command before overwriting the file.

(Based on Sed - Delete a line containing a specific string)

Your script should work if fixed like so:

while
 last_line=$(tail -1 "./file.txt")
 [[ "$last_line" =~ ^$ ]] || [[ "$last_line" =~ ^[[:space:]]+$ ]]
do
 sed -i '$d' "./file.txt"
done

Your script had two main problems: (1) you never updated $last_line, so the loop's guard would always evaluate the same thing; (2) your [[ "$last_line" =~ $ ]] test matched any line, since any line has an end. (This is the reason why your script emptied your file completely.) You probably want to match against ^$ instead, which matches only empty lines. Additionally, I simplified the sed command to delete the last line in the loop's body (simply $d does the job).

However, this script is unnecessarily complicated. sed is there for just that kind of thing! This one-liner will do the same thing as the above script:

sed -i ':a;/^[ \n]*$/{$d;N;ba}' ./file.txt

Roughly,

1. Match current line against ^[ \n]*$. (i.e, can only contain whitespaces and newlines)
2. If it doesn't match, just print it. Read in next line and continue with step 1.
3. If it does match,
   • If we are at the end of the file, delete it.
   • If we are not at the end of the file, append the next line to the current line, inserting a newline character between the two, and go back to step 1 with this new, longer line.

There are lots of awesome sed tutorials on the Internet. For example, I can recommend this one. Happy learning! :-)

Update: And of course, if you additionally want to remove the last (non-blank) line of the file after having truncated the trailing blank lines, you can just use another sed -i '$d' ./file.txt after either your script or the above one-liner. I intentionally did not want to include that in the sed one-liner since I thought that removing trailing blank lines is quite a reusable piece of code that may be interesting for other people; but removing the last non-blank line is really specific to your use case, and trivial anyway once you removed the trailing blank lines.