Ubuntu – How to delete words from txt file, that exists on another txt file

Your script should work if fixed like so:

while
 last_line=$(tail -1 "./file.txt")
 [[ "$last_line" =~ ^$ ]] || [[ "$last_line" =~ ^[[:space:]]+$ ]]
do
 sed -i '$d' "./file.txt"
done

Your script had two main problems: (1) you never updated $last_line, so the loop's guard would always evaluate the same thing; (2) your [[ "$last_line" =~ $ ]] test matched any line, since any line has an end. (This is the reason why your script emptied your file completely.) You probably want to match against ^$ instead, which matches only empty lines. Additionally, I simplified the sed command to delete the last line in the loop's body (simply $d does the job).

However, this script is unnecessarily complicated. sed is there for just that kind of thing! This one-liner will do the same thing as the above script:

sed -i ':a;/^[ \n]*$/{$d;N;ba}' ./file.txt

Roughly,

1. Match current line against ^[ \n]*$. (i.e, can only contain whitespaces and newlines)
2. If it doesn't match, just print it. Read in next line and continue with step 1.
3. If it does match,
   • If we are at the end of the file, delete it.
   • If we are not at the end of the file, append the next line to the current line, inserting a newline character between the two, and go back to step 1 with this new, longer line.

There are lots of awesome sed tutorials on the Internet. For example, I can recommend this one. Happy learning! :-)

Update: And of course, if you additionally want to remove the last (non-blank) line of the file after having truncated the trailing blank lines, you can just use another sed -i '$d' ./file.txt after either your script or the above one-liner. I intentionally did not want to include that in the sed one-liner since I thought that removing trailing blank lines is quite a reusable piece of code that may be interesting for other people; but removing the last non-blank line is really specific to your use case, and trivial anyway once you removed the trailing blank lines.

Using awk

awk '!x[$0]++' infile.txt > outfile.txt

the way it works is that it keeps count of the lines in an array, and if the current count is zero, ie the first occurance, it prints the line, otherwise it continues to the next one.