I have a very large file (~400 GB), and I need to remove the last 2 lines from it. I tried to use sed
, but it ran for hours before I gave up. Is there a quick way of doing this, or am I stuck with sed
?
Linux – Efficiently remove the last two lines of an extremely large text file
linuxsedtext manipulation
Best Answer
I haven't tried this on a large file to see how fast it is, but it should be fairly quick.
To use the script to remove lines from the end of a file:
It seeks to the end of the file, checks to make sure the last character is a newline, then reads each character one at a time going backwards until it's found three newlines and truncates the file just after that point. The change is made in place.
Edit: I've added a Python 2.4 version at the bottom.
Here is a version for Python 2.5/2.6:
Here's a Python 3 version:
Here is a Python 2.4 version: