Sed in-place line deletion on full filesystem

edfilessedtext processing

Due to an application bug as yet undiagnosed, I have several hundred servers with a full disk. There is one file that has been filled up with duplicate lines—not a log file, but a user environment file with variable definitions (so I can't just delete the file).

I wrote a simple sed command to check for the erroneously added lines and delete them, and tested it on a local copy of the file. It worked as intended.

However, when I tried it on the server with the full disk, I got approximately the following error (it's from memory, not copy and paste):

sed: couldn't flush /path/to/file/sed8923ABC: No space left on deviceServerHostname

Of course, I know there's no space left. That's why I'm trying to delete stuff! (The sed command I'm using will reduce a 4000+ line file to about 90 lines.)

My sed command is just sed -i '/myregex/d' /path/to/file/filename

Is there a way I can apply this command despite the full disk?

(It must be automated, since I need to apply it to several hundred servers as a quick-fix.)

(Obviously the application bug needs to be diagnosed, but in the meantime the servers aren't working correctly….)

Update: The situation I faced was resolved by deleting something else that I found out I could delete, but I'd still like the answer to this question, which would be helpful in the future and for other people.

/tmp is a no-go; it's on the same filesystem.

Before I freed up disk space, I did test and find out that I could delete the lines in vi by opening the file and running :g/myregex/d and then successfully save the changes with :wq. It seems it should be possible to automate this, without resorting to a separate filesystem to hold a temp file…. (?)

Best Answer

The -i option doesn't really overwrite the original file. It creates a new file with the output, then renames it to the original filename. Since you don't have room on the filesystem for this new file, it fails.

You'll need to do that yourself in your script, but create the new file on a different filesystem.

Also, if you're just deleting lines that match a regexp, you can use grep instead of sed.

grep -v 'myregex' /path/to/filename > /tmp/filename && mv /tmp/filename /path/to/filename

In general, it's rarely possible for programs to use the same file as input and output -- as soon as it starts writing to the file, the part of the program that's reading from the file will no longer see the original contents. So it either has to copy the original file somewhere first, or write to a new file and rename it when it's done.

If you don't want to use a temporary file, you could try caching the file contents in memory:

file=$(< /path/to/filename)
echo "$file" | grep -v 'myregex' > /path/to/filename
Related Question