Linux – Best Way to Remove 4 Million Small Text Files from a Folder

I realize that any question asking how to delete files must be taken with great care. My first answer was too hasty I didn't take the fact that the filelist could be malformed to be used with egrep. I edited the answer to reduce that risk.

That should work for the files that has no space in the name:

First rebuild your filelist to be sure to match the exact file name:

sed -e 's,^,^,' -e 's,$,$,'  filelist  > newfilelist 

build the rm commands

cd your_directory
ls | egrep -vf newfilelist   | xargs -n 1 echo rm  >  rmscript

Check if the rm script suits you (You can do it with "vim" or "less").
Then perform the action :

sh -x rmscript

If the files have spaces in their name (if the files have the " in the name then this will not work) :

ls | egrep -vf newfilelist  | sed 's,^\(.*\)$,rm "\1",' > rmscript

of course the filelist should not be in the same directory!

EDITED :

The Nathan's file list contained names that were matching all the files in the directory (like "html" matches "bob.html"). So nothing was deleted because egrep -vf absorbed all the stream. I added a command to put a "^" and a "$" around each file name. I was lucky here that Nathan's file list was correct. Would have it been DOS formatted with CR-LF ended lines or with additional spaces, no files would have been preserved by the egrep and all been deleted.

The ls command, or even TAB-completion or wildcard expansion by the shell, will normally present their results in alphanumeric order. This requires reading the entire directory listing and sorting it. With ten million files in a single directory, this sorting operation will take a non-negligible amount of time.

If you can resist the urge of TAB-completion and e.g. write the names of files to be zipped in full, there should be no problems.

Another problem with wildcards might be wildcard expansion possibly producing more filenames than will fit on a maximum-length command line. The typical maximum command line length will be more than adequate for most situations, but when we're talking about millions of files in a single directory, this is no longer a safe assumption. When a maximum command line length is exceeded in wildcard expansion, most shells will simply fail the entire command line without executing it.

This can be solved by doing your wildcard operations using the find command:

find <directory> -name '<wildcard expression>' -exec <command> {} \+

or a similar syntax whenever possible. The find ... -exec ... \+ will automatically take into account the maximum command line length, and will execute the command as many times as required while fitting the maximal amount of filenames to each command line.