rm – Deleting Billions of Files from a Directory While Seeing the Progress

progress-informationrm

I have a directory of 30 TB having billions of files in it which are formally all JPEG files. I am deleting each folder of files like this:

sudo rm -rf bolands-mills-mhcptz

This command just runs and doesn't show anything whether it's working or not.

I want to see as it's deleting files or what is the current status of the command.

Best Answer

You can use rm -v to have rm print one line per file deleted. This way you can see that rm is indeed working to delete files. But if you have billions of files then all you will see is that rm is still working. You will have no idea how many files are already deleted and how many are left.

The tool pv can help you with a progress estimation.

http://www.ivarch.com/programs/pv.shtml

Here is how you would invoke rm with pv with example output

$ rm -rv dirname | pv -l -s 1000 > logfile
562  0:00:07 [79,8 /s] [====================>                 ] 56% ETA 0:00:05

In this contrived example I told pv that there are 1000 files. The output from pv shows that 562 are already deleted, elapsed time is 7 seconds, and the estimation to complete is in 5 seconds.

Some explanation:

pv -l makes pv to count by newlines instead of bytes
pv -s number tells pv what the total is so that it can give you an estimation.
The redirect to logfile at the end is for clean output. Otherwise the status line from pv gets mixed up with the output from rm -v. Bonus: you will have a logfile of what was deleted. But beware the file will get huge. You can also redirect to /dev/null if you don't need a log.

To get the number of files you can use this command:

$ find dirname | wc -l

This also can take a long time if there are billions of files. You can use pv here as well to see how much it has counted

$ find dirname | pv -l | wc -l
278k 0:00:04 [56,8k/s] [     <=>                                              ]
278044

Here it says that it took 4 seconds to count 278k files. The exact count at the end (278044) is the output from wc -l.

If you don't want to wait for the counting then you can either guess the number of files or use pv without estimation:

$ rm -rv dirname | pv -l > logfile

Like this you will have no estimation to finish but at least you will see how many files are already deleted. Redirect to /dev/null if you don't need the logfile.

Nitpick:

do you really need sudo?
usually rm -r is enough to delete recursively. no need for rm -f.

Related Solutions

Bash – Delete All Files with a Particular Extension in a Folder

Yes, rm *.xvg will only delete the files with the specified extension in your current directory.

A good way to make sure you are indeed in the directory you want delete your files is to use the pwd command which will display your current directory and then do an ls to verify you find the files you are expecting.

If you are bit apprehensive about issuing the rm command, there are 2 things you can do:

type ls *.xvg to see a list of what files would be affected by this command.
Unless you have a lot of files, you could always also use the -i command line switch for rm (also exists for cp and mv). Using rm -i *.xvg would prompt you for each individual file if it was ok to delete it, so you could be sure nothing you didn't expect was getting deleted. (This will be tedious if you have a lot of files though :)

Shell – How to Remove All Sub-Directories from a Directory

In BASH you can use the trailing slash (I think it should work in any POSIX shell):

rm -R -- */

Note the -- which separates options from arguments and allows one to remove entries starting with a hyphen - otherwise after expansion by the shell the entry name would be interpreted as an option by rm (the same holds for many other command line utilities).

Add the -f option if you don't want to be prompted for confirmation when deleting non-writeable files.

Note that by default, hidden directories (those whose name starts with .) will be left alone.

An important caveat: the expansion of */ will also include symlinks that eventually resolve to files of type directory. And depending on the rm implementation, rm -R -- thelink/ will either just delete the symlink, or (in most of them) delete the content of the linked directory recursively but not that directory itself nor the symlink.

If using zsh, a better approach would be to use a glob qualifier to select files of type directory only:

rm -R -- *(/) # or *(D/) to include hidden ones

or:

rm -R -- *(-/)

to include symlinks to directories (but because, this time, the expansion doesn't have trailing /s, it's the symlink only that is removed with all rm implementations).

With bash, AT&T ksh, yash or zsh you can do:

set -- */
rm -R -- "${@%/}"

to strip the trailing /.

Best Answer

Related Solutions

Bash – Delete All Files with a Particular Extension in a Folder

Shell – How to Remove All Sub-Directories from a Directory

Related Question