Understanding the ‘-mindepth’ Option in Find Command

find

I am using the find command of bash, and I am trying to understand it since it is part of a code that I am using. In the code, the command in question is –

find -L $RAW_DIR -mindepth 2 -maxdepth 2 -name "*.bam" -o -name "*.sam" | wc -l

I have been trying to understand this command by searching its components. In esence, I thinking it is trying to find the number of files that end with .bam or .sam. I think -maxdepth 2 means to search for these files in this folder and its immediate subfolder.

What I do not understand is what mindepth -2 does in this case. I looked up mindepth, and the explanation given everywhere is –

"Do not apply any tests or actions at levels less than levels (a non-negative integer). '-mindepth 1' means process all files except the command line arguments."

To me this explanation is not very clear. Just like maxdepth -2means search for subfolders upto a depth of 2, what does mindepth -2 correspondingly mean, in simple language?

Also, if mindepth is just the opposite of maxdepth in terms of the direction (which would make intuitive sense), then how do I understand the fact that executing the command above on a folder that does have a .bam file leads to the output 0, whereas omitting the mindepth part of the command leads to the output 1 ?

Best Answer

Depth 0 is the command line arguments, 1 the files contained within them, 2 the files contained within depth 1, etc.

-mindepth N tells to process only files that are at depth >= N, similar to how -maxdepth M tells to process only files are at depth <= M. So if you want the files that are at depth 2, exactly, you need to use both.

Your command would match $RAW_DIR/foo/bam.bam, but not $RAW_DIR/bar.bam.

Try, e.g.

$ mkdir -p a/b/c/d
$ find ./a -maxdepth 2
./a
./a/b
./a/b/c
$ find ./a -mindepth 2
./a/b/c
./a/b/c/d
$ find ./a -maxdepth 2 -mindepth 2
./a/b/c

maxdepth with a negative argument doesn't mean anything:

$ find ./a -maxdepth -2
find: Expected a positive decimal integer argument to -maxdepth, but got ‘-2’

The reason why the find command is slow

That is a really interesting issue... or, honestly, mallicious:

The command

find . -mindepth 2 -mtime +5 -print -delete

is very different from the usual tryout variant, leaving out the dangerous part, -delete:

find . -mindepth 2 -mtime +5 -print

The tricky part is that the action -delete implies the option -depth. The command including delete is really

find . -depth -mindepth 2 -mtime +5 -print -delete

and should be tested with

find . -depth -mindepth 2 -mtime +5 -print

That is closely related to the symtoms you see; The option -depth is changing the tree traversal algorithm for the file system tree from an preorder depth-first search to an inorder depth-first search.
Before, each file or directory that was reached was immediately used, and forgotten about. Find was using the tree itself to find it's way. find will now need to collect all directories that could contain files or directories still to be found, before deleting the files in the deepest directoies first. For this, it needs to do the work of planing and remembering traversal steps itself, and - that's the point - in a different order than the filesystem tree naturally supports. So, indeed, it needs to collect data over many files before the first step of output work.

Find has to keep track of some directories to visit later, which is not a problem for a few directories.
But maybe with many directories, for various degrees of many.
Also, performance problems outside of find will get noticable in this kind of situation; So it is possible it's not even find that's slow, but something else.

The performance and memory impact of that depends on your directory structure etc.

The relevant sections from man find:

See the "Warnings":

ACTIONS
    -delete
           Delete  files;  true if removal succeeded.  If the removal failed,
           an error message is issued.  If -delete fails, find's exit  status
           will  be nonzero (when it eventually exits).  Use of -delete auto‐
           matically turns on the -depth option.

           Warnings: Don't forget that the find command line is evaluated  as
           an  expression,  so  putting  -delete  first will make find try to
           delete everything below the starting points you  specified.   When
           testing  a  find  command  line  that you later intend to use with
           -delete, you should explicitly specify -depth in  order  to  avoid
           later  surprises.  Because -delete implies -depth, you cannot use‐
           fully use -prune and -delete together.
    [ ... ]

And, from a section further up:

 OPTIONS
    [ ... ]
    -depth Process each directory's contents  before  the  directory  itself.
           The -delete action also implies -depth.

The faster solution to delete the files

You do not really need to delete the directories in the same run of deleting the files, right? If we are not deleting directories, we do not need the whole -depth thing, we can just find a file and delete it, and go on to the next, as you proposed.

This time we can use the simple print variant for testing the find, with implicit -print.

We want to find only plain files, no symlinks, directories, special files etc:

find . -mindepth 2 -mtime +5 -type f

We use xargs to delete more than one file per rm process started, taking care of odd filenames by using a null byte as separator:

Testing this command - note the echo in front of the rm, so it prints what will be run later:

find . -mindepth 2 -mtime +5 -type f -print0 | xargs -0 echo rm

The lines will be very long and hard to read; For an initial test it could help to get readable output with only three files per line by adding -n 3 as first arguments of xargs

If all looks good, remove the echo in front of the rm and run again.

That should be a lot faster;

In case we are talking about millions of files - you wrote it's 600 million files in total - there is something more to take into account:

Most programs, including find, read directories using the library call readdir (3). That usually uses a buffer of 32 KB to read directories; That becomes a problem when the directories, containing huge lists of possibly long filenames, are big.

The way to work around it is to directly use the system call for reading directory entries, getdents (2), and handle the buffering in a more suitable way.

For details, see You can list a directory containing 8 million files! But not with ls..

(It would be interesting if you can add details to your question on the typical numbers of files per directroy, directories per directory, max depth of paths; Also, which filesystem is used.)

(If it is still slow, you should check for filesystem performance problems.)

find -maxdepth 0 Not Returning Any Output – Why?

Let us suppose that we have file1 in the current directory. Then:

$ find . -maxdepth 0 -name "file1"
$ find . file1 -maxdepth 0 -name "file1"
file1

Now, let's look at what the documentation states:

-maxdepth 0 means only apply the tests and actions to the command line arguments.

In my first example above, only the directory . is listed on the command line. Since . does not have the name file1, nothing is listed in the output. In my second example above, both . and file1 are listed on the command line, and, because file1 matches -name "file1", it was returned in the output.

In other words, -maxdepth 0 means do not search directories or subdirectories. Instead, only look for a matching file among those explicitly listed on the command line.

In your examples, only directories were listed on the command line and none of them were named file1. Hence, no output.

In general, many files and directories can be named on the command line. For example, here we try a find command with nine files and directories on the command line:

$ ls
d1  file1  file10  file2  file3  file4  file5  file6  file7
$ find d1 file1 file10 file2 file3 file4 file5 file6 file7 -maxdepth 0 -name "file1"
file1

Overlapping paths

Consider:

$ find . file1 -maxdepth 0 -iname file1
file1
$ find . file1 file1 -maxdepth 0 -iname file1
file1
file1
$ find . file1 file1 -maxdepth 1 -iname file1
./file1
file1
file1

find will follow each path specified on the command line and look for matches even if the paths lead to the same file, as in . file, or even if the paths are exact duplicates, as in file1 file1.

Best Answer

Related Solutions

Find Command – Can the ‘find’ Command Work More Efficiently to Delete Many Files?

The reason why the find command is slow

The faster solution to delete the files

find -maxdepth 0 Not Returning Any Output – Why?

Overlapping paths

Related Question