I have an internal HDD which contains a directory, which contains a series of folders, which contain a total of about about 10 million small text files. Whenever I open this directory in Windows Explorer, the hard drive goes absolutely nuts and sounds like it's having a heart attack.
Resource Monitor shows that it's due to a roughly 11.5 MB/sec access to the drive's NTFS Master File Table (see below screenshot):
Indexing has been disabled for this directory, and all its subdirectories, as it is known that drive indexing and large numbers of small files are a volatile combination. Likewise, the directory only contains a couple folders (each of which house a large number of files), so none of the files in question are ever being displayed onscreen.
As such, I am a bit baffled as to why this intense reading occurs. It slows down any other processes which are working with the HDD to a near-standstill whenever it happens, so I have recently started navigating the drive and its contents via Command Prompt/PowerShell, which, for whatever reason, does not seem to trigger the NTFS reading-frenzy.
I'm not really a tech person, and so I don't know the details of how Windows 7 accesses drives to display files and folders, so I have a couple questions:
-
Why is the NTFS Master File Table being accessed, even though none of the files in question ever appear on screen, and none of them are ever being opened?
-
Is it absolutely necessary?
-
If not, is there a way to disable it? If so, what negative consequences would result?
-
Is there a more efficient (in terms of reducing disk wear and unnecessary file table access) way to set up a directory whose sole purpose is to house a large number of files?
Additional info: The drive is healthy (no S.M.A.R.T errors, and CrystalDiskInfo says there have been no problems), and is not a boot drive.
Best Answer
A good way to store this many files is by adding some directory structure to it.
For example see the "algorithm" described here: https://serverfault.com/a/95454/5109