A larger cache won't help, not with any standard file copy app.
No matter how large the Windows file cache is, no sane file copy tool will close the input files for the copy task (allowing you to delete them) before all the data has been written to the destination. That is true even if the entire input data happens to have been read into the cache.
The reason is that the data in the cache is not safe - it could disappear at any time. Any RAM used by the cache that has not been modified since it was read from disk is considered discardable by the memory manager. That is, if something else needs the RAM, RAM used by the cache can be grabbed from the cache and "repurposed", ie given to the something else, at any moment. Of course it can - after all, there isn't supposed to be any data anywhere that's only in the cache. (Unless it's been modified since being read from disk. In that case it is automatically queued for writeback, which will happen within four seconds; it can't be repurposed until the writeback is complete.)
So with a standard copy program, you're going to have to wait for the spinning disks to write the data before you can delete the source files - regardless of how much is buffered in a cache.
Note that a copy program (or any other app) cannot even find out whether something is in the cache. There's no interface for that. Even if there was, the information would be considered "stale" the moment it was retrieved, even before the app looked at it. The file cache is supposed to work automatically, transparent to apps, and part of transparency is that there are very few controls on it.
Think of it this way: You need a safe intermediate copy of your results - something functionally equivalent to another copy of your source files in a different directory, maybe even a separate drive letter, before you can safely delete the originals. The Windows file cache will never give you that. Even if Windows does decide to slurp the original files in their entirety into the file cache (which is very unlikely), the file cache does not give you that.
(You may be wondering "so what good is it anyway?" The main goal of the file cache is to make repeated access to numerous small files (and file system metadata) faster. And it does that quite well.)
SuperFetch
TL,DR version: SuperFetch doesn't give you that either.
You are correct in your expressed doubt about SuperFetch. While SuperFetch is effective at what it tries to do, it won't help this case. What SuperFetch does is to keep track of files that are frequently accessed, say on every boot, and try to read them into RAM in advance of need.
This distinction is important if you want to understand Windows caching in general. The Windows file cache (which is what I've described in the previous section) is reactive, meaning that it never caches anything until a program has actually tried to read it. It has been in the Windows NT family since first release (NT 3.1).
SuperFetch is a separate mechanism, originally added with Vista. It is proactive, trying to pre-fetch things that have been observed to have been accessed often in the past.
SuperFetch manages its RAM separately from the Windows file cache. SuperFetch uses "lower-priority" pages on the Windows standby page list - and funny thing, it just leaves them on that list, so they remain part of "available" RAM. (So all other things being equal, you won't notice a difference in "Available" RAM with our without SuperFetch enabled, but you will notice a difference in the amount of reported "Cache".) The RAM assigned to the file cache is in a working set and therefore takes a little longer to repurpose, if that need arises. Because of this design the SuperFetch cache is even more quickly "discardable" than the Windows file cache, so it does not give you a safe temporary copy of anything, any more than the file cache does.
So what can you do?
To solve your problem, I would be looking at dedicated hardware. Maybe get a single not-so-expensive-or-fast SSD of say 120 GB and copy the data from your SSD array to that, then copy it from there to the hard drive. This would, alas, mean that you can't start writing to the hard drives until all of your data has been copied to the temporary, so this will take longer than what you're doing now. But the source data will be freed sooner.
A simpler idea would be to get more hard drives and put them in a larger stripe set to increase write throughput.
A dedicated copy program, one that knows the structure of your data, might help.
btw, I hope your FEA software is using mapped file access when creating the data set - it's far faster than traditional read/write calls.
Best Answer
You have a memory leak caused by a driver. Look at the high value of nonpaged kernel memory. In your case this is over 5.6 GB. You can use poolmon to see which driver is causing the high usage.
Install the Windows WDK, run poolmon, sort it via P after pool type so that non paged is on top and via B after bytes to see the tag which uses most memory. Run poolmon by going to the folder where WDK is installed, go to Tools (or C:\Program Files (x86)\Windows Kits\8.1\Tools\x64) and click poolmon.exe.
Now look which pooltag uses most memory as shown here:
Now open a cmd prompt and run the findstr command. To do this, open cmd prompt and type "cd C:\Windows\System32\drivers" to go to the drivers directory, without quotes. Then type
findstr /s __ *.*
, where__
is the tag that you see in poolmon.After doing this to see which driver uses this tag:
Now, go to the drivers folder (C:\Windows\System32\drivers) and right-click the driver in question (
intmsd.sys
in the above image example). Click Properties, go to the details tab to find the Product Name. Look for an update for that product.If you can't find a driver to the pooltag, look in the pooltag.txt if the tag is used by a Windows driver.
Here you need to capture a grow of the pool usage with xperf. First, you have to install the Windows Performance Toolkit. Next open a cmd prompt (cmd.exe) as admin and run this:
Now open it in WPA.exe, load the debug symbols and look for the tag that you saw in poomon under
AIFO
(allocated insde freed outside) and expend the stack. From the function names you may have any idea what is going on.In this example the
FILE
tag usage comes from a tool called locate32 which scans the HDD to build up its search index.