"Why is half my RAM of 8GB is missing?"
It is not missing, nor is it even unused. Approximately 4GiB are in active use, and an additional 3.3GiB is inactive, but also in use. The 3.3GiB is ready to be paged to disk, but you have no space allocated for it, so it remains in physical RAM. "Modified" memory is effectively no different than "In Use" if starved of disk pages.
Neither a 64-bit OS nor PAE enables any 32-bit process to use more than 4GiB (2 user/2 kernel on 32-bit/PAE, 4GiB user on x64. See: Pushing the Limits of Windows: Virtual Memory) of memory whether that memory is currently paged into physical RAM or onto disk.
So, for Firefox, which does not have a 64-bit version, this means that Firefox is ultimately limited to 2GiB (PAE) or 4GiB (x64) of user space memory regardless of installed RAM. Even if you had 64GiB physical RAM installed, if Firefox exceeds usage, it will exhaust its memory limit.
This looks excessive and possibly indicative of a memory or handle leak. Worse, in your case, the pagefile is not large enough to move any of the leaking process to disk, so you have essentially forced it to leak exclusively into physical RAM.
In TechNet article Having a problem with Excessive "modified" memory usage in Win7 x64, upwards of 3.6GB, any suggestions?, a user with similar symptoms (large shared/modified block) finds:
Modified memory is memory that was allocated by some application and then removed from the application's working set, usually because it hasn't been used for a long time. The fact that most of your memory is in this state means two things:
- Some app (or multiple apps) allocated a lot of memory, and is not actively using most of it. Often (but not always) this is due to a memory leak in the app.
- The pagefile is not large enough for the system to move all this unused memory to disk.
In the end, as suggested by magicandre1981, the issue was the Dell Wireless (Broadcom) LAN tray app:
Thanks everyone, I was having the same issue and added handles and GDI and saw BCMWLTRY.EXE, the Dell wireless tray util, is creating handles nonstop. Thanks for the advice.
Update: So I disabled the service that started BCMWLTRY.EXE + rebooted and I'm now not seeing the handle leaks. WLTRAY.EXE is still running but it's not leaking handles.
Likewise, in Excessive modified memory in shared memory,
Dell wlan utility was the reason. Killing the bcmwltry.exe process in task manager released memory. But this process initiated again after few sec.. It was kinda crappy - i couldnt uninstall it. Luckily this notebook needed OS change anyway so i`m not even trying to fix this.
...
Same exact problem here. Killing and disabling DW WLAN (bcmwltry.exe) freed up 5 GB of memory. It leaks.
The tray app BCMWLTRY.EXE is likewise implicated in Pagefile keeps growing until I run out of memory
Additional references for BCMWLTRY.EXE/Dell Wireless Tray problems (WLTRAY.EXE), diagnostics, and resolutions:
So, it's not actually the driver in these cases, it's the tray app that comes with the driver. I would agree that given the OEM of your wireless card, that BCMWLTRY.EXE is running, and the rather large number of nearly identical problems, this is part one of the problem.
You can also add the "Handles" and "Page Faults" columns to Task Manager to check for excessive usage of either, in the case that there are more errant programs. You can use SysInternals VMMap to view allocation of virtual memory from the point of view of each process. Handle leaks are insidious because they are also likely leaking Windows objects. Microsoft makes available a tool called Application Verifier that, among other things, will help track down leaking handles.
So the answer is: You likely have a memory leak, it is likely in BCMWLTRY.EXE (or some other app leaking handles or similar shared resources), and the OS has been starved of disk backed pages to more "gracefully"* handle such a condition.
* There's nothing particularly "graceful" about a memory leak, but a ~4GiB pagefile would have allowed the process to run up to its inherit limit (acting as a quota) and then crash, creating a obvious pattern in the event log of a specific process crashing followed by a release of all the resources and, if restarted, repeat. Without the inherent 32-bit limit address space limit, it could have chewed through a lot of disk space first. As it is, all the processes are now resource constrained, so it's less clear if it's being closed due to its own leak (Firefox), or because another app (BCMWLTRY).
Best Answer
This is just business as usual, and everything is working correctly. Windows isn't holding cached files in memory over other data - Cached files means that at one point, Windows had to load them into memory or read them off the disk, and something higher-priority has not come along since that time. Just as soon as you resume using the computer and programs request memory, Windows will happily remove the cached files to make room.
If you look at RAMMap, you'll notice that most of those cached files are allocated under "Standby" - that means that it's being held in memory because Windows needed it in the past, might need it again, and will quite happily discard it if something else actually needs the space.
Basically what you're seeing here is your program requests a large data file to be loaded, so Windows loads it into memory. Managing memory is a very complex process, and what Windows is doing under the covers here is making a judgement call: It looks at the current processes, and sees that Chrome is taking up, say 5GB of memory (lots of tabs!), but that most of that memory hasn't been touched in the last hour. At this point, it has a choice: It can leave Chrome in memory, and not cache the files. This means that the Cataloger process could take hours instead of minutes to process a file, especially if it is jumping around in the file a lot; or, it can page out the chrome tabs and load the Cataloger file into memory, and finish that process quickly.
Now, you'll feel the pain when Windows has to page chrome back into memory, but Windows won't do that until you request it (i.e., bring chrome back into the foreground and select a tab that was paged out), and what ultimately needs to be evaluated by you is if the pain of that is greater or less than the pain of your Cataloger task having to wait on it's memory. You can try running the program as a service and tell Windows to optimize for responsiveness, or try lowering the process priority, but I'm pretty sure what you're going to want is for the Cataloger process to finish as fast as possible - While it's maxing out your disk I/O, your entire system is going to be incredibly sluggish. Everything that requires disk I/O will be put into a queue (even webbrowsing - it has a cache it uses too!). Programs will open slow, new tabs will open fast, but actually using them for something will be slow.
If the program is just sequentially reading file after file and then not touching them again, I'll agree that Windows doesn't need to be caching the files. The problem here is Windows doesn't know that, because nobody is telling it that they don't need to be cached.
If you have access to the source code for the Cataloger program (or can request modifications be made to it), it can be configured to open files with
FILE_FLAG_NO_BUFFERING, which will cause Windows to skip the disk cache for files opened by the program in this manner.If that's not an option, then unless the Cataloger process has to be run on your system, I'd look around to see if there's a spare computer that isn't being used, and see if it can be set up as a dedicated machine for cataloging these files. If there's not one already, simply increasing your RAM might be another option; memory prices are quite cheap these days, and developers know this - most things are optimized for speed and responsiveness at the cost of memory. 8GB isn't what it used to be.
Another option is to move all of the files that it uses onto a separate harddrive, and specifically turn disk-caching off for that drive.