MongoDB page faults even when all data is in resident in RAM

The real reason why you can't do as you ask (limit the memory) is because MongoDB doesn't manage the memory it uses directly - it lets the OS do it. MongoDB just memory maps all its data and then has the OS page it in and out of memory as needed. As a result, there is no direct management of the amount used possible until MongoDB implements this in a completely different way, or the OS allows it (not possible in Linux since the 2.4 days).

The only way to truly segregate resources at present is to use a virtualization solution and isolate MongoDB in its own VM. Yes, there are overheads involved (though hypervisors have gotten a lot better), but at the moment that is the price to be paid for that level of resource control.

In terms of the OOM Killer, even with no other processes on the host, as long as your data set and indexes overall exceed available memory, MongoDB can hit OOM Killer issues. This is because of how the data gets paged out of memory - if there is no memory pressure (nothing else wants Resident memory), and you keep adding/touching new data and indexes, then eventually it will grow to consume all available RAM. Hence the recommendation to always configure some swap when running MongoDB:

https://docs.mongodb.com/manual/administration/production-notes/#swap

Of course, LRU data will be paged out first, other processes can take up the res mem also, but the concept still applies unless you load your data set into memory and then it stays static. Best thing to do if you are worried is get it into MMS and track the usage over time:

http://mms.mongodb.com

Update: August 2015

Since I wrote this answer things have moved on somewhat and the information is a little out of date. For example, Linux now has cgroups and related technologies (Docker containers for example) that have matured to the point that they allow you to better isolate and limit the resources (including memory) consumed by any process in a production environment, even one that uses memory mapping like MongoDB.

Additionally, with the advent of new storage engines beyond MMAP like WiredTiger in MongoDB 3.0+ you can use built-in functionality to limit the cache size for MongoDB. Hence, the RAM requirements now really do depend on how you choose to configure MongoDB, what environment you run it in, and what storage engine you choose.

The resident memory size represents the number of pages in memory actually touched by the mongod process. If that is significantly lower than the available memory and data exceeds the available memory (yours does), then it could be a case of simply not having actively touched enough pages yet.

To determine if this is the case, you should run free -m, the output should look something like this:

free -m
             total       used       free     shared    buffers     cached
Mem:          3709       3484        224          0         84       2412
-/+ buffers/cache:        987       2721
Swap:         3836        156       3680

In my example, cached is not close to the total, which means that not only has mongod not touched enough pages, the filesystem cache has not yet even been filled by pages being read from the disk in general.

A quick remedy for this would be the touch command (added in 2.2) - it should be used with caution on large data sets as it will attempt to load everything into RAM even if the data is far too large to fit (causing a lot of disk IO and page faults). It will certainly fill up the memory effectively though :)

If your cached value is close to the total available, then your issue is that a large number of pages being read into memory from disk are not relevant to (and hence not touched by) the mongod process. The usual candidate for this kind of discrepency is readahead. I've already covered that particular topic elsewhere in detail, so I'll just link those two answers for future reading if necessary.