MongoDB – how to reduce numYields

There is no priority system currently for writes (or reads, though you can send reads to secondaries) - the closest thing you will get is yielding. For long running operations and for operations that it predicts will page in data from disk, MongoDB will yield the lock and allow other operations through, essentially interleaving operations:

If you wanted to make sure that the less important writes are throttled somewhat you could rate limit them by ensuring they are replicated W=2, REPLICAS_SAFE, or similar writes (depending on your driver). See here for the command behind such implementations on the MongoDB side - take a look at your driver docs for the relevant equivalent there.

http://www.mongodb.org/display/DOCS/getLastError+Command#getLastErrorCommand-%7B%7Bw%7D%7D

There would then be a slight delay as the write waits for replication out to the secondaries, allowing your other, more important writes their shot at the write lock.

http://www.mongodb.org/display/DOCS/How+does+concurrency+work

In terms of the future, with 2.2 due out shortly, you will get database level locking, so as long as your 2 different profiles/priorities are in different databases you should have no lock contention (IO/RAM contention may still exist, of course).

Finally, in terms of other things to look at, for the line by line type read, I would look at capped collections and tailable cursors - see if they fit your use case:

http://www.mongodb.org/display/DOCS/Tailable+Cursors

The real reason why you can't do as you ask (limit the memory) is because MongoDB doesn't manage the memory it uses directly - it lets the OS do it. MongoDB just memory maps all its data and then has the OS page it in and out of memory as needed. As a result, there is no direct management of the amount used possible until MongoDB implements this in a completely different way, or the OS allows it (not possible in Linux since the 2.4 days).

The only way to truly segregate resources at present is to use a virtualization solution and isolate MongoDB in its own VM. Yes, there are overheads involved (though hypervisors have gotten a lot better), but at the moment that is the price to be paid for that level of resource control.

In terms of the OOM Killer, even with no other processes on the host, as long as your data set and indexes overall exceed available memory, MongoDB can hit OOM Killer issues. This is because of how the data gets paged out of memory - if there is no memory pressure (nothing else wants Resident memory), and you keep adding/touching new data and indexes, then eventually it will grow to consume all available RAM. Hence the recommendation to always configure some swap when running MongoDB:

https://docs.mongodb.com/manual/administration/production-notes/#swap

Of course, LRU data will be paged out first, other processes can take up the res mem also, but the concept still applies unless you load your data set into memory and then it stays static. Best thing to do if you are worried is get it into MMS and track the usage over time:

http://mms.mongodb.com

Update: August 2015

Since I wrote this answer things have moved on somewhat and the information is a little out of date. For example, Linux now has cgroups and related technologies (Docker containers for example) that have matured to the point that they allow you to better isolate and limit the resources (including memory) consumed by any process in a production environment, even one that uses memory mapping like MongoDB.

Additionally, with the advent of new storage engines beyond MMAP like WiredTiger in MongoDB 3.0+ you can use built-in functionality to limit the cache size for MongoDB. Hence, the RAM requirements now really do depend on how you choose to configure MongoDB, what environment you run it in, and what storage engine you choose.