MongoDB: queue vs. realtime write/update

mongodb

On a large mongoDB I have 2 different use-cases for reading and 2 different types for write/update.

  • read 1: find by index -> no question on this. this works like a charm – as long as the index fits in the memory/ram.
  • read 2: read line by line, like a batch script which is reading a log-file one by one.
  • write 1: insert – append a new dataset or document.
  • write 2: update data, directly addresses or found by index.

Now I wonder if mongoDB is able to "priorize" different write-tasks. I have one collection which does not require the data to be written and available instantly in "real-time". It's enough to use a slow queque and process these writing tasks without hurry.
However, there are also write statements that should be processed asap and be available with a minimum delay (if any).

Does mongoDB allow me to differ between high (asap) and low prio (queque) writing/updating?

Best Answer

There is no priority system currently for writes (or reads, though you can send reads to secondaries) - the closest thing you will get is yielding. For long running operations and for operations that it predicts will page in data from disk, MongoDB will yield the lock and allow other operations through, essentially interleaving operations:

If you wanted to make sure that the less important writes are throttled somewhat you could rate limit them by ensuring they are replicated W=2, REPLICAS_SAFE, or similar writes (depending on your driver). See here for the command behind such implementations on the MongoDB side - take a look at your driver docs for the relevant equivalent there.

http://www.mongodb.org/display/DOCS/getLastError+Command#getLastErrorCommand-%7B%7Bw%7D%7D

There would then be a slight delay as the write waits for replication out to the secondaries, allowing your other, more important writes their shot at the write lock.

http://www.mongodb.org/display/DOCS/How+does+concurrency+work

In terms of the future, with 2.2 due out shortly, you will get database level locking, so as long as your 2 different profiles/priorities are in different databases you should have no lock contention (IO/RAM contention may still exist, of course).

Finally, in terms of other things to look at, for the line by line type read, I would look at capped collections and tailable cursors - see if they fit your use case:

http://www.mongodb.org/display/DOCS/Tailable+Cursors

Related Question