Mongodb – Copy Remote Database MongoDB

You are not missing the point :)

The key is how many reads and how many writes do you have to do when you store/update/fetch various entities in your collections?

You are presumably constantly (daily?) creating new posts and frequently creating new comments, and very frequently querying posts.

Actions like updating authors' email are very rare compared to the above. You want to optimize the performance of reads and writes that happen hundreds or thousands of times per in your application, and worry less about performance of very infrequent actions.

Having said that, I would store the basic author information embedded in the post, but I would only have information there that must be displayed with every post. If there are additional details about the author I want to keep in its own collection, I would expect that would be details that the user needs to click additional link to see (allowing for time to do another read).

A capped collection would allow you to maintain a relatively small list, though you would want to have a decent buffer, or be very confident in terms of the size of your documents. The other option would be a TTL collection, but that would be more prone to fragmentation since it will be doing a lot of deletes (this will mean having to deal with said fragmentation regularly with repair, compact or a resync).

Based on what you outline, this sounds very like what people have done when looking to implement a message queue with MongoDB - you use the capped or TTL collection for your immediate needs, then you decide what level of durability you want for the documents long term. That is, store them in another collection, database, or even another instance of MongoDB entirely before they are removed. At a larger scale I have even seen people have long term and short term shards, but that should be a long way down the line from your initial attempts.

If you do go down the path of a capped collection, with "long term" storage elsewhere, it is often beneficial to have the capped collection on one disk and the other data elsewhere (capped collections tend to have sequential access patterns, whereas regular collections tend towards random access). That means using different databases, different disk mount points, and the directoryperdb option.

There are a lot of potential options here but thankfully several of the approaches have been publicised and documented. To start I would take a look at these:

http://www.mongodb.com/presentations/mongodb-message-queue http://captaincodeman.com/2011/05/28/simple-service-bus-message-queue-mongodb/ http://blog.mongodb.org/post/29495793738/pub-sub-with-mongodb

Hopefully that's enough to get you started.