MongoDB replicate data item across multiple shards

mongodbreplicationsharding

Is it possible to replicate a data item across multiple shards?

My end goal is to determine how up-to-date the data in a replica slave is. I'm only concerned with subsets of the data, so I don't think examining oplogs will work. Instead, I plan on having a counter increment in the master, every time a modification to the appropriate data occurs. Then, I can fetch the slave's replicated counter to see how current it is.

The problem occurs when I try to shard the data. I need the counter replicated across every replica, not just the replica set that owns the counter data item. So, I need some way to replicate the counter to every master shard, where it will then be replicated to each slave. Is this possible?

Modifications to the counter would still go through the master shard that "owns" the counter. Data reads would need to aggregate the counter value from each shard/replica, taking the minimum value. I read through the MongoDB documentation, and I really don't think it's possible. But perhaps one of you MongoDB experts/developers can shed some hope.

Someone on SO mentioned replSetGetStatus, which may be an option. However, I would like separate counters for different subsets of the data, where the replica status corresponds to all the data.

Best Answer

  1. Create 'c' -database
  2. Add c -collection
  3. Shard this collection with _id
  4. Set this collection balacing:false
  5. Pre-split this collection with _id values: 0,1,2,3,.. (or your real shard names). How many shards you have; one per shard
  6. With moveChunk command, move one chunk to every shard, so that right _id is on right shard.

Now if you update this c.c's counter-key with $inc -operator (what is atomic operation) and without _id is set but multi:true, you can always "query" min value of that counter-key and answer tells what shard and what was value.

db.getSiblingDB('c').c.update({},{$inc:{"counter":1}},{multi:true})

Related Question