Mongodb – The reading with secondaryPreferred when chunk migration occurred often fail

mongodb

We have some problems regarding balancer configuration.
Our application which is set up with secondaryPreferred often fails to read documents from the secondary when balancer( chunk migration ) is running.
So I changed the Balancer setting to the appropriate setting as below, but I could observe the unexpected behavior.

use config
db.settings.update(
   { "_id" : "balancer" },
   { $set : { "_secondaryThrottle" : true ,
              "writeConcern": { "w": 6} } },
   { upsert : true }
)

->  { "_id" : "balancer",   "stopped" : false,   "_secondaryThrottle" : true,   "writeConcern" : { "w" : 6 } }

1. Environment

Our cluster is as below.

( 1 primary, 5 secondary, one arbiter ) * 6 masters on Sharded cluster
on MongoDB 3.2.4 on Google Cloud Platform Mongod instance type is
"n1-standard-16 (16 vCPUs, 60 GB memory) with 500GB persistent disk"

2. What did I expect?

I expect that the application with reading as secondaryPreferred can read from the Secondary-mongod without failing. I think that the Balancer can wait for all completion of chunk-migration because I set up "writeConcern:6". So my expectations are as described above.

3. Test conditions and results

I configured as above settings and restarted balancer and execute the application.
The application( secondaryPreferred ) failed with high frequency to read from the secondary-mongod.

Please let me ask a question.

  • Q1. What is the meaning of writeConcern in balancer setting? It seems
    that writeConcern does not work as shown in the above test.
  • Q2. How can I read without failing when chunk migration occurred?
  • Q3. What is the purpose of _secondaryThrottle? I have read the
    official document of v3.2. And official document described that
    _secondaryThrottle is true as the default value and has the same meaning as writeConcern: 2. But we can configure settings as below.
    If I set up _secondaryThrottle:false, does it makes the writeConcern
    invalid right? Is "_secondaryThrottle" the flag for enabling
    writeConcern?

e.g. { "_id" : "balancer", "stopped" : false, "_secondaryThrottle" :
false, "writeConcern" : { "w" : 6 } }

sincerely

Best Answer

Nowadays, balancer doesn't honor _secondaryThrottle, nor w: when w:<=1. But, this case that w:6 means that when data is removed (range deleter at end of chunk move), we are removing documents one by one AND every document delete must have 6 acknowledge. If moved chunk had 100.000 documents, then there is 100.000 individual deletes with acknowledges. (slooooow)

If your query at secondary hits documents in the range what is under range delete, there will be a collision.

Related Question