MongoDB Data loss following machine PowerOff with writeConcernMajorityJournalDefault true

mongodb

Background:

I'm using mongodb 4.2.0
I have deployed a mongo cluster which contain 5 configs, 3 querys and 3 – shards. Each shard consist of 4 replicas and 1 arbiter.
All members are set on VMs.
ReplicaSets writeConcernMajorityJournalDefault flag is true.
Test conducted with pymongo

The Test:

Before implementing the cluster on the production environment, I've conducted several "stress tests". I have created a simple script that performs many inserts to the cluster and returns the amount of successful inserts.

When I run and stop the script everything is just fine. The number of inserts_count is identical to the count of documents in the collection.

BUT, When I run the script and then PowerOff the Primary member, I'm facing a hitch. My script's insert_count is bigger (10-20) than the count of documents in my collection. I assume that I'm losing data.

I got successful insert acknowledge even though my replicaSet is set writeConcernMajorityJournalDefault true.

Raising the primary doesn't help to retrieve the lost data.

I think the data was still in memory!

Conclusion:

I believe that there is some malfunction with the journaling setting.

Please help me understand what am I doing wrong.

P.S:

I tried insert with {w: majority, j: true, wtimeout: 5000} parameters.

Same results :/

Best Answer

How much time your query was running?? and from where you get next count as you mentioned after stopping primary you get this count is lesser?? also If you can share your script.

It will be obvious that If you are stopping the Script the data will not going to be insert as you expected.

Related Solutions

MongoDB store weekly data

have you looked at having Historic Players and Historic Team collections?

Then if you wanted stats about the players at the time, you'd probably want to copy out the current players on the team in your historic games documents. You'd probably want to store that as an array in the Historic Games table as the current players list might be different than the players from last week/year.

Keep the position of the players in the document in mind so you can end up using proper indexes.

historic players
_id: jfoi23j8xx
PlayerID:Bo Jackson
weekinfo<array or tags> 
[week 1- hits:30, misses:3, bunts:7, TeamID: LA Raiders], 
[week2 - hits:17, misses:2, bunts:3, TeamID: Kansas City Royals]


historic teams:
_id:10u40o321js
TeamID: LA Raiders
Weekinfo<array or tags>
[week 1 - consolidated hits: 73, misses: 12],
[week 2 - consolidated hits:65, misses: 16]

Something like that might be a good start. Does it meet your requirements? You might want to have your app handle the aggregation but that's up to you.

MongoDB hashtable namespace index max chain reached error after machine crash

OK I found my own answer.

Because the journal file seems corrupted, this solution will abandon all changes in the journal file so the database might be restored to a slightly earlier state before the crash

According to MongoDB documentation,

At default intervals of 60 seconds, MongoDB asks the operating system to flush the shared view to disk.

the period of losing data should be within less than 60 seconds before crash by default.

The steps are as follows:

Turn off journal in the config. For me it's in /etc/mongod.conf and change journal=true to journal=false.
In the MongDB data directory (e.g. /var/lib/mongodb/, or for me it's /edx/var/mongo/mongodb/), remove both journal folder and mongod.lock file.
Run mongod with --repair option like sudo -u mongodb mongod -f /etc/mongod.conf --repair and wait. This operation may need a lot of disk space.
Turn on journal to journal=true as opposite to step 1, and start the server normally. E.g. sudo service mongod start.

Best Answer

Related Solutions

MongoDB store weekly data

MongoDB hashtable namespace index max chain reached error after machine crash

Related Question