Mongodb – Mongo E11000 duplicate key error collection after mongo 3.4 to 3.6 upgrade

mongodbmongodb-3.4mongodb-3.6upgrade

I have a Mongo replica set on Ubuntu with three members configured for my information system and am having issues when I try to upgrade MongoDB from version 3.4 to 3.6

After setting the "featureCompatibilityVersion" to 3.4 on the master node and checking it on the two slaves, I started upgrading the slaves one by one (I gracefully shut the DB down with db.shutdownServer(), update the packages using apt and start the mongod service). That all went fine and the slaves transitioned back to secondary states. I then shut the master down, repeated all the steps for the upgrade from the official guide (https://docs.mongodb.com/manual/release-notes/3.6-upgrade-replica-set/) and tried starting the mongod service again.

However, while I was upgrading the master node, the slave that was elected as a new master crashed with the error:

[rsSync] Fatal assertion 40454 DuplicateKey: E11000 duplicate key error collection: admin.system.users index: user_1_db_1 dup key: { : "john", : "admin" } at src/mongo/db/system_index.cpp 134

I didn't notice this and when the old master started up again it too crashed with the same error. In my frantic trying to get the members up again, the third member had also been elected as a master for a short time before crashing with the same error.

When I managed to get the first member up again as a standalone, I checked the system.users collection and found that there were indeed two users with the same username for the admin database (I redacted the parts with …):

{ "_id" : ObjectId("5...8"), "user" : "john", "db" : "admin", "credentials" : { "MONGODB-CR" : "f...c" }, "roles" : [ { "role" : "read", "db" : "admin" } ] }

and

{ "_id" : "admin.john", "user" : "john", "db" : "admin", "credentials" : { "MONGODB-CR" : "f...c" }, "roles" : [ { "role" : "read", "db" : "admin" } ] }

My question is how is that possible? Shouldn't mongo prevent someone from adding an identical user? Or did mongo try to change that collection after the upgrade and failed? I am using authSchema version 3 if it makes any difference. Thanks!

Best Answer

First, per the documentation you should always upgrade the Secondaries first, finishing with the primary node as per MongoDB upgrade process.

Second, since 3.6.13, MongoDB added a field userId adding a UUID into any user created to avoid _id collision such as what you seem to have encountered. See https://jira.mongodb.org/browse/SERVER-38984.

Related Question