MongoDB- could not find member to sync from

mongodb

Node logs under Recovery mode, even after restarting:

    2018-04-24T12:37:52.915+0530 I REPL [replication-0] We are too stale to use J-DB-02:27017 as a sync source. Blacklisting this sync source because our last fetched timestamp: 5ade01e0:66 is before their earliest timestamp: 5aded094:198 for 1min until: 2018-04-24T12:38:52.915+0530 2018-04-24T12:37:52.916+0530 I REPL [replication-0] could not find member to sync from 2018-04-24T12:37:52.916+0530 E REPL [rsBackgroundSync] too stale to catch up -- entering maintenance mode 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] Our newest OpTime : { ts: Timestamp 1524498912000|102, t: 68 } 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] Earliest OpTime available is { ts: Timestamp 1524551828000|408, t: 68 } 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] going into maintenance mode with 0 other maintenance mode tasks in progress 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] transition to RECOVERING

Here is the status of replica set from primary server

    WAPRS01:PRIMARY> db.oplog.rs.stats().maxSize
    WAPRS01:PRIMARY> rs.status();
    {
            "set" : "WAPRS01",
            "date" : ISODate("2018-04-24T07:17:15.276Z"),
            "myState" : 1,
            "term" : NumberLong(68),
            "heartbeatIntervalMillis" : NumberLong(2000),
            "optimes" : {
                    "lastCommittedOpTime" : {
                            "ts" : Timestamp(1524498912, 102),
                            "t" : NumberLong(68)
                    },
                    "appliedOpTime" : {
                            "ts" : Timestamp(1524554235, 160),
                            "t" : NumberLong(68)
                    },
                    "durableOpTime" : {
                            "ts" : Timestamp(1524554235, 92),
                            "t" : NumberLong(68)
                    }
            },
            "members" : [
                    {
                            "_id" : 0,
                            "name" : "J-DB-01:27017",
                            "health" : 1,
                            "state" : 3,
                            "stateStr" : "RECOVERING",
                            "uptime" : 569,
                            "optime" : {
                                    "ts" : Timestamp(1524498912, 102),
                                    "t" : NumberLong(68)
                            },
                            "optimeDurable" : {
                                    "ts" : Timestamp(1524498912, 102),
                                    "t" : NumberLong(68)
                            },
                            "optimeDate" : ISODate("2018-04-23T15:55:12Z"),
                            "optimeDurableDate" : ISODate("2018-04-23T15:55:12Z"),
                            "lastHeartbeat" : ISODate("2018-04-24T07:17:13.532Z"),
                            "lastHeartbeatRecv" : ISODate("2018-04-24T07:17:13.493Z"),
                            "pingMs" : NumberLong(0),
                            "configVersion" : 4
                    },
                    {
                            "_id" : 1,
                            "name" : "J-DB-02:27017",
                            "health" : 1,
                            "state" : 2,
                            "stateStr" : "SECONDARY",
                            "uptime" : 299382,
                            "optime" : {
                                    "ts" : Timestamp(1524554232, 270),
                                    "t" : NumberLong(68)
                            },
                            "optimeDurable" : {
                                    "ts" : Timestamp(1524554232, 270),
                                    "t" : NumberLong(68)
                            },
                            "optimeDate" : ISODate("2018-04-24T07:17:12Z"),
                            "optimeDurableDate" : ISODate("2018-04-24T07:17:12Z"),
                            "lastHeartbeat" : ISODate("2018-04-24T07:17:13.454Z"),
                            "lastHeartbeatRecv" : ISODate("2018-04-24T07:17:13.864Z"),
                            "pingMs" : NumberLong(0),
                            "syncingTo" : "J-DB-03:27017",
                            "configVersion" : 4
                    },
                    {
                            "_id" : 2,
                            "name" : "J-DB-03:27017",
                            "health" : 1,
                            "state" : 1,
                            "stateStr" : "PRIMARY",
                            "uptime" : 613420,
                            "optime" : {
                                    "ts" : Timestamp(1524554235, 160),
                                    "t" : NumberLong(68)
                            },
                            "optimeDate" : ISODate("2018-04-24T07:17:15Z"),
                            "electionTime" : Timestamp(1524254862, 1),
                            "electionDate" : ISODate("2018-04-20T20:07:42Z"),
                            "configVersion" : 4,
                            "self" : true
                    },
                    {
                            "_id" : 3,
                            "name" : "J-DB-04:27017",
                            "health" : 1,
                            "state" : 7,
                            "stateStr" : "ARBITER",
                            "uptime" : 300361,
                            "lastHeartbeat" : ISODate("2018-04-24T07:17:14.863Z"),
                            "lastHeartbeatRecv" : ISODate("2018-04-24T07:17:14.229Z"),
                            "pingMs" : NumberLong(0),
                            "configVersion" : 4
                    }
            ],`enter code here`
            "ok" : 1
    }
    WAPRS01:PRIMARY>

db.isMaster():

WAPRS01:PRIMARY> db.isMaster();
{
        "hosts" : [
                "J-DB-01:27017",
                "J-DB-02:27017",
                "J-DB-03:27017"
        ],
        "arbiters" : [
                "J-DB-04:27017"
        ],
        "setName" : "WAPRS01",
        "setVersion" : 4,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "J-DB-03:27017",
        "me" : "J-DB-03:27017",
        "electionId" : ObjectId("7fffffff0000000000000044"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1524559624, 465),
                        "t" : NumberLong(68)
                },
                "lastWriteDate" : ISODate("2018-04-24T08:47:04Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 1000,
        "localTime" : ISODate("2018-04-24T08:47:04.511Z"),
        "maxWireVersion" : 5,
        "minWireVersion" : 0,
        "readOnly" : false,
        "ok" : 1
}

Best Answer

As I am able to see from your rs.status() , there is secondary replica is showing in RECOVERING state. And as MongoDB jira blog After node goes into RECOVERING due to being too stale to sync from its source, it will never recover even if there is a valid sync source it could use.

As per MongoDB documentation here A replica set member becomes “stale” when its replication process falls so far behind that the primary overwrites oplog entries the member has not yet replicated. The member cannot catch up and becomes “stale.” When this occurs, you must completely resynchronize the member by removing its data and performing an initial sync.

For further your ref here , here , here, here and here

Related Solutions

Mongodb config servers not in sync

Restoring config servers, particularly if you have had some sort of catastrophic event is tricky, but not impossible. But, before we go any further, a big bold caveat:

BACK UP EVERYTHING

That means taking a back up of all three config servers. I am going to give you some advice, and it is generally correct, but please, please take a back up of every current config server instance before you overwrite/replace anything

As a quick explanation, config servers are not configured as a replica set - each config server instance is supposed to be identical (at least for all the collections that matter) to the others. Hence, any healthy config server can be used to replace a non-healthy config server and you can then follow the tutorial you mentioned to get back to a good config.

The key to recovery is to identify the healthy config server and then use that to replace the others - you then end up with 3 identical config servers.

There is more than one way to do this, they basically fall into three categories:

1) Use the error message

The error message that is printed out actually lets you know which config server it believes is health, though that is not obvious from the messaging. Here's how to read it generically:

ERROR: config servers not in sync! config servers <healthy-server> and <out-of-sync-server> differ

Basically the first one in the list is the healthy one, in your case that would be mongocfg1.testing.com:27000. That is our first candidate for a healthy config database.

2) Use dbhash to compare all three and pick the ones that agree

On each config server switch to the config database using use config, run db.runCommand("dbhash") and compare the hashes for the collections below:

chunks
databases
settings
shards
version

You are looking for two servers that agree, and using that as the basis to determine that the version of the config database on those hosts is basically trustworthy and should be used to seed the rest.

3. Manually inspect the collections in the config database

Finally, take a look at the config database, and pay attention to the collections listed in the second option above. This is a straight judgement call based on your familiarity with your data.

Hopefully all three methods point you at the same host (or hosts). That config server should be used to seed the other two (after you have taken backups so you can go back). That is basically your best bet. Should that fail, then you may want to try one of the other versions (from the backups) - always making sure that when you start them, all three are identical.

Finally, always ensure that all mongos processes are using the same config server string, and that all 3 servers are always listed in the same order on every process - not doing so across all mongos processes can lead to (very) odd results.

Mongodb – “ERROR: could not read from config file” MongoDB

First, assuming you are actually specifying where to read the file from, make sure that you have permission read that file with the current user (cat /usr/local/Cellar/mongodb/2.4.6/mongod.conf - or use less/vi/editor of choice). Assuming that works (and if it does not, adjust your permissions), then the next thing you need to do is make sure you are actually pointing at the correct file.

However, if you are not specifying where to read the file from, by default, if you just run mongod using the brew installation it will attempt to read from:

/usr/local/etc/mongod.conf

I verified this by installing 2.4.6 with brew and then checking the logs when it starts up:

Tue Oct 22 17:17:30.695 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/usr/local/etc/mongod.conf", dbpath: "/usr/local/var/mongodb", logappend: "true", logpath: "/usr/local/var/log/mongodb/mongo.log" }

You can either modify that file (/usr/local/etc/mongod.conf) to look the way you want in your example, and make sure you have permissions to get to it, or you can run this instead to specify the original file:

mongod -f /usr/local/Cellar/mongodb/2.4.6/mongod.conf

Best Answer

Related Solutions

Mongodb config servers not in sync

Mongodb – “ERROR: could not read from config file” MongoDB

Related Question