MongoDB – Secondary and Arbiter Stuck in Startup

configurationmongodbmongodb-3.0replication

I'm using MongoDB version 3.0.0. I'm trying to setup mongodb replication on our machine. Replication was initially setup but due to some changes on the VM the entire thing crashed. When I tried to set it up again, the secondary and Arbiter became stuck in StartUp Mode.

In the mongoDB conf file I have set

replSet=ReplicaSet1

I added 2 machines using the command

rs.add("10.235.96.12:27017")
rs.add("10.235.96.12:27017")

But after this on the primary when I do a rs.status() secondary and arbiter are still shown in StartUp

ReplicaSet1:PRIMARY> rs.status()
{
    "set" : "ReplicaSet1",
    "date" : ISODate("2015-07-31T04:45:57.260Z"),
    "myState" : 1,
    "members" : [
            {
                    "_id" : 0,
                    "name" : "BOSPROD9:27017",
                    "health" : 1,
                    "state" : 1,
                    "stateStr" : "PRIMARY",
                    "uptime" : 63104,
                    "optime" : Timestamp(1438257913, 1),
                    "optimeDate" : ISODate("2015-07-30T12:05:13Z"),
                    "electionTime" : Timestamp(1438254975, 2),
                    "electionDate" : ISODate("2015-07-30T11:16:15Z"),
                    "configVersion" : 7,
                    "self" : true
            },
            {
                    "_id" : 1,
                    "name" : "10.235.96.12:27017",
                    "health" : 1,
                    "state" : 0,
                    "stateStr" : "STARTUP",
                    "uptime" : 62663,
                    "optime" : Timestamp(0, 0),
                    "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                    "lastHeartbeat" : ISODate("2015-07-31T04:45:56.520Z"),
                    "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                    "pingMs" : 0,
                    "configVersion" : -2
            },
            {
                    "_id" : 2,
                    "name" : "10.235.96.13:27017",
                    "health" : 1,
                    "state" : 0,
                    "stateStr" : "STARTUP",
                    "uptime" : 60043,
                    "lastHeartbeat" : ISODate("2015-07-31T04:45:55.786Z"),
                    "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                    "pingMs" : 0,
                    "configVersion" : -2
            }
    ],
    "ok" : 1
}

When I try to do a rs.conf on the Secondary or Arbiter I get the message

2015-07-31T05:23:07.927+0000 E QUERY    Error: Could not retrieve replica set config: {
    "info" : "run rs.initiate(...) if not yet done for the set",
    "ok" : 0,
    "errmsg" : "no replset config has been received",
    "code" : 94
}

I did try multiple times to clean all the local.0, local.1 files in all the machines and also removed all the journal files from the journal folder as well. Still I'm getting the same issue. Can someone tell me what is it I'm doing wrong here?

I have set replset on all the members. Tried starting in standalone mode and it works.

Best Answer

From "BOSPROD9", try to connect with the mongoshell to the other serves:

$ mongo --host 10.235.96.12 --port 27017

$ mongo --host 10.235.96.13 --port 27017

(Telnet is not the same.) If this doesn't work, it might be firewall or BindIP.

Check bind_ip (should be 0.0.0.0, change in mongodb.conf is it's 127.0.0.1):

$ netstat -nap | grep :27017 | grep LISTEN
tcp        0      0 0.0.0.0:27018           0.0.0.0:*               LISTEN      -

Try to look at the log-files on 10.235.96.12 and 10.235.96.13, why they are stuck. Did they receive the configuration?

Try to reconfigure this way:

mongo> var cfg = {_id:"ReplicaSet1",members:[{_id:0, host:"BOSPROD9:27017"},{_id:1, host:"10.235.96.12:27017"},{_id:2, host:"10.235.96.13:27017",arbiterOnly:true}]};
mongo> rs.reconfig(cfg);

Solution:

Use hostnames instead of ip's: give all servers a hostname, update the hosts-files and use hostnames in the rs-configuration. It seems that mongodb advices not to use ip addresses, but hostnames.

Related Question