Mongodb replication failed with socketexception and listener

I'm trying to set up mongodb replication using two servers in different data center. For the purposes of this question, I'll say their IP address are 1.1.1.1 and 2.2.2.2. I installed a clean version of centos8 and mongo with these commands on each server:

yum install -y yum-utils
cat <<EOT >> /etc/yum.repos.d/mongodb-org-4.4.repo
[mongodb-org-4.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/8Server/mongodb-org/4.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc
EOT
yum install -y mongodb-org-4.4.1 mongodb-org-server-4.4.1 mongodb-org-shell-4.4.1 mongodb-org-mongos-4.4.1 mongodb-org-tools-4.4.1
mkdir -p /data/db
sed -i "s/bindIp:.*/bindIp: 0.0.0.0/g" /etc/mongod.conf
systemctl start mongod

This worked great. I can do ssh root@1.1.1.1 and then I can run mongo --host 2.2.2.2 to enter the mongo cli of server 2.2.2.2 from server 1.1.1.1. I can also ssh root@2.2.2.2 and run mongo --host 1.1.1.1 to enter the mongo cli of server 1.1.1.1 from server 2.2.2.2.

Next, I ssh root@1.1.1.1. Then I ran the command:

 mongod --auth --replSet "rs0" --bind_ip localhost,2.2.2.2

But this failed and gave this output:

[root@li1953-23 ~]# mongod --auth --replSet "rs0" --bind_ip localhost,2.2.2.2
{"t":{"$date":"2020-11-15T01:27:26.192+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2020-11-15T01:27:26.197+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2020-11-15T01:27:26.198+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2020-11-15T01:27:26.198+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2020-11-15T01:27:26.199+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":130556,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"li1953-23"}}
{"t":{"$date":"2020-11-15T01:27:26.199+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","openSSLVersion":"OpenSSL 1.1.1c FIPS  28 May 2019","modules":[],"allocator":"tcmalloc","environment":{"distmod":"rhel80","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2020-11-15T01:27:26.199+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"CentOS Linux release 8.2.2004 (Core) ","version":"Kernel 4.18.0-193.14.2.el8_2.x86_64"}}}
{"t":{"$date":"2020-11-15T01:27:26.199+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"localhost,2.2.2.2"},"replication":{"replSet":"rs0"},"security":{"authorization":"enabled"}}}}
{"t":{"$date":"2020-11-15T01:27:26.200+00:00"},"s":"E",  "c":"STORAGE",  "id":20568,   "ctx":"initandlisten","msg":"Error setting up listener","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Cannot assign requested address"}}}
{"t":{"$date":"2020-11-15T01:27:26.200+00:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":10000}}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"REPL",     "id":4784907, "ctx":"initandlisten","msg":"Shutting down the replica set node executor"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"CONTROL",  "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"FTDC",     "id":4784926, "ctx":"initandlisten","msg":"Shutting down full-time data capture"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"STORAGE",  "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"STORAGE",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2020-11-15T01:27:26.201+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":48}}

I am unable to continue with the mongodb tutorial on how to set up replication until I remedy this problem. What did I do wrong and how do I fix?

EXTRA – 2020-11-16 06:30

I also tried to ssh root@1.1.1.1 and then mongo to enter cli of mongo instance at 1.1.1.1. Then I ran this command:

rs.initiate( {
_id :  "rs0",
members: [
{ _id:  0, host:  "1.1.1.1:27017" },
{ _id:  1, host:  "2.2.2.2:27017" }
]
})

Which gives the error:

{
        "ok" : 0,
        "errmsg" : "This node was not started with the replSet option",
        "code" : 76,
        "codeName" : "NoReplicationEnabled"
}

EXTRA 2020-11-16 0900

I made some slight progress by using --bind-ip 0.0.0.0 and --db-path /var/lib/mongo. What I did was

ssh root@1.1.1.1
systemctl stop mongod
rm -f /tmp/mongodb-27017.sock #this file doesn't seem to delete itself after shutdown
chown -R mongod:mongod /var/lib/mongo* #for some reason, permissions are set to root after
mongod --auth --dbpath /var/lib/mongo --replSet "rs0" --bind_ip 0.0.0.0

My terminal window for 1.1.1.1 reports mongod is running, and I must leave this window open to keep things active. I also make sure mongod is running on my 2.2.2.2 with the usual systemctl start mongod.

Now I open a new terminal window and do this:

ssh root@1.1.1.1
mongo #enter mongo cli
rs.initiate( {
_id :  "rs0",
members: [
{ _id:  0, host:  "1.1.1.1:27017" },
{ _id:  1, host:  "2.2.2.2:27017" }
]
})

But this results in the follow error:

rs.initiate( { _id :  "rs0", members: [ { _id:  0, host:  "1.1.1.1:27017" }, { _id:  1, host:  "2.2.2.2:27017" } ] });
{
        "ok" : 0,
        "errmsg" : "replSetInitiate quorum check failed because not all proposed set members responded affirmatively: 2.2.2.2:27017 failed with not running with --replSet",
        "code" : 74,
        "codeName" : "NodeNotFound"
}