We have a chat application that runs on MongoDB. For the amount of users that we currently have, it is sufficient to have a 2 node+Arbiter set up during peak hours.
However, during off-peak hours, we would like to turn off the secondary node and the arbiter to save on cloud computing costs.
Is this possible? Simply shutting down the secondary+arbiter makes Mongo unable to elect a new primary and the app fails.
If the best/only solution is to remove the secondary and arbiter from the replica set, what are the consequences of doing when adding them back on the next peak hours cycle?
Best Answer
MongoDB replication is to provide automatic fail over during disaster recovery, minimize downtime during maintenance, to use the secondary for analytical use cases and so on. In your scenario, you don't want to avail these benefits during off-peak hours and as far as I know there is no perfect workaround to do these tasks without downtime.
Yes, election process will not happen in a replica set with single node. It needs at least two mongod nodes which might not be present in your case if you remove secondary and an arbiter.
You can remove the secondary and arbiter from the replica set by following the steps in this link mongo-replica-set-to-single-server.
While adding back the replica sets, you will have to restart mongod instances, issue rs.reconfig() method and it can force the current primary to step down which causes an election. During this window time, your app will not read/write anything from/into database which can last up to 10-30 seconds (for re-configuring replicas), another 20-30 seconds (for restarting mongod instances) based on network bandwidth and replica set configuration. Hence, it is not recommended during peak hours.