Mongodb – Replacing a member of a replica set with the same host name

MongoDB replication is to provide automatic fail over during disaster recovery, minimize downtime during maintenance, to use the secondary for analytical use cases and so on. In your scenario, you don't want to avail these benefits during off-peak hours and as far as I know there is no perfect workaround to do these tasks without downtime.

Simply, shutting down the secondary + arbiter makes Mongo unable to elect a new primary and the app fails

Yes, election process will not happen in a replica set with single node. It needs at least two mongod nodes which might not be present in your case if you remove secondary and an arbiter.

If the best/only solution is to remove the secondary and arbiter from the replica set

You can remove the secondary and arbiter from the replica set by following the steps in this link mongo-replica-set-to-single-server.

what are the consequences of doing when adding them back on the next peak hours cycle?

While adding back the replica sets, you will have to restart mongod instances, issue rs.reconfig() method and it can force the current primary to step down which causes an election. During this window time, your app will not read/write anything from/into database which can last up to 10-30 seconds (for re-configuring replicas), another 20-30 seconds (for restarting mongod instances) based on network bandwidth and replica set configuration. Hence, it is not recommended during peak hours.

This is basically the rolling upgrade procedure from the docs, with the only difference being that you will temporarily be going up to a 4 member replica set based on what you have described. Hence it's basically a viable strategy if you have concerns about 3.0.

Couple of things I will note:

First, the priority zero step shows good caution but is not really gaining you anything. There is no realistic way for that node to become primary before it catches up (not considered up to date, and if the other two members fail there will not be enough votes to elect it primary with 2 out of 4). There's no harm in doing it, it just doesn't add much in a 4 member set.

The other thing that I will note is that the addition of a new set member is not actually needed. You can stop your current secondary and simply update its binaries to 3.0, restart it and it is good. Do the same for the arbiter, then step down the primary and repeat. No resync is then necessary because 3.0 is fine as a straight drop-in replacement for 2.6.