Help in understanding Hinted Handoffs and Data replication in Cassandra

cassandra

We have a 3 node Cassandra cluster with RF = 2. The read and write consistency is set to ONE. We are using Vnodes as well.
Lets label these nodes as N1, N2 and N3. Let's say that N3 goes down. I was under the impression that whenever a node goes down, the other nodes would store hints and whenever N3 comes up, the hints would be sent to N3, thereby ensuring that the data is consistent across replicas. However, as I was going through the docs, I came across the parameter '' which defaults to 3 hours.
So, if a node is dead for more than 3 hours, it is considered permanently dead and no hints are stored. So far, so good.

So, my understanding now is that if a node is down for say 10 hours, then the hints for the first 3 hours would be transferred to this node when it comes back up, but the writes for this 7 hour duration would be lost for this node. Moreover, if a read query is fired for a particular token range, and since this node is also elligible to serve the read requests for a token range, it would return null instead of the actual data that is stored in some other node. Is my understanding correct? What, then, should be done?

Best Answer

Your understanding is correct that beyond the default 3 hours, hints are no longer stored. This is to avoid overloading the other alive nodes for prolonged period of time.

If any node happens to be down for a longer period than the hint_window, its absolutely necessary to rebuild the node or even remove all the files from its data directory and let it bootstrap as a brand new node. You may have to use replace_address if you are bringing up the same node.