Have 3 node cluster. RF=3. Table with gc_grace_seconds = 120 secs. max_hints duration as default at 3 hours.
inserted a row
nodetool flush on all the 3 nodes
bring down node 3
delete the inserted row
nodetool flush on node 1 and node 2.
hints directory on node 1 and node 2 show the hints files.
Now I bring up the node 3. the hints are sent to the node 3 and deleted entry is made on the node 3.
Why is this behaviour? Is'nt the hint supposed to be expired with the gc_grace_seconds over? hints should not be sent to node 3. Am I missing something here?
help appreciated
Best Answer
Hint and gcgrace(you can set in table level) are two different parameters in Cassandra. irrespective of gcgrace seconds a hint will be stored in system.hints table for the >max_hint_window_in_ms< period mentioned in cassandra.yaml and will be removed after that. max_hint_window_in_ms is your SLA to bring the down node back online and run repair or the deleted data will comeback as a zombie.