“Cluster nodes” command showing different status on different Redis Cluster nodes

redis

I had created a redis cluster of 6 instances running on 2 RHEL-7.4 servers some time ago. Today I found that 3 of those instances(on 10.32.129.79) were down. Since I've restarted those 3 instances, I've observed that the command "cluster nodes" is not giving the same output on different nodes.
Given below is one of the output:

dcfc502ffae3810b61368e774187406f774114d8 10.32.129.77:6379@16379 myself,master - 0 1571299699000 1 connected 0-5460
274c70586d2ac7ce2723d8a23b4f0126ddae4174 10.32.129.77:6378@16378 master - 0 1571299701559 8 connected 5461-10922
e8b9f87f369da10a3b5f4b9453aaae24c744a156 10.32.129.79:6380@16380 slave,fail 66649746bbd41f06748063cd268f71f4be063aca 1571290310404 1571290310404 5 connected
66649746bbd41f06748063cd268f71f4be063aca 10.32.129.77:6380@16380 master - 0 1571299700000 2 connected 10923-16383
40503860d939a2d70e47c48050e6eaaf109e7f3f 10.32.129.79:6379@16379 slave,fail 274c70586d2ac7ce2723d8a23b4f0126ddae4174 1571290310404 1571290310404 8 connected
8e5c582a49a868ace78d10b8b0d24c4647e1b2dc 10.32.129.79:6378@16378 slave,fail dcfc502ffae3810b61368e774187406f774114d8 1571290310404 1571290310404 6 connected

And given below is the other one:

274c70586d2ac7ce2723d8a23b4f0126ddae4174 10.32.129.77:6378@16378 master,fail? - 1571297986302 1571297981292 8 connected 5461-10922
40503860d939a2d70e47c48050e6eaaf109e7f3f 10.32.129.79:6379@16379 slave 274c70586d2ac7ce2723d8a23b4f0126ddae4174 0 1571299542894 8 connected
66649746bbd41f06748063cd268f71f4be063aca 10.32.129.77:6380@16380 master - 0 1571299538884 2 connected 10923-16383
e8b9f87f369da10a3b5f4b9453aaae24c744a156 10.32.129.79:6380@16380 myself,slave 66649746bbd41f06748063cd268f71f4be063aca 0 1571299541000 5 connected
8e5c582a49a868ace78d10b8b0d24c4647e1b2dc 10.32.129.79:6378@16378 slave dcfc502ffae3810b61368e774187406f774114d8 0 1571299543895 6 connected
dcfc502ffae3810b61368e774187406f774114d8 10.32.129.77:6379@16379 master,fail? - 1571297991309 1571297985000 1 connected 0-5460

Is there any way of recovering from this inconsistent state without losing my data? Also, in which type of scenarios can this happen?

Redis Version: 5.0.4

Best Answer

It turns out there was a firewall issue on both the servers. After stopping the firewall(systemctl stop firewalld), the cluster is consistent.