Hmm, the two data nodes on one server means that if you lose 192.168.0.1 (node 3/node 4) you lose the whole cluster (possibly why you are experiencing the failure).
In fact the fact that you have effectively a single data node (unless I'm missing something) means you don't really have a cluster - you only have one copy of data.
The management node is not essential for normal mysql operations, except on:
- Installation and other cluster-wide maintenance operations
- It is by default the arbitrator node, which means that it holds the responsibility to decide if a particular partition continues running in the case of a split brain.
Split brain is an important concept in clustering, which happens when apparently, a group of nodes identify some other group of nodes as down (a group can be a single node). However, you cannot be sure if they are really down or the communication among them is down. In order to maintain consistency, an algorithm must decide what to do. If not, your separate partitions of the cluster could end up with different states/data sets. Typically, and that is what happens, if your partition has more than 50% of the nodes, you continue, if not, you stop yourself.
In your particular case, without an arbitrator, as 1 node is only 50% of the data nodes, it decides to "kill themself" -it is not a failure, it is a design decision. With the arbitrator, we know that the other node will not continue by himself, as the arbitrator cannot see it, so it decides to continue with 50% of the nodes (1, that has all the data, as the default is noofreplicas = 2) + arbitrator. In general, you want to run with an odd number of nodes to minimize the possibility of a full cluster shutdown.
Typically, on an NDB setup, your SQL nodes are also a primary and a secondary arbitrator node (you can modify its weights). You normally have more than one API node, to avoid a single point of failure. So the most typical setup is 2 API/SQL/arbitrator nodes and 2 NDB nodes. That way, in most cases, 2 nodes have to go down for the whole cluster to fail.
Best Answer
MySQL Cluster automatically replicates data nodes, based on the number of replicas and the number of data nodes you specify. Your tables are automatically sharded across these nodes, and updates are replicated between them for HA.
Worth watching the short demo to see more: http://www.oracle.com/pls/ebn/swf_viewer.load?p_shows_id=11464419
For more detail - take a look at the 'Cluster overview' section of the docs: http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-basics.html