Mysql – Problem with MySQL Cluster in practice

MySQLmysql-clusterndbcluster

I am researching MySQL cluster. For the purpose, I do a demo with 4 virtual machines. I have some problems in practice.

Topology

My topology contains four nodes
Node 1: management node, ip address 192.168.56.205
Node 2: sql node, ip address 192.168.56.206
Node 3: data node 1, ip address 192.168.56.207
Node 4: data node 2, ip address 192.168.56.208

Other information

  • Linux OS: Centos 6.4, 32 bits
  • Linux kernel: 2.6.32-358.el6.i686
  • MySQL Cluster version: MySQL-Cluster-gpl-7.3.6-2.el6.i686.rpm-bundle.tar
  • RAM 128MB
  • Turned off firewall iptables
  • Disabled selinux

Installation

  • On all four nodes, I install MySQL-Cluster-server-gpl-7.3.6-2.el6.i686.rpm and MySQL-Cluster-shared-compat-gpl-7.3.6-2.el6.i686.rpm
  • Except Node 2 – sql node, I install more MySQL-Cluster-client-gpl-7.3.6-2.el6.i686.rpm

Configuartion

On node 1 – management node

  • Make directory /var/lib/mysql-cluster, assign owner:group is mysql:mysql

  • Make /var/lib/mysql-cluster/config.ini:

    [ndbd default]
    NoOfReplicas=2
    DataMemory=80M
    IndexMemory=18M

    [tcp default]
    portnumber=2202

    [ndb_mgmd]
    hostname=192.168.56.205
    datadir=/var/lib/mysql-cluster

    [ndbd]
    hostname=192.168.56.207
    datadir=/usr/local/mysql/data

    [ndbd]
    hostname=192.168.56.208
    datadir=/usr/local/mysql/data

    [mysqld]
    hostname=192.168.56.206

On node 2 – sql node

[mysqld]
datadir = /var/lib/mysql
port = 3306
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
ndbcluster

[mysql_cluster]
ndb-connectstring=192.168.56.205

On node 3 and node 4 – data nodes

  • Make directory /usr/local/mysql/data, assign owner:group is mysql:mysql

  • Config /etc/my.cnf:

    [mysqld]
    ndbcluster

    [mysql_cluster]
    ndb-connectstring=192.168.56.205

Start up

On management node, I run:
ndb_mgmd -f /var/lib/mysql-cluster/config.ini
On data node 1 và 2, I run:
ndbd
Finally, on sql node, I run:
mysqld_safe &

Result

ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2 (not connected, accepting connect from 192.168.56.207)
id=3    @192.168.56.208  (mysql-5.6.19 ndb-7.3.6, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @192.168.56.205  (mysql-5.6.19 ndb-7.3.6)

[mysqld(API)]   1 node(s)
id=4 (not connected, accepting connect from any host)

Error:

Node 2: Forced node shutdown completed. Occured during startphase 0. Initiated by signal 11.

There is ndb_2_error.log in /usr/local/mysql/data of node 2

The content of ndb_2_error.log:

Time: Monday 28 July 2014 - 09:45:01
Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine running this node (Internal error, programming error or missing error message, please report a bug)
Error: 6050
Error data: Job Handling
Error object: /export/home/pb2/build/sb_0-12598553-1404293345.9/rpm/BUILD/mysql-cluster-gpl-7.3.6/mysql-cluster-gpl-7.3.6/storage/ndb/src/kernel/vm/WatchDog.cpp
Program: ndbd
Pid: 1709
Version: mysql-5.6.19

Warning for three nodes: node 2, node 3 and node 4

2014-07-28 09:47:23 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.56.208. Returned error: 'No free node id found for mysqld(API).'
2014-07-28 09:47:23 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.56.207. Returned error: 'No free node id found for mysqld(API).'
2014-07-28 09:47:23 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.56.206. Returned error: 'No free node id found for mysqld(API).'

Thanks for your time.

Best Answer

Without looking in detail to your configuration, I would bet that your problem is the kernel doing an Out of memory (OOM) kill: killing some of your ndbd, etc. processes. Those are usually logged to /var/log/messages, so you can confirm it easily.

128MB of ram, 32-bits

I am not 100% sure if you can run an NDB cluster with those spects, but I can tell you that, in practice, you do not want. Here it is a guide on how much memory consumption you can more or less expect from NDB. Remember that NDB preallocates most of its memory at first execution, ant that it is a memory-focused transactional engine. Increase its available ram and/or reduce some of its configuration parameters.

Related Question