MySQL Cluster 7.3 – NDB engine – Could not connect to storage engine

mysql-clusterndbcluster

Setup

I have installed three Oracle Linux 7.3. 2 for data nodes, 1 for management and SQL API node. The machines can ping each other, SSH working, "etc/hosts" files are properly filled:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.16.1.101    mysqld_mgmt_1
172.16.1.103    mysql_data_1
172.16.1.104    mysql_data_2

The management node has the following config in the "/var/lib/mysql-cluster/config.ini" file:

# Options affecting ndbd processes on all data nodes:
[NDBD DEFAULT]
NoOfReplicas=2                      # Number of replicas

[ndb_mgmd]
# Management process options:
hostname=mysqld_mgmt_1              # Hostname of the manager
datadir=/var/lib/mysql-cluster      # Directory for the log files

[ndbd]
hostname=mysql_data_1               # Hostname of the first data node
datadir=/var/lib/mysql-cluster      # Remote directory for the data files
ServerPort=50501

[ndbd]
hostname=mysql_data_2               # Hostname of the second data node
datadir=/var/lib/mysql-cluster      # Remote directory for the data files
ServerPort=50502

[mysqld]
# SQL node options:
hostname=mysqld_mgmt_1              # In our case the MySQL server/client is on the same Droplet as the cluster manager

The SQL API node (same with the management) has the following config in the "/etc/my.cnf" file:

[mysqld]
ndbcluster                      # Run NDB storage engine
ndb-connectstring=172.16.1.101

[mysql_cluster]
ndb-connectstring=mysqld_mgmt_1 # IP address for server management node

The two data nodes has the same config file ("/etc/my.cnf") on each node:

[mysqld]
ndbcluster
ndb-connectstring=172.16.1.101

[mysql_cluster]
ndb-connectstring=172.16.1.101     # IP address of Management Node 1

Starting the cluster:

1. management:          ndb_mgmd -f /var/lib/mysql-cluster/config.ini
                        or with a new config to erase the cache:
                        ndb_mgmd --initial --config-file=/var/lib/mysql-cluster/config.ini
2. data node 1 and 2:   ndbd
3. mysql server:        service mysql start 

Status of the cluster:

[root@mysqld-mgmt-1 ~]# ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: mysqld_mgmt_1:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @172.16.1.103  (mysql-5.6.37 ndb-7.3.18, Nodegroup: 0, *)
id=3    @172.16.1.104  (mysql-5.6.37 ndb-7.3.18, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @172.16.1.101  (mysql-5.6.37 ndb-7.3.18)

[mysqld(API)]   1 node(s)
id=4 (not connected, accepting connect from 172.16.1.101)

I can login to MySQL from command line on the server node (and from MySQL Workbench as well from another VM):

[root@mysqld-mgmt-1 ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.37-ndb-7.3.18-cluster-gpl MySQL Cluster Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show engines;
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine             | Support | Comment                                                        | Transactions | XA   | Savepoints |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| ndbcluster         | YES     | Clustered, fault-tolerant tables                               | YES          | NO   | NO         |
| MRG_MYISAM         | YES     | Collection of identical MyISAM tables                          | NO           | NO   | NO         |
| MyISAM             | YES     | MyISAM storage engine                                          | NO           | NO   | NO         |
| ndbinfo            | YES     | MySQL Cluster system information storage engine                | NO           | NO   | NO         |
| CSV                | YES     | CSV storage engine                                             | NO           | NO   | NO         |
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears) | NO           | NO   | NO         |
| ARCHIVE            | YES     | Archive storage engine                                         | NO           | NO   | NO         |
| FEDERATED          | NO      | Federated MySQL storage engine                                 | NULL         | NULL | NULL       |
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                             | NO           | NO   | NO         |
| InnoDB             | DEFAULT | Supports transactions, row-level locking, and foreign keys     | YES          | YES  | YES        |
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables      | NO           | NO   | NO         |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
11 rows in set (0.01 sec)

Testing

I created a sample table without "engine" parameter, just to try things. I wanted to alter table to use ndb engine, but I have the following problem:

mysql> use clusterdb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select * from simples;
+----+
| id |
+----+
|  1 |
|  2 |
|  3 |
|  4 |
+----+
4 rows in set (0.00 sec)

mysql> alter table simples engine=ndb;
ERROR 157 (HY000): Could not connect to storage engine

mysql> show warnings;
+---------+------+---------------------------------------------------------------------------------+
| Level   | Code | Message                                                                         |
+---------+------+---------------------------------------------------------------------------------+
| Warning | 1296 | Got error 4009 'Cluster Failure' from NDB. Could not acquire global schema lock |
| Error   |  157 | Could not connect to storage engine                                             |
| Error   | 1499 | Too many partitions (including subpartitions) were defined                      |
+---------+------+---------------------------------------------------------------------------------+
3 rows in set (0.00 sec)

mysql> show global status like 'ndb_number_of%';
+--------------------------------+-------+
| Variable_name                  | Value |
+--------------------------------+-------+
| Ndb_number_of_data_nodes       | 2     |
| Ndb_number_of_ready_data_nodes | 0     |
+--------------------------------+-------+
2 rows in set (0.00 sec)

Also, creating table with "engine=ndb" parameter also fails and give the same error. I see the db working with InnoDB engine fine, but I need a fault tolerant (like) database.

Best Answer

OK, so I managed to resolve the problem.

Since it's an Oracle Linux 7.3 there is a so called Security-Enhanced Linux (SELinux) option. I set this to "disabled" and restarted the node:

[root@mysqld-mgmt-1 ~]# vi /etc/selinux/config

SELINUX=disabled

[root@mysqld-mgmt-1 ~]# shutdown -r now

After the node restart, I started the services again and used the ndb_mgm to list the nodes:

ndb_mgm> show
Connected to Management Server at: mysqld_mgmt_1:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @172.16.1.103  (mysql-5.6.37 ndb-7.3.18, Nodegroup: 0, *)
id=3    @172.16.1.104  (mysql-5.6.37 ndb-7.3.18, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @172.16.1.101  (mysql-5.6.37 ndb-7.3.18)

[mysqld(API)]   1 node(s)
id=4    @172.16.1.101  (mysql-5.6.37 ndb-7.3.18)
Related Question