Mysql – Adding Memcached plugin to Mysql Cluster Fails to start

I was able to setup a new mysql cluster, with total 6 VMs. New requirement is to add memcache plugin to mysql cluster, memcache failing to start with this error message WARNING -- Failed to allocate nodeid for API at 192.168.1.100. Returned error: 'No free node id found for mysqld(API).' .

Here is my existing setup.

192.168.1.100 Management1
192.168.1.101 Management2
192.168.1.102 datanode1
192.168.1.103 datanode2
192.168.1.104 sqlnode1
192.168.1.105 sqlnode2

I am trying to run memcached on existing sql nodes

sqlnode1 & sqlnode2

Here is the configuration on management nodes Management1, Management2

[ndb_mgmd default]
DataDir=/mysql/data

[ndbd default]
NoOfReplicas=2
LockPagesInMainMemory=1
DataMemory=25G
IndexMemory=5G
NoOfFragmentLogFiles=300
DataDir=/mysql/data
MaxNoOfConcurrentOperations=1000000
SchedulerSpinTimer=400
SchedulerExecutionTimer=100
RealTimeScheduler=1
MaxNoOfTables=1024
MaxNoOfOrderedIndexes=256

[ndb_mgmd]
HostName=192.168.1.100
NodeId=51

[ndb_mgmd]
HostName=192.168.1.101
NodeId=52

[ndbd]
HostName=192.168.1.102
NodeId=53

[ndbd]
HostName=192.168.1.103
NodeId=54

[mysqld]
HostName=192.168.1.104
NodeId=55

[mysqld]
HostName=192.168.1.105
NodeId=56

[mysqld]
HostName=192.168.1.104
NodeId=57

[mysqld]
HostName=192.168.1.105
NodeId=58

Here is configuration file /etc/my.conf on data & sql nodes datenode1, datanode2, sqlnode1, sqlnode2.

[mysqld]
ndbcluster
ndb-connectstring=192.168.1.100,192.168.1.101
datadir=/mysql/data
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[mysql_cluster]
ndb-connectstring=192.168.1.100,192.168.1.101

current status of cluster.

#ndb_mgm -e show
Connected to Management Server at: 192.168.1.100:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=53    @192.168.1.102  (mysql-5.7.19 ndb-7.5.7, Nodegroup: 0)
id=54    @192.168.1.103 (mysql-5.7.19 ndb-7.5.7, Nodegroup: 0, *)

[ndb_mgmd(MGM)] 2 node(s)
id=51    @192.168.1.100  (mysql-5.7.19 ndb-7.5.7)
id=52    @192.168.1.101  (mysql-5.7.19 ndb-7.5.7)

[mysqld(API)]   4 node(s)
id=55    @192.168.1.104  (mysql-5.7.19 ndb-7.5.7)
id=56    @192.168.1.105  (mysql-5.7.19 ndb-7.5.7)
id=57 (not connected, accepting connect from 192.168.1.104)
id=58 (not connected, accepting connect from 192.168.1.105)

#

here is the error when I try to add memcache plugin to mysql cluster.
created ndbmemcache configuration in database, didnt notice any error.

# mysql -u root -p < /usr/share/mysql/memcache-api/ndb_memcache_metadata.sql
Enter password:
#

when I start memcached server, it fails with error below.

#/usr/sbin/memcached -E /usr/lib64/mysql/ndb_engine.so -u root -e "connectstring=192.168.1.100:1186,192.168.1.101:1186;debug=true" -scheduler=S:c1,f0,t1
main --> ndb_initialize()
main --> connect_to_primary_cluster()
27-Oct-2017 15:35:04 UTC NDB Memcache 5.7.19-ndb-7.5.7 started [NDB 7.5.7; MySQL 5.7.19]
Contacting primary management server (192.168.1.100:1186,192.168.1.101:1186) ...
main --> ClusterConnectionPool::connect()
FAILED.
Could not connect to NDB.  Shutting down.
main --> ndb_destroy()
Failed to initialize instance. Error code: 255
#

cluster management logs ndb_51_cluster.log show this Warning message.

2017-10-27 15:33:44 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017-10-27 15:35:06 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017-10-27 15:35:09 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017-10-27 15:35:12 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017-10-27 15:35:15 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017-10-27 15:35:18 [MgmtSrvr] WARNING  -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).

'

Can someone please suggest how can I fix this issue, I already have additional sql client nodes, but not able to start memcache process.

Thanks,