Mysql – ndbcluster: Error: 1114, The table is full, when using table

mysql-5.6ndbcluster

I have searched for a while about this error. I have a MySQL cluster running with 2 data nodes. The cluster houses a few other databases. I have a dump of a database in –tab format from another server. I have modified the SQL to use ndbcluster as the table engine and can import the SQL successfully. When I try to import data to the tables using mysqlimport, a few tables will populate but then I get the following message:

mysqlimport: Error: 1114, The table 'doc_archive' is full, when using table: doc_archive

Searching around for this error would suggest that I am out of memory on my data nodes, however that is not the case as you can see:

ndb_mgm> all report memory
Node 31: Data usage is 45%(179865 32K pages of total 393216)
Node 31: Index usage is 67%(88761 8K pages of total 131200)
Node 32: Data usage is 45%(179863 32K pages of total 393216)
Node 32: Index usage is 67%(88761 8K pages of total 131200)

Importing the database up until it is "full" only increases the data usage on each data node by 2%, and the index usage only goes up 3%.

Here is a copy of my config.ini from the management node:

[ndbd default]
# Options affecting ndbd processes on all data nodes:

NoOfReplicas=2    # Number of replicas

DataMemory=12G    # How much memory to allocate for data storage

IndexMemory=1G   # How much memory to allocate for index storage
            # For DataMemory and IndexMemory, we have used the
            # default values. Since the "world" database takes up
            # only about 500KB, this should be more than enough for
            # this example Cluster setup.

MaxNoOfOrderedIndexes=8192
MaxNoOfUniqueHashIndexes=512
MaxNoOfAttributes=100000
MaxNoOfExecutionThreads=8
MaxNoOfConcurrentOperations=256000
MaxNoOfTables=2048
MaxBufferedEpochs=100000
TimeBetweenEpochsTimeout=32000
TransactionDeadlockDetectionTimeout=1000000
TransactionInactiveTimeout=10000000
MaxNoOfTriggers=3072 # Default is 768

datadir=/var/lib/mysql-cluster   # Directory for this data node's data files
serverport=1187

NoOfFragmentLogFiles=1125
RedoBuffer=32M

[ndb_mgmd]
# Management process options:
nodeid=30
hostname=192.168.2.30
datadir=/var/lib/mysql-cluster

[ndbd]
nodeid=31
hostname=192.168.2.31

[ndbd]
nodeid=32
hostname=192.168.2.32

[mysqld]
nodeid=33
hostname=192.168.2.33

[mysqld]
nodeid=34
hostname=192.168.2.34

Where else should I be looking? There is plenty of disk space and memory to spare.

Best Answer

I had a similar issue. Except for me it was complaining about a large table being full, even though all nodes were using 0% data and index memory.

What fixed it for me was setting MaxNoOfExecutionThreads=4 (a value of 8 caused the table to be full).

I don't know why this worked, but it did. I wish the ndbmtd executable would provide more descriptive output than just (the table is full), such as describing exactly which internal memory or resource is full, because neither my memory nor disk were anywhere being full.

Hope this helps!

Related Question