Mysql – Database Design – Creating Multiple databases to avoid the headache of limit on table size

innodbMySQLmysql-5.5windows

I need to design database to store information contained in millions of log files generated by devices on trial on network.

My first approach was storing all the information in tables contained in a single database. But this approach seems to fail as the data to be stored is too large. So, I thought of creating separate database for each device and storing each device log files in separate database. My question is if this approach can spare me of the headache involved with table size limit. Below are given my platform specification

  • Operating System – Windows 7
  • File System – NTFS
  • Storage Engine – InnoDB
  • Table Type – InnoDB

I want to build my database design in a way so that I need not to apply shrink of tables procedures in future

Best Answer

In my answer to your previous question, I mentioned how the size of an individual table is 2TB when you use innodb_file_per_table. There may be a way to surpass the table limit by doing just the the opposite, leaving innodb_file_per_table disabled.

With innodb_file_per_table disabled, there are three ways you can go about this:

TECHNIQUE #1 : Use multiple system tablespaces in ext3

You could chain InnoDB tablespaces 2TB at a time. Start off with this

innodb_data_file_path=ibdata1:10M:autoextend:max:2048G

When ibdata1 is about to hit 2TB, add another ibdata file

innodb_data_file_path=ibdata1:2048G:ibdata2:10M:autoextend:max:2048G

When ibdata2 is about to hit 2TB, add another ibdata file

innodb_data_file_path=ibdata:2048G:ibdata2:2048G:ibdata3:10M:autoextend:max:2048G

and so on...

UPDATE 2013-06-01 20:45 EDT

I just discovered something with reference to setting the max filesize for a system tablespace. Here is the situation:

Client has the following InnoDB files

[root@l*****]# ls -l ibd*
-rw-rw---- 1 s-em7-mysql s-em7-mysql     362807296 Jun  2 00:15 ibdata1
-rw-rw---- 1 s-em7-mysql s-em7-mysql 2196875759616 Jun  2 00:15 ibdata2

and has this setting

innodb_data_file_path=ibdata1:346M;ibdata2:500M:autoextend:max:10240000M

MySQL was reporting table full errors.

I tried creating a new file ibdata3 with

innodb_data_file_path=ibdata1:346M;ibdata2:2048G;ibdata3:10M:autoextend

That value did not work

I tried using new sizes for the max value

  • 2046G
  • 2045G
  • 2046000M

After trying for 15 min with different combinations, none of these values worked. Looks like mysqld was never going to startup again. I decided to try the exact number of bytes for the filesize of ibdata2:

innodb_data_file_path=ibdata1:346M;ibdata2:2196875759616;ibdata3:10M:autoextend

When I started mysql, LO AND BEHOLD, MySQL was up.

  • ibdata3 was created at 10M
  • ibdata3 was soon growing in 8M chunks
  • Client was happy once again

Moral of the story : At times, you may need to use the exact filesize for a system tablespace in innodb_data_file_path

TECHNIQUE #2 : RAW DISK PARTITION

According to MySQL 5.0 Certification Study Guide, Page 428

Any raw partitions in the configuration must exist but must have the modifier newraw listed after the size of the file specification. newraw tells InnoDB to initialize the partition when the server starts up. New partitions are treated as read-only after initialization. After InnoDB initializes the tablespace, stop the server change newraw to raw in the partition specfication, and restart the server. For example, to use a 10GB Unix partition named /dev/hdc6, begin with a configuration like this:

[mysqld]
innodb_data_home_dir=
innodb_data_file_path=/dev/hdc6:10Gnewraw

Start the server and let InnoDB initialize the tablespace. Then stop the server and change the configuration from newraw to raw:

[mysqld]
innodb_data_home_dir=
innodb_data_file_path=/dev/hdc6:10Graw

After changing the configuration, restart the server.

Just create a RAID10 Disk with whatever size you need. Just don't put any filesystem on it.

TECHNIQUE #3 : Use ext4 with TECHNIQUE #1

You could chain InnoDB tablespaces 16TB at a time. Start off with this

innodb_data_file_path=ibdata1:10M:autoextend:max:16384G

When ibdata1 is about to hit 16TB, add another ibdata file

innodb_data_file_path=ibdata1:16384G:ibdata2:10M:autoextend:max:16384G

When ibdata2 is about to hit 16TB, add another ibdata file

innodb_data_file_path=ibdata:16384G:ibdata2:16384G:ibdata3:10M:autoextend:max:16384G

And so on...