Mysql – how to dump via thesqldump hugh table into chunks

MySQLmysqldump

i have very hugh table , around 100 M records and 100 GB in a dump file , when i try to restore it it to a different DB i get sql query lost connection , i want to try and dump this table into chunks (something like 10 chinks of 10 GB) where each chink will be in seperate table.
what i managed to optimized so far is this :

mysqldump --skip-triggers --compact --no-create-info --single-transaction --quick --max_allowed_packet 1G  -h {host} -u {user} -P 3306 -p{pwasword} {my_schema} {}> /mnt/datadir/{table_name}.sql

and now the output is that i have 1 file {table_name}.sql i na size of 100 GB i want to get 10 files in sizes of 10 GB each

Best Answer

You can use primary key to specify to dump only id in those range

mysqldump -u root -p db_name table_name --where='id  < 10000' > test1.sql
mysqldump -u root -p db_name table_name --where='id between 10000 and 20000' > test2.sql
.....
 mysqldump -u root -p db_name table_name --where='id between 900000 and 1000000' > test11.sql

Related Solutions

Mysql – Users complain that system runs slow when thesqldump is in progress

There is a known issue in Windows, that when you push a large file to another server all the memory ends up getting allocated to the System cache instead of the user processes. You can look in the Physical Memory (MB) section of task manager to see how much memory is allocated to the system cache.

This can be solved by backing up to a local disk, then having the remote machine pull that file.

MySQL – How to Recover or Restore Corrupted InnoDB Data Files

Check the MySQL error log. It looks like trying to read table 44 is actually crashing the server.

"Lost connection to MySQL server during query" is (or should be) heart-stopping time for a MySQL DBA because it often means that whatever your query just did has actually crashed the server.

The subsequent messages seem to bear this out:

mysqldump: Got error: 2003: Can't connect to MySQL server on 'localhost' (10061) when
trying to connect
Operation failed with exitcode 2

A quick test of mysqldump on Windows 7 against a MySQL server that is not running returns exactly that same error and exits with %ERRORLEVEL% set to 2. The nature of error 10061 can be found with the perror utility.

C:\>perror 10061
Win32 error code 10061: No connection could be made because the target machine actively refused it.

...so, check the MySQL error log. What's likely happening is that you're hitting corruption severe enough that in spite of innodb_force_recovery, the MySQL Server is crashing. I will then usually restart on its own, but this takes time so may not complete before mysqldump gets an IP connection refused on the subsequent connection attempts. This should all show up in the error log.

I assume that when the dump does not yield any error, the table's data is non-corrupted.

Incorrect assumption. If the dump does not yield any error, it could also mean that that innodb_force_recovery allowed the server to retrieve some of the data. A "successful" dump of a table only means it's not corrupted so severely that innodb_force_recovery can't survive it.

1 (SRV_FORCE_IGNORE_CORRUPT)

Let the server run even if it detects a corrupt page. Try to make SELECT * FROM tbl_name jump over corrupt index records and pages, which helps in dumping tables.

^{http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html}

Again, you should see chatter from InnoDB in the MySQL error log.

Final suggestion, don't use workbench. Use the mysql command line client to explore the server schemata and use mysqldump directly from the command line to try to restore individual schemas or individual databases.

update/additional:

how do you recover stored procedures? I get an error because they access corrupt tables

It shouldn't be because the procedures access corrupt tables, because stored procedures and functions aren't validated against their referenced tables in the way views are (drop a table that a view references, and then SHOW CREATE VIEW doesn't work -- the same isn't the case with procs or functions)... MySQL doesn't care whether tables referenced by stored procedures and functions even exist (except at runtime)... however, unlike view definitions, which are stored in individual files, the definitions of procedures and functions are stored in the proc table in the mysql schema, which is a MyISAM table... so I would expect that you'd see error messages referencing that table and CHECK TABLE mysql.proc; and REPAIR TABLE mysql.proc; might get you going again. Note that the version of MySQL Server that you're using for your recovery needs to be the same release series (e.g. 5.5) or the table definition of mysql.proc will be wrong... that shouldn't be the issue here, since if I read the question correctly, the crashed server and the recovery server are both 5.5.31.

If you can SELECT * FROM mysql.proc then you can retrieve the definitions that way if for some reason the SHOW CREATE PROCEDURE statement doesn't work.

And what options for a (partial) rescue are there for the tables that yielded an error in the dumps?

It depends on the nature and scope of the errors encountered. One option, discussed here, involves repeated INSERT ... SELECT ... LIMIT ... OFFSET, extracting blocks of rows from readable pages into a MyISAM table (because you can't write to an InnoDB table while innodb_force_recovery is on). There's also the Percona Data Recovery Tool for InnoDB, though I have never had occasion to use it.

Best Answer

Related Solutions

Mysql – Users complain that system runs slow when thesqldump is in progress

MySQL – How to Recover or Restore Corrupted InnoDB Data Files

Related Question