Mysql – thesqldump: ERROR 2006: MySQL server has gone away when attempting to push data to RDS instance

amazon-rdsMySQLmysqldump

I'm attempting to migrate a MySQL database (10M+ records) from an EC2 instance into an RDS Instance using a script:

#!/bin/bash

mysqldump --databases my_db \
    --compress \
    --order-by-primary \
    --single-transaction \
    -u dba \
    -ppassword | mysql \
        --host=host.us-east-1.rds.amazonaws.com \
        --port=3306 \
        --max_allowed_packet=500M \
        -u dba \
        -ppassword

The script connects and begins to transfer data as expected, but then errors out after about 5 minutes with the following error:

ERROR 2006 (HY000) at line 1406: MySQL server has gone away
mysqldump: Got errno 32 on write

The script works fine for smaller databases, and for the larger database it transfers several million records before the error.

I'm not sure how to get past this issue. I'd love to get some suggestions (parameter adjustments on EC2 or RDS, how to get more detail on error etc.)

Best Answer

The error

mysqldump: Got errno 32 on write

simply means you ran out of space.

The question is: Where is the space issue coming from ???

--order-by-primary

Using this requires creating a temp table in the folder configured in tmpdir

Bigger mysqldumps need a much larger tmpdir when using --order-by-primary

If you get rid of --order-by-primary, you can handle bigger dumps.

You just asked

what are the implications of removing --order-by-primary ?

There are improvements to the dump's creation

IMPROVEMENT #1 : Faster Dump

Every time you dump a table using --order-by-primary, a temp table must be created. That can pause the mysqldump on the source side. Such pausing will never occur anymore

IMPROVEMENT #2 : No bloated indexes

When you load a table in PRIMARY KEY order, it can generate up to 45% additonal wasted space in the indexes due to lopsided page splits in your BTREE indexes.

IMPROVEMENT #3 : AUTO_INCREMENT PRIMARY KEYs

If the majority of your tables has a single integer that is auto incremented and is the PRIMARY KEY, the table is already ordered. No need for the overkill of --order-by-primary

Concerning your next comment

I removed --order-by-primary, got the same error at the same time :(

Please make sure you have extra diskspace on the RDS Instance. Why ?

It is possible for a dump to be bigger when reloaded, especially when exacerbated by having keys ordered.