You should be very happy the mysqldump is two-thirds the size of the data. Why?

You should want to move data and indexes without copying unused space at the same time. What aspects would you have to consider by simply importing a mysqldump?

Aspect #1

InnoDB indexes tend to get very bloaty because Secondary Indexes have keys into the Clustered Index. This always results in a double index lookup. This can be overlooked with large, write-heavy datasets.

Aspect #2

InnoDB tablespaces tend to get very bloaty because of MVCC being created and discarded without an automatic reclaiming of disk space:

• With innodb_file_per_table disabled, the system tablespace file ibdata1 would steadily grow with 0% chance of reclaiming of disk space. You would have to convert InnoDB storage engine to use innodb_file_per_table.
• With innodb_file_per_table enabled, you could go to the individual InnoDB and run of the the following to physically shrink the InnoDB table mydb.mytable:
  • OPTIMIZE TABLE mydb.mytable;
  • ALTER TABLE mydb.mytable ENGINE=InnoDB;

In light of this, you should keep in mind that a mysqldump is a logical representation of the data and indexes. There is no wasted space to be concerned with. You should not have to scp or rsync bloated InnoDB data or logs. Importing a mysqldump into a fresh installation of MySQL should be what you what want.

I have written about this many times:

• MySql - Clean ibdata1 (April 19, 2012)
• How do you remove fragmentation from InnoDB tables? (April 11, 2012)
• Is innodb_file_per_table advisable? (April 01, 2012)
• How to Clean a mysql innodb storage engine (October 29, 2010)

Once you have loaded the test database with the mysqldump, you should run CHECKSUM TABLE on random tables of various sizes. If all tables have the same checksum value, you could certify as good. Make a disklevel backup of the data on the prod server as well.

You can then do this in production

• shutdown mysql
• hose everything (including ibdata1, ib_logfile0, ib_logfile1) in /var/lib/mysql except for /var/lib/mysql/mysql
• Set innodb_data_file_path=ibdata1:10M:autoextend in /etc/my.cnf
• restart mysql (this recreates ibdata1, ib_logfile0, ib_logfile1)
• reload mysqldump

Your InnoDB architecture and data will then be lean and mean.

Here are a few things you could change

• innodb_log_buffer_size : You have 4M. The default is 8M. Try raising it to 8M.
• innodb_log_file_size : You have 128M. Given the following info you supplied
  • you write on avg 93MB per hour to the innodb log files, based on 25% of the stat
  • peak time would be about 372 MB an hour
  • half is 186 MB round off to 192MB
  • set innodb_log_file_size to 192M
• innodb_io_capacity seems a little high, make it 200 or 300 (depending on hardware)
• innodb_read_io_threads and innodb_write_io_threads should be maxed out at 64. Sometimes, less is better in a write-heavy environment. Use 8 or 16 in such cases.

To change innodb_log_file_size to 192M, add this to /etc/my.cnf

[mysqld]
innodb_log_buffer_size=8M
innodb_log_file_size=192M
innodb_open_files=300
innodb_read_io_threads=64
innodb_write_io_threads=64

then do the following

mysql -u... -p... -e"FLUSH TABLES;"
service mysql stop
mv /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile0000
mv /var/lib/mysql/ib_logfile1 /var/lib/mysql/ib_logfile0001
service mysql start

Test this during slow times and see if this improves things

UPDATE 2012-10-19 12:31 EDT

To put your situation into perspective think about what the purposes of ibdata1 and ib_logfile0/1 are. Rather than rewriting everything, please read my most recent post about ibdata1 and InnoDB Log Files : What exactly are iblog files in mysql

Once you have read my other post, now consider this: What would increase I/O on these files?

• Updating Secondary Indexes in ibdata1's Insert Buffer
• Cleaning up the Rollback Segments/Undo Space

IMHO I would increase the InnoDB Buffer Pool from 384M to 1G.

I would also recommend upgrading RAM to 8G