MySQL backup of incremental changesets

backupMySQLreplication

With a MySQL database, is there a way of logging and/or backing up incremental changesets?

We have a number of large databases (100-200 GB), for which we take daily backups using mysqldump. These backups are a snapshot of the entire database, which can be restored as whole. However, taking/restoring from these backups is not a quick process.

As such, I'd like to be able to take backups of incremental changes – i.e. a backup of what's changed in a given day. Then, assuming a database was in a state consistent with a previous snapshot, we could restore the set of incremental changes to get it to the same/current state.

To some extent, this could be done by operating on mysqldump files – i.e. comparing today's snapshot dump with the previous day's snapshot dump to create a .sql backup of commands to run to convert from the previous snapshot state to today's snapshot state. However, I'd rather not write this myself if there's already a better way of doing this.

Also, I'm conscious that, to some degree this is analogous to MySQL master-slave replication, with the exceptions that the changesets are grouped by time period (i.e. a 24hr window) and not automatically read into another database / server. Furthermore, the binary logs are for the entire database server, not for individual databases.

Additional Comments:

With some further research on this, I've also come across the incremental backup functions in MySQL Enterprise, but we're not using the enterprise version.

Best Answer

It is probably your only option to use the Percona Xtrabackup tool. Fortunately, this is a free tool that has very good documentation.

This tool takes backups in a physical way, rather than logical backups that mysqldump produces. This means that restore time is simply the amount of time required to copy the backup files into the MySQL datadir and not the time to process all the SQL statements in a mysqldump backup. Backup time is generally faster as well, and allows for the possibility for incremental backup.

Related Solutions

MySQL incremental backups per database (not global)

You can read binlogs with the mysqlbinlog tool, and produce a stream of logical changes, like a mysqldump backup. The mysqlbinlog has an option --database=dbname so you can extract only changes pertaining to the named database.

Read more about using this option here: http://dev.mysql.com/doc/refman/5.6/en/mysqlbinlog.html#option_mysqlbinlog_database

Two caveats:

This assumes that you have a continuous set of binary logs back either to the last full backup or to the last incremental dump.
It's not really a filter for changes to dbname. It's a filter for all binlog events applied to any database while dbname was selected as the default database. So if you have an event where someone updated dbname2.tablename while dbname was the default, then the dump will include that change to dbname2.

This makes this method unsafe to use this as a per-database incremental backup method, unless you have 100% certainty that no such transactions occurred.

P.S.: Your comments about LSN are a bit incorrect. Mysqldump knows nothing about the LSN, which is a feature of the InnoDB redo log. Mysqldump does examine the binary log position, and can output that position if you use the option --master-data.

SQL Server Backup – Incremental Backup of SQL Server 2008 R2

I think what you are doing is you are writing your full and differential backups to the same physical file, which just appends each differential (and perhaps even additional full backups) to the same physical file container.

What you should be doing instead is using a different physical file name each time. I like embedding the date (and time, if we're talking about backups that are taken more often, like log backups) into the filename, like so:

BACKUP DATABASE foo TO DISK = N'c:\backups\foo_full_20150526.bak'
  WITH INIT;

BACKUP DATABASE foo TO DISK = N'c:\backups\foo_diff_20150527.diff'
  WITH INIT, DIFFERENTIAL;

And yes, you can do this dynamically:

DECLARE @path SYSNAME = N'c:\backups\foo_diff_'
  + CONVERT(CHAR(8), GETDATE(), 112) + N'.diff';

BACKUP DATABASE foo TO DISK = @path WITH INIT, DIFFERENTIAL;

Best Answer

Related Solutions

MySQL incremental backups per database (not global)

SQL Server Backup – Incremental Backup of SQL Server 2008 R2

Related Question