Linux tar listed-incremental untrustworthy

backuptar

I'm looking for some way to troubleshoot incremental backups with tar on linux.

I've got huge datasets (about 3TB) that need backing up to tape.
For this I use the linux tar command and mt / mtx on LTO4 device.

As the backup is very very very very slow (I suppose I'll have to put that in another question) there is no choice for me but to backup with incrementals to avoid it running during production time.

basically, it goes like this:

  1. A full tar is made: (with level 0)

    tar --new-volume-script=changetape.sh \
  --exclude=.zfs \
  --listed-incremental=file.index \
  --label=full_DS01 \
  -cvf /dev/nst0 \
  /mnt/storage/sharename >>filelisting.log 2>>errlisting.errlog

  2. Incremental tar per day:

    mtx -f /dev/sg1 load X #(load correct tape)
mt -f /dev/nst0 eod #(forward to end of data to write a new incremental tar)
tar --exclude=.zfs \
  --listed-incremental=file.index \
  --label=incremental_DS01 \
  -czvf /dev/nst0 \
  /mnt/storage/sharename >>filelisting.log 2>>errlisting.errlog

  3. Repeat incremental tars.
    using the same process as in 2 so I always use the level 0 listed-incremental wich is adapted every time.
    This should mean that I always backup the files that have changed since my previous incrmental update. (as opposite to differential where you always compare with the full update)

Problem:

After a few iterations, the incrementals start to fail, meaning: the backup runs but it seems to think that ALL the files have been changed and it does a full backup in fact.

This happened in different datasets.

How can I troubleshoot this? And how is this possible? All seems to run well and then it thinks that all files have changed?

for clarity:

  1. the dataset is a readonly mountpoint
  2. the folder structure is the same as in the full backup
  3. the upper folder name has not changed

How can I solve this issue?

Best Answer

The incremental features in gtar have a conceptional problem and cannot work correctly in all cases. When I tried to check the incremental features on star in 2004, gtar failed early and could not be tested at all.

Did you try star's incremental backups?

Freebsd uses star for zfs backups and this works fine since freebsd fixed some old kernelbugs.

gtar has several issues with incremental backups:

  • Important larger meta data is not inside the backup archives but rather in a separate file that is created as a result of using the gtar vendor specific option -g. In case of a complete system crash, it is very likely that this file is lost (at least in it's most recent version). The existence of this file separated from the backups is however not the real problem. The real problem is that this file cannot track renames.

  • The backup archives from gtar do not contain any information about renamed files. There is just a trace that allows to assume that a file name that is not mentioned in the new archive but present on the current filesytem must have disappeared and thus removed.

  • gtar needs to archive a complete directory tree with all content if a directory at top level was renamed. This makes gtar incrementals huge and this may easily overflow the target filesystem during a restore operation.

  • gtar has problems with non-directory files that change the file type between two incrementals.

  • gtar is unable to handle a directory rename followed by creating a new directory of the previous name.

  • I thus was unable to run the initial hand crafted test with gtar I created in September 2004 for star.

Star on the other side uses a basic algorithm that is known to be correct since 35 years, as it was developed for ufsdump/ufsrestore in 1981. Star's incremental backups work the following way:

  • There is a dumpdates files that records no more than time stamp, level and filesystem for every full or incremental dump.

  • Each tar archive contains all meta data that is needed to track all changes in a filesystem between two backups. In addition to all changed files, star archives a list of filenames for all of the changed directories together with the related inode numbers. This permits to track all file removals and all file renames at any time for any incremental backup.

  • When star restores a filesystem from backups, it creates a data base with entries for all extracted objects that contains the file name, the original inode number from the backuped filesystem and new inode number that is used on the extracted filesystem. This allows star to track all changes between two incrementals.

  • When you rename a directory at the top level of a filesystem, star just needs to backup the root directory and the renamed directory with it's meta data, but not a single file from the content of that directory.

  • Star was used to run daily cumulative incremental backups and restores on the Berlios filesystems (typically 2-10GB of changed data per day) during 10 years and never failed.

Before you start to use gtar for your backups,, you may like to run similar restore tests first.

BTW: here is a script that verifies how GNU tar fails with incremental restores: Is it possible to use tar for full system backups?

Related Question