Backup – Create a Tar Archive Split into Blocks of a Maximum Size

backuptar

I need to backup a fairly large directory, but I am limited by the size of individual files. I'd like to essentially create a tar.(gz|bz2) archive which is split into 200MB maximum archives. Clonezilla does something similar to this by splitting image backups named like so:

sda1.backup.tar.gz.aa
sda1.backup.tar.gz.ab
sda1.backup.tar.gz.ac

Is there a way I can do this in one command? I understand how to use the split command, but I'd like to not have to create one giant archive, then split it into smaller archives, as this would double the disk space I'd need in order to initially create the archive.

Best Answer

You can pipe tar to the split command:

tar cvzf - dir/ | split --bytes=200MB - sda1.backup.tar.gz.

On some *nix systems (like OS X) you may get the following error:

split: illegal option -- -

In that case try this (note the -b 200m):

tar cvzf - dir/ | split -b 200m - sda1.backup.tar.gz.

If you happen to be trying to split the file to fit on a FAT32 formatted drive, use a byte limit of 4294967295. For example:

tar cvzf - /Applications/Install\ macOS\ Sierra.app/ | \
split -b 4294967295 - /Volumes/UNTITLED/install_macos_sierra.tgz.

When you want to extract the files use the following command (as of @Naftuli Kay commented):

cat sda1.backup.tar.gz.* | tar xzvf -

Related Solutions

Bash – Tar piped to split piped to scp

If the recipient has split, then you can do:

tar -cvf - ~/batch/ | gzip |
  ssh recipient 'cd /destination &&
    split --bytes=1024m - batch.tar.gz.seg'

Tar directory into independent archive files no larger than a certain size

tar can cope with partial archives after splitting. When you try to restore part of such an archive, it will skip over whatever it can't use at the start, and tell you about any partial file at the end; everything in between will be restored properly. You can instruct tar itself to split archives as it creates them, using the tape length options; see Create a tar archive split into blocks of a maximum size for details.

There are utilities which do better than that though, and produce parts of archives which stand alone (as long as the size limit is sufficient to store the largest file in the archive); unfortunately the ones I know of don't meet all your requirements. On most platforms, there's zipsplit which can split zip files, but it only copes with archives up to 2GB in size. On Plan 9, there's tarsplit which splits tarballs, but I'm not sure it can be easily ported to whatever system you're using (I suspect you're not using Plan 9...).

Best Answer

Related Solutions

Bash – Tar piped to split piped to scp

Tar directory into independent archive files no larger than a certain size

Related Question