I have some directories with over 100Gb of data. I'm trying to archive them into smaller volumes i.e. 10Gb each that are independent / standalone.
The problem is if I use tar
+ split
, it results in multiple tar parts that are not independent. I cannot just extract files from one of the parts, unless I cat / combine all back into a single large file first.
I've also tried using tar -c -L1000M ...
to split volumes, but that doesn't work either and there's a problem with long filenames getting truncated.
Tried star
as well, but seems like its split volumes are not independent either; while 7zip does not preserve permissions in unix.
The reason I wish to have independent split archives is for safety purposes, in case one of the split files are corrupted, I can still retrieve data from the other archives. It is also much faster if I wish to only extract specific files/folders, without needing to combine all the archives back into a single large volume.
How best do I achieve this? Thank you.
SOLUTION FOUND
I have found a solution using tar, as suggested by @Haxiel's answer. The answer has been posted below.
Note that there may still be a file or two that lost if it crosses the boundary of a volume and you don't have the next volume available, but at least the separate volumes could be independently extracted even if the other parts are missing.
Best Answer
This is not a perfect solution, but GNU
tar
's multi-volume archives seem to be close to what you're looking for. This option is already mentioned in your question, but I would like to add a reference from the GNUtar
manual that clarifies why this is a possible option:With this definition, the only files that would be a problem are the ones that are split across the size boundary. Files that are fully contained within a single volume could be treated as independent of the other volumes.
For each volume, it is possible to identify the split files using the
-v
option.The files that are fully contained can be extracted as you would with a single archive.
tar
seems to complain about the split file being incomplete, but it is able to extract the complete files without any problems.The split files can also be extracted as a single unit from multiple volumes by using the
-M
option, which will prompt you to provide the name of the next volume. The usage is documented here. Instead, if you prefer to concatenate the volumes to a single archive, you can consider the tarcat utility as well.