See rdup-simple (from rdup). You said you wanted compression, but in case you change your mind, I strongly recommend rsnapshot.
By the way, if two hardlinks point to a file, it's the same file. You can't compress only one of the hardlinks since it's the same underlying data in the file-system.
Since tar files are a streaming format — you can cat two of them together and get an almost-correct result — you don't need to extract them to disk at all to do this. You can decompress (only) the files, concatenate them together, and recompress that stream:
xzcat *.tar.xz | xz -c > combined.tar.xz
combined.tar.xz will be a compressed tarball of all the files in the component tarballs that is only slightly corrupt. To extract, you'll have to use the --ignore-zeros option (in GNU tar), because the archives do have an "end-of-file" marker that will appear in the middle of the result. Other than that, though, everything will work correctly.
GNU tar also supports a --concatenate mode for producing combined archives. That has the same limitations as above — you must use --ignore-zeros to extract — but it doesn't work with compressed archives. You can build something up to trick it into working using process substitution, but it's a hassle and even more fragile.
If there are files that appear more than once in different tar files, this won't work properly, but you've got that problem regardless. Otherwise this will give you what you want — piping the output through xz is how tar compresses its output anyway.
If archives that only work with a particular tar implementation aren't adequate for your purposes, appending to the archive with r is your friend:
tar cJf combined.tar.xz dummy-file
for x in db-*.tar.xz
do
mkdir tmp
pushd tmp
tar xJf "../$x"
tar rJf ../combined.tar.xz .
popd
rm -r tmp
done
This only ever extracts a single archive at a time, so the working space is limited to the size of a single archive's contents. The compression is streaming just like it would have been had you made the final archive all at once, so it will be as good as it ever could have been. You do a lot of excess decompression and recompression that will make this slower than the cat versions, but the resulting archive will work anywhere without any special support.
Note that — depending on what exactly you want — just adding the uncompressed tar files themselves to an archive might suffice. They will compress (almost) exactly as well as their contents in a single file, and it will reduce the compression overhead for each file. This would look something like:
tar cJf combined.tar.xz dummy-file
for x in db-*.tar.xz
do
xz -dk "$x"
tar rJf combined.tar.xz "${x%.xz}"
rm -f "${x%.xz}"
done
This is slightly less efficient in terms of the final compressed size because there are extra tar headers in the stream, but saves some time on extracting and re-adding all the files as files. You'd end up with combined.tar.xz containing many (uncompressed) db-*.tar files.
Best Answer
You can use
ncduitself!This shows the uncompressed sizes of the files.
In the case you say you care about, namely many uncompressible files, it should reflect what you need pretty well:
To make the file sizes accessible to
ncdu, they need to be in a file system. So we need to mount the archive as a file system.We use a fuse user-space filesystem implementation,
archivemount:Install the fuse file system:
mkdira directory,mountthe archive to it,cdinto it, and runncdu:Now you can use
ncdujust normally:Now, what you are really interested in is the compressed size of the files, not uncompressed: You want to see which files take up the most space in the actual archive.
Strictly speaking, that's not possible because the archive is compressed as a whole. An individual file has no "compressed size".
So the compressed size of individual files can only be approximated.
One approximation would be the size of individually compressed files.
Another would be a fraction of the compressed size assuming all files compress by the same ratio. There are certainly other ways.
The first seems to be ok. To implement it, there is no way around actually unpacking and recompressing the individual files, so I see no reason to not just do that, unpack to the filesystem, and use ncdu on the files.