Linux – use tar to extract and immediately compress files from tarball

gziplinuxtar

I have a compressed tarball (eg foo.tar.gz) that I wish to extract all the files from, but the files within the tarball are not compressed. That is to say, the contents of foo.tar.gz are un-compressed txt files.

There is not enough space on my filesystem to extract the files directly, so I wish to extract these files and immediately compress them as they are written to the disk. I can't simply extract the files and then gzip the extracted files because, as I've said, there's not enough space on the filesystem. I would also like to ensure that the original filenames, including their directories are faithfully preserved on disk. So if one of the files in the tarball is /a/b/c/foo.txt, at the end of the process I would like to have /a/b/c/foo.txt.gz

How can I accomplish this?

Best Answer

It won't be fast, especially for a large tarball with lots of files, but in bash you can do this:

tar -tzf tarball.tgz | while IFS= read -r file; do
    tar --no-recursion -xzf tarball.tgz -- "$file"
    gzip -- "$file"
done

The first tar command extracts the names of the files in the tarball, and passes those names to a while read ... loop. The file name is then passed to a second tar command that extracts just that file, which is then compressed before the next file is extracted. The --no-recursion flag is used so trying to extract a directory doesn't extract all the files under that directory, which is what tar would normally do.

You'll still need enough free space to store somewhat more than the original size of the compressed tarball.

Related Solutions

Tar files from another directory without copying them

You want to create a tar file away from the place the files you need to tar reside?

There are many ways to do this.

If it is to be created locally (= on the same machine) :

tar czvf /path/to/destination/newfile.tar.gz  ./SOURCEDIR_OR_FILES

You can add additionnal files or directories to tar at the end of that command.

If it is to be created remotely (ie, you want to create the tar file on a remote host from the one containing the data to be tared):

tar czvf - ./SOURCEDIR_OR_FILES | ssh user@host 'cat > newfile.tar.gz'

The later version is very versatile. For example you can also "duplicate" a directory + subdirs using the same technique:

Duplicate a directory+subdirs to another local directory:

tar cf - ./SOURCEDIR_OR_FILES | ( cd LOCAL_DEST_DIR && tar xvf - )

Duplicate a directory+subdirs to another remote directory:

tar cvf - ./SOURCEDIR_OR_FILES | ssh user@host 'cd REMOTE_DEST_DIR && tar xf - '

Drop the 'v' if you don't need it to display files as they are tar-ed (or untarred): it will then go much faster, but won't say much unless there is an error.

I use "./..." for the source to force tar to store it as a RELATIVE path. In some cases you'll want to add additionnal path information:

For example to tar the crontab files, including the one in /etc, you could do:

cd / ; tar czf all_crons.tgz  ./etc/*cron*  ./var/spool/cron

I use on purpose the relative path: some OLD versions of tar may be dangerous and extract files with their original GLOBAL path, meaning you could do : cd /safedir ; tar xvf sometar and have the files with global names overwrite files at their original path, which is OUTSIDE of /safedir and not underneath it! Very dangerous, and still possible as there are old production servers out there. Better to be used to use relative paths all the time, even if you use a more recent tar.

How to securely extract an untrusted tar file

You don't need the paranoia at all. GNU tar — and in fact any well-written tar program produced in the past 30 years or so — will refuse to extract files in the tarball that begin with a slash or that contain .. elements, by default.

You have to go out of your way to force modern tar programs to extract such potentially-malicious tarballs: both GNU and BSD tar need the -P option to make them disable this protection. See the section Absolute File Names in the GNU tar manual.

The -P flag isn't specified by POSIX,¹ though, so other tar programs may have different ways of coping with this. For example, the Schily Tools' star program uses -/ and -.. to disable these protections.

The only thing you might consider adding to a naïve tar command is a -C flag to force it to extract things in a safe temporary directory, so you don't have to cd there first.

Asides:

Technically, tar isn't specified by POSIX any more at all. They tried to tell the Unix computing world that we should be using pax now instead of tar and cpio, but the computing world largely ignored them.

It's relevant here to note that the POSIX specification for pax doesn't say how it should handle leading slashes or embedded .. elements. There's a nonstandard --insecure flag for BSD pax to suppress protections against embedded .. path elements, but there is apparently no default protection against leading slashes; the BSD pax man page indirectly recommends writing -s substitution rules to deal with the absolute path risk.

That's the sort of thing that happens when a de facto standard remains in active use while the de jure standard is largely ignored.

Best Answer

Related Solutions

Tar files from another directory without copying them

How to securely extract an untrusted tar file

Related Question