Maximum number of files you can store in a tar archive under Solaris 11

gnulimitsolaristarzfs

I need to store a large number of files (> 100,000) in a tar archive and have run into the error: /bin/sh: /bin/tar: cannot execute [Arg list too long].

Are there limits with UNIX Solaris tar or GNU tar? Or, is this a ZFS issue? I am running Solaris 11.3 with ZFS.

Best Answer

This is not a limitation of tar. You are exceeding the maximum size of arguments that can be passed to a program. If the list of file names is available in a file, then you can use GNU tar's -T option:

tar -c -f output.tar -T filelist.txt

Tar can also receive the file list from standard input by using - as the file name. For example, if you want to create an archive containing all files and subdirectories of mydir, you can pass the output of find to tar:

find mydir | tar -c -f output.tar -T -

Related Solutions

Shell – How to hook into Tar with BASH

You want to use the --to-command option of tar:

tar xf tarfile.tar --to-command='sh -c "mkdir -p $(dirname $TAR_FILENAME) && base64 -d > $TAR_FILENAME"'

This is described in the manual here.

backup-specs is part of the backup system the manual claims is distributed with tar, however my system lacks this and I don't think actually uses it, at least not anymore.

How to create a gnu tarball that can be extracted by solaris tar

GNU tar formats suggests tar -c --format=posix or tar -c --format=pax is the only way to create a portable tar file containing path names longer than 255 characters.

The Solaris tar man page doesn't mention supporting that format, but the Solaris pax man page does.

So first, I'd try to create the archive on your Cygwin (or Linux) system using tar -c --format=posix -f dir.tar dir, and to extract it on your Solaris system using pax -r -x pax -f dir.tar, or likely just pax -r -f dir.tar.

Alternatively, if you have cpio installed on both systems, it seems like the crc format supports 1,023 character paths on both systems. That would look something like find -L dir | cpio -o -H crc > dir.cpio to create and cpio -i -d < dir.cpio to extract it.

Finally, you could try adding the -i flag to ignore directory checksum errors when extracting on Solaris, e.g. tar -x -i -f dir.tar. Checksumming problems says this can be needed if any of the file names in the tar archive are non-ASCII.

Best Answer

Related Solutions

Shell – How to hook into Tar with BASH

How to create a gnu tarball that can be extracted by solaris tar

Related Question