tar – What Does the -f Parameter Do in the tar Command

If you want to exclude an entire directory, your pattern should match that directory, not files within it. Use --exclude=/data/sub1 instead of --exclude='/data/sub1/*'

Be careful with quoting the patterns to protect them from shell expansion.

See this example, with trouble in the final invocation:

$ for i in 0 1 2; do mkdir -p /tmp/data/sub$i; echo foo > /tmp/data/sub$i/foo; done
$ find /tmp/data
/tmp/data
/tmp/data/sub2
/tmp/data/sub2/foo
/tmp/data/sub0
/tmp/data/sub0/foo
/tmp/data/sub1
/tmp/data/sub1/foo
$ tar -zvcf /tmp/_data.tar /tmp/data --exclude='/tmp/data/sub[1-2]'
tar: Removing leading `/' from member names
/tmp/data/
/tmp/data/sub0/
/tmp/data/sub0/foo
$ tar -zvcf /tmp/_data.tar /tmp/data --exclude=/tmp/data/sub[1-2]
tar: Removing leading `/' from member names
/tmp/data/
/tmp/data/sub0/
/tmp/data/sub0/foo
$ echo tar -zvcf /tmp/_data.tar /tmp/data --exclude=/tmp/data/sub[1-2]
tar -zvcf /tmp/_data.tar /tmp/data --exclude=/tmp/data/sub[1-2]
$ tar -zvcf /tmp/_data.tar /tmp/data --exclude /tmp/data/sub[1-2]
tar: Removing leading `/' from member names
/tmp/data/
/tmp/data/sub2/
/tmp/data/sub2/foo
/tmp/data/sub0/
/tmp/data/sub0/foo
/tmp/data/sub2/
tar: Removing leading `/' from hard link targets
/tmp/data/sub2/foo
$ echo tar -zvcf /tmp/_data.tar /tmp/data --exclude /tmp/data/sub[1-2]
tar -zvcf /tmp/_data.tar /tmp/data --exclude /tmp/data/sub1 /tmp/data/sub2

It seems tar wants to know all the file names upfront. So it is less on-the-fly and more after-the-fly. cpio does not seem to have that problem:

| cpio -vo 2>&1 > >(gzip > /tmp/arc.cpio.gz) | parallel rm