How to execute more than one tar commands for parallel execution

solaristar

I want to tar four directories which contain large number of small files using a shell script. Because of this script takes too long to execute so I want to make these 4 tar commands run in parallel using shell script hoping I can make better use of resources available.

Commands that I am currently using:

tar cf - /ebs/uat/uatappl | gzip -c > /ebs/backup/uatappl.tar.gz
tar cf - /ebs/uat/uatcomn | gzip -c > /ebs/backup/uatcomn.tar.gz
tar cf - /ebs/uat/uatora | gzip -c > /ebs/backup/uatora.tar.gz
tar cf - /ebs/uat/uatdata | gzip -c > /ebs/backup/uatdata.tar.gz

Best Answer

You can put all the tars in background like this:

tar cf - /ebs/uat/uatappl | gzip -c > /ebs/backup/uatappl.tar.gz &
tar cf - /ebs/uat/uatcomn | gzip -c > /ebs/backup/uatcomn.tar.gz &
tar cf - /ebs/uat/uatora | gzip -c > /ebs/backup/uatora.tar.gz &
tar cf - /ebs/uat/uatdata | gzip -c > /ebs/backup/uatdata.tar.gz &

But be aware you must have enough processor power and fast disk, otherwise the concurrency will make total execution longer than consecutive one

Related Solutions

TAR gives error – tar: You may not specify more than one `-Acdtrux’ or `–test-label’ option

As the message says, you can't combine c and r; the former means "create an archive", the latter "append to an archive", so they can't be used simultaneously.

You can simply do

tar cpvzf ALPHA.tar.gz https-ALPHA

Archive big data into multiple parts

If you have enough space to store the compressed archive, then the archive could be created and split in one go (assuming GNU split):

tar -c -vz -f - directory | split --additional-suffix=.gz.part -b 1G

This would create files called xaa.gz.part, xab.gz.part etc., each file being a 1G compressed bit of the tar archive.

To extract the archive:

cat x*.gz.part | tar -x -vz -f -

If the filesystem can not store the compressed archive, the archive parts needs to be written to another filesystem, alternative to some remote location.

On that remote location, for example:

ssh user@serverwithfiles tar -c -vz -f - directory | split --additional-suffix=.gz.part -b 1G

This would transfer the compressed archive over ssh from the machine with the big directory to the local machine and split it.

Best Answer

Related Solutions

TAR gives error – tar: You may not specify more than one `-Acdtrux’ or `–test-label’ option

Archive big data into multiple parts

Related Question