Prevent Tar from Using Too Much CPU and Disk

limittar

I want to backup 1 terabyte of data to an external disk.

I am using this command: tar cf /media/MYDISK/backup.tar mydata

PROBLEM: My poor laptop freezes and crashes whenever I use 100% CPU or 100% disk (if you want to react about this please write here).
So I want to stay at around 50% CPU and 50% disk max.

My question: How to throttle CPU and disk with the tar command?

Rsync has a –bwlimit option, but I want an archive because 1) there are many small files 2) I prefer to manage a single file rather a tree. That's why I use tar.

Best Answer

You can use pv to throttle the bandwidth of a pipe. Since your use case is strongly IO-bound, the added CPU overhead of going through a pipe shouldn't be noticeable, and you don't need to do any CPU throttling.

tar cf - mydata | pv -L 1m >/media/MYDISK/backup.tar

Related Solutions

Unable to back-up data from android tablet to laptop using tar!

Yes, I've got the solution. See there were 2 options for connecting tablet. 1) Media device (MTP) 2) Camera (PTP)

I'd been using the 1st one where via command line I was failing to do & via GUI, I wasn't able to retain the time stamp. This option uses mtp to connect. enter image description here

Then yesterday, it stroked my mind & I connected via the 2nd option , i.e. camera. Now via GUI (shotwell software), I was able to import the files. Then in nautilus, I found it was connected not via mtp but gphoto2. enter image description here

Now I was able to use the command line & copy via cp -a retaining the time stamp & it worked. Now I know all command line commands like tar will work as with any external HDD or pen-drive. A great relief!

How to Create SHA1 Checksums of Files Inside a Tar Archive

Too easy :

tar xvJf myArchive.tar.xz --to-command=sha1sum

The result is like this :

z/
z/DOCUMENTATION
3c4d9df9bcbd1fb756b1aaba8dd6a2db788a8659 *-
z/getnameenv.sh
1b7f1ef4bbb229e4dc5d280c3c9835d9d061726a *-

Or create "tarsha1.sh" with :

#!/bin/bash

sha1=`sha1sum`
echo -n $sha1 | sed 's/ .*$//'
echo " $TAR_FILENAME"

Then use it this way :

tar xJf myArchive.tar.xz --to-command=./tarsha1.sh

The result is like this :

3c4d9df9bcbd1fb756b1aaba8dd6a2db788a8659 z/DOCUMENTATION
1b7f1ef4bbb229e4dc5d280c3c9835d9d061726a z/getnameenv.sh

Best Answer

Related Solutions

Unable to back-up data from android tablet to laptop using tar!

How to Create SHA1 Checksums of Files Inside a Tar Archive

Related Question