I often work on full disk images, I often need to read and write full disk images on not so capable laptops using a USB-3 disk as my temporary medium.
Duplicating a raw image is probably the fastest way, but I have to deal with limited storage space available.
What I need is high throughput and low CPU usage – compression ratio is not that important for me.
I'd like to have a tool that can very lightly compress the images created – mostly encoding empty space on the disk so that I can only store useful data, while making it as fast as possible – hopefully almost as fast as a raw dd duplication.
I tied to use pxz -1, as it can use multiple CPU cores to speed up the compression, hoping for the fastest compression to still be useful. That doesn't however seem to be as fat as I'd like.
Compressing a 500 GB disk image was estimated to take 21 hours:
mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G | pxz -1 > Lenovo-Win8-sda.dd.xz
0:09:21 [9.26MiB/s] [6.54MiB/s] [> ] 0% ETA 21:34:46
No compression at all predicted 3 hours to complete the imaging:
mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G > Lenovo-Win8-sda.dd
0:00:20 [80.2MiB/s] [49.8MiB/s] [> ] 0% ETA 2:50:20
Using lbzip2 -1 for compresson seems to be slightly faster than pxz -1 with ETA at 15 hours, which is still very slow.
mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G | lbzip2 -1 > Lenovo-Win8-sda.dd.bz2
0:00:22 [9.07MiB/s] [9.76MiB/s] [> ] 0% ETA 14:33:38
Are there any faster compression tools for Linux?
Best Answer
LZ4
apt install liblz4-toolThen you can compress with
lz4command and decompress withlz4 -d.It defaults to the fastest compression mode.
500 GB from internal hard drive to an external drive over USB 3.0 is estimated to take between 2 and 3 hours: