Triple compression and I only save 1% in space

bzip2compressiongziptar

I've been trying to save space on my linux server, and I had a folder containting, in subfolders, 22GB of images.

So I decided to compress them.

First I used tar:

tar -zcf folder.tar folder

Then gzip

gzip folder

And finally, for good measure, just in case, bzip2

bzip2 folder

And after all that, the total of all the folder.tar.gz.bzip2s, came to still 22GB! With, using finer precision, a 1% space saving!

Have I done something wrong here? I would expect many times more than a 1% saving!

How else can I compress the files?

Best Answer

Compression ratio is very dependent of what you're compressing. The reason text compresses down so well is because it doesn't even begin to fully utilize the full range of numbers representable in the same binary space. So formats that do (e.g compressed files) can store the same information in less space just by virtue of using all those binary numbers that mean nothing in textual encodings and can effectively represent whole progressions of characters in a single byte and get a good compression ratio that way.

If the files are already compressed, you're typically not going to see much advantage to compressing them again. If that actually saved you additional space it's probably an indication that the first compression algorithm kind of sucks. Judging from the nature of the question I'm going to assume a lot of these are media files and as such are already compressed (albeit with algorithms that prioritize speed of decompression) and so you're probably not going to get much from them. Sort of a blood from a stone scenario: they're already as small as they could be made without losing information.

If I'm super worried about space I just do a "bzip2 -9" and call it good. I've heard good things about the ratio on XZ though. I haven't used XZ myself (other than to decompress other people's stuff), but it's supposed to have a better ratio than bzip2 but take a little longer to compress/decompress.

Related Solutions

Does gz compression ratio improve over time

It does up to a certain point and this evens out. The compression algorithms have a restriction on the size of the blocks they look at (bzip2) and/or on the tables they keep with information on previous patterns (gzip).

In the case of gzip, once a table is full old entries get pushed out, and compression no further improves. Depending on the your compression quality factor (-0 to -9) and the repetitiveness of your input this filling up can of course can take a while and you might not notice.

compression gzip bzip2 xz zstd – Compression Tool with an Arbitrarily Large Dictionary

It's at least available with the xz command. The xz manpage has:

The following table summarises the features of the presets:
Preset    DictSize    CompCPU     CompMem     DecMem
    -0    256 KiB        0          3 MiB      1 MiB

[...]

    -9     64 MiB        6        674 MiB     65 MiB
Column descriptions:

DictSize is the LZMA2 dictionary size. It is waste of memory to use a dictionary bigger than the size of the uncompressed file. This is why it is good to avoid using the presets -7 ... -9 when there's no real need for them. [...]

As documented in the Custom compressor filter chains section, you can simply supply manually the dictionary size to xz with for example --lzma2=dict=150MiB (we have insight information telling 150MiB is enough, else in doubt the file size would have to be used).

xz -9 --lzma2=dict=150MiB test300.bin

While doing this the xz process on amd64 stayed most of the time at about 1.6g usage of resident memory.

$ ls -l test*
-rw-r--r--. 1 user user 157286400 Jan 19 16:03 test150.bin
-rw-r--r--. 1 user user 157317764 Jan 19 16:03 test300.bin.xz

Best Answer

Related Solutions

Does gz compression ratio improve over time

compression gzip bzip2 xz zstd – Compression Tool with an Arbitrarily Large Dictionary

Related Question