Explaining performance irregularities when unpacking .tar.gz archives

archivingmactarwindows 10

This question is about the performance of unpacking, not "how to" unpack.
I notices this just a few days ago, when I started extracting several .tar.gz (will refer to them as archives now) on my rather powerful Windows 10 machine. Unpacking a 25 GB archive took more than an entire day. The content was mostly mp3 audio files of some 2-digit KB sizes, so I had plenty of files to unpack. My Windows machine features a new i7 Processor, 16 GB of RAM and I was unpacking from a 1TB NVMe to an external 500 GB SSD connected via USB-C, which is usually quite fast.

Just for the sake of curiosity I tried unpacking the same archive on my MacBook, which has smaller specs (an older i5 Processor, 16 GB RAM and 500 GB internal NVMe). For some reason, unpacking took only around 45 minutes. I was shocked about this huge difference, for which I honestly could not come up with any explanation. Searching the web, people stated that this would be caused by a slower hard-drive (which is likely not true in my case, I tried unpacking on Windows from my internal NVMe to itself, which was also extremely slow).

How can one explain this large difference? I suspect it has to do with the file systems, but I cannot make sense of it.

Best Answer

It seems that Windows Defender will scan each new file whenever the program issues a "close" call – which happens synchronously, i.e. the call will not return and the thread will not continue until Defender completes the scan. This was discussed last year in a Linux.conf.au talk:

"NTFS really isn't that bad" - Robert Collins (LCA 2020)

File system and OS kernel design may also be a factor; since each new file needs its metadata to be written and directory metadata to be updated, operation latency matters more than just the raw disk throughput. (For example, when writing a large file the OS can just issue a bunch of writes at once and collect the results later; but metadata updates might involve reading some data and waiting for the read to complete before it can be written back.)

Related Solutions

Find out the size of a .tar.gz archive in the terminal without unpacking

Sure. Just use -tv to list the contents with their sizes. E.g.

% tar -tvzf sometools2.tar.gz 
-rw-r--r-- madler/admin   3442 2005-02-27 21:40 pngdat.c
-rw-r--r-- madler/admin  24938 2005-02-27 21:39 infgen.c

If you want to add up the sizes (like du), you can use awk:

% tar -tvzf sometools2.tar.gz|awk '{ s += $3 } END { print s }'
28380

For an imbedded .tar.gz file, you would need to do those individually when you find them by sending them to stdout with -O:

% tar -tvzf imbed.tar.gz 
-rw-r--r-- madler/staff    505 2012-02-12 00:06 lucas.c
-rw-r--r-- madler/staff  27913 2005-03-20 11:10 lzwtry.c
-rw-r--r-- madler/staff   8314 2005-02-27 21:42 sometools2.tar.gz
% tar -xOzf imbed.tar.gz sometools2.tar.gz | tar -tvzf - | awk '{ s += $3 } END { print s }'
28380

You can write a script to find those in the -tv output and then extract them, and even do it recursively. I will leave that as an exercise for the reader.

Note, these options are for GNU tar, which is what is on both Linux and Mac OS X. The options for BSD tar may be different.

Google-chrome – Chromium: prevent unpacking tar.gz

Your web server is likely sending the .tar.gz file with a content-encoding: gzip header, causing the web browser to assume a gzip layer was applied only to save bandwidth, and what you really intended to send was the .tar archive. Chrome un-gzips it on the other side like it would with any other file (.html, .js, .css, etc.) that it receives gzipped (it dutifully doesn't modify the filename though).

To fix this, make sure your web server serves .tar.gz files without the content-encoding: gzip header.

More Info: https://code.google.com/p/chromium/issues/detail?id=83292

Best Answer

Related Solutions

Find out the size of a .tar.gz archive in the terminal without unpacking

Google-chrome – Chromium: prevent unpacking tar.gz

Related Question