Why is the Physical Size of a .dylib File Smaller Than Its Logical Size?

librariesmacos

There is a behavior I cannot explain in OSX Lion (might be on other versions too).

When I copy or duplicate some applications, for example iTunes, that contain some dynamic libraries, the physical size of those libraries changes (the size grows). This can be because of disk fragmentation. But it is not only that, because what is surprising is that the original physical size is smaller than the logical size (where usually the physical size is always bigger than the logical one, except when there is compression involved).

To illustrate my case:

antoine@amarante:/Applications$ ls -ls iTunes.app/Contents/MacOS/libgnsdk_submit.1.8.2.dylib 
384 -rwxrwxr-x+ 1 root  admin  558704  8 jul  2011 iTunes.app/Contents/MacOS/libgnsdk_submit.1.8.2.dylib

ls considers a block size of 512 bytes, 384 blocks make 196608 bytes compared to the 558704 bytes reported logical size. Or, as shown in a get info window from the Finder:

558.704 octets (197 Ko sur disque)

This is also shown after some copy operations where I copied over the whole iTunes application. The first iTunes copie is a duplicate done through the Finder and iTunes-cp is a copy done with the CLI cp -a command.

antoine@amarante:/Applications$ ls -ls iTunes*.app/Contents/MacOS/libgnsdk_dsp.1.8.2.dylib 
11632 -rwxr-xr-x+ 1 antoine  admin  5955104  8 jul  2011 iTunes copie.app/Contents/MacOS/libgnsdk_dsp.1.8.2.dylib
11632 -rwxrwxr-x+ 1 antoine  admin  5955104  8 jul  2011 iTunes-cp.app/Contents/MacOS/libgnsdk_dsp.1.8.2.dylib
 3944 -rwxrwxr-x+ 2 root     admin  5955104  8 jul  2011 iTunes.app/Contents/MacOS/libgnsdk_dsp.1.8.2.dylib

We see that the copied files have a physical size larger (11632*512 = 5955584 bytes) than the logical size (5955104), except for the original file which is a lot smaller. When I copy the single .dylib file on its own, the behavior is the same. This happens for a few dynamic library files, not for all.

How can the original library file be smaller than the copies? Is there some compression involved? Or are my original files corrupted in some way? How can I check that?

Best Answer

There is in fact compression in HFS+ on the mac on 10.6 and later. It's transparent, and can be difficult to even detect with standard OS tools.

And, it isn't always done. For backward compatibility the default is not to compress, but they do compress the OS when it's installed.

http://hints.macworld.com/article.php?story=20090902223042255 has many gory details.
http://reviews.cnet.com/8301-13727_7-20015100-263.html lists how various command line tools view the sizes of compressed files.

Tool list, in case of link breakage:

hfsdebug
afsctool
stat -f %f - reports "user flags" on files - one of the flags means compression. I get "32" on a compressed file and "0" on an uncompressed file. Probably the 32 maps to a bit field.
ditto - has support for writing compressed files

Related Solutions

Macos – Why is the Terminal.app ‘open’ command slower than opening a file via the Finder

I have previously experienced issues with quarantined files that were downloaded from the web and have the com.apple.quarantine extended attribute set.

To see if this is the case for your file, ls -l@ filename.ext in Terminal.

To remove the extended attribute, run xattr -d com.apple.quarantine filename.ext

Macos – Mac OS X not reporting directory sizes correctly

The differences came from different reasons: different ways of counting, different tools, compression and what looks like a bug.

The first difference in size you see seems to be a bug in the Finder. The file sizes shown by the Finder are somehow calculated in real time and cached in .DS_Store files. For some reason, while duplicating a big application/folder, the Finder calculates its size during the copy process and caches the, then incomplete, size. It then shows that size as greyed in the Finder windows, grey meaning the Finder knows the content has changed since it's last size calculation but it hasn't recalculate it yet.

The only way I've found to make it recalculate correctly the size is by deleting the .DS_Store file in the Application folder, then quitting the Finder (from the Activity Monitor for example) and relaunching it again (from the Dock Icon). If you don't delete the .DS_Store file, it still stays out greyed. Maybe waiting some time (hours, days, reboot, ...) will make the Finder do it by itself.

After that, you should see that all sizes reported by the Finder are the same.

So yes, it looks like a Finder bug, at least in OSX Lion (tested with 10.7.4 here, Finder version 10.7.3). You can also see this thread which reports same kind of behavior.

Then, let's consider the du tool. At first, I thought the difference we see could be explained by the difference between logical and physical sizes of items being copied. Logical size is the real size of the item, meaning every single bit of information it contains added up together. Physical size is the size of the item on the disk, where each information bit is written on a disk sector.

For example a file containing a single character would end up having a 1 byte logical size, but a 512 bytes or even 4096 bytes physical size when actually written to disk. The physical size is usually larger than the logical size (and depends on the actual sector/block size of the disk or the file system). This is explained into more details in this other thread. Logical size could be larger in the case of sparse files, but HFS+ doesn't seem to support such a feature.

du shows only the physical size (and you can tell it what a BLOCKSIZE is). You can see that the size reported by du is always bigger (or, exceptionally, the same) as the original. This is because of file system and disk space fragmentation. When you copy over a file (actually here a bunch of files, as an Application is a directory) new sectors are being allocated on the disk and, as fragmentation occurs, the number of blocks used is usually higher than that of the original item. Some people call that File Slack.

Now, back to the Finder. If you open up the get info window of the Applications you duplicated, you'll see that the Finder is actually reporting both the Logical and the Physical size of the item you selected. Which then makes sense. You'll even be able to compare the Physical size reported by the Finder and the one reported by du if you do a bit of math.

Why doing some math? Because the Finder shows the file sizes in kB, MB or GB where du reports them in kiB, MiB or GiB. Those are the IEC binary prefixes which should be used to calculate and display units of digital information.

But, actually, I'm not sure File Slack is involved here, there's something else. HFS+ volumes allow compression, done transparently, and Apple uses that for the original items installed by the OS. Then, when files are copied using standard tools, compression is not used anymore (as a default, to be backward compatible). If you want to keep compression on those files, you need to use the ditto command instead of cp or any Finder action. This is explained in this review.

Here is the output of copying iTunes.app using the different techniques. You'll see that ditto makes the Application exactly the same size, preserving compression, where cp doesn't. And you can even remove the binary for the arch you don't need, then reducing the whole size):

antoine@amarante:/Applications$ du -ms iTunes.app/
281 iTunes.app/
antoine@amarante:/Applications$ cp -a iTunes.app/ iTunes-copy.app/
antoine@amarante:/Applications$ ditto iTunes.app/ iTunes-ditto.app
antoine@amarante:/Applications$ ditto --arch x86_64 iTunes.app/ iTunes-64.app
antoine@amarante:/Applications$ du -ms iTunes*
236 iTunes-64.app
289 iTunes-copy.app
281 iTunes-ditto.app
281 iTunes.app

Thanks to @DanPritts for his answer on my complementary post.

Best Answer

Related Solutions

Macos – Why is the Terminal.app ‘open’ command slower than opening a file via the Finder

Macos – Mac OS X not reporting directory sizes correctly

Related Question