Windows – Why do big gaps remain after defragmentation in Windows XP

IIRC, the NTFS filesystem attempts to allocate the file in contiguous storage. However it can only do that if the filesystem knows the size of the file. If you open a file and start writing to it, it'll write to the "best" place to fit the file (typically towards the outside of the platter). But that "best" place might not be big enough to fit the file.

If the application tells NTFS the actual size of the file (with SetEndOfFile()), NTFS can do a better job of finding contiguous space for the file (the SetEndOfFile API causes NTFS to allocate storage for the entire file).

What probably happened is that the defrag operation forced Windows to throw out some system restore snapshots. It'd be a pathological case of fragmentation to cause metadata overhead to be a full 10% of your drive space on top of what Windows normally uses. even then, I'm not sure it's possible.

I don't see anything in Defraggler's version history or documentation that indicates that it's able to correctly defragment files to prevent the purging of shadow copies. In fact, this thread from Defraggler's support forum indicates that they know that it's happening (there's a post from a board admin labeled "Official Piriform Bug Fixer" in thread) but don't indicate whether or not they're going to fix it.

Shadow copies may be lost when you defragment a volume: The reason this happens is that by default, VSS operates with 16 KB clusters by default, while most NTFS volumes are formatted with 4 KB clusters. So if a defrag operation is moving data that isn't a multiple of a 16 KB cluster (or the "distance" it's moved isn't a multiple of 16 KB), then VSS will track it as a change and might purge all your snapshots.

MSDN: Defragmenting Files:

When possible, move data in blocks aligned relative to each other in 16-kilobyte (KB) increments. This reduces copy-on-write overhead when shadow copies are enabled, because shadow copy space is increased and performance is reduced when the following conditions occur:

• The move request block size is less than or equal to 16 KB.
• The move delta is not in increments of 16 KB.

Vista's built in defrag doesn't do this:

One change that’s not obvious to users is our shadow copy optimization during defragmentation. Defrag has special heuristics to move file blocks in a way that will minimize the copy-on-write activity and shadow copy storage area consumption. Without this optimization, the defragmentation process would accelerate the deletion of older shadow copies.