Avoiding fragmentation
The secret is to not write uncompressed files on the disk to begin with.
Indeed, after you compress an already existing large file it will become horrendously fragmented due to the nature of the NTFS in-place compression algorithm.
Instead, you can avoid this drawback altogether by making OS compress a file's content on-the-fly, before writing it to the disk. This way compressed files will be written to the disk as any normal files - without unintentional gaps. For this purpose you need to create a compressed folder. (The same way you mark files to be compressed, you can mark folders to be compressed.) Afterwards, all files written to that folder will be compressed on the fly (i.e. written as streams of compressed blocks). Files compressed this way can still end up being somewhat fragmented, but it will be a far cry from the mess that in-place NTFS compression creates.
Example
NTFS compressed 232Mb system image to 125Mb:
- In-place compression created whopping 2680 fragments!
- On-the-fly compression created 19 fragments.
Defragmentation
It's true that NTFS compressed files can pose a problem to some defragment tools. For example, a tool I normally use can't efficiently handle them - it slows down to a crawl. Fret not, the old trusty Contig from Sysinternals does the job of defragmenting NTFS compressed files quickly and effortlessly!
It seems that:
Percent file fragmentation = Percent free space (after defragmentation) - Percent free space (before defragmentation)
In other words, the percent free space added by performing full defragmentation.
As you can see, 7 % = 31 % - 24 %:
C:\Windows\system32>defrag c: -w -v
Windows Disk Defragmenter
Copyright (c) 2006 Microsoft Corp.`
Defragmentation report for volume C:
Volume size = 74.53 GB
Cluster size = 4 KB
Used space = 55.96 GB
Free space = 18.57 GB
Percent free space = 24 %
File fragmentation
Percent file fragmentation = 7 %
Total movable files = 203,178
Average file size = 342 KB
Total fragmented files = 3,641
Total excess fragments = 32,280
Average fragments per file = 1.19
Total unmovable files = 26
Free space fragmentation
Free space = 18.57 GB
Total free space extent = 22,689
Average free space per extent = 858 KB
Largest free space extent = 4.09 GB
Folder fragmentation
Total folders = 40,150
Fragmented folders = 155
Excess folder fragments = 567
Master File Table (MFT) fragmentation
Total MFT size = 203 MB
MFT record count = 205,234
Percent MFT in use = 98
Total MFT fragments = 3
Note: On NTFS volumes, file fragments larger than 64MB are not included in the fragmentation statistics
Defragmentation report for volume C:
Volume size = 74.53 GB
Cluster size = 4 KB
Used space = 50.90 GB
Free space = 23.62 GB
Percent free space = 31 %
File fragmentation
Percent file fragmentation = 0 %
Total movable files = 203,178
Average file size = 342 KB
Total fragmented files = 3
Total excess fragments = 1,152
Average fragments per file = 1.00
Total unmovable files = 26
Free space fragmentation
Free space = 23.62 GB
Total free space extent = 10,109
Average free space per extent = 2 MB
Largest free space extent = 6.99 GB
Folder fragmentation
Total folders = 40,150
Fragmented folders = 1
Excess folder fragments = 0
Master File Table (MFT) fragmentation
Total MFT size = 203 MB
MFT record count = 205,234
Percent MFT in use = 98
Total MFT fragments = 3
Note: On NTFS volumes, file fragments larger than 64MB are not included in the fragmentation statistics
C:\Windows\system32>
On the other hand, when, after defragmenting a drive, the Percent free space does not change, nor the Percent file fragmentation does not change, even if the Total fragmented files number does change:
C:\Windows\system32>defrag d: -w -v -f
Windows Disk Defragmenter
Copyright (c) 2006 Microsoft Corp.
Defragmentation report for volume D:
Volume size = 932 GB
Cluster size = 4 KB
Used space = 841 GB
Free space = 90.89 GB
Percent free space = 9 %
File fragmentation
Percent file fragmentation = 2 %
Total movable files = 400,108
Average file size = 2 MB
Total fragmented files = 7,671
Total excess fragments = 1,420,753
Average fragments per file = 4.70
Total unmovable files = 7
Free space fragmentation
Free space = 90.89 GB
Total free space extent = 213,633
Average free space per extent = 446 KB
Largest free space extent = 27.46 GB
Folder fragmentation
Total folders = 16,189
Fragmented folders = 526
Excess folder fragments = 2,304
Master File Table (MFT) fragmentation
Total MFT size = 421 MB
MFT record count = 405,678
Percent MFT in use = 94
Total MFT fragments = 3
Note: On NTFS volumes, file fragments larger than 64MB are not included in the fragmentation statistics
Defragmentation report for volume D:
Volume size = 932 GB
Cluster size = 4 KB
Used space = 841 GB
Free space = 90.89 GB
Percent free space = 9 %
File fragmentation
Percent file fragmentation = 2 %
Total movable files = 400,108
Average file size = 2 MB
Total fragmented files = 353
Total excess fragments = 1,226,342
Average fragments per file = 4.19
Total unmovable files = 7
Free space fragmentation
Free space = 90.89 GB
Total free space extent = 260,094
Average free space per extent = 366 KB
Largest free space extent = 27.46 GB
Folder fragmentation
Total folders = 16,189
Fragmented folders = 1
Excess folder fragments = 0
Master File Table (MFT) fragmentation
Total MFT size = 421 MB
MFT record count = 405,678
Percent MFT in use = 94
Total MFT fragments = 3
Note: On NTFS volumes, file fragments larger than 64MB are not included in the fragmentation statistics
C:\Windows\system32>
Best Answer
The defragmenter does not have anywhere to move the big files to: all free space is somewhat evenly distributed in small chunks, so there wouldn't be any improvement if $VERYBIGFILE changed from, say, 20 close pieces to 10 scattered-all-over pieces.
One of the steps done by the defragmenter is file compaction, in which contiguous but separate files are shuffled around to form a single block of data. Try running this a few times, and it should get better over time.
You can also try
contig -v somebigfile
to defragment files one by one.