200GB lost on APFS boot disk

apfsdisk-utilityhigh sierra

Recently (last month) I upgraded from Sierra to High Sierra on one of my Macs. I now am confronted with 200GB of the 500GB 'lost'. That is, when running OmniDiskSweeper as root, I find 255GB files, but according to df, 460GB is in use. Removing large files (e.g. I removed a 33GB directory with rm) does not increase free space. I've run repair on the volume (system in target disk mode, repair executed from another Mac).

The disk was full yesterday (unexepected) and the system died. What pushed it over the edge was CrashPlan crashing and writing 12.5GB cores. I used the string command to try to find where the core dumps came from but string gave errors like "malformed object LC_SEGMENT_64 fileoff field plus filesize extends past the end of the file". That sounded like file system corruption to me, hence the attempt to repair.

I first tried to remove those core dumps to free up data, and that worked to give me 40GB. But I removed more from the disk and I never got that space back. Not even the repair gave me my disk space back. I now have a file system with 210GB space missing.

I cannot use DiskWarrior as it doesn't handle APFS. So, what can I do?

Best Answer

The reasons for this problem were twofold, it has turned out.

About half of the missing space was in SIP-protected folders that are not found by programs like du or OmniDiskSweeper, not even when they are run as root. A safe boot fixed that.
The other half was indeed (see How to thin your local Time Machine Snapshots on macOS High Sierra mentioned in a comment to my question) in time machine local snapshots.

The LC_SEGMENT error may have been a red herring. Checking the APFS system with the machine in TDM attached to another Mac did not notice any problems with the file system.

Time Machine does provide a risk for boot file systems that are pretty full and that have a volatile content (large size of additions and removals). And these days, you cannot turn local snapshots off anymore.

Related Solutions

Finding lost partitions

As the gpt command reveals you have two recovery partitions (and a lot of unallocated disk space after partition 4):

partition 3 with a wrong size (1269544 blocks) and a correct partition type (426F6F74-0000-11AA-AA11-00306543ECAC)
partition 4 with the correct size (1269536 blocks) and a wrong partition type & visibility

The second Recovery HD (slightly visible below the Macintosh HD in your screenshot) is blocking the expansion of the volume Macintosh HD. The easiest procedure to solve your problem is to boot to Internet Recovery Mode and completely reformat the internal drive and reinstall OS X:

According to this Apple support document your MacBook Pro (Mid 2010)* can be upgraded to use Internet Recovery after applying a firmware upgrade. Since the firmware upgrade was published in 2012 I assume that it was applied already.

So to answer the first part of your first question: After booting to Internet Recovery Mode you may install Mac OS X Lion. In Internet Recovery Mode usually the Mac OS X the Mac originally was shipped with will be installed. For Snow Leopard Macs like your MBP Mid 2010 Mac OS X Lion will be installed.

To initiate the Internet Recovery Mode restart your Mac and press alt cmd R.
The prerequisites for an Internet Recovery are listed here: Recovery/Internet Recovery. In a few words you need the latest firmware update installed, either ethernet or WLAN (WPA/WPA2) and a router with DHCP activated for a successful internet recovery.
On a 50 Mbps-line it takes about 4 min (presenting a small animated globe) to boot into the Recovery Netboot image which is loaded from an Apple server.

After booting successfully open Disk Utility and repartition your disk to 1 JHFS+ partition with a GUID partition table. Quit Disk Utility and start 'Reinstall OS X'. Depending on your internet connection the reinstall will take some time.

The second part of question 1 as well as the questions 2 and 3 are obsolete after following the above instructions and successfully reinstalling Mac OS X Lion.

*: A MacBook Pro (Late 2010) never was produced/sold, so I assume it's a MacBook Pro (Mid 2010)

Create an APFS RAM disk

It works if you create a JHFS+ volume first and convert it to APFS in a second step:

DISK_ID=$(hdiutil attach -nomount ram://$((<number_of_blocks>)))
diskutil eraseDisk JHFS+ "RAM Disk" $DISK_ID
diskutil apfs convert $(tr -d ' '<<<${DISK_ID}s2)

If the RAM disk has a size of 2 GiB (4 * 1024 * 1024)(block_size) or smaller no EFI partition is created and the 3rd command is: diskutil apfs convert $(tr -d ' '<<<${DISK_ID}s1) or more generally:

DISK_ID=$(hdiutil attach -nomount ram://$((<number_of_blocks>)))
SIZE=$(diskutil info $DISK_ID | awk -F'[^0-9]*' '/Disk Size/ {print$4}')
diskutil eraseDisk JHFS+ "RAM Disk" $DISK_ID
if [ $SIZE -le 2147483648 ]; then diskutil apfs convert $(tr -d ' '<<<${DISK_ID}s1); else diskutil apfs convert $(tr -d ' '<<<${DISK_ID}s2); fi

Result:

...
/dev/disk2 (disk image):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        +4.3 GB     disk2
   1:                        EFI EFI                     209.7 MB   disk2s1
   2:                 Apple_APFS Container disk3         4.0 GB     disk2s2

/dev/disk3 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +4.0 GB     disk3
                                 Physical Store disk2s2
   1:                APFS Volume RAM Disk                32.8 KB    disk3s1

Best Answer

Related Solutions

Finding lost partitions

Create an APFS RAM disk

Related Question