MacOS – Catalina: Purpose of /System/Volumes/Data/Previous Content/

Catalina introduces a new file system layout. Where Mojave and earlier had one filesystem that combines the system and user data, Catalina has a read-only system volume and a read-write user volume interleaved on a folder by folder basis using firmlinks.

The easy way to move forward is just save your additional files to /usr/local and other traditional places where Apple expects user modifications to their default system to be saved.

• Check out What's New in Apple File Systems from #WWDC19
• PDF summary of the above video presentation

Some of the implementation is quite normal for Unix/Linux like sparse files not being allocated and copy on write and cloning of an entire file system / snapshots. Other items like Firmlinks that act as “wormholes” between two containers / filesystems to present an unified file tree, System Integrity Protection and APFS specific features are quite new still to everyone.

You can see this better with df or diskutil apfs list command line tools than the Disk Utility view.

There is a particular fsck error:

** Checking the object map.
error: (oid 0xd31c1) om: btn: found zeroed-out block
   Object map is invalid.

Here, om refers to the object map of the macOS volume, and btn refers to a B-tree node in that object map. Evidently, part of the node has been zeroed-out, leading to some or all of the dentries for /Users/jivan being inaccessible.

I developed some tools to inspect the APFS container, in the hopes that older versions of the object map and other file-system structures were intact (as referenced by an older APFS transaction), thereby allowing me to access my files. Using these tools, I indeed found that a few nodes in the file-system root B-tree for my main APFS volume had been zeroed out. Thanks to APFS's copy-on-write/transaction-based behaviour, I was able to search the entire disk for older versions of these missing nodes, and successfully found recent instances of them — except for the particular leaf node that contains the file-system records for /Users/jivan, so its contents cannot be directly determined. Just my luck(!) However, I was able to see that /Users/jivan had an ID of 0xb54a8, and thus search for nodes which contained dentries for items whose parent ID was also 0xb54a8; these nodes were then the ones which listed the contents of /Users/jivan.

In order to more easily do an automated recovery, I reconstructed the missing internal node of the file-system B-tree, and then used my apfs-recover tool to actually get each file. For example, to recover /Users/jivan/Documents/my file.pdf, I can do:

apfs-recover /dev/disk2s2 0 "/Users/jivan/Documents/my file.pdf" > "~/Desktop/my file.pdf"

Rather than run such a command for each file, I wrote a Bash script, pull.sh, which, when given a target recovery directory and a file which lists paths to files to attempt to recover, runs apfs-recover for each such filepath and outputs the result to a corresponding path in the recovery directory. For example, if the contents of filepaths.txt are

/Users/jivan/Documents/my doc.pdf
/Users/jivan/Pictures/my pic.jpg

then running pull.sh ~/Desktop/RECOVERY filepaths.txt recovers the files to the following paths:

~/Desktop/RECOVERY/Users/jivan/Documents/my doc.pdf
~/Desktop/RECOVERY/Users/jivan/Pictures/my pic.jpg

I added the desired entries in filepaths.txt with some programmatic assistance, and was then able to successfully recover the vast majority of my files. For any particularly important files which this script fails to recover (due to bugs in the software I've written or additional malformed/missing APFS structures on the affected disk) I'll have to dig deeper, but this is effectively solved now.

All tools mentioned are available in the Git repo.