Ext3 File System – Best Mount Options to Minimize Data Loss

First, you're right to suspect that “all data” doesn't mean the whole file. In fact, that layer of the filesystem operates on fixed-size file blocks, not on whole files. At that level, it's important to keep a bounded amount of data, so working on whole files (which can be arbitrary large) wouldn't work.

Second, there's a misconception in your question. The journaling behavior isn't something you can observe by looking at the directory contents with ls, it works at a much lower level. With normal tools, you'll always see that the file is there. (It would be catastrophic if creating a file didn't appear to, y'know, create it.) What happens under the hood is that the file can be stored in different ways. At first, the first few blocks are saved in the journal. Then, as soon as efficiently possible, the data is moved to its final location. It's still the same file in the same directory, just stored differently.

The only way you can observe journaling behavior is if you go and see exactly what the kernel is writing to the disk, or if you analyse the disk content after a crash. In normal operation, the journal is an implementation detail: if you could see it in action (other than performance-wise), it would be severely broken.

For more information about filesystem journals, I recommend starting with the Wikipedia article. In ext3 terms, a data=journal ensures that if the system crashes, each file is in a state that it had at some point before the crash (it's not always the latest state because of buffering). The reason this doesn't happen automatically is that the kernel reorders disk writes for efficiency (it can make a big difference). This is called a “physical journal” in the Wikipedia article. The other two modes (data=ordered and data=writeback) are forms of “logical journal”: they're faster, but they can lead to corrupted files. The journal limits the risk of corruption to a few files containing garbage; ext3 always uses a full journal for metadata. Without a journal for metadata, metadata can get lost, leading to major filesystem corruption. Furthermore, without a journal, recovery after a crash requires a full filesystem integrity check, whereas with a journal recovery means replaying a few journal entries.

Note that even with a journal, typical unix filesystems don't guarantee global filesystem consistency, only per-file consistency at most. That is, suppose you write to file foo, then you write to file bar, then the system crashes. It's possible for bar to have the new contents but foo to still have the old contents. To have complete consistency, you need a transactional filesystem.

This is an attempt to summarize from the chat troubleshooting session.

The setup turns out to be physical disk -> mdraid raid1 -> LVM. So there are several layers to work through. The old setup was (due to unfortunate prior recovery efforts) not available.

However, the NAS gui had been used to create another volume on a different disk, and thankfully the GUI created the new volume exactly the same way. So it was possible to discover the setup from the new disk:

• mdadm -E new-disk provided the offset to the start of the data, under the mdraid layer (2048 sectors).
• dmsetup table provided the start block of the logical volume (relative to the start of the physical volume) (1152 sectors)
• There is a magic number (0x53ef) in the third sector of an ext4 volume. Using dd and xxd, we verified that the magic number is present at that offset on the disk we're trying to recover data from.

Armed with the start sector of the ext4 filesystem, you can use a read-only loop device to recover the data:

# losetup /dev/loop0 -o $((512*(1152+2048))) -r /dev/sda1
# mount -text4 -o ro /dev/loop0 /mnt

And then copy it off.