SD Card Filesystem – Corruption-Proof Filesystem for Embedded Linux

embeddedfailure-resistancefilesystemslinux

Recently we had a rather unpleasant situation with our customer – Raspberry Pi based "kiosk" used to display remote sensing data (nothing more fancy than a kiosk mode browser displaying a self-updating webpage from the data-collection server) failed to boot due to filesystem corruption. Ext4, Manual fsck required, the system will be a part of tomorrow's important presentation, service required immediately. Of course we can't require the customer to shut down the system nicely when switching it off for the night; the system must simply withstand such mistreatment.

I'd like to avoid such situations in the future, and I'd like to move the OS to a filesystem that would prevent this. There's a bunch of filesystems intended for MTD devices, where getting them to run on SD card (a standard block device) requires some serious hoop-jumping. There are also some other filesystems (journalling etc) that boast good resistance against corruption. I still need to see some reasonable comparison of their pros and cons.

Which filesystem available in Linux would provide best resistance against corruption on unexpected power failures and not require jumping through impossible hoops like yaffs2 in order to install to SD.

Wear-balancing is a plus, but not a requirement – SD cards usually have their own mechanisms, if less than perfect, though the system should be "gentle for flash" (systems like NTFS can murder an SD card within a month).

Best Answer

The best resistance against corruption on a single SD card would be offered by BTRFS in RAID1 mode with automatic scrub run every predefined period of time.

The benefits:

  1. retaining ability to RW to the filesystem
  2. modern, fully featured filesystem with very useful options for an RPi, like transparent compression and snapshots
  3. designed with flash memory in mind (among other things)

Here is how to do it:

I run my RaspberryPi on ArchARM linux and my card is in the SD reader, so modify those instructions accordingly for other distros and /dev interfaces.

Here is an example partition layout:

/dev/mmcblk0p1: fat32 boot partition
/dev/mmcblk0p2: to be used as btrfs partition
/dev/mmcblk0p3: to be used as btrfs partition (mirrored with the above)
/dev/mmcblk0p4 (optional): swap

To get btrfs into RAID1, you create the filesystem like so:

mkfs.btrfs -m raid1 -d raid1 /dev/mmcblk0p2 /dev/mmcblk0p3

Then you rsync -aAXv to it your previously backed up system.

To get it to boot from BTRFS in raid1, you need to modify initramfs. Therefore, you need to do the following while you still have your system running on your old filesystem.

Raspberry does not normally use mkinitcpio so you must install it. Then, you need to add “btrfs” to MODULES array in mkinitcpio.conf and recreate initramfs with

mkinitcpio -g /boot/initrd -k YOUR_KERNEL_VERSION

To know what to type instead of YOUR_KERNEL_VERSION, run 

ls /lib/modules

If you update the kernel, you MUST recreate initramfs BEFORE you reboot.

Then, you need to modify RPi’s boot files.

In cmdline.txt, you need to have

root=/dev/mmcblk0p2 initrd=0x01f00000 rootfstype=btrfs

and in config.txt, you need to add

initramfs initrd 0x01f00000

Once you’ve done all that and successfully booted into your btrfs RAID1 system, the only thing left is to set up periodic scrub (every 3-7 days) either with systemd timer (preferred), or cron (dcron) like so:

btrfs scrub start /

It will run on your filesystem comparing checksums of all the files and fixing them (replacing with the correct copy) if it finds any corruption.

The combination of BTRFS RAID1, single medium and Raspberry Pi make this pretty arcane stuff. It took some time and work to put all the pieces together, but here it is.

Related Question