RAID6 scrubbing mismatch repair

corruptionmdadmraid

You can initiate a scrub of a mdadm array with echo 'check' > /sys/block/mdX/md/sync_action, and if a bad sector is found, it'll rewrite it automatically (from a mirror or from parity information for RAID5/6).

However if all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch. In this case repair is complicated because mdadm cannot tell which mirror contains the correct data (RAID1/10) or whether it is data or parity that is corrupted (RAID5).

In theory this is not the case with RAID6 if I understand RAID6 correctly. Because double-parity exists, it should be possible to pinpoint where a single corruption is, whether it is data or parity.

Is my understanding correct, should this be possible in theory?
If correct, is mdadm able to repair this inconsistent data without guessing which block is corrupted?

Best Answer

It is possible in theory: the data+parity gives you three opinions on what the data should be; if two of them are consistent, you can assume the third is the incorrect one and re-write it based on the first two.

Linux RAID6 does not do this. Instead, any time there is a mismatch, the two parity values are assumed to be incorrect and recalculated from the data values. There have been proposals to change to a "majority vote" system, but it hasn't been implemented.

The mdadm package includes the raid6check utility that attempts to figure out which disk is bad in the event of a parity mismatch, but it has some rough edges, is not installed by default, and doesn't fix the errors it finds.

Related Solutions

Bit Rot Detection – Detection and Correction with mdadm

I don't have enough rep to comment, but I want to point out that the mdadm system in Linux DOES NOT correct any errors. If you tell it to "fix" errors during a scrub of, say, RAID6, if there is an inconsistency, it will "fix" it by assuming the data portions are correct and recalculating the parity.

Ubuntu – mdadm – RAID5 array size vs. actual disk size mismatch

fdisk is the wrong tool for disks >2TB. Use parted or gdisk instead.

It appears that /dev/sdc1 and /dev/sdd1 are 2TB partitions, so that's what limits your array size. For the other disks, they have GPT so I assume they are 3TB already, but you should check.

Basically you have to stop the array, enlarge each partition to 3TB (without changing the starting offset), then start it again and follow it up with a grow:

mdadm --grow /dev/md0 --size=max

If you can't stop the array, you'll have to fail each 2TB partition individually, repartition and re-add it. This might go faster if you add a write-intent bitmap first.

mdadm --grow /dev/md0 --bitmap=internal

Then for each disk individually,

mdadm /dev/md0 --fail /dev/disk1 # check mdstat for [UUUU] first
mdadm /dev/md0 --remove /dev/disk1
parted /dev/disk -- mklabel gpt mkpart primary 1mib -1mib
mdadm /dev/md0 --re-add /dev/disk1
mdadm --wait /dev/md0 # must wait for sync

Once that's done you can remove the bitmap again (keeping it may harm performance).

mdadm --grow /dev/md0 --bitmap=none
mdadm --grow /dev/md0 --size=max

Finally do your resize2fs or whatever.

Best Answer

Related Solutions

Bit Rot Detection – Detection and Correction with mdadm

Ubuntu – mdadm – RAID5 array size vs. actual disk size mismatch

Related Question