The simple answer to the question in the title is "Yes". But what you really want to do is the next step, which is getting the existing data mirrored.
It's possible to convert the existing disk, but it's risky, as mentioned, due the the metadata location. Much better to create an empty (broken) mirror with the new disk and copy the existing data onto it. Then, if it doesn't work, you just boot back to the un-mirrored original.
First, initialize /dev/sdb1
as the new /dev/md0
with a missing drive and initialize the filesystem (I'm assuming ext3, but the choice is yours)
mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sdb1 missing
mkfs -text3 /dev/md0
Now, /dev/sda1
is most likely your root file system (/
) so for safety you should do the next step from a live CD, rescue disk or other bootable system which can access both /dev/sda1
and /dev/md0
although I have successfully done this by dropping to single user mode.
Copy the entire contents of the filesystem on /dev/sda1
to /dev/md0
. For example:
mount /dev/sda1 /mnt/a # only do this if /dev/sda1 isn't mounted as root
mount /dev/md0 /mnt/b
cd /mnt/a # or "cd /" if it's the root filesystem
cp -dpRxv . /mnt/b
Edit /etc/fstab
or otherwise ensure that on the next boot, /dev/md0
is mounted instead of /dev/sda1
. Your system is probably set to boot from /dev/sda1
and the boot parameters probably specify this as the root device, so when rebooting you should manually change this so that the root is /dev/md0
(assuming /dev/sda1
was root). After reboot, check that/dev/md0
is now mounted (df
) and that it is running as a degraded mirror (cat /proc/mdstat
). Add /dev/sda1
to the array:
mdadm /dev/md0 --add /dev/sda1
Since the rebuild will overwrite /dev/sda1
, which metadata version you use is irrelevant. As always when making major changes, take a full backup (if possible) or at least ensure that anything which can't be recreated is safe.
You will need to regenerate your boot config to use /dev/md0
as root (if /dev/sda1
was root) and probably need to regenerate mdadm.conf
to ensure /dev/md0
is always started.
This is a fundamental problem with RAID5—bad blocks on rebuild are a killer.
Oct 2 15:08:51 it kernel: [1686185.573233] md/raid:md0: device xvdc operational as raid disk 0
Oct 2 15:08:51 it kernel: [1686185.580020] md/raid:md0: device xvde operational as raid disk 2
Oct 2 15:08:51 it kernel: [1686185.588307] md/raid:md0: device xvdd operational as raid disk 1
Oct 2 15:08:51 it kernel: [1686185.595745] md/raid:md0: allocated 4312kB
Oct 2 15:08:51 it kernel: [1686185.600729] md/raid:md0: raid level 5 active with 3 out of 4 devices, algorithm 2
Oct 2 15:08:51 it kernel: [1686185.608928] md0: detected capacity change from 0 to 2705221484544
⋮
The array has been assembled, degraded. It has been assembled with xvdc, xvde, and xvdd. Apparently, there is a hot spare:
Oct 2 15:08:51 it kernel: [1686185.615772] md: recovery of RAID array md0
Oct 2 15:08:51 it kernel: [1686185.621150] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Oct 2 15:08:51 it kernel: [1686185.627626] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Oct 2 15:08:51 it kernel: [1686185.634024] md0: unknown partition table
Oct 2 15:08:51 it kernel: [1686185.645882] md: using 128k window, over a total of 880605952k.
The 'partition table' message is unrelated. The other messages are telling you that md is attempting to do a recovery, probably on to a hot spare (which might be the device that failed out before, if you've attempted to remove/re-add it).
⋮
Oct 2 15:24:19 it kernel: [1687112.817845] end_request: I/O error, dev xvde, sector 881423360
Oct 2 15:24:19 it kernel: [1687112.820517] raid5_end_read_request: 1 callbacks suppressed
Oct 2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423360 on xvde).
Oct 2 15:24:19 it kernel: [1687112.821837] md/raid:md0: Disk failure on xvde, disabling device.
Oct 2 15:24:19 it kernel: [1687112.821837] md/raid:md0: Operation continuing on 2 devices.
And this here is md attempting to read a sector from xvde (one of the remaining three devices). That fails [bad sector, probably], and md (since the array is degraded) can not recover. It thus kicks the disk out of the array, and with a double-disk failure, your RAID5 is dead.
I'm not sure why its being labeled as a spare—that's weird (though, I guess I normally look at /proc/mdstat
, so maybe that's just how mdadm labels it). Also, I thought newer kernels were much more hesitant to kick out for bad blocks—but maybe you're running something older?
What can you do about this?
Good backups. That's always an important part of any strategy to keep data alive.
Make sure that the array gets scrubbed for bad blocks routinely. Your OS may already include a cron job for this. You do this by echoing either repair
or check
to /sys/block/md0/md/sync_action
. "Repair" will also repair any discovered parity errors (e.g., the parity bit doesn't match with the data on the disks).
# echo repair > /sys/block/md0/md/sync_action
#
Progress can be watched with cat /proc/mdstat
, or the various files in that sysfs directory. (You can find somewhat up-to-date documentation at the Linux Raid Wiki mdstat article.
NOTE: On older kernels—not sure the exact version—check may not fix bad blocks.
One final option is to switch to RAID6. This will require another disk (you can run a four- or even three-disk RAID6, you probably don't want to). With new enough kernels, bad blocks are fixed on the fly when possible. RAID6 can survive two disk failures, so when one disk has failed, it can still survive a bad block—and thus it'll both map out the bad block and continue the rebuild.
Best Answer
As for overwriting each disk with random data, it's redundant. Since you're going to build a new encrypted RAID, the resync will overwrite everything anyway.
As for the method of overwriting,
/dev/urandom
is horribly slow, people who try to use it for wiping terabytes usually cancel halfway through because it just takes way too long. Encrypting the device with a random key and then wiping that with/dev/zero
is faster, andshred -n 1
is much faster still. So if you must have random data on your disk, I recommend you use those methods instead.Now to your RAID, I'd do the following: (in this example
loop2
is the 2TB disk)Add the new 1TB disk partition to the RAID-1. Wait until the sync is finished. This way, your 1TB of data spans three disks.
remove the 2TB disk from the RAID-1 array. Your 1TB is still preserved redundantly on the two 1TB disks.
wipe the 2TB disk
repartition the disk to 2TB. Note that you will need a boot partition if you have no other boot device, as you can not boot from encrypted devices.
create a new RAID-1 array using that partition, and
missing
for the 2nd device. The 2nd device will be added later on. Also, the size will be limited for now, to be grown later, as we're not sure of the size of the 2x1TB disks at this point. (If you are sure, feel free to use a different size here, but you won't be able to add the 2x1TB later if you make it too large).encrypt it, mkfs, copy data (use your preferred cipher and settings, LVM, filesystem, copy methods, ...)
Now your data is redundant on the old RAID 1 (unencrypted), and the new non-redundant RAID 1 (encrypted). But at this point you have to break redundancy in order to add the 2x1TB disks to the new RAID 1.
At this point you could also wipe the 2x1TB disks if you like to waste your time. It's not necessary as the RAID-1 will sync the random data from the 2TB disk over to the 1TB disks.
Combine 2x1TB using mdadm, either
linear
or0
, depending on your preference.Add it to the RAID 1 and grow the RAID 1 at the same time.
Grow the cryptsetup and filesystem also.
Aaand you're done.
Edit:
Here's an example
/etc/mdadm.conf
to go with this setup (usemdadm --detail --scan
to get a starting point):In particular:
DEVICE
lines (if you must have them, make sure it includesmd*
devices)ARRAY
lines is important - the RAID0 array must be built first, which happens if it's listed first in the file.