RAID Array Not Auto Assembling – Boot Problems Solution

I had just asked this question where I was unable to boot after installing a new RAID1 Array. I was able to get to the terminal, but once I sorted that out I realized that my problem isn't so much an fstab boot problem as an mdadm auto assemble problem.

I have three RAID1 arrays on my system, with /dev/md1 mounted at / and /dev/md0 mounted as swap, and these currently run without problems. I did not create these arrays. I created a new RAID1 array, /dev/md2 which I formatted to ext4 using this guide, and in doing so I created a new partition (the only one) as md2p1 (the guide also created a similarly named partition although fdisk never explicitly asked for a name).

Upon creating this new array I was able to mount manually using

mount -t ext4 /dev/md2p1 /srv/Waveforms

And this worked fine. I was able to access the directory and added about 700 GB worth of data to it. After doing this, I get

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md2 : active raid1 sdc1[0] sdd1[1]
      1953423552 blocks [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
      961136576 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      15624128 blocks [2/2] [UU]

unused devices: <none>

, so the computer clearly recognizes the array. I then used

sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf 

and the file now contains

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=afa7ccee:df4dfa79:a84dbc05:35401226
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=a0c526cc:6de93504:c8b94301:85474c49
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=1ac720e1:192b2c38:f6e0357b:f3e0074f

# This file was auto-generated on Thu, 10 Mar 2011 00:57:55 -0700
# by mkconf $Id$
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=afa7ccee:df4dfa79:a84dbc05:35401226
ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=a0c526cc:6de93504:c8b94301:85474c49
ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=1ac720e1:192b2c38:f6e0357b:f3e0074f

Two things to note here. The original file had metadata=00.90, but I modified this to be metadata=0.90 as this solved a minor issue where the metadata was not recognized (a quick google search will explain this).

The second think to note is that auto is set to yes, meaning that the system should automatically assemble all the arrays on boot. This must be the case, as the fact that I am able to boot at all must mean that /dev/md1 has been assembled.

Anyways, now the trouble. Upon reboot, my machine hangs and tells me

fsck from util-linux-ng 2.17.2
/dev/md1: clean, 3680768/60071936 files, 208210802/240284144 blocks

My fstab currently reads

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/md1 during installation
UUID=1d3cb392-f522-485b-8516-a7791fc23c4d /               ext4    errors=remount-ro 0       1
# swap was on /dev/md0 during installation
UUID=6eb8e6f2-3166-4f77-883c-26268d636b0b none            swap    sw              0       0
/dev/md2p1 /srv/Waveforms            ext4    defaults,noauto              0       0

with

blkid
/dev/sda1: UUID="afa7ccee-df4d-fa79-a84d-bc0535401226" TYPE="linux_raid_member" 
/dev/sda2: UUID="a0c526cc-6de9-3504-c8b9-430185474c49" TYPE="linux_raid_member" 
/dev/sdb1: UUID="afa7ccee-df4d-fa79-a84d-bc0535401226" TYPE="linux_raid_member" 
/dev/sdb2: UUID="a0c526cc-6de9-3504-c8b9-430185474c49" TYPE="linux_raid_member" 
/dev/sdc1: UUID="1ac720e1-192b-2c38-f6e0-357bf3e0074f" TYPE="linux_raid_member" 
/dev/sdd1: UUID="1ac720e1-192b-2c38-f6e0-357bf3e0074f" TYPE="linux_raid_member" 
/dev/md0: UUID="6eb8e6f2-3166-4f77-883c-26268d636b0b" TYPE="swap" 
/dev/md1: UUID="1d3cb392-f522-485b-8516-a7791fc23c4d" TYPE="ext4" 
/dev/md2p1: UUID="867ee91e-527e-435b-b6bc-2f6d89d2d8c6" TYPE="ext4" 

I had previously used UUID=867ee91e-527e-435b-b6bc-2f6d89d2d8c6 in lieu of /dev/md2p1, but that gave me no results. I have also tried options as defaults, defaults+noatime,errors=remount-ro with this md2p1, but none worked.

I am able to boot by modifying fstab to exclude my new md2p1 line. After boot with this configuration, I get

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[0] sdb1[1]
      15624128 blocks [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
      961136576 blocks [2/2] [UU]

So the system has not assembled md2. I can then run

sudo mdadm --assemble --scan
[sudo] password for zach: 
mdadm: /dev/md2 has been started with 2 drives.

whence

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md2 : active raid1 sdc1[0] sdd1[1]
      1953423552 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      15624128 blocks [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
      961136576 blocks [2/2] [UU]

and I can now manually mount as above. So the problem seems to be that the RAID1 array is not assembling on boot, and thus is not recognized by fstab, and thus I cannot boot at all except in a recovery mode.

I've found this post, but I don't think it applies to me, as the answer seems to be to set auto to yes, which then automatically assembles the arrays on boot. My configuration is already set to do this, so I'm at a loss.

There is an answer in that post that does seem applicable, but I don't understand what his solution was. This is the post by Deplicator, which says

After a reboot I could never see /dev/md0. Running the mdadm –detail
–scan again (without putting the result in a file) I would see

ARRAY /dev/md/ubuntu:0 metadata=1.2 name=ubuntu:0 UUID=a8a570c6:96f61865:05abe131:5c2e2f7e

and manually mounting /dev/md/ubuntu:0 would work. In the end, that
was what I put in the fstab file too.

What was put into fstab? The problem seems to be that I am not assembling md2 on boot, and thus I will hang every time fstab tries to mount the md2p1 partition. This may in fact be related to md2 being partitioned while the others are not, but I fail to see why this should be the case.

Edit: Just in case

uname -a
Linux ccldas2 2.6.32-74-server #142-Ubuntu SMP Tue Apr 28 10:12:19 UTC 2015 x86_64 GNU/Linux