AppleRAID 0+1 failure: [corestorage] LVG missing and cannot repair mirror

disk-utility

I just recently created a corestorage LVG and an encrypted volume that used an AppleRAID 0+1 array, but something has gone horribly wrong.

I had four 1TB disks, and created two 2TB stripes, cbs1 and cbs2, and mirrored them in Disk Utility. Then I created a [corestorage] LVG and fed it the UUID of the mirrored device, and created an encrypted jHFS+ filesystem at 100% of the capacity of the 2TB mirror.

This worked great, and behaved nicely for over a week. Today I had a software update that resulted in a reboot, and my RAID didn't show up. The LVG I created using the AppleRAID is no longer showing up in diskutil cs list and the mirror was set to auto repair and was making progress (it looked like cbs1 vanished but it was still there, so I thought it was an intermittent hardware issue on boot) and I let it make progress on the rebuild until it was complete, but it didn't end up the way I thought it would.

I now saw in diskutil ar list that I had a missing or damaged mirror device, and so in an effort to be smart, I created another stripe of two other 1TB disks, called it cbs3 and attempted to add that as a spare or a member to the UUID of the mirror. HOWEVER, this fails with an error of -69848: Couldn't modify RAID set in both cases.

diskutil ar list now gives me the following:

AppleRAID sets (4 found)
===============================================================================
Name:                 cbs1
Unique ID:            1E127495-1FDE-4521-A941-5E5DC31245DD
Type:                 Stripe
Status:               Online
Size:                 2.0 TB (1999721791488 Bytes)
Rebuild:              manual
Device Node:          -
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk6s2   AA501823-4955-473B-A27E-0EEA9E75072A  Online     999860895744
1  disk7s2   A6486548-3E07-4C0F-8F88-BE909D23DC22  Online     999860895744
===============================================================================
===============================================================================
Name:                 Untitled
Unique ID:            1E127495-1FDE-4521-A941-5E5DC31245DD
Type:                 Mirror
Status:               Online
Size:                 2.0 TB (1999721758720 Bytes)
Rebuild:              automatic
Device Node:          -
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
-  -none-    1E127495-1FDE-4521-A941-5E5DC31245DD  Missing/Damaged
1  -none-    2B52C67D-801E-4481-A76B-F98FF28C13B5  Online     1999721758720
===============================================================================
===============================================================================
Name:                 cbs2
Unique ID:            2B52C67D-801E-4481-A76B-F98FF28C13B5
Type:                 Stripe
Status:               Online
Size:                 2.0 TB (1999721791488 Bytes)
Rebuild:              manual
Device Node:          -
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk8s2   890A3BB9-94CB-4E02-8580-5F16D94890FF  Online     999860895744
1  disk10s2  71C4DE7E-6EC2-4660-9CDA-0ED31574925D  Online     999860895744
===============================================================================
===============================================================================
Name:                 cbs3
Unique ID:            A153709E-88C8-4281-8268-F68C26077E6A
Type:                 Stripe
Status:               Online
Size:                 2.0 TB (1999721791488 Bytes)
Rebuild:              manual
Device Node:          disk14
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk2s2   C1227A8B-8215-4E7E-99C8-C8491B7D994F  Online     999860895744
1  disk4s2   A65356C3-270C-404B-82AC-9EAB042B9300  Online     999860895744
===============================================================================

This is doubly confusing because there is no DevNode for the mirror'ed volume, probably due to it previously being fed to corestorage. The mirror device doesn't show up in Disk Utility.app, but I see it as above in the CLI.

Flailing a little, I tried to repair the mirror using both UUIDs to no avail:

lindsay:~$ sudo diskutil ar repairmirror 1E127495-1FDE-4521-A941-5E5DC31245DD /dev/disk14
Password:
Started RAID operation
Can't add spare to the RAID set
Error: -69848: Couldn't modify RAID
lindsay:~$ sudo diskutil ar repairmirror 2B52C67D-801E-4481-A76B-F98FF28C13B5 /dev/disk14
Started RAID operation
Can't add spare to the RAID set
Error: -69848: Couldn't modify RAID 

Uh oh!

Is it possible to import the corestorage LVG that I created against the mirror'ed device while the mirror is degraded?

Best Answer

Two sets use the same UUID!

Stripe-1
Name:                 cbs1
Unique ID:            1E127495-1FDE-4521-A941-5E5DC31245DD

Stripe-2
Name:                 cbs2
Unique ID:            2B52C67D-801E-4481-A76B-F98FF28C13B5

Create a mirror
Name:                 Untitled
Unique ID:            1E127495-1FDE-4521-A941-5E5DC31245DD

That mirror has the same UUID as Stripe-1. You have defined a loop.

Maybe that's the source of your problem. You need a disktool to directly modify the contents of your disks to create unique UUIDs.