LVM Data Recovery – Is pvcreate Destructive When Recovering an LVM2 Volume Group

data-recoveryjfslinuxlvm

I don't want to lose any data, so I'm asking if the solution Novell offers is safe to try. If not, are there safe alternatives?

Step-by-tedious-step:

I have two volume groups: main and Rand; Rand is what I boot from while main is an older group.
main/home stopped mounting due to a possible bad superblock. (this error)
I found this Novell link suggesting vgcfgrestore yesterday, so I tried it. No bueno.
I then ran fsck.jfs on /dev/main/home, which allowed it to mount. Success!
This morning, I see errors. df -h shows /dev/mapper/Rand-root has 0 bytes free. Deleting a debian .iso–and more– fails to change that. (20+ gigs were free yesterday.)
vgscan, pvscan–a lot of utilities fail to work due to a "disk full" error.
I reboot. df -h still reports 0 bytes free, but vgscan and pvscan work now.
Something one of those utilities returned led me to try vgcfgrestore Rand. No change in df -h and now main/home (mounted at /mnt/10.10/) starts spewing I/O errors.
Reboot. A BIOS/SMART error on a disk along with a pvscan error saying can't find device with uuid="uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82", which blkid identifies as /dev/sdb5.

fdisk -l shows:

Disk /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00039f8a

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          32      248832   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sdb2              32       14594   116969473    5  Extended
/dev/sdb5              32       14594   116969472   8e  Linux LVM
Note: sector size is 4096 (not 512)

The Novell link says my problem fits Symptom 2 and provides a solution, but says nothing about fdisk reporting a partition error. (Which may be causing the SMART error.)

The Novell solution says to first identify the device, then run pvcreate with the UUID and device as parameters, then vgcfgrestore, vgscan, vgchange -ay, and fsck.

If I try this, is there a chance pvcreate will damage anything?

Also, for the pvcreate command, should I use /dev/sdb or /dev/sdb5 as the device?

Output:

~ » sudo vgscan                                                                                                                                                                                                                                                                                                   steven@Rand
[sudo] password for steven: 
  Reading all physical volumes.  This may take a while...
  Found volume group "Rand" using metadata type lvm2
  Couldn't find device with uuid uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82.
  Found volume group "main" using metadata type lvm2
------------------------------------------------------------
~ » sudo pvscan                                                                                                                                                                                                                                                                                                   steven@Rand
  Couldn't find device with uuid uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82.
  PV /dev/sdb5        VG Rand   lvm2 [111.55 GiB / 0    free]
  PV unknown device   VG main   lvm2 [1.36 TiB / 0    free]
  PV /dev/sda1        VG main   lvm2 [465.76 GiB / 461.76 GiB free]
  Total: 3 [1.93 TiB] / in use: 3 [1.93 TiB] / in no VG: 0 [0   ]
------------------------------------------------------------
~ » blkid                                                                                                                                                                                                                                                                                                         steven@Rand
/dev/sdb1: UUID="ba9a3955-0b9c-4660-9852-0f9f405d2f8e" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sdb5: UUID="uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82" TYPE="LVM2_member" 
/dev/sde1: LABEL="My Book" UUID="A2CA0AEBCA0ABC13" TYPE="ntfs" 
/dev/sdf1: UUID="5F8C6ED4773C3763" TYPE="ntfs" 
------------------------------------------------------------

~ » sudo lvs

  Couldn't find device with uuid uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82.
  LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  root   Rand -wi-ao 106.98g                                      
  swap_1 Rand -wi-ao   4.56g                                      
  home   main -wi---   1.35t                                      
  root   main -wi---   2.00g                                      
  swap   main -wi---   4.00g                                      
  tmp    main -wi--- 512.00m                                      
  usr    main -wi---   6.00g                                      
  var    main -wi---   2.00g                                      
------------------------------------------------------------
~ » sudo lvscan
  ACTIVE            '/dev/Rand/root' [106.98 GiB] inherit
  ACTIVE            '/dev/Rand/swap_1' [4.56 GiB] inherit
  Couldn't find device with uuid uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82.
  inactive          '/dev/main/swap' [4.00 GiB] inherit
  inactive          '/dev/main/root' [2.00 GiB] inherit
  inactive          '/dev/main/usr' [6.00 GiB] inherit
  inactive          '/dev/main/var' [2.00 GiB] inherit
  inactive          '/dev/main/tmp' [512.00 MiB] inherit
  inactive          '/dev/main/home' [1.35 TiB] inherit
------------------------------------------------------------
~ » sudo pvs
  Couldn't find device with uuid uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82.
  PV             VG   Fmt  Attr PSize   PFree  
  /dev/sda1      main lvm2 a-   465.76g 461.76g
  /dev/sdb5      Rand lvm2 a-   111.55g      0 
  unknown device main lvm2 a-     1.36t      0 
------------------------------------------------------------
~ » blkid
/dev/sdb1: UUID="ba9a3955-0b9c-4660-9852-0f9f405d2f8e" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sdb5: UUID="uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82" TYPE="LVM2_member" 
/dev/sde1: LABEL="My Book" UUID="A2CA0AEBCA0ABC13" TYPE="ntfs" 
/dev/sdf1: UUID="5F8C6ED4773C3763" TYPE="ntfs" 
------------------------------------------------------------
~ » sudo vgs
  Couldn't find device with uuid uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82.
  VG   #PV #LV #SN Attr   VSize   VFree  
  Rand   1   2   0 wz--n- 111.55g      0 
  main   2   6   0 wz-pn-   1.82t 461.76g
------------------------------------------------------------
~ » sudo vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "Rand" using metadata type lvm2
  Couldn't find device with uuid uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82.
  Found volume group "main" using metadata type lvm2
------------------------------------------------------------
~ » sudo pvscan
  Couldn't find device with uuid uZ1fiS-5Wo4-VNzC-gzs0-ekVz-Bepn-1MZe82.
  PV /dev/sdb5        VG Rand   lvm2 [111.55 GiB / 0    free]
  PV unknown device   VG main   lvm2 [1.36 TiB / 0    free]
  PV /dev/sda1        VG main   lvm2 [465.76 GiB / 461.76 GiB free]
  Total: 3 [1.93 TiB] / in use: 3 [1.93 TiB] / in no VG: 0 [0   ]
------------------------------------------------------------

Best Answer

pvcreate writes PV metadata onto the device/partition, I think most would call that "destructive" however, since it's part of the LVM planning and layout it's also "constructive". pvcreate could be destructive to data areas if any of the following parameters were changed to increase the metadata size or location.

Depending on the command line options passed to pvcreate, one can
write multiple copies of the metadata via --[pv]metadatacopies
change the metadata size via --metadatasize
change the data alignment via --dataalignment
shift the start of the data area an additional alignment_offset via --dataalignmentoffset
recreate a previous PV by specifying the UUID --uuid

Also from the pvcreate man page.
To see the location of the first Physical Extent of an existing Physical Volume use pvs -o +pe_start

Typically, the metadata is written in the first few blocks of the device, up to the first usable PE, shown by pvs -o +pe_start and can be partially viewed by the following cmd.

dd if=/dev/sdb5 bs=4096 count=4 |less

Using less instead of od because much of the metadata is clear text and less does a good job of handling both binary and text data.

Personally, I always set metadata copies to be more than one.

Related Solutions

LVM2 to RAID5 Migration – How to Migrate an Entire Volume Group

First, you create the raid array. Assuming the new drives are sdc, sdd, and sde, and you don't already have any raid arrays, and you have created a single raid partition on each, do:

sudo -s
mdadm --create /dev/md0 -n 3 -l raid5 /dev/sd[cde]1
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Then you add it to the vg, move the logical volumes over, and remove the existing pvs:

pvcreate /dev/md0
vgextend vg0 /dev/md0
pvmove -i 10 /dev/sda2 /dev/md0
pvmove -i 10 /dev/sdb1 /dev/md0
vgreduce vg0 /dev/sda2 /dev/sdb1

Now you will need to transfer your /boot partition, rebuild your initramfs, and reinstall grub to get the system able to boot from the new disks:

mount --move /boot /mnt
rmdir /boot
cp -a /mnt /boot
umount /mnt
update-initramfs -u
dpkg-reconfigure grub-pc

A menu will ask which disks grub should be installed to. Select sdc, sdd, and sde. Now you can shutdown and remove the old disks.

Linux – LVM: PV missing after reboot

Does the LV become mountable if you do a sudo vgscan and sudo vgchange -ay? If those commands result in errors, you probably have a different problem and should probably add those error messages in your original post.

But if the LV becomes ready for mounting after those commands, read on...

The LVM logical volume pathname (e.g. /dev/mapper/vgNAME-lvNAME) in /etc/fstab alone won't give the system a clue that this particular filesystem cannot be mounted until networking and iSCSI have been activated.

Without that clue, the system will assume that filesystem is on a local disk and will attempt to mount it as early as possible, normally before networking has been activated, which will obviously fail with an iSCSI LUN. So you'll need to supply that clue somehow.

One way would be to add _netdev to the mount options for that filesystem in /etc/fstab. From this Ubuntu help page it appears to be supported on Ubuntu. This might actually also trigger a vgscan or similar detection of new LVM PVs (+ possibly other helpful stuff) just before the attempt to mount any filesystems marked with _netdev.

Another way would be to use the systemd-specific mount option x-systemd.requires=<iSCSI initiator unit name>. That should achieve the same thing, by postponing any attempts to mount that filesystem until the iSCSI initiator has been successfully activated.

When the iSCSI initiator activates, it will automatically make any configured LUNs available, and as they become available, LVM should auto-activate any VGs on them. So, once you get the mount attempt postponed, that should be enough.

The lack of PARTUUID is a clue that the disk/LUN does not have a GPT partition table. Since /dev/sdc is listed as TYPE="LVM2_member" it actually does not have any partition table at all. In theory, it should cause no problems for Linux, but I haven't personally tested an Ubuntu 18.04 system with iSCSI storage, so cannot be absolutely certain.

The problem with disks/LUNs with no partition table is that other operating systems won't recognize the Linux LVM header as a sign that the disk is in use, and will happily overwrite it with minimal prompting. If your iSCSI storage administrator has accidentally presented the storage LUN corresponding to your /dev/sdc to another system, this might have happened.

You should find the LVM configuration backup file in /etc/lvm/backup directory that corresponds to your missing VG, and read it to find the expected UUID of the missing PV. If it matches what blkid reports, ask your storage administrator to double-check his/her recent work for mistakes like described above. If it turns out the PV has been overwritten by some other system, any remaining data on the LUN is likely to be more or less corrupted and it would be best to restore it from backup... once you get a new, guaranteed-unconflicted LUN from your iSCSI admin.

If it turns out the actual UUID of /dev/sdc is different from expected, someone might have accidentally run a pvcreate -f /dev/sdc somehow. If that's the only thing that has been done, that's relatively easy to fix. (NOTE: check man vgcfgrestore chapter REPLACING PHYSICAL VOLUMES for updated instructions - your LVM tools may be newer than mine.) First restore the UUID:

pvcreate --restorefile /etc/lvm/backup/<your VG backup file> --uuid <the old UUID of /dev/sdc from the backup file> /dev/sdc

Then restore the VG configuration:

vgcfgrestore --file /etc/lvm/backup/<your VG backup file> <name of the missing VG>

After this, it should be possible to activate the VG, and if no other damage has been done, mount the filesystem after that.

Best Answer

Related Solutions

LVM2 to RAID5 Migration – How to Migrate an Entire Volume Group

Linux – LVM: PV missing after reboot

Related Question