Centos – LVM volume is inactive after reboot of CentOS

centoslvmrhel

I've reinstalled a Linux server from CentOS 6 to 7. The server has 3 drives – a system SSD drive (it hosts everything except /home) and two 4TB HDD drives that host /home. Everything uses LVM. The two 4TB drives are mirrored (using the raid option within LVM itself), and they are completely filled with the /home partition.

The problem is that although the 4TB disks are recognized fine, and LVM sees the volume without problems, it does not activate it automatically. Everything else is activated automatically. I can activate it manually, and it works.

I have an image of the old system drive in /home. That contains LVM volumes too. If I mount it with kpartx, and LVM picks those up and activates them. But I can see no difference between those volumes and the inactive ones.

The root filesystem is LVM too, and that activates just fine.

I see a peculiar thing though: executing lvchange -aay tells me that I need to specify which drives I want to activate. It doesn't do it automatically either. If I specify lvchange -ay lv_home – that works.

I cannot find anything that could be responsible for this behavior.

Added: I noticed that the old system (which used init) had vgchange -aay --sysinit in its startup scripts. The new one uses systemd, and I don't see the vgchange call in its scripts. But I also don't know where to put it.

Added 2: Starting to figure out systemd. I found where the scripts are located and started understanding how they are called. Also found that I could see the executed scripts with systemctl -al. This shows me that after starting lvmetad it calls pvscan for each known udev block device. However at that point there is just one registered udev block device, and that is one of the recognized lvm volumes. The hard drives are there too, but under different paths and much longer names. The recognized block device is something like 8:3, while the hard drives are like /device/something/. I'm not at the server anymore, so I cannot write it precisely (will fix this later).

I think that it has something to do with udev and device detection/mapping. I will continue in the evening and will study udev then.

If all else fails, I found the script that calls pvscan and checked that I can modify it to scan all the devices all the time. That fixes the problem, but it looks like a rather ugly hack, so I'll try to figure out the real root cause.

Added 3: OK, I still don't know why this happens, but at least I've made a fairly passable workaround. I made another systemd service that calls the pvscan once, right after starting lvmetad. The other call for the specific device is still there, and I think it's actually udev that calls it (that's the only place where I found reference to it). Why it doesn't call it for the other hard drives – I have no idea.

Best Answer

I did it! I did it! I fixed it properly (I think).

Here's the story:

After some time the server turned out to be faulty and had to be scrapped. I kept disks and got everything else new. Then I reinstalled CentOS again on the SSD and then I attached the HDDs. LVM worked nicely, the disks were recognized, the configuration kept. But the same problem came up again - after a reboot, the volume was inactive.

However this time I chanced to notice something else - the bootloader passes the following parameters to the kernel:

crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

Hmm, wait a minute, those look FAMILIAR!

Quick google query, and there we are:

rd.lvm.lv=

only activate the logical volumes with the given name. rd.lvm.lv can be specified multiple times on the kernel command line.

Well now. THAT explains it!

So, the resolution was (gathered from several more google queries):

Modify /etc/defaults/grub to include the additional volume in the parameters: crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rd.lvm.lv=vg_home/lv_home rhgb quiet
Reconfigure grub with grub2-mkconfig -o /boot/grub2/grub.cfg
Reconfigure initramfs with mkinitrd -f -v /boot/initramfs-3.10.0-327.18.2.el7.x86_64.img 3.10.0-327.18.2.el7.x86_64. Note: your values may vary. Use uname -r to get that kernel version. Or just read up on mkinitrd. (Frankly, I don't know why this step is needed, but apparently it is - I tried without it and it didn't work)
And finally, reinstall grub: grub2-install /dev/sda
Reboot, naturally.

TA-DA! The volume is active on reboot. Add it to fstab and enjoy! :)

Related Solutions

Linux – LVM: PV missing after reboot

Does the LV become mountable if you do a sudo vgscan and sudo vgchange -ay? If those commands result in errors, you probably have a different problem and should probably add those error messages in your original post.

But if the LV becomes ready for mounting after those commands, read on...

The LVM logical volume pathname (e.g. /dev/mapper/vgNAME-lvNAME) in /etc/fstab alone won't give the system a clue that this particular filesystem cannot be mounted until networking and iSCSI have been activated.

Without that clue, the system will assume that filesystem is on a local disk and will attempt to mount it as early as possible, normally before networking has been activated, which will obviously fail with an iSCSI LUN. So you'll need to supply that clue somehow.

One way would be to add _netdev to the mount options for that filesystem in /etc/fstab. From this Ubuntu help page it appears to be supported on Ubuntu. This might actually also trigger a vgscan or similar detection of new LVM PVs (+ possibly other helpful stuff) just before the attempt to mount any filesystems marked with _netdev.

Another way would be to use the systemd-specific mount option x-systemd.requires=<iSCSI initiator unit name>. That should achieve the same thing, by postponing any attempts to mount that filesystem until the iSCSI initiator has been successfully activated.

When the iSCSI initiator activates, it will automatically make any configured LUNs available, and as they become available, LVM should auto-activate any VGs on them. So, once you get the mount attempt postponed, that should be enough.

The lack of PARTUUID is a clue that the disk/LUN does not have a GPT partition table. Since /dev/sdc is listed as TYPE="LVM2_member" it actually does not have any partition table at all. In theory, it should cause no problems for Linux, but I haven't personally tested an Ubuntu 18.04 system with iSCSI storage, so cannot be absolutely certain.

The problem with disks/LUNs with no partition table is that other operating systems won't recognize the Linux LVM header as a sign that the disk is in use, and will happily overwrite it with minimal prompting. If your iSCSI storage administrator has accidentally presented the storage LUN corresponding to your /dev/sdc to another system, this might have happened.

You should find the LVM configuration backup file in /etc/lvm/backup directory that corresponds to your missing VG, and read it to find the expected UUID of the missing PV. If it matches what blkid reports, ask your storage administrator to double-check his/her recent work for mistakes like described above. If it turns out the PV has been overwritten by some other system, any remaining data on the LUN is likely to be more or less corrupted and it would be best to restore it from backup... once you get a new, guaranteed-unconflicted LUN from your iSCSI admin.

If it turns out the actual UUID of /dev/sdc is different from expected, someone might have accidentally run a pvcreate -f /dev/sdc somehow. If that's the only thing that has been done, that's relatively easy to fix. (NOTE: check man vgcfgrestore chapter REPLACING PHYSICAL VOLUMES for updated instructions - your LVM tools may be newer than mine.) First restore the UUID:

pvcreate --restorefile /etc/lvm/backup/<your VG backup file> --uuid <the old UUID of /dev/sdc from the backup file> /dev/sdc

Then restore the VG configuration:

vgcfgrestore --file /etc/lvm/backup/<your VG backup file> <name of the missing VG>

After this, it should be possible to activate the VG, and if no other damage has been done, mount the filesystem after that.

CentOS – Best Practices for Running Filesystem Check on LVM Volume

How to run fsck against a LVM filesystem?

It's a good idea to make a snapshot or backup of the drive first.
1. and 2. depending which drive to fix.
Booting to emergency mode by adding systemd.unit=emergency.target to grub line is a good start. / is still mounted.

If fsck is the way, what switches does one pass to it?

You could check the root partition with a dry run (fsck -N) to see if errors are present. You'd need to boot from an external device to actually fix this one. (fsck -y is useful if there is a lot to fix, skips the annoying "Fix (y)?" question on each error found.)

Is there something built in to lvm to automatically check the fs?
(What is the best practice set of commands to make certain that filesystems are checked when they reside on LVM?)

No. AFAIK /etc/fstab - 6th field is setting this. (see man fstab)

The sixth field, (fs_passno), is used by the fsck(8) program to determine the order in which filesystem checks are done at reboot time. The root filesystem should be specified with a fs_passno of 1, and other filesystems should have a fs_passno of 2. Filesystems within a drive will be checked sequentially, but filesystems on different drives will be checked at the same time to utilize parallelism available in the hardware. If the sixth field is not present or zero, a value of zero is returned and fsck will assume that the filesystem does not need to be checked.

If so, which logfile entry actually proves a check was done?

This is stored to the filesystem. For ext2/ext3/ext4 it could look like this (fsck in 13, mount count will be set to 0. tune2fs -C <n> set mount count manually)

sudo tune2fs -l /dev/mapper/vg0-lv0 | grep -i "mount count" 

Mount count:              17
Maximum mount count:      30

(Other filesystems eg. XFS, btrfs work differently but you get the idea)

What lvm commands should be run before running the fsck?

none

Best Answer

Related Solutions

Linux – LVM: PV missing after reboot

CentOS – Best Practices for Running Filesystem Check on LVM Volume

Related Question