If I understood correctly, you have already fixed the volume, even though you have a lost+found
directory which may or may not have critical files.
What is going on now that's blocking the VM from booting? It still can't find the boot device?
Your fdisk -l
output seems a bit off to me. Have you considered the possibility that only the partition table was damaged? In this scenario, your snapshot may be helpful, and in the best case you won't even need a(nother) fsck. But we'll need something to try to find the partition offsets - I've used testdisk successfully more than once.
In the worst case scenario, if you need to scrape anything from the volume, forensic tools like PhotoRec or Autopsy/The Sleuth Kit may prove useful.
If none of this works, give us a lsblk -o NAME,RM,SIZE,RO,TYPE,MAJ:MIN -fat
too (these flags are just to show as much information as possible), and relevant dmesg
output, if any.
Does the LV become mountable if you do a sudo vgscan
and sudo vgchange -ay
? If those commands result in errors, you probably have a different problem and should probably add those error messages in your original post.
But if the LV becomes ready for mounting after those commands, read on...
The LVM logical volume pathname (e.g. /dev/mapper/vgNAME-lvNAME
) in /etc/fstab
alone won't give the system a clue that this particular filesystem cannot be mounted until networking and iSCSI have been activated.
Without that clue, the system will assume that filesystem is on a local disk and will attempt to mount it as early as possible, normally before networking has been activated, which will obviously fail with an iSCSI LUN. So you'll need to supply that clue somehow.
One way would be to add _netdev
to the mount options for that filesystem in /etc/fstab
. From this Ubuntu help page it appears to be supported on Ubuntu. This might actually also trigger a vgscan
or similar detection of new LVM PVs (+ possibly other helpful stuff) just before the attempt to mount any filesystems marked with _netdev
.
Another way would be to use the systemd-specific mount option x-systemd.requires=<iSCSI initiator unit name>
. That should achieve the same thing, by postponing any attempts to mount that filesystem until the iSCSI initiator has been successfully activated.
When the iSCSI initiator activates, it will automatically make any configured LUNs available, and as they become available, LVM should auto-activate any VGs on them. So, once you get the mount attempt postponed, that should be enough.
The lack of PARTUUID is a clue that the disk/LUN does not have a GPT partition table. Since /dev/sdc
is listed as TYPE="LVM2_member"
it actually does not have any partition table at all. In theory, it should cause no problems for Linux, but I haven't personally tested an Ubuntu 18.04 system with iSCSI storage, so cannot be absolutely certain.
The problem with disks/LUNs with no partition table is that other operating systems won't recognize the Linux LVM header as a sign that the disk is in use, and will happily overwrite it with minimal prompting. If your iSCSI storage administrator has accidentally presented the storage LUN corresponding to your /dev/sdc
to another system, this might have happened.
You should find the LVM configuration backup file in /etc/lvm/backup
directory that corresponds to your missing VG, and read it to find the expected UUID of the missing PV. If it matches what blkid
reports, ask your storage administrator to double-check his/her recent work for mistakes like described above. If it turns out the PV has been overwritten by some other system, any remaining data on the LUN is likely to be more or less corrupted and it would be best to restore it from backup... once you get a new, guaranteed-unconflicted LUN from your iSCSI admin.
If it turns out the actual UUID of /dev/sdc
is different from expected, someone might have accidentally run a pvcreate -f /dev/sdc
somehow. If that's the only thing that has been done, that's relatively easy to fix. (NOTE: check man vgcfgrestore
chapter REPLACING PHYSICAL VOLUMES for updated instructions - your LVM tools may be newer than mine.) First restore the UUID:
pvcreate --restorefile /etc/lvm/backup/<your VG backup file> --uuid <the old UUID of /dev/sdc from the backup file> /dev/sdc
Then restore the VG configuration:
vgcfgrestore --file /etc/lvm/backup/<your VG backup file> <name of the missing VG>
After this, it should be possible to activate the VG, and if no other damage has been done, mount the filesystem after that.
Best Answer
I have found a convenient way to do this: two SystemD services:
/mnt/systemd/system/loops-setup.service
/mnt/systemd/system/loops-fsck.service