Linux – Why are there two different values for the size of a drive and how to reconcile them with device-mapper and LVM

device-mapperhard-diskhdparmlinuxlvm

I have a 1 TB drive attached via USB. It contains an LVM physical volume filling the entire device (without any partition table). When I tried to extend a logical volume by using the entire PV, the device-mapper started to complain that the section allocated by LVM on the PV is larger than the device. The error message from the device-mapper (as shown by dmesg) reports a size of 1953320367 [dm] sectors:

device-mapper: table: 254:0: sdf too small for target: start=1821353984, len=132169728, dev_size=1953320367

But LVM created a PV with 238467 physical extents, which is 1953521664 [lvm] sectors (which is about 100 MB more):

$ pvdisplay /dev/sdf
  --- Physical volume ---
  PV Name               /dev/sdf
  VG Name               apu-vg1
  PV Size               931.51 GiB / not usable 1.71 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              238467
  Free PE               100
  Allocated PE          238367
  PV UUID               LrKDDW-4dXz-kDgh-CK78-OWhY-4sCH-rKT0e4

Now if I run hdparm -gI on the device, I can see two values for the size of the device. Under geometry, there's the same value that is reported by the device-mapper as the device's size. But under LBA48 user addressable sectors, there's the value 1953525168 [lba], which is less than one PE larger than the combines size of the PEs of the PV. This makes me think that this is the value that LVM sees:

$ hdparm -Ig /dev/sdf

/dev/sdf:
 geometry      = 121588/255/63, sectors = 1953320367, start = 0

ATA device, with non-removable media
          Model Number:       ST1000LM024 HN-M101MBB                  
          Serial Number:      S2RUJ9BC702524      
          Firmware Revision:  2AR10001
          Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
          Used: unknown (minor revision code 0x0028) 
          Supported: 8 7 6 5 
          Likely used: 8
Configuration:
          Logical         max     current
          cylinders       16383   16383
          heads           16      16
          sectors/track   63      63
          --
          CHS current addressable sectors:   16514064
          LBA    user addressable sectors:  268435455
          LBA48  user addressable sectors: 1953525168
...

Now to my questions:

Why are there two different values for the device's size used by different parts of the kernel?
And, how do I handle/fix this situation so that the LVM will not make PVs that are larger than the space the device-mapper is willing/able to access?

Best Answer

The key problem here is your Free PE value. Notice how it says that only 100 extents are free? That means that you won't be able to extend it. If you do a vgscan, pvscan, and then a vgdisplay (also vgs) you will probably see that your device is already part of a vg (and likely lv(s) as well), which is why the extents are not free.

Could you provide the output from:

vgscan;pvscan;vgdisplay;vgs

you should be able to get an additional 400MB based on the size of your extents and how many are free.

Related Solutions

How to shrink a physical volume

If you just need another separate filesystem, you don't have to create a new partition: you can create another LVM logical volume within the existing volume group using the unallocated space in the PV. But if you specifically need a new partition, then you must do as you originally planned.

GParted is a GUI tool that can do the entire PV + partition resizing operation for you. You'll most likely find it in your distribution's package collection.

But if you cannot or don't want to use it for some reason, read on...

Reducing a LVM physical volume that has unallocated space in it is pretty simple. For example, to reduce the size of /dev/sda1 to 40 GiB:

# pvresize --setphysicalvolumesize 40G /dev/sda1

This command is safe, as it will refuse to do anything if the part of the PV it would cut away contains allocated extents.

But that's just the first step. The second step will be to reduce the size of the partition itself.

To be safe, it is recommended that you initially shrink the PV a bit more than you plan to actually shrink the partition, so that after shrinking the partition you can explicitly extend the PV to cover the full size of the partition. If the extend operation fails, you'll immediately know that something went wrong in resizing the partition and won't be leaving hidden errors to trap you or someone else in the future.

You can shrink a partition using a plain Linux fdisk command, but you must keep in mind a few things:

When using fdisk to resize a partition, make sure to set it to display all partition sizes and starting locations in sectors first. During the history of Linux fdisk, its behavior has varied here: modern versions do it by default, older versions may need a -u or -u=sectors option to do so.
When resizing an existing partition, you cannot move the starting point of the partition. (If you move the starting point, it will no longer be a simple resize operation, but a more complex partition move operation.)

When using fdisk, any changes are written to the disk only when you use the w command. So I would suggest proceeding like this:

use sfdisk -d /dev/sda > /safe/location/partition_table_backup.dump to backup the existing partition table.
start fdisk /dev/sda in interactive mode
use the p command to view the current partition table and verify the "start" and "end" values are displayed as sectors. Make a note of the "start" value.
use the d command to delete the current partition entry for sda1.
use the n command to recreate the primary partition entry 1 for sda1 in desired size, taking care to specify the location of the first sector exactly as it was (i.e. the same number as was displayed in the "start" value of the p command).
use the t command to set the type of the partition 1 to "Linux LVM".
double-check your work
use the w command to write the updated partition table and exit fdisk.

Pay attention to the message displayed by fdiskat the end. If it says "The kernel still uses the old table", something is blocking the on-line partition table update. You can try partprobe /dev/sda to tell the kernel to reread the partition table (using a newer interface than some older versions of fdisk use, so it might help and certainly won't hurt) and then look at cat /proc/partitions to see if the partition is now recognized by the kernel at the expected size.

If the partition resize is successful, you can now run pvresize /dev/sda1. If you followed my advice and resized the PV initially a bit smaller than the desired new size of the partition, it should now successfully extend the PV a bit to fully cover the resized partition automatically. If this command reports an error, you've probablu made a mistake in specifying the new size and should use sfdisk /dev/sda < /safe/location/partition_table_backup.dump to restore the partition table to what it initially was, and then try again.

(On very old systems, you might actually have to deactivate the VG before running partprobe or even reboot to allow the system re-read the updated partition table.)

Once the resizing operation is complete, you can now use fdisk again to create the second partition you needed. Yes, you could also do it together while resizing the partition, but doing just one thing at a time is safer and ensures you can easily roll back your changes if needed.

Linux – LVM: PV missing after reboot

Does the LV become mountable if you do a sudo vgscan and sudo vgchange -ay? If those commands result in errors, you probably have a different problem and should probably add those error messages in your original post.

But if the LV becomes ready for mounting after those commands, read on...

The LVM logical volume pathname (e.g. /dev/mapper/vgNAME-lvNAME) in /etc/fstab alone won't give the system a clue that this particular filesystem cannot be mounted until networking and iSCSI have been activated.

Without that clue, the system will assume that filesystem is on a local disk and will attempt to mount it as early as possible, normally before networking has been activated, which will obviously fail with an iSCSI LUN. So you'll need to supply that clue somehow.

One way would be to add _netdev to the mount options for that filesystem in /etc/fstab. From this Ubuntu help page it appears to be supported on Ubuntu. This might actually also trigger a vgscan or similar detection of new LVM PVs (+ possibly other helpful stuff) just before the attempt to mount any filesystems marked with _netdev.

Another way would be to use the systemd-specific mount option x-systemd.requires=<iSCSI initiator unit name>. That should achieve the same thing, by postponing any attempts to mount that filesystem until the iSCSI initiator has been successfully activated.

When the iSCSI initiator activates, it will automatically make any configured LUNs available, and as they become available, LVM should auto-activate any VGs on them. So, once you get the mount attempt postponed, that should be enough.

The lack of PARTUUID is a clue that the disk/LUN does not have a GPT partition table. Since /dev/sdc is listed as TYPE="LVM2_member" it actually does not have any partition table at all. In theory, it should cause no problems for Linux, but I haven't personally tested an Ubuntu 18.04 system with iSCSI storage, so cannot be absolutely certain.

The problem with disks/LUNs with no partition table is that other operating systems won't recognize the Linux LVM header as a sign that the disk is in use, and will happily overwrite it with minimal prompting. If your iSCSI storage administrator has accidentally presented the storage LUN corresponding to your /dev/sdc to another system, this might have happened.

You should find the LVM configuration backup file in /etc/lvm/backup directory that corresponds to your missing VG, and read it to find the expected UUID of the missing PV. If it matches what blkid reports, ask your storage administrator to double-check his/her recent work for mistakes like described above. If it turns out the PV has been overwritten by some other system, any remaining data on the LUN is likely to be more or less corrupted and it would be best to restore it from backup... once you get a new, guaranteed-unconflicted LUN from your iSCSI admin.

If it turns out the actual UUID of /dev/sdc is different from expected, someone might have accidentally run a pvcreate -f /dev/sdc somehow. If that's the only thing that has been done, that's relatively easy to fix. (NOTE: check man vgcfgrestore chapter REPLACING PHYSICAL VOLUMES for updated instructions - your LVM tools may be newer than mine.) First restore the UUID:

pvcreate --restorefile /etc/lvm/backup/<your VG backup file> --uuid <the old UUID of /dev/sdc from the backup file> /dev/sdc

Then restore the VG configuration:

vgcfgrestore --file /etc/lvm/backup/<your VG backup file> <name of the missing VG>

After this, it should be possible to activate the VG, and if no other damage has been done, mount the filesystem after that.

Best Answer

Related Solutions

How to shrink a physical volume

Linux – LVM: PV missing after reboot

Related Question