Ubuntu – In Ubuntu ISO MBR, why does a GPT partition overlap the boot partition, and make MBR uneditable

fdiskgptlive-usbmbrpartitioning

Question and context:

In the MBR partition table of distros like for example ubuntu-16.04-desktop-amd64.iso, why does a GPT partition appear to overlap the primary boot partition? This seems to generate errors if I try, in any way, to edit the partition table.

I want to edit the MBR to add USB persistence (as described elsewhere), and must add a partition or enlarge the boot partition.

I believe this used to work, but the overlapped GPT #2 partition in this and similar distros appears to badly confuse fdisk, sfdisk, parted, gparted, and partprobe.

My machine is a MBR, and not a GPT bios.

What am I missing?

Here is the distro's MBR partition table (directly from the ISO file):

cat ubuntu-16.04-desktop-amd64.iso | xxd | head -32 | tail -5

gives:

00001b0: 28db 2b00 0000 0000 708e 0e0e 0000 8000  (.+.....p.......
00001c0: 0100 0058 e0fa 0000 0000 6048 2c00 00fe  ...X......`H,...
00001d0: ffff effe ffff 4411 2c00 8012 0000 0000  ......D.,.......
00001e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001f0: 0000 0000 0000 0000 0000 0000 0000 55aa  ..............U.

(To decode this see https://en.wikipedia.org/wiki/Master_boot_record.)

  • The table starts at address 01be, where you see "80", for the #1 primary boot partition.

  • Then notice the "ef" at address 01d2 which denotes the #2 primary partition as being type=GPT.

Here's a breakdown of the table by partition (little endian):

partition#1 (normal MBR):
  80        = 'boot' partition flag
  00 01 00  = starting HSC (head, sector, cylindar)
  00        =  partition type ("Empty partition entry")
  58 e0 fa  = last     HSC (head, sector, cylindar)

  0000 0000 = LBA (logical block address) of first absolute sector in the 
  6048 2c00 = number of sectors in partition

partition #2 (GPT):
  00        = non-boot partition
  fe ff ff  = starting HSC (head, sector, cylindar)
  ef        = partition type ("EFI system partition")
  fe ff ff  = last HSC

  4411 2c00 = LBA (logical block adr) of first abs sector in part.
  8012 0000 = number of sectors in partition

Attempts to edit partition table:

  • fdisk reports that these partitions overlap. Note how the beginning of sdb2 [2927216] is inside of sdb1 [0-2955679] from

    sudo fdisk -l /dev/sdb

    gives:

      Disk /dev/sdb: 14.5 GiB, 15527313408 bytes, 30326784 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: dos
  Disk identifier: 0x40a863e7

  Device     Boot   Start     End Sectors  Size Id Type
  /dev/sdb1  *          0 2955679 2955680  1.4G  0 Empty
  /dev/sdb2       2927216 2931951    4736  2.3M ef EFI (FAT-12/16/32)

  • From what I've read, it's ok to have a GPT partition, but not for partitions to overlap.

  • When I try to dd the iso to a usb stick, the trouble begins. The stick boots Ubuntu live ok, but I can not edit the partition table without errors (that is while not booted and not mounted), no matter which partition editor I try.

  • Furthermore, I can't even delete the partition table and return the USB stick to normal without the following lengthy procedure: 

    dd if=/dev/zero of=/dev/sdb bs=[something like 512 or 2048; doesn't matter] count=[some large number like 100000^]

    ^Smaller numbers don't always seem to fix this. I suspect you need to not only wipe out the primary fs blocks, but the secondary ones also.

    and then use fdisk to make a new ms-dos partition table on the stick.

    Note, if I don't do the dd above to wipe before creating a new partition table, I get the following error message or similar when I try to edit the partition map (from either debian8 or ubuntu-16): 

    Libparted Warning The driver descriptor says the physical block size
is 2048 bytes, but Linux says it is 512 bytes.

  • Also from ubuntu-12, I get this most enlightening message: 

    gparted -l /dev/sdb:  libparted : 2.3 Could not stat device -l -- No
such file or directory. /dev/sdb contains GPT signatures, indicating
that it has a GPT table.  However, it does not have a valid fake msdos
partition table, as it should.  Perhaps it was corrupted -- possibly
by a program that doesn't understand GPT partition tables.  Or perhaps
you deleted the GPT table, and are now using an msdos partition table.
Is this a GPT partition table?  Both the primary and backup GPT tables
are corrupt.  Try making a fresh table, and using Parted's rescue
feature to recover partitions.

So how do I edit the distro's MBR?

BTW, I'm NOT asking how to fix the overlap, as this is the way the distro actually is, and from the looks of it, has been for some time, (I also looked at Ub-v16, 14 and 12), rather I'm looking to figure out how to edit it if possible.

Ideas:

  • Could this be caused by too many applications of isohybrid (used to add a ms-dos MBR to a CDROM 9660 iso so it can boot on a USB)?

  • Is there an other, possibly newer partition table editor to use?

  • Is there a different, perhaps older, Ubuntu distro which doesn't have this issue?

Best Answer

Ubuntu's .iso image files use a Frankenstein's Monster format that's designed to support multiple boot methods and devices:

  • Boot media
    • Writing the image to a DVD for booting using ISO-9660 and El Torito
    • Copying the image "raw" to a USB flash drive or similar media for booting using conventional disk partitioning systems
  • Boot mode
    • BIOS-mode booting
    • EFI-mode booting

To work with so many formats and boot methods, the developers play games with the data structures. These images really should not be treated like "normal" disk images, and in particular, you should most emphatically NOT attempt to modify the images in any way unless you're a top-level expert in these data structures. FWIW, I wrote the GPT fdisk (gdisk, cgdisk, and sgdisk) partitioning tool, and I would not attempt what you say you're attempting!

Instead, if you need to modify the installation medium, you should do one of two things:

  • Use a tool like Rufus, Pen Drive Linux, or UNetbootin to write the image file to a USB flash drive, then modify the USB flash drive. The reason this works is that these tools don't do a simple dd-like copy of the image's contents to disk; instead, they take the files from the image and copy them to a conventionally partitioned disk with a pre-existing filesystem (or sometimes one that the tool creates). The result is a more normal disk that can be modified with normal partition table editors and other utilities.
  • Prepare your own custom image. I don't have pointers to documentation on how to do this, but the Ubuntu images are of course created by automated tools, not by hand-editing partition tables with hex editors. If you track down the tools and scripts used to create the images, you may be able to create an image that's modified in a way you would want. This approach is used by people who create "spins" of Ubuntu, so you might want to start by looking for documentation on how to do this.

Of these two approaches, the first is probably the better one for what you want. In fact, I'm pretty sure some tools of that sort offer an option to do precisely what you want. (I don't recall which tool(s) offer such a feature, though.) The second approach is more likely to be useful for tasks like adding packages to a custom installation image.

If you were ever able to do this sort of thing before, chances are it's changed because the developers have found a need to make a more monstrous hack of a data structure in order to work around problems with particular systems. For instance, if Brand X's computers can't parse the Frankenstein's Monster partition table, the developers might tweak it so that it works with Brand X computers, even if that means the data structure is even weirder than it had been before. This is just speculation on my part, though, and I certainly can't speak to the specifics you've described.

Related Question