Nvme devices, in line with many other MTD devices in Linux kernel, rely on asynchronous initialization. After the controller is discovered by kernel, the block layer will initiate the partition scan, but the kernel will not wait for it (by default). This means that an attempt to mount a root file system on such a device will either fail or will be racy, because by the time kernel attempts to mount the root fs the partition layout may not yet be known.
To make sure kernel actually waits for root device to appear instead of rushing forward to an inevitable panic, it must be passed a rootwait option on the boot loader/embedded command line (rootdelay with explicit timeout is another option, but rootwait is clearly simpler and safer).
The problem only manifests itself with direct booting kernels, because initramfs activation ensures completion of initialization tasks (or may be simply introduces enough delay, not sure about it) for partition scan to finish and all partitions to become registered and available.
Table of contents
- Please use a method of writing to the USB which re-checks the written data. This is probably quite difficult, sorry.
- How to verify using
cmp command after dd.
- The initramfs read from your USB is damaged.
- Why I blame the initramfs read from the USB.
1. Please use a method of writing to the USB which re-checks the written data. This is probably quite difficult, sorry.
You are having boot problems that look very much like bad data on the USB. Therefore, please use a USB writing method that re-checks the written data.
dd on its own, does not re-check the written data. Please re-check the data manually using cmp, by following the exact instructions in the next section. Unfortunately I couldn't think of any easier method.
GNOME Disks can be used to write a USB (the Fedora install instructions tell you how). But it doesn't seem to re-check the written data.
The official Fedora install instructions "default" to Fedora Media Writer. At least the version of Fedora Media Writer I have, does automatically re-check the written data after writing it to the USB. I don't know an easy way to install Fedora Media Writer on your Ubuntu OS :-(.
https://docs.fedoraproject.org/f28/install-guide/install/Preparing_for_Installation.html
Ubuntu's "Startup Disk Creator" did not work at all for me. I think it is only willing to write Ubuntu ISOs, not Fedora ones.
2. How to verify using cmp command after dd
In principle this could be a simple cmp command. Very sadly, there are some difficulties that will return useless results, so I need to explain these.
I believe cmp might also show differences if you ever allowed the written USB to be mounted on a Linux system :-(. This is likely to happen if you ever plugged it in again to a Linux system after writing it! (Or successfully boot the Fedora Live USB??) You have to be careful with it!
So you want to run dd to write the data, and then cmp command to verify the USB data, without re-plugging the USB (or rebooting) in between.
First run your dd command. Remember to take great care and don't wipe your internal hard drive :).
Then remember you should run sync, to make sure the data is finished writing.
Then, you will be able to run echo 3 | sudo tee /proc/sys/vm/drop_caches. This step is required, to make sure that cmp won't just be reading back from caches in system RAM.
Then you can run
# cmp Fedora-Workstation-Live-28.iso /dev/sdb
I.e. where sdb is the name of your USB device, and the .iso file is the name of your ISO file. This command could show IO errors, or differences, if the problem is that your USB stick needs replacing.
In principle you might want to go further, and test the USB keeps its data correctly after it is removed and loses power. So this would require an alternative method which disables the auto-mount of your graphical login. It can be very hard to know how to disable auto-mount temporarily :-(. I think the simplest way is to log in on a text console and run your commands from there. Make sure you do not switch back to a graphical login before you have finished! For some information about logging in on a text console, see here.
3. The initramfs read from your USB is (probably) damaged
From the evidence so far, it looks like the initramfs read from your USB is damaged. You could try to verify the USB data, on the same system that you wrote the USB. This could be useful to confirm some sort of problem writing to your USB.
For completeness, the full list of possibilities is :-
- a problem with writing to your USB
- a problem in the ISO image you wrote to it
- a problem in your computer e.g. EFI/BIOS problems during the early boot
- "incompatibility" between your computer and this version of Linux. (That is, if someone thinks your computer is not to blame, they would have to blame the specific version of Linux :).
4. Why I blame the initramfs read from the USB
I found other unsolved mysteries described with the same error message and very similar conditions:
[1.81660] ---[end Kernel] panic - not syncing: VFS: unable to mount root fs on unknown-block(0,0).
Unfortunately, this isn't the real, specific error.
I have double-checked; Fedora-Workstation-Live 28 uses an initramfs. This is sometimes also called a type of initrd.
This error would be shown if you set up a boot with both a kernel and an initramfs, but some error prevents the initramfs being loaded.
For example, one way to trigger this error, would be if you pressed a special key at the boot menu to start a temporary edit of the boot options. In theory you might have started editing the boot options, and unintentionally removed an initrd option. So I have to mention this as a possibility :). Note this type of edit is temporary. So you can just boot again, and make sure you're not making an edit that disables the initramfs :).
In this case the kernel may "fall through" to an old-style boot without an initramfs, but then it doesn't have any "block device" that it was specified to mount as the root fs. "(0,0)" means "unknown"; this device number is never a valid block device.
But I don't think that's what you (and some others) did.
Therefore, I think there is probably a more specific error earlier in the boot process. Unfortunately, if there was an error message, it is possible you can't see it because it scrolled off the top of the screen, if there were too many messages since then :).
An image search for "unable to mount root fs on unknown-block(0,0)." shows that is very common for the kernel call trace to fill the screen. When this happens you cannot see if there is any previous error message.
I found an example of such an error, posted regarding a different Linux:
https://bbs.archlinux.org/viewtopic.php?id=220178
Initramfs unpacking failed: junk in compressed archive
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
...
--- [end Kernel] panic - not syncing: VFS: unable to mount root fs on unknown-block(0,0).
EDIT: called it :-). "Initramfs unpacking failed:" was the real error. "XZ-compressed data is corrupt" makes it clear the initramfs is not being read correctly from the USB
Best Answer
I generally use one of the following alternatives:
rsync -aHAX(add v for verbosity) makes sure that you preserve any link structure and x-attrs in the target folder while copying. Don't forgetameans archive and preserves time, ownership and permissions already.tar cvf(don't compress to save time, just tar them up) is what I use if the first one doesn't meet what I need for whatever reason and I have no time, but I always try the first one.To check that everything went as it should, you can run
diff -r <folder1> <folder2>afterwards if you want.