Linux – What would cause the video driver to stop working after every update

arch linuxgraphicsupgradevideo

I am not sure if this is a systemd issue (like order of loading modules) or a video driver issue. It does seem like every time

I update boot-related stuff from a chrooted environment using my Arch ISO USB Stick, everything works.
It is when I update from a running native system that things go sour AND I think when boot-related things are in that update.

UPDATE 2016-04-18

I changed root (arch-chroot) into the machine and tried to rebuild the kernel with mkinitcpio -p linux. I get the following error:

ERROR: '/lib/modules/4.4.5-1-ARCH' is not a valid kernel module directory

`uname -r` (check kernel version) yields

4.3.3-2-ARCH

The folders in `/lib/modules/`

4.5.0-1-ARCH
extramodules-4.5-ARCH

Obviously when the kernel version numbers do not match the module version numbers, this is not good. Why would this happen? This would mean there is a problem with the way the system updates itself (pacman). Or did I do something wrong?

`pacman -Qo /lib/modules/4.5.0-1-ARCH` (check package owner) yields

/usr/lib/modules/4.5.0-1-ARCH/ is owned by linux 4.5-1
/usr/lib/modules/4.5.0-1-ARCH/ is owned by linux-headers 4.5-1

If you give me suggestions in the comments, I'll try to be prompt at adding information.

When Problem Happens

After running yaourt -Syau --noconfirm and rebooting, upon next boot my screen resolution gets messed up (appears the driver is working) and USB devices (keyboard+mouse) become nonresponsive. Could this be due to Grub?

Error after grub boot screen is

Failure to Load Kernel Modules

Machine Information

Bootloader is Grub 2
OS Arch Linux
Graphics GeForce 9500 GT (rev a1) with driver nouveau (acquired using lspci -v | grep VGA -A 10)

I can chroot into this machine by booting from an Arch ISO USB stick. Maybe it is helpful to know that I have recovered the system by waiting a few days for another update and then updating the system from a chrooted environment. Suddenly, everything starts working again. This has happened three times already. It just happened today and it is getting irritating.

Possibly Related Questions

Best Answer

It is important to note immediately that it is not just the video that has been disrupted. The modules/drivers were not permitted to load because of version-matching restraints.

Update

This can be caused by having an error in /etc/fstab in the /boot mount for those who keep boot on a separate partition (usually EFI main boards). If during an update this partition is not mounted, the modules could (likely would) get upgraded while the kernel, which resides in /boot, would not. dkms normally prevents kernel-module mismatches.

I did indeed have kernel + module mismatch that led to the error message

ERROR: '/lib/modules/4.4.5-1-ARCH' is not a valid kernel module directory

mentioned in the question.

To fix it, I needed to boot from an Arch USB Stick and mount the OS and BOOT partitions to /mnt and /mnt/boot, respectively. I had to change root over to the machine arch-chroot /mnt /bin/bashand update my kernel manually (and forcefully).

pacman -Sy linux --force

This brought my system's kernel up to date with the modules (4.4.5-1-ARCH) such that they match (as they absolutely should!)

As to why there was a kernel + module mismatch, I don't know. I am hoping [dkms][1] will prevent this in the future. In my case, the modules weren't built when the new kernel was installed, rather, the modules were updated when the kernel was not.

Dynamic Kernel Module Support (DKMS) is a program/framework that enables generating Linux kernel modules whose sources generally reside outside the kernel source tree. The concept is to have DKMS modules automatically rebuilt when a new kernel is installed.

*Lerhaupt, Gary (September 1, 2003). "Kernel Korner - Exploring Dynamic Kernel Module Support (DKMS)". Linux Journal. Retrieved March 12, 2016.

Booting safely on a USB image

As you can see here, it seems like your system went into some trouble when trying to boot from the USB drive:

ERROR: '/dev/disk/by-label/ARCH_201409' device did not show up after 30 seconds...
Falling back to interactive prompt
You can try to fix the problem manually, log out when you are finished

While your BIOS successfully detects a bootable medium and initiates a boot sequence, Arch does not boot correctly due to some kind of disk error. As you said in chat, your formatted and sent your images on your key quite a few times, which could lead to broken images or partition tables. When sending a bootable ISO image onto a USB drive, it's usually a good idea to make sure the drive is really clean before proceeding:

$ fdisk /dev/sdX # sdX being your USB drive (NOT partition!)
Command (m for help): d

Use d repeatedly until there is no partition left. Then, recreate a clean partition taking up the whole device:

$ fdisk /dev/sdbX
Command (m for help): n
Partition type:
    p   primary (1 primary, 0 extended, 3 free)
    e   extended
Select (default p): p
Partition number (1-4, default 1): 1
# ...

Then will follow some size-related options, just choose the default every time. When you're done, write the changes to disk using w. Now that the partition table is a little cleaner, you may format...

$ mkfs.ext4 /dev/sdX1

... and send your Arch ISO onto the drive:

$ dd if=/path/to/arch.iso of=/dev/sdX # Again, the device, NOT the partition.

Note: it is very important that both the Arch ISO and the hard system share the same architecture!

At this point, you may reboot your machine and get to the live system without too much trouble. Just make sure your USB drive comes first in your BIOS boot sequence.

Chrooting into the old/broken system

Now this is a little trickier, I'll mostly reuse the contents of the Arch Wiki. You have two options here:

Use the Arch scripts (recommended).
Chroot everything yourself, manually.

In the first case, all you have to do is mount your "homemade" partitions: those your created when you installed the system:

$ mount -o exec /mnt /dev/sda1      # / partition.
$ mount /mnt/boot /dev/sda2         # /boot partition.
$ mount /mnt/home /dev/sda3         # /home partition.
$ # ... and so on.

Once you're done, just use arch-chroot to get inside:

$ arch-chroot /mnt /bin/bash

Now, if you want to chroot everything yourself, you're going to have a little more work to do. First, mount the previous systems, then add:

$ mount -t proc proc /mnt/proc/     # procfs
$ mount --rbind /sys /mnt/sys/      # sysfs
$ mount --rbind /dev /mnt/dev/      # /dev
$ mount --rbind /run /mnt/run/      # /run

You might also want a working DNS resolver (yet it's unlikely to have been damaged):

$ cp /etc/resolv.conf /mnt/etc/resolv.conf

Finally, get inside:

$ chroot /mnt /bin/bash

Investigating

Basically, your system just went out of power. As you said, there was no significant task running (at least, no upgrade), so the loss must have been limited. First things first, check your logs. Have a look in /var/log and use journalctl to find information about what happened before the shutdown.

Reinstalling GRUB

In chat, you said that your system was no longer available in the boot menu, probably something related to the latest update you made. Let's reinstall it:

$ pacman -S grub # Should not do anything though.
$ grub-install --recheck /dev/sdX # Your hard drive.
$ grub-mkconfig -o /boot/grub/grub.cfg

Ubuntu – GRUB Confuses Kernels/Initramfs of Ubuntu and Arch Linux Installation

I finally solved it by nuking the Arch partition and its boot files in my Ubuntu's /boot directory from orbit. Ubuntu is fine again now, all remaining GRUB entries are working again.

Here's a list of what I did:

Delete Arch's initramfs files:
```
sudo rm /boot/initramfs-linux*
```
Delete Arch's vmlinuz file:
```
sudo rm vmlinuz-linux
```
Format the Arch partition (/dev/sda8) using GParted
Update GRUB's configuration:
```
sudo update-grub
```
Reboot and enjoy!