Ubuntu – Blacklist a Nvidia gpu for qemu/kvm passthrough

elementarykvmnvidiapciqemu

I am trying to run Windows in a virtual machine while giving the VM a direct passthrough to the GPU for better performance.

I have an integrated intel GPU (I will use this one for the host) and a Nvidia GTX980 (I want this one for the VM). I use Elementary OS 0.3.2 Freya 64 bit.

I have followed this guide but am now stuck at step 2. I cannot get the Nvidia gpu to be blacklisted.

To start with I do lspci -nn | grep NVIDIA

This results in the following output

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

Next I added this line to the /etc/initramfs-tools/modules file.

pci_stub ids=10de:13c0,10de:0fbb

And I then reloaded using update-initramfs -u and afterwards rebooted.

After the reboot when I run dmesg | grep pci-stub I get the following output:

[    2.029626] pci-stub: add 10DE:13C0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.029630] pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.029637] pci-stub 0000:01:00.1: claimed by stub

As you can see neither the audio or video are claimed by stub.

I have also tried adding this option directly to the grub file in etc/default/grub so the GRUB_CMDLINE_LINUX_DEFAULT line looks like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on pci-stub.ids=10de:13c0,10de:0fbb"

But this also resulted in the Nvidia card not being blacklisted.

Anyone got any idea what might be causing this?

Best Answer

I have run into similar problems like you (Lubuntu 16.04). This comes due to drivers/modules binding the devices to them before pci-stub is able to do this. You have at least two options in here:

The first and easiest one would be to blacklist the modules that claim the device. Type in lspci -knn | grep VGA -A 5 to see all your VGA pci devices and their device-number and their kernel modules.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:128b] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8c93]
    Kernel driver in use: nouveau
    Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] GK208 HDMI/DP Audio Controller [1462:8c93]
--
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. GM204 [GeForce GTX 970] [19da:1366]
    Kernel driver in use: nouveau
    Kernel modules: nvidiafb, nouveau
02:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. GM204 High Definition Audio Controller [19da:1366]

Now you need to check which driver is in use. For example nouveau grabbed my VGA-device 02:00.0 which i want to use for my VM, so I blacklist that one in:

sudo nano /etc/modprobe.d/blacklist.conf blacklist nouveau

and your are done.

In my case this would cause a problem since I have two nVidia VGA's installed (01:00.0 and 02:00.0) both running with the same driver. In my case I do not blacklist the driver.

I do manually unbind nouveau from my 02:00.0 VGA card, since i wanted to use that card for my VM-guest and the 01:00.0 VGA for my Linux host. Thanks to this guide i found out how to do so: https://lwn.net/Articles/143397/

Type in sudo tree /sys/bus/pci/drivers/nouveau. Exchange nouveau with whatever module grabbed your device.

You should recieve a list like this:

/sys/bus/pci/drivers/nouveau
├── 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:03.0/0000:01:00.0
├── 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:05.0/0000:02:00.0
├── bind
├── module -> ../../../../module/drm
├── new_id
├── remove_id
├── uevent
└── unbind

We see that driver nouveau has to devices binding to it: 0000:01:00.0 and 0000:02:00.0

To unbind and bind my graphic-card I first need to turn off lightdm.service. Therefor I open the console outside of the desktop environment with CTRL+ALT+F2 for example. Login as root and type systemctl stop lightdm.service

Now I can unbind the module from the graphics-card:

echo -n "0000:02:00.0" > /sys/bus/pci/drivers/nouveau/unbind

and bind it to whatever module I want (pci-stub or vfio-pci). I used vfio-pci.

echo -n "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

After that, you can start your desktopmanager again: systemctl start lightdm.service

If everything worked you should find your device binded to the module you specified by looking up with lspci -knn | grep VGA -A 5 again.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:128b] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8c93]
    Kernel driver in use: nouveau
    Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] GK208 HDMI/DP Audio Controller [1462:8c93]
--
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. GM204 [GeForce GTX 970] [19da:1366]
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau
02:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. GM204 High Definition Audio Controller [19da:1366]

Unfortunately this workaround loses effect after reboot. Yet i did not find out on how to make it persistent. Maybe anybody else can give me a hint. Something like a startscript would be possible, i guess. But it would be better beeing able to bind the device to a specific module without having to unbind it first. Imagine i would like to use the nvidia driver one day. In that case unbinding from nouveau would be useless since the graphics card would be bind to the nvidia module.

Related Question