Ubuntu – Headless Passthrough: QEMU-KVM NVidia

I have run into similar problems like you (Lubuntu 16.04). This comes due to drivers/modules binding the devices to them before pci-stub is able to do this. You have at least two options in here:

The first and easiest one would be to blacklist the modules that claim the device. Type in lspci -knn | grep VGA -A 5 to see all your VGA pci devices and their device-number and their kernel modules.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:128b] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8c93]
    Kernel driver in use: nouveau
    Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] GK208 HDMI/DP Audio Controller [1462:8c93]
--
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. GM204 [GeForce GTX 970] [19da:1366]
    Kernel driver in use: nouveau
    Kernel modules: nvidiafb, nouveau
02:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. GM204 High Definition Audio Controller [19da:1366]

Now you need to check which driver is in use. For example nouveau grabbed my VGA-device 02:00.0 which i want to use for my VM, so I blacklist that one in:

sudo nano /etc/modprobe.d/blacklist.conf blacklist nouveau

and your are done.

In my case this would cause a problem since I have two nVidia VGA's installed (01:00.0 and 02:00.0) both running with the same driver. In my case I do not blacklist the driver.

I do manually unbind nouveau from my 02:00.0 VGA card, since i wanted to use that card for my VM-guest and the 01:00.0 VGA for my Linux host. Thanks to this guide i found out how to do so: https://lwn.net/Articles/143397/

Type in sudo tree /sys/bus/pci/drivers/nouveau. Exchange nouveau with whatever module grabbed your device.

You should recieve a list like this:

/sys/bus/pci/drivers/nouveau
├── 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:03.0/0000:01:00.0
├── 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:05.0/0000:02:00.0
├── bind
├── module -> ../../../../module/drm
├── new_id
├── remove_id
├── uevent
└── unbind

We see that driver nouveau has to devices binding to it: 0000:01:00.0 and 0000:02:00.0

To unbind and bind my graphic-card I first need to turn off lightdm.service. Therefor I open the console outside of the desktop environment with CTRL+ALT+F2 for example. Login as root and type systemctl stop lightdm.service

Now I can unbind the module from the graphics-card:

echo -n "0000:02:00.0" > /sys/bus/pci/drivers/nouveau/unbind

and bind it to whatever module I want (pci-stub or vfio-pci). I used vfio-pci.

echo -n "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

After that, you can start your desktopmanager again: systemctl start lightdm.service

If everything worked you should find your device binded to the module you specified by looking up with lspci -knn | grep VGA -A 5 again.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:128b] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8c93]
    Kernel driver in use: nouveau
    Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] GK208 HDMI/DP Audio Controller [1462:8c93]
--
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. GM204 [GeForce GTX 970] [19da:1366]
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau
02:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. GM204 High Definition Audio Controller [19da:1366]

Unfortunately this workaround loses effect after reboot. Yet i did not find out on how to make it persistent. Maybe anybody else can give me a hint. Something like a startscript would be possible, i guess. But it would be better beeing able to bind the device to a specific module without having to unbind it first. Imagine i would like to use the nvidia driver one day. In that case unbinding from nouveau would be useless since the graphics card would be bind to the nvidia module.

Thanks Christian for the feedback. I did try to use spice with GL, but kept getting an error trying to start the VM with it enabled and didn't go down that rabbit hole yet. I had passed through that GPU to my VM and no matter what I tried to do, updating Windows or drivers, trying to delete the other display and video through virt-manager (couldn't seem to delete them all), OpenGL wasn't showing as installed. Maybe if I'd deleted the display and video from the virsh xml file it would have worked, but I decided to start from scratch and got it to work finally so here are the steps I took if anyone else runs into this in the future.

Pre-requisites

Ubuntu 18.04 with QEMU (I have 3.1.0, would likely be similar with 2.11 or others). There are a few steps needed for Nvidia cards, not sure what would be needed for AMD. This guide is what I mostly mimicked to get everything working including my steps below. This was another one with similar info on Nvidia card setups.

As a note - when I first physically added the GPU, I could no longer access my machine over my network (headless) because adding the GPU changed the network interface which broke my netplan. Could have connected a monitor, but I have an IPMI port on my MB so connected that way and got the new network interface to update netplan.

Then

Create Windows 10 VM in virt-manager - at the end choose advanced options

In Options - use OVMF bios and Q35 chipset. SeaBIOS doesn't work, had to "sudo apt install ovmf". OVMF can't use IDE drives, so changed to SATA to start (SCSI didn't work either), made Virtio later.

In Options - delete all display and video devices

In Options - pass through PCI devices of GPU video (have monitor directly connected to Nvidia video card) AND audio (failed with only video)

In Options - pass through usb keyboard/mouse (since deleted all display and video options to be safe, couldn't interact remotely)

Start VM and load Windows 10 - on first startup it dropped me straight to the UEFI shell which I had to exit out of and took me to the BIOS where I could go to the boot menu and choose the Win10 iso I'd attached as a CD over SATA to boot from.

After Windows loaded and I'd shut the VM down, I edited the XML per the above guides to be sure I didn't have the Code 43 error with the GPU (specifically adding the 3 lines where noted - "vendor_id state='on' value='1234567890ab'/", "hidden state='on'/", and "ioapic driver='kvm'/").

Load Virtio drivers and utility

Load latest Nvidia drivers

Check for OpenGL and it was finally there.

Installed Sketchup and it ran just fine.

I usually use remote desktop for Windows VM's but I still wanted to try to have the ability to connect from virt-manager, so after it was all working, I did go back and add a VNC display and Virtio video to the VM so I can connect and interact through virt-manager or the like. This only seems to work if the Nvidia monitor is physically disconnected, otherwise I get a "guest has not initialize the display (yet)" which I'm sure there is a solution for, but I don't plan on having a monitor connected anyway so I have not investigated that.

Thanks for the input here and my previous post Christian, I'll add a short note to the old one as well linking here as they are related.