Ubuntu – 19.10 Ubuntu host entirely freezes during Windows 10 qemu install

driverskvmnvidiaqemuvirtualization

On Ubuntu 19.10 as a host, I'm trying to do GPU passthrough to Windows 10 as guest. I successfully isolated my GTX 1060 and passed to the insides of a VM in virt-manager (libvirt doc) (home page).

When I try to install Windows, it first copies the files during the installation process. Then it restarts the VM, and when the Windows continues the installation process, Ubuntu freezes forever. I rebooted Ubuntu, tried again, and the same happens at the same spot.

You can see, in details, all I did to enable passthrough here: KVM GPU passthrough: group 15 is not viable. Please ensure all devices within the iommu_group are bound to their vfio bus driver.'

But, in short: I edited /etc/initramfs-tools/modules and /etc/modules to include all the 4 pci devices on my iommu group 15. However, only the GTX 1060 appeared as being used by vfio-pci when I ran lspci -nnv after a reboot. So I manually unbinded them and inserted into vfio-pci with a little script I wrote:

echo -n "0000:07:00.1" > /sys/bus/pci/drivers/snd_hda_intel/unbind
echo -n "0000:07:00.1" > /sys/bus/pci/drivers/vfio-pci/bind

echo -n "0000:07:00.2" > /sys/bus/pci/drivers/xhci_hcd/unbind
echo -n "0000:07:00.2" > /sys/bus/pci/drivers/vfio-pci/bind

echo -n "0000:07:00.3" > /sys/bus/pci/drivers/nvidia-gpu/unbind
echo -n "0000:07:00.3" > /sys/bus/pci/drivers/vfio-pci/bind

after that, I can sucessfully start my Windows 10 VM installation, but it freezes during installation. Is it possibly related to what I did?

PS: I cannot simply block the noveau driver because the host's GPU is also from NVIDIA. So I need to manually unbind each device.

Best Answer

You will almost certainly need to blacklist nouveau and nvidia device drivers, and set up a startup script that does the above binding to vfio-pci, and then explicitly loads nouveau/nvidia kernel drivers to let them bind do the hardware devices you didn't already explicitly bind to vfio-pci in your script prior to that point. Once the host side driver has tainted the device, the device tends not to work for pci passthrough any more on most setups.

Related Solutions

KVM GPU Passthrough – Group 15 IOMMU Issue

I discovered that there's a way to manually unbind kernel modules for specific devices in the pci so I did this little script

echo -n "0000:07:00.1" > /sys/bus/pci/drivers/snd_hda_intel/unbind
echo -n "0000:07:00.1" > /sys/bus/pci/drivers/vfio-pci/bind

echo -n "0000:07:00.2" > /sys/bus/pci/drivers/xhci_hcd/unbind
echo -n "0000:07:00.2" > /sys/bus/pci/drivers/vfio-pci/bind

echo -n "0000:07:00.3" > /sys/bus/pci/drivers/nvidia-gpu/unbind
echo -n "0000:07:00.3" > /sys/bus/pci/drivers/vfio-pci/bind

It hangs for a while (like 2 minutes) because of the echo -n "0000:07:00.3" > /sys/bus/pci/drivers/nvidia-gpu/unbind line but when it finishes, this is the output of lspci -nnv:

7:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660] [10de:2184] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: NVIDIA Corporation TU116 [GeForce GTX 1660] [10de:1324]
    Flags: bus master, fast devsel, latency 0, IRQ 11
    Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
    Memory at d0000000 (64-bit, prefetchable) [size=256M]
    Memory at e0000000 (64-bit, prefetchable) [size=32M]
    I/O ports at f000 [size=128]
    Expansion ROM at 000c0000 [disabled] [size=128K]
    Capabilities: <access denied>
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau

07:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
    Subsystem: NVIDIA Corporation Device [10de:1324]
    Flags: fast devsel, IRQ 83
    Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: <access denied>
    Kernel driver in use: vfio-pci
    Kernel modules: snd_hda_intel

07:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1) (prog-if 30 [XHCI])
    Subsystem: NVIDIA Corporation Device [10de:1324]
    Flags: fast devsel, IRQ 46
    Memory at e2000000 (64-bit, prefetchable) [size=256K]
    Memory at e2040000 (64-bit, prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: vfio-pci

07:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
    Subsystem: NVIDIA Corporation Device [10de:1324]
    Flags: fast devsel, IRQ 58
    Memory at f7084000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: <access denied>
    Kernel driver in use: vfio-pci
    Kernel modules: i2c_nvidia_gpu

As you can see, all them are using vfio-pci. Then I simply added the GPU to virt-manager and it worked. However I'm still investigating why, in the middle of the windows 10 installation, the entire ubuntu froze forever.

UPDATE:

manually unbinding works to unding the GPU but if you have to unbind this means that the linux driver for the GPU already touched the GPU, so now the GPU knows it was on linux. When you bind it to the VM and start the VM, the Windows driver for the GPU will read the GPU state and know somebody (linux) messed with it before and thus will refuse to work because NVIDIA sucks.

Don't unbind manually, or at least try but it will probably not work. Instead make sure linux drivers never touch the GPU

Related Question

KVM GPU Passthrough – Group 15 IOMMU Issue