Linux – Implementing PCI-Passthrough with Linux-KVM on Debian

debianlinux-kvmvirtualizationvt-d

I am attempting to use PCI-Passthrough to attach an old video card (Radeon 4770) to a virtual machine. I am using Linux-KVM to run my virtual machines on a Debian Linux (Wheezy, 3.2.0-4-amd64) host.

Question

To clarify, I am not sure what the correct 'path' is for implmenting PCI-Passthrough with Linux KVM. At this stage I suspect the correct action is to add CONFIG_DMAR, CONFIG_DMAR_DEFAULT_ON, and CONFIG_PCI_STUB to the "Bus options (PCI etc.)" section of the kernel source and recompile.

But I am not sure if this is an exhaustive list of necessary additions before recompliling. Or if recompiling the kernel is necessary–perhaps there is an easier method?

Of the guides I have referenced, only linux-kvm.org explicitly mentions compiling is necessary. Linux-KVM is already installed and functioning as a hypervisor.

Research

At this point I think my issue is related to my kernel. My primary resource has been the guide at linux-kvm.org (http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM). However, I have found other resources which indicate slightly different methods that are (seemingly) distribution specific:

Fedora–https://docs.fedoraproject.org/en-US/Fedora/13/html/Virtualization_Guide/chap-Virtualization-PCI_passthrough.html

SUSE–"openSUSE: Virtualization with KVM" (Link omitted due to low-relevancy and 2-link limit)

The Fedora guide works until referencing setsebool which appears to be RedHat-specific. The SUSE guide indicates graphics-card assignment is not supported by SUSE, however I am referencing it as well because it indicated I should find a CONFIG_DMAR_DEFAULT_ON string within /boot/config-`uname -r`. The linux-kvm.org site also references CONFIG_DMAR_DEFAULT_ON, so this appears to be a common and necessary component.

Note: I have not found restrictions for graphics cards in guides for Fedora or Debian. The referenced SUSE document is dated 2006-2013.

I cannot find CONFIG_DMAR_DEFAULT_ON in /boot/config-`uname -r` on my system. Further research suggests that CONFIG_DMAR, CONFIG_DMAR_DEFAULT_ON, and CONFIG_PCI_STUB are Linux kernel configuration items that are relevant to the instructions on linux-kvm.org. As such I believe that I need to recompile my host's kernel with these 3 (at least) kernel config items. Booting with intel_iommu=on as a kernel parameter to my host-OS appears to be insufficient.

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

VT-d/IOMMU/KVM Support Confirmation

My research indicates that PCI-Passthrough requires both CPU and Motherboard support for VT-d.

VT-d

I have confirmed that my processor, a non-k INTEL i7-3770 (per ark.intel.com/products/65719), supports VT-d:

Intel® Virtualization Technology for Directed I/O (VT-d) ‡
Yes

My Asrock Z77 Extreme4 motherboard also supports VT-d (per page 62 of the User Manual):

VT-d
Use this to enable or disable Intel
®
VT-d technology (Intel
®
Virtualization
Technology for Directed I/O). The default value of this feature is [Disabled].

IOMMU

I verified that my system has IOMMU support:

dmesg | grep -e DMAR -e IOMMU | grep -e "DRHD base" -e "enabled"
[    0.000000] Intel-IOMMU: enabled

KVM

KVM is installed and functional, aside from PCI-Passthrough support:

lsmod | grep kvm
kvm_intel             121968  0 
kvm                   287749  1 kvm_intel

I have ensured that VT-d is enabled via my motherboard's BIOS. As such, I do not suspect hardware/BIOS issues that would prevent the use of VT-d. Regardless, I am unable to successfully detach my video card from my host and reassign it to a virtual machine.

Closing Thoughts

Finally I would like to mention that I also tried testing:

echo "8086 10b9" \> /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:01:00.0" \> /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" \> /sys/bus/pci/drivers/pci-stub/bind
echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/remove_id
kvm -m 512 -boot c -net none -hda debian-7.1.0-amd64-netinst.iso -device pci-assign,host=01:00.0

and got the following error after trying to create the target VM:

Failed to assign device "(null)" : Device or resource busy
*** The driver 'pci-stub' is occupying your device 0000:01:00.0.
***
*** You can try the following commands to free it:
***
*** $ echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
*** $ echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/unbind
*** $ echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind
*** $ echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/remove_id
***
kvm: -device pci-assign,host=01:00.0: Device 'pci-assign' could not be initialized

I am guessing this is because the host still will not relinquish control of the video card and is likely due to the kernel not being compiled with the appropriate configuration items.

This is new territory for me so please forgive my inexperience. I would greatly appreciate any feedback whatsoever, even if it is simply confirmation that I am on the right track. Please let me know if I have made a glaring oversight or am over-thinking. Constructive criticism of my question is welcome as well. Let me know if I have not provided enough information to "help you help me" (or if I've included too much!). I would be more than happy to help make my question clearer or easier to answer.

Thank you in advance,

Best Answer

Article mentioned for PCI-passthrough, the Kernel config is required

make menuconfig
set "Bus options (PCI etc.)" -> "Support for DMA Remapping Devices" to "*"
set "Bus options (PCI etc.)" -> "Enable DMA Remapping Devices" to "*"
set "Bus options (PCI etc.)" -> "PCI Stub driver" to "*"

optional setting: 
set "Bus options (PCI etc.)" -> "Support for Interrupt Remapping" to "*"

However, I could not find these options under "Bus options" and instead for it under "Device Drivers --> IOMMU Hardware Support".

After following the article mentioned, I still had issues boot up the guest machine and getting errors that "Device 'pci-assign' could not be initialized". I managed to get it boot by executing:

 echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

Refer to the following article if you're still having issues: http://spica-and-roid.blogspot.com.au/2012/07/howto-kvm-passthrough.html

Related Question