KVM GPU Passthrough

Requirements

  • AMD-V/VT-d capable CPU & motherboard
  • Two GPUs (I am using two different NVIDIA GPUs
  • Enough RAM and disk space for a Windows VM

Guide

  1. Install two GPUs into your system.

  2. Locate the GPU you wish to passthrough to the VM.
    lspci -nn | grep -i nvidia

    Copy the device id’s which looks like this:

    1
    2
    06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)
    06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
  3. Edit the default grub config (/etc/default/grub) to enable iommu and disable the secondary GPU that you are going to passthrough to the VM.
    GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pci-stub.ids=10de:1380,10de:0fbc"

  4. Update grub.
    sudo grub-mkconfig -o /boot/grub/grub.cfg

  5. Next, install qemu.
    sudo pacman -S qemu

  6. Enable KVM modules.
    /etc/modules-load.d/20-kvm.conf

    1
    2
    3
    4
    5
    6
    pci_stub
    vfio
    vfio_iommu_type1
    vfio_pci
    kvm
    kvm_intel

    Reboot.

  7. Create a vfio bind script at /usr/bin/vfio-bind.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    #!/bin/bash
    modprobe vfio-pci
    for dev in "[email protected]"; do
    vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
    device=$(cat /sys/bus/pci/devices/$dev/device)
    if [ -e /sys/bus/pci/devices/$dev/driver ]; then
    echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
    fi
    echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
    done

    chmod 755 /usr/bin/vfio-bind

  8. Bind the two devices (this needs to be done everytime you reboot).
    vfio-bind 0000:06:00.0 0000:06:00.1

  9. Get the latest virtio drivers for Windows.
    https://fedoraproject.org/wiki/Windows_Virtio_Drivers#Direct_download

  10. Create a blank image for Windows.
    qemu-img create -f raw -o size=120G windows8.1.raw

  11. Start the VM (change the device ids and image paths).

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    #!/bin/bash
    rm /tmp/windows_vars.fd
    cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/windows_vars.fd
    QEMU_PA_SAMPLES=128
    export QEMU_AUDIO_DRV=pa
    sudo vfio-bind 0000:06:00.0 0000:06:00.1
    sudo qemu-system-x86_64 -enable-kvm -m 8G -cpu host -smp sockets=1,cores=6,threads=1 -machine type=pc,accel=kvm -mem-prealloc \
    -balloon none \
    -vga none \
    -device qxl \
    -soundhw hda \
    -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
    -drive if=pflash,format=raw,file=/tmp/windows_vars.fd \
    -device vfio-pci,host=06:00.0,multifunction=on \
    -device vfio-pci,host=06:00.1 \
    -drive id=cd0,if=none,format=raw,readonly,file=/home/USER/windows8.1.iso -device driver=ide-cd,bus=ide.0,drive=cd0 \
    -drive id=virtiocd,if=none,format=raw,file=/home/USER/virtio-win-0.1.102.iso -device driver=ide-cd,bus=ide.1,drive=virtiocd \
    -drive id=disk0,if=none,format=raw,cache=writeback,file=/home/USER/windows8.1.img -device virtio-scsi-pci,id=scsi0 -device scsi-hd,drive=disk0 \
    -boot menu=on

The -device qxl argument will allow you to click into the qemu window and control the keyboard/mouse. It is good enough to get Windows installed but too laggy for anything else.

Notes

  • You can bind a USB hub or USB keyboard/mouse to the VM but then it will be completely bound to the VM and you will not be able to use Windows like a second PC on a second monitor. Instead, you can install Synergy and then you’ll be able to move the mouse just like you would if it were a physical machine set up with Synergy.
  • I have noticed some annoying lag spikes when the Windows VM consumes a lot of CPU resources. The only way I’ve found to fix this is to give the VM less cores. You could also try linux-ck which is what I am testing now. The lag spikes I’m getting aren’t specific to running a Windows VM. It also occurs when compiling code or when Bitcoin loads more than a weeks worth of data. The audio will stop for a second or two and the mouse will lag. It’s very annoying.

Tweaks

References