DMAR errors with VT-d enabled

English Virtualization
#1

I’m opening this thread to find out more about some errors when booting with kernel parameter iommu=on (I need VT-d for two VMs on this server).

Specifically, when enabling iommu=on the kernel module for the RAID card (a Highpoint RocketRAID 2642) the logs state


2014-04-25T09:37:53.595431+01:00 kvmhost kernel:     8.490167] dmar: DRHD: handling fault status reg 2
2014-04-25T09:37:53.595431+01:00 kvmhost kernel:     8.490171] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 41a6aa000
2014-04-25T09:37:53.595431+01:00 kvmhost kernel:     8.490171] DMAR:[fault reason 01] Present bit in root entry is clear

resulting in the kernel module not loading and the RAID array not showing up.

I have requested to close the thread in the Install/Boot/Login forum I had opened for this issue https://forums.opensuse.org/showthread.php/497371-Highpoint-RocketRAID-2642-not-working-on-13-1 as it looks to be VT-d related.

I’m not knowledgeable enough to interpret this and I was hoping someone would be able to help and tell me if

  1. this is related to a kernel issue, motherboard BIOS issue or RAID driver issue
  2. if there is a workaround or any other troubleshooting steps I can take

This is on OpenSUSE 13.1 with kernel 3.11. Mainboard and RAID card BIOS have been updated to the latest version, respectively 2.1 for the Super Micro X9SCM-F mainboard and 1.4 for the Highpoint RocketRAID 2642 card.

Any feedback is very much appreciated. Thanks!

#2

Recommend you start here.
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Looks like it steps you through how to test, and if your hardware supports to configure.

TSU

#3

Hi Tsu, thanks for your reply.

IOMMU is enabled and working, I can successfully passthrough the onboard SATA controllers and several USB devices to guests.

The instructions and troubleshooting on the page you linked relate to assigning devices to KVM guests. What I’m trying to have working here is the RAID controller for use by the host, the device should not be passed to any guests.

I have been able to reproduce this issue on another mainboard that supports VT-d from another manufacturer, so it may not be related to the VT-d support in the mainboard BIOS (which I understood can be dodgy at times).

Is there a way to disable DMAR on a per-device basis or am I wrong in understanding this error message as a DMA remapping issue?

#4

There appears to be a kernel parameter intel_iommu=pt. This would,

set up pass through mode in
context mapping entry. This disables DMAR in linux kernel; but KVM still runs on
VT-d and interrupt remapping still works.

In this mode, kernel uses swiotlb for DMA API functions but other VT-d
functionalities are enabled for KVM. KVM always uses multi level translation
page table in VT-d. By default, pass though mode is disabled in kernel.

This is useful when people don’t want to enable VT-d DMAR in kernel but still
want to use KVM and interrupt remapping for reasons like DMAR performance
concern or debug purpose.

quoting from http://lkml.iu.edu/hypermail/linux/kernel/0904.1/03905.html.

Setting intel_iommu=pt does allow the RAID controller kernel module to start properly. Need to do further testing though to confirm that passthrough of SATA and USB still works for the guest VMs.

I’m not overly keen on this kind of “blanket” workaround and would prefer a solution or workaround that only affects the specific device I’m having issues with. Though I would think that this would have the most impact for passthrough of VGA devices, which I’m not doing on this unit.

Xen does seem to have more control over regarding VT-d as listed on http://wiki.xen.org/wiki/VTd_HowTo but I’d like to stick to KVM.

Anyhow, ideas and feedback still welcome. Thanks.