Everything Breaks when trying to passthrough HBA controller to TrueNAS

OpenZFS Proxmox
1

I just got myself a SFF-8087 8 Drive HBA card and am having an issue trying to bring the drive over to TrueNAS. I followed a guide for enabling iommu for pcie pass-through. Basic rundown

  1. Edit Grub to add amd_iommu=on iommu=pt
  2. edit modules add
  3. vfio
  4. vfio_iommu_type1
  5. vfio_pci
  6. vfio_virqfd
  7. reload/reboot
  8. attach card to TrueNAS (here is where issues start)

After attaching the card to TrueNAS the device id starts off as orange. After I rebooted the VM it went white but then the issues started.

I checked on the node bigbertha to see if the drives that are connected to the card have disappeared (3 drives on the card + 1nvmeSSD + 1 sada hdd on motherboard) bigbertha now has access to ZERO drives. It is as if the card decided to take control of all drives on the system and when i transferred it over to TRUENAS, it tried to bring every drive into the VM.

I don’t know how I broke it this badly. Had to remove the card and remove it from BigNASty (TrueNAS) for proxmox to boot again properly.

link to card: 9211-8i 6Gbps HBA LSI FW:P20 IT Mode ZFS FreeNAS unRAID+2* SFF-8087 SATA From US | eBay

2

it is as if the card decided to take control of all drives on the system and when i transferred it over to TRUENAS, it tried to bring every drive into the VM.

This makes me suspect your IOMMU groups are too broad. What CPU and motherboard do you have on the host? Have you used PCIe passthrough before with this host?

This is a script I’ve used (on Linux) in the past to list IOMMU groups:

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Unless you use some nasty hacks the Linux kernel can only pass devices to a guest at the granularity of an IOMMU group. You can use this script to verify that your LSI HBA is in its own IOMMU group. If it shares an IOMMU group with the NVMe drive, or SATA controller, then you wouldn’t be able to pass it to a guest by itself. (You can try using different PCIe slots on the motherboard. Usually slots electrically connected via the CPU, versus the platform chipset, are at least in distinct IOMMU groups.)