SR-IOV on ARM64 system with SMMU

Mon Feb 20 11:21:43 PST 2023

Hi Robin,

thanks for your email. the platform doesn't support ACS, so I added  
pcie ACS override patch:

martin at ubuntu:~$ sudo dmesg | grep ACS
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow 
non-IOMMU protected peer-to-peer DMA

but, after applying this patch, the PCIe VFs and PF are still grouped in 
the same IOMMU group 10:

martin at ubuntu:~$ find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/20/devices/15340000.vic
/sys/kernel/iommu_groups/10/devices/0005:01:00.2
/sys/kernel/iommu_groups/10/devices/0005:01:00.0
/sys/kernel/iommu_groups/10/devices/0005:00:00.0
/sys/kernel/iommu_groups/10/devices/0005:01:00.3
/sys/kernel/iommu_groups/10/devices/141a0000.pcie

martin at ubuntu:~$ sudo lspci -vvt
-+-[0005:00]---00.0-[01-ff]--+-00.0  Micron NVMe SSD Controller PCC
  |                           +-00.2 Micron NVMe SSD Controller PCC
  |                           \-00.3  Micron NVMe SSD Controller PCC
  +-[0001:00]---00.0-[01-ff]----00.0  Marvell Technology Group Ltd. 
Device 9171
  \-[0000:00]-

martin at ubuntu:~$ ls -lah /sys/kernel/iommu_groups/10/devices/
total 0
drwxr-xr-x 2 root root 0 Feb 17 17:58 .
drwxr-xr-x 3 root root 0 Feb 17 17:58 ..
lrwxrwxrwx 1 root root 0 Feb 17 18:09 0005:00:00.0 -> 
../../../../devices/platform/141a0000.pcie/pci0005:00/0005:00:00.0
lrwxrwxrwx 1 root root 0 Feb 17 18:09 0005:01:00.0 -> 
../../../../devices/platform/141a0000.pcie/pci0005:00/0005:00:00.0/0005:01:00.0
lrwxrwxrwx 1 root root 0 Feb 17 18:09 0005:01:00.2 -> 
../../../../devices/platform/141a0000.pcie/pci0005:00/0005:00:00.0/0005:01:00.2
lrwxrwxrwx 1 root root 0 Feb 17 18:09 0005:01:00.3 -> 
../../../../devices/platform/141a0000.pcie/pci0005:00/0005:00:00.0/0005:01:00.3
lrwxrwxrwx 1 root root 0 Feb 17 18:09 141a0000.pcie -> 
../../../../devices/platform/141a0000.pcie

IOMMU Group 10:
     0005:00:00.0 PCI bridge [0604]: NVIDIA Corporation Device 
[10de:1ad0] (rev a1)
     0005:01:00.0 Non-Volatile memory controller [0108]: Micron NVMe SSD 
Controller PCC [144d:a824]
     0005:01:00.2 Non-Volatile memory controller [0108]: Micron NVMe SSD 
Controller PCC [144d:a824]
     0005:01:00.3 Non-Volatile memory controller [0108]: Micron NVMe SSD 
Controller PCC [144d:a824]

martin at ubuntu:~$ lspci
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 
(rev 13)
0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)   // 
this is PCIe bridge
0005:01:00.0 Non-Volatile memory controller: Micron NVMe SSD Controller 
PCC   // this is PF
0005:01:00.2 Non-Volatile memory controller: Micron NVMe SSD Controller 
PCC   // this is VF1
0005:01:00.3 Non-Volatile memory controller: Micron NVMe SSD Controller 
PCC   //this is VF2

so after creating VFs,  bind the vfio-pci driver to the relevant 
device(s), here I use VF1

martin at ubuntu:~$ sudo modprobe -v vfio-pci
insmod /lib/modules/5.10.104-tegra/kernel/drivers/vfio/vfio.ko
insmod /lib/modules/5.10.104-tegra/kernel/drivers/vfio/vfio_iommu_type1.ko
insmod /lib/modules/5.10.104-tegra/kernel/drivers/vfio/vfio_virqfd.ko
insmod /lib/modules/5.10.104-tegra/kernel/drivers/vfio/pci/vfio-pci.ko

martin at ubuntu:~$ sudo modprobe -r vfio_iommu_type1

martin at ubuntu:~$ sudo modprobe -v vfio_iommu_type1 
allow_unsafe_interrupts=1
insmod 
/lib/modules/5.10.104-tegra/kernel/drivers/vfio/vfio_iommu_type1.ko 
allow_unsafe_interrupts=1

martin at ubuntu:~$ echo vfio-pci > 
/sys/bus/pci/devices/0005\:01\:00.2/driver_override

martin at ubuntu:~$ sudo chmod 777 -R /sys/bus/pci
martin at ubuntu:~$ echo 0005:01:00.2 > /sys/bus/pci/drivers/nvme/unbind
martin at ubuntu:~$ echo 144d a824 > /sys/bus/pci/drivers/vfio-pci/new_id
martin at ubuntu:~$ sudo echo 0005:01:00.2 > /sys/bus/pci/drivers_probe
martin at ubuntu:~$
martin at ubuntu:~$

after that, the VF1 disappears in the nvme list, but "virsh 
nodedev-list" still doesn't have VF PCIe, and I added VF PCIe bus in the 
vm.xml, after start vm,
I got this error,

martin at ubuntu:~$ virsh start --domain vm1
error: Failed to start domain vm1
error: internal error: Found invalid device link '141a0000.pcie' in 
'/sys/bus/pci/devices/0005:01:00.2/iommu_group/devices' //141a0000 is 
PCIe bridge

I checked the IOMMU group 10, VFs, PF and a PCIe bridge are still 
grouped in the same group.

do you know what else I missed?

kind regards,
Martin

On 20.02.23 2:15 PM, Robin Murphy wrote:
> Hi Martin,
>
> On 2023-02-17 15:42, Martin Bayern wrote:
>> Hello,
>>
>> Recently I faced the challenge of enabling SR-IOV on an ARM based 
>> system, PCIe devices have SR-IOV capability, I enabled it and created 
>> its VF, "lspci -nn" can also list these PF and VF PCIe nodes, but the 
>> command  'virsh nodedev-list' lists no PCIe node devices, and the 
>> hypervisor cannot detach the VF from the host, attaches to the 
>> virtual machine. I checked many blogs and papers, it seems that for 
>> ARM-based systems, we should use VFIO-PCI or pci-stub module,I tried 
>> these commands:
>>
>>
>> sudo modprobe -v vfio-pci
>> sudo modprobe -r vfio_iommu_type1
>> sudo modprobe -v vfio_iommu_type1 allow_unsafe_interrupts=1
>>
>> After that, the problem persists, do you know what else I should 
>> check and enable? Does this require ARM64 chipset support SR-IOV? If 
>> yes, how can I check if the CPU supports it?
>
> If you can successfully enable VFs, then the system supports SR-IOV as 
> far as is relevant. Beyond that it's just regular VFIO usage - there's 
> nothing Arm-specific about that (note that if you have a sufficiently 
> modern system you shouldn't need allow_unsafe_interrupts either). 
> Taking the above at face value it looks like you're missing the steps 
> to actually bind the vfio-pci driver to the relevant device(s) - see 
> here:
>
> https://www.kernel.org/doc/html/latest/driver-api/vfio.html?highlight=vfio#vfio-usage-example 
>
>
> However it's also possible that you won't get usefully-assignable 
> groups because the system doesn't support PCIe ACS (and therefore 
> can't prevent peer-to-peer traffic between your VFs and other devices 
> the host is still using).
>
> Thanks,
> Robin.