SR-IOV on ARM64 system with SMMU

[Robin Murphy]

Hi Martin,

On 2023-02-17 15:42, Martin Bayern wrote:
> Hello,
> 
> Recently I faced the challenge of enabling SR-IOV on an ARM based 
> system, PCIe devices have SR-IOV capability, I enabled it and created 
> its VF, "lspci -nn" can also list these PF and VF PCIe nodes, but the 
> command  'virsh nodedev-list' lists no PCIe node devices, and the 
> hypervisor cannot detach the VF from the host, attaches to the virtual 
> machine. I checked many blogs and papers, it seems that for ARM-based 
> systems, we should use VFIO-PCI or pci-stub module,I tried these commands:
> 
> 
> sudo modprobe -v vfio-pci
> sudo modprobe -r vfio_iommu_type1
> sudo modprobe -v vfio_iommu_type1 allow_unsafe_interrupts=1
> 
> After that, the problem persists, do you know what else I should check 
> and enable? Does this require ARM64 chipset support SR-IOV? If yes, how 
> can I check if the CPU supports it?

If you can successfully enable VFs, then the system supports SR-IOV as 
far as is relevant. Beyond that it's just regular VFIO usage - there's 
nothing Arm-specific about that (note that if you have a sufficiently 
modern system you shouldn't need allow_unsafe_interrupts either). Taking 
the above at face value it looks like you're missing the steps to 
actually bind the vfio-pci driver to the relevant device(s) - see here:

https://www.kernel.org/doc/html/latest/driver-api/vfio.html?highlight=vfio#vfio-usage-example

However it's also possible that you won't get usefully-assignable groups 
because the system doesn't support PCIe ACS (and therefore can't prevent 
peer-to-peer traffic between your VFs and other devices the host is 
still using).

Thanks,
Robin.