how test the Translation Support for SMMUv3?

[Nicolin Chen]

On Sun, Dec 31, 2023 at 10:18:12PM +0800, Ben wrote:
 
> I am trying your patchset on FVP (Fixed Virtual Platforms) but failed.
> 
> Here is the Host side running on FVP (platform is  rdn1egde).
> 
> master:~# echo 0000:05:00.0 > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind
> master:~# echo 0abc aced > /sys/bus/pci/drivers/vfio-pci/new_id
> 
> when i want to run the QEMU to launch a VM, some failed, like below:
> 
> root at master:/# cat qemu-iommufd.sh
> ./build/qemu-system-aarch64 -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel /Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=/busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id="test0"
> root at master:/# sh qemu-iommufd.sh
> WARNING: Image format was not specified for '/busybox_arm64.ext4' and probing guessed raw.
>          Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
>          Specify the 'raw' format explicitly to remove the restrictions.
> qemu-system-aarch64: -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id=test0: vfio 0000:05:00.0: vfio /sys/bus/pci/devices/0000:05:00.0/vfio-dev: failed to load "/sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev"
> 
> It looks cannot find the /sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev for this device.
> 
> root at master:/# ls -l /sys/bus/pci/devices/0000\:05\:00.0/vfio-dev/vfio0/
> total 0
> lrwxrwxrwx 1 root root    0 Dec 31 13:29 device -> ../../../0000:05:00.0
> drwxr-xr-x 2 root root    0 Dec 31 13:29 power
> lrwxrwxrwx 1 root root    0 Dec 31 13:29 subsystem -> ../../../../../../../../class/vfio-dev
> -rw-r--r-- 1 root root 4096 Dec 31 13:20 uevent
> 
> any suggestion on that?

CONFIG_VFIO_DEVICE_CDEV=y

Do you have this enabled in kernel config?

> BTW, another questions,
> 1. does it the device which assigned to VM by VFIO can leverage the nested IOMMU?

I think so, as long as it's behind an IOMMU hardware that supports
nesting (and requiring both host kernel and VMM/qemu patches).

> how about the virtual device emulated  by QEMU without  assigned via VFIO?

The basic nesting feature is about 2-stage translation setup (STE
configuration in SMMU term) and cache invalidation. An emulated
device doesn't exist in the host kernel, so there is no nesting
IMHO.

> 2. when fill the S1 and S2 page table for device on nested IOMMU scenario?
> does it a shadow page table for vIOMMU on VM? and will trap into hypervisor
> to refill the real S1 and S2 page table? I am not clear the workflow for your
> patchset.

S2 page table is created/filled at VM creating stage. It's basically
managed by the hypervisor or host kernel. S1 page table on the other
hand is created inside the guest memory and thus managed by the guest
OS. As I mentioned the above, nesting is all about STE configuration
besides cache invalidation. VMM traps the S1 page table pointer from
the guest and forwards the pointer to the host kernel to then setup
the STE of the device's for a 2-stage translation mode.

Nicolin