how test the Translation Support for SMMUv3?

[Ben]

At 2023-12-28 05:30:56, "Nicolin Chen" <nicolinc at nvidia.com> wrote:
>On Wed, Dec 27, 2023 at 03:46:41PM +0800, Ben wrote:
>> Hi Nicolin,
>> I saw your pachset on Add Nested Translation Support for SMMUv3
>> [LWN.net]<https://lwn.net/Articles/931502/>, i have built the
>> kernel and qemu you provided in cover letter.
>> I run a QEMU system as Host, and launch a VM by kvmtool. But i
>> cannot find a IOMMU on VM, how do you test your patchset?
>
>Make sure you build the kernel with CONFIG_IOMMUFD, and QEMU
>likely.
>
>> would you like provide your steps?
>
># A set of sample commands
>echo 0002:01:00.0 | tee /sys/bus/pci/devices/0002\:01\:00.0/driver/unbind; dmesg | tail
>echo 8086 0111 | tee /sys/bus/pci/drivers/vfio-pci/new_id
>
>qemu-1214 -object iommufd,id=iommufd0 \
>-machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
>-cpu host -m 256m -nographic  -kernel /root/Image -bios /root/AAVMF_CODE.fd \
>-initrd /root/buildroot-20200422-aarch64-qemu-test-rootfs.cpio \
>-object memory-backend-ram,size=256m,id=m0 -numa node,cpus=0,nodeid=0,memdev=m0 \

>-device vfio-pci,host=0002:01:00.0,rombar=0,iommufd=iommufd0,id="test0"


Thanks！

I am trying your patchset on FVP (Fixed Virtual Platforms) but failed.

Here is the Host side running on FVP (platform is  rdn1egde).

master:~# echo 0000:05:00.0 > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind
master:~# echo 0abc aced > /sys/bus/pci/drivers/vfio-pci/new_id

when i want to run the QEMU to launch a VM, some failed, like below:

root at master:/# cat qemu-iommufd.sh 
./build/qemu-system-aarch64 -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel /Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=/busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id="test0"
root at master:/# sh qemu-iommufd.sh 
WARNING: Image format was not specified for '/busybox_arm64.ext4' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
qemu-system-aarch64: -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id=test0: vfio 0000:05:00.0: vfio /sys/bus/pci/devices/0000:05:00.0/vfio-dev: failed to load "/sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev"

It looks cannot find the /sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev for this device. 

root at master:/# ls -l /sys/bus/pci/devices/0000\:05\:00.0/vfio-dev/vfio0/
total 0
lrwxrwxrwx 1 root root    0 Dec 31 13:29 device -> ../../../0000:05:00.0
drwxr-xr-x 2 root root    0 Dec 31 13:29 power
lrwxrwxrwx 1 root root    0 Dec 31 13:29 subsystem -> ../../../../../../../../class/vfio-dev
-rw-r--r-- 1 root root 4096 Dec 31 13:20 uevent

any suggestion on that?

BTW, another questions, 
1. does it the device which assigned to VM by VFIO can leverage the nested IOMMU? how about the virtual device emulated  by QEMU without  assigned via VFIO? 
2. when fill the S1 and S2 page table for device on nested IOMMU scenario? does it a shadow page table for vIOMMU on VM? and will trap into hypervisor to refill the real S1 and S2 page table? I am not clear the workflow for your patchset.