[PATCH v1 4/5] iommu/arm-smmu-v3: Keep track of attached ssids

[Michael Shavit]

Oh nice, this is exactly what I was looking for (minus the missing ATC
inv, but that's somewhat easier to reason from code)! Thanks for the
detailed guide Jean!
I finally got around to trying it out and was able to see the page
fault followed by invalidations on this patch-series as it is. This
will be super useful to start refactoring SVA for a v2 of this patch.

On Fri, May 12, 2023 at 3:59 AM Jean-Philippe Brucker
<jean-philippe at linaro.org> wrote:
>
> On Thu, May 11, 2023 at 11:26:48PM +0800, Michael Shavit wrote:
> > > You should be getting rid of mm->pasid in this series as well.
> > >
> > > When each domain keeps track of what STE/CD entries that point to it then
> > > *ALL* invalidation should iterate over the list of pointing entires
> > > and generate the correct invalidation for that pointer.
> > >
> >
> > Completely agree. The arm_smmu_atc_inv_domain_ssid function introduced
> > by this patch is a stopgap to decompose this patch from the SVA
> > refactor that's required to stop using ssid in these calls.
> > I also agree that such a refactoring probably belongs in the same
> > patch series. @Jean-Philippe Brucker and others: is there any way I
> > can about testing or at least exercising the SVA flow without physical
> > hardware that supports SVA?
>
> Yes, there is a model with a simple test device that supports PASID and
> I/O page faults. It's not completely straightforward to setup and the
> driver needs to be rewritten from scratch, but it's the best we have at
> the moment.  I'd like to do something equally useful for QEMU, so we can
> have proper regression tests, but that requires a lot of preliminary work
> to add PASID+PRI to PCI, virtio and IOMMUs.
>
> You'll need a kernel with the driver and a rootfs with the smmute
> tool [1]; the RevC model [2] and a boot-wrapper [3].
>
>   $ ${BOOTWRAPPER}/configure --host=aarch64-linux-gnu
>      --with-dtb=${KERNEL}/arch/arm64/boot/dts/arm/fvp-base-revc.dts
>      --with-kernel-dir=${KERNEL}
>      --with-initrd=${BUILDROOT}/images/rootfs.cpio
>      --with-cmdline="console=ttyAMA0"
>      --enable-psci --enable-gicv3
>   $ make        # produces linux-system.axf
>
> Run the model:
>   $ FVP_Base_RevC-2xAEMvA
>      -C bp.secure_memory=0
>      -C 'pctl.startup=0.*.*.*'
>      -C bp.refcounter.non_arch_start_at_default=1
>      -C cache_state_modelled=0
>      -C bp.vis.disable_visualisation=1
>      -C bp.virtio_net.enabled=1
>      -C bp.virtio_net.hostbridge.userNetPorts=8022=22
>      -C bp.virtio_net.hostbridge.userNetworking=1
>      -C pci.pci_smmuv3.mmu.SMMU_IDR0=135100351
>      -C pci.pci_smmuv3.mmu.SMMU_IDR3=4116
>      -C pci.pci_smmuv3.mmu.SMMU_IDR5=8389749
>      -C cluster0.NUM_CORES=4
>      -C cluster1.NUM_CORES=4
>      -a 'cluster*.cpu*=linux-system.axf'
>
> Then run a job using the tool. The process allocates two buffers and
> passes their VA to the device (via the kernel driver). The device memcpies
> one buffer to the other:
>
>    $ smmute -u mmap
>    ... Success
>
> With smmu and iommu trace events enabled, a trace should contain
> smmu_mm_invalidate and dev_fault/dev_page_response events.
>
> It's not entirely representative of SVA flow, where an assignable device
> interface is mapped into the process and the process launches jobs
> directly without going through the kernel (that would now use
> drivers/misc/uacce), but it does exercise IOMMU SVA: sva_bind(), device
> accessing the process address space with PASID and some IOPFs, which I
> think is what you're looking for. However this model doesn't have a PCI
> test device so you won't be able to test ATC invalidations with PASID.
>
> Other useful tests would be enabling lockdep (some intricate locking
> between IOPF, the driver and mm), killing bound processes (-k), triggering
> invalid accesses to verify TLB invalidations (-f tlb, I think). There is a
> lot more to test, like thp and oom, but I don't have those in this branch.
>
> Thanks,
> Jean
>
> [1] https://jpbrucker.net/git/linux/log/?h=sva/smmute-revc
> [2] https://developer.arm.com/downloads/-/arm-ecosystem-models
> [3] https://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git/