RFC iommutests_: Testing software for everything IOMMU

[Joel Granados]

Hello everyone

This is a Request For Comment (RFC) describing a new testing infra
called iommutests. It is motivated by interest expressed during the "IO
Page Fault for all" [1] talk in LPC 2024 [2]. At the end of that
presentation in the "VFIO/IOMMU/PCI MC" microconference, There was a
clear "yes please" after asking if it made sense to have an
infrastructure that could evaluate the full spectrum of IOMMU
interactions — from host software to IOMMU in hardware and in emulated
environments.

What is iommutests?
--------------------
The primary goal of iommutests [3] is to provide a modular framework to
test IOMMU-mediated Direct Memory Access (DMA). It is meant to run
PASS/NO_PASS tests for the interaction between kernel subsystems and
hardware/virtual devices. All this in the hope of clarifying which IOMMU
features are working correctly and which are not. You can find the
project in github : https://github.com/SamsungDS/iommutests

* Test Orchestration:
  This component is implemented in pytest [4], a testing framework where
  tests can be parametrized, filtered and implemented concisely and
  easily. Additionally, pyudev [5] is employed for device enumeration
  and monitoring, as well as querying device properties and attributes.

* Test Executables:
  For the creation and execution of tests, iommutests leans heavily on
  libvfn [6], a zero-dependency C library designed for interacting with
  PCIe-based devices from user-space using the Linux kernel user API.
  The libvfn library can be used to abstract away common lower-level
  interactions which can then be re-used through out.

A Working Demonstration
-----------------------
To better illustrate how everything fits together, There’s a demo script
[7], with some demo notes [8] explaining the setup and output. It does
the following:

1. Sets up a custom qemu [9] virtual machine environment using a custom
   test device.
2. Builds both libvfn and iommutests.
3. Runs a pair of example tests (one of them always fails to show how
   how that would look like)

This demo shows how everything comes together to give the pass/no-pass
IOMMU testing results. It is there for anyone interested in trying it
out or contributing. To run it, you need to pass it a local ssh key and
linux kernel compiled with VFIO support

Custom qemu device: pci-ats-testdev
-------------------------------------
To support IOMMU testing under qemu, the pci-ats-testdev [10]
(different from pci-testdev [11]) was used to emulate DMA transactions.
It is a full fledged pci device capable of executing emulated DMA
accesses. It was originally intended to test Linux kernel interactions
with devices that had a working Address Translation Cache (ATC) but can
become a platform capable of testing anything PCI/IOMMU related if
needed.

Feedback
--------
This is a first draft, and many implementation details are still open to
refinement. I would appreciate your thoughts on any part of the project
— its design, scope, implementation language choices, or usability.
These are however some of the questions that are still outstanding from
my POV:

Q1: Beyond binary testing:
    Would iommutests be used for something other than pass/no-pass
    tests? Like performance? Stress testing?

Q2: Kernel Integration:
    Should iommutests interact with the already existing IOMMU linux
    kernel unit tests? Since it is an orchestration framework, then it
    can execute the unit tests in some way. This could actually be the
    next step, if found useful.

Thanks for your time
Best regards,

-- 

Joel Granados

[1] https://www.youtube.com/watch?v=UFrAjJ5TUf4
[2] https://lpc.events/event/18/timetable/#20240918
[3] https://github.com/SamsungDS/iommutests
[4] https://docs.pytest.org/en/stable/
[5] https://github.com/pyudev/pyudev
[6] https://github.com/SamsungDS/libvfn
[7] https://github.com/SamsungDS/iommutests/blob/master/docs/demo0_debian.sh
[8] https://github.com/SamsungDS/iommutests/blob/master/docs/demo0.md
[9] https://github.com/Joelgranados/qemu/tree/pcie-testdev
[10] https://github.com/Joelgranados/qemu/blob/pcie-testdev/hw/misc/pcie-ats-testdev.c
[11] https://github.com/Joelgranados/qemu/blob/pcie-testdev/hw/misc/pci-testdev.c

PS:
  Apologies for the long CC list. I always try to include the ppl that I
  think will be interested, but its not always easy to know who you are
  :). In this case, you are included because you appear as "M" under a
  subsystem that contains the string "IOMMU" in the MAINTAINERS file. Feel
  free to shoot me a mail if you don't want to be included in the future,
  I'll try my best to remove you.