[PATCH v2 00/19] Update SMMUv3 to the modern iommu API (part 1/3)

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgg at nvidia.com]
> Sent: 13 November 2023 17:53
> To: iommu at lists.linux.dev; Joerg Roedel <joro at 8bytes.org>;
> linux-arm-kernel at lists.infradead.org; Robin Murphy
> <robin.murphy at arm.com>; Will Deacon <will at kernel.org>
> Cc: Michael Shavit <mshavit at google.com>; Nicolin Chen
> <nicolinc at nvidia.com>; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi at huawei.com>
> Subject: [PATCH v2 00/19] Update SMMUv3 to the modern iommu API (part
> 1/3)
> 
> The SMMUv3 driver was originally written in 2015 when the iommu driver
> facing API looked quite different. The API has evolved, especially lately,
> and the driver has fallen behind.
> 
> This work aims to bring make the SMMUv3 driver the best IOMMU driver
> with
> the most comprehensive implementation of the API. After all parts it
> addresses:
> 
>  - Global static BLOCKED and IDENTITY domains with 'never fail' attach
>    semantics. BLOCKED is desired for efficient VFIO.
> 
>  - Support map before attach for PAGING iommu_domains.
> 
>  - attach_dev failure does not change the HW configuration.
> 
>  - Fully hitless transitions between IDENTITY -> DMA -> IDENTITY.
>    The API has IOMMU_RESV_DIRECT which is expected to be
>    continuously translating.
> 
>  - Safe transitions between PAGING -> BLOCKED, do not ever temporarily
>    do IDENTITY. This is required for iommufd security.
> 
>  - Full PASID API support including:
>     - S1/SVA domains attached to PASIDs
>     - IDENTITY/BLOCKED/S1 attached to RID
>     - Change of the RID domain while PASIDs are attached
> 
>  - Streamlined SVA support using the core infrastructure
> 
>  - Hitless, whenever possible, change between two domains
> 
>  - iommufd IOMMU_GET_HW_INFO, IOMMU_HWPT_ALLOC_NEST_PARENT,
> and
>    IOMMU_DOMAIN_NESTED support
> 
> Over all these things are going to become more accessible to iommufd, and
> exposed to VMs, so it is important for the driver to have a robust
> implementation of the API.
> 
> The work is split into three parts, with this part largely focusing on the
> STE and building up to the BLOCKED & IDENTITY global static domains.
> 
> The second part largely focuses on the CD and builds up to having a common
> PASID infrastructure that SVA and S1 domains equally use.
> 
> The third part has some random cleanups and the iommufd related parts.
> 
> Overall this takes the approach of turning the STE/CD programming upside
> down where the CD/STE value is computed right at a driver callback
> function and then pushed down into programming logic. The programming
> logic hides the details of the required CD/STE tear-less update. This
> makes the CD/STE functions independent of the arm_smmu_domain which
> makes
> it fairly straightforward to untangle all the different call chains, and
> add news ones.
> 
> Further, this frees the arm_smmu_domain related logic from keeping track
> of what state the STE/CD is currently in so it can carefully sequence the
> correct update. There are many new update pairs that are subtly introduced
> as the work progresses.
> 
> The locking to support BTM via arm_smmu_asid_lock is a bit subtle right
> now and patches throughout this work adjust and tighten this so that it is
> clearer and doesn't get broken.
> 
> Once the lower STE layers no longer need to touch arm_smmu_domain we
> can
> isolate struct arm_smmu_domain to be only used for PAGING domains, audit
> all the to_smmu_domain() calls to be only in PAGING domain ops, and
> introduce the normal global static BLOCKED/IDENTITY domains using the
> new
> STE infrastructure. Part 2 will ultimately migrate SVA over to use
> arm_smmu_domain as well.
> 
> All parts are on github:
> 
>  https://github.com/jgunthorpe/linux/commits/smmuv3_newapi

Hi Jason,

I had a go with the above branch on our HiSilicon D06 board(SMMUv3).

Basically covered the following functionality test runs:
-Host kernel: boot with DOMAIN_DMA.
-Host kernel: boot with DOMAIN_IDENTITY.
-Host kernel: ACC dev SVA test run with uadk/uadk_tool benchmark.

And with Qemu branch: https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_cdev_v7

-Guest boot with a n/w VF dev assigned, legacy VFIO mode.
-Guest boot with a n/w VF dev assigned, IOMMUFD mode.
-Device Hot plug(add/del) on both VFIO and IOMMUFD modes.

All the tests seems to be fine so far.

FWIW:
Tested-by: Shameer Kolothum <shameerali.kolothum.thodi at huawei.com>

Thanks,
Shameer