[RFC PATCH v3 0/5] iommu/arm-smmu-v3: Use pinned KVM VMID for stage 2

[Shameer Kolothum]

Hi All,

Changes from RFCv2[0]

-Based on feedback received, removed generic KVM APIs for pinned_vmid_
 {get,put}() and are now implemented directly by KVM ARM.
-struct kvm * is now associated with struct iommufd_device and the association
 is established when idev is first allocated during iommufd_device_bind().
-This is now based on Nicolin's recent series[1], which decouples the vmid
 from S2 parent domains and is assigned during viommu/vSMMU alloc time.
-Added additional checks to make sure when BTM is enabled, kvm is mandatory.
-At the moment, kvm_get_kvm_safe()/kvm_put_kvm() is used inside the smmuv3
 driver and not in iommufd as iommufd is not a user of kvm and is just
 passed through.

ToDo:
-May be we need to check both KVM and SMMUv3 has the same number of VMID
bits supported. Not sure we have systems like that out there.

Why we require this(Thanks to Jean for the nice write-up below):

---
* When enabling BTM in the SMMU, all TLB invalidations to the
  inner-shareable domain issued by the CPU are taken into account by the
  SMMU. That includes for example the TLBI IPAS2E1IS from
  __kvm_tlb_flush_vmid_range().

* BTM is enabled globally in the SMMU CR2 register. If we enable BTM for
  host SVA, then it also affects KVM.

* Stage-1 TLB entries in the SMMU have a bit (ASET) saying "this entry
  is private and does not participate in BTM", which we set for private
  SMMU address spaces.

  Annoyingly, the stage-2 TLB entries do not have it. With BTM all VMIDs
  are shared between CPU and SMMU.

* So, if the SMMU driver allocates VMID privately and we enable BTM, then
  CPU invalidations will remove unrelated SMMU TLB entries. Instead, the
  SMMU driver needs to coordinate with KVM on VMID allocation.

* Private stage-2 address spaces in the SMMU would need to allocate VMIDs
  that aren't used by KVM, but that's not a use-case at the moment:

  - For assigning devices to a host process or to a VM, we use private
    stage-1 mappings. stage-2 will be used to enable nesting translation,
    and will typically mirror the KVM stage-2 since it pins the guest
    address space.

  - If the SMMU doesn't support stage-1, the driver falls back to stage-2
    for private address spaces. For such an implementation we disable BTM.
---

This is sanity tested on a HiSilicon platform and the complete branch is
available here[2].

Please take a look and let me know your feedback.

Thanks,
Shameer
[0] https://lore.kernel.org/kvmarm/20240208151837.35068-1-shameerali.kolothum.thodi@huawei.com/
[1] https://lore.kernel.org/linux-iommu/cover.1741150594.git.nicolinc@nvidia.com/
[2] https://github.com/hisilicon/kernel-dev/tree/smmuv3_vmid-v1-with-rmr-vmid-v3-ext

Jean-Philippe Brucker (1):
  iommu/arm-smmu-v3: Enable broadcast TLB maintenance

Shameer Kolothum (4):
  KVM: arm64: Introduce support to pin VMIDs
  iommufd/device: Associate a kvm pointer to iommufd_device
  iommu/arm-smmu-v3-iommufd: Pass in kvm pointer to viommu_alloc
  iommu/arm-smmu-v3-iommufd: Use KVM VMID for s2 stage

 arch/arm64/include/asm/kvm_host.h             |  3 +
 arch/arm64/kvm/vmid.c                         | 76 ++++++++++++++++++-
 .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     | 54 ++++++++++++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 24 +++++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  3 +
 drivers/iommu/iommufd/device.c                |  5 +-
 drivers/iommu/iommufd/iommufd_private.h       |  2 +
 drivers/iommu/iommufd/viommu.c                |  3 +-
 drivers/vfio/iommufd.c                        |  2 +-
 include/linux/iommu.h                         |  4 +-
 include/linux/iommufd.h                       |  4 +-
 11 files changed, 168 insertions(+), 12 deletions(-)

-- 
2.47.0