[RFC PATCH v2 0/7] iommu/arm-smmu-v3: Use pinned KVM VMID for stage 2

[Jean-Philippe Brucker]

Hi Shameer,

On Thu, Feb 08, 2024 at 03:18:30PM +0000, Shameer Kolothum wrote:
> Hi,
> 
> On an ARM64 system with a SMMUv3 implementation that fully supports
> Broadcast TLB Maintenance(BTM) feature as part of the Distributed
> Virtual Memory(DVM) protocol, the CPU TLB invalidate instructions are
> received by SMMUv3. This is very useful when the SMMUv3 shares the
> page tables with the CPU(eg: Guest SVA use case). For this to work,
> the SMMUv3 must use the same VMID that is allocated by KVM to configure
> the nested stage 2(S2) translations.

The series makes sense to me. Maybe a little more detail to help the KVM
maintainers understand why we need something like this even though the s2
page tables aren't shared between CPU and SMMU:

* When enabling BTM in the SMMU, all TLB invalidations to the
  inner-shareable domain issued by the CPU are taken into account by the
  SMMU. That includes for example the TLBI IPAS2E1IS from
  __kvm_tlb_flush_vmid_range().

* BTM is enabled globally in the SMMU CR2 register. If we enable BTM for
  host SVA, then it also affects KVM.

* Stage-1 TLB entries in the SMMU have a bit (ASET) saying "this entry
  is private and does not participate in BTM", which we set for private
  SMMU address spaces.

  Annoyingly, the stage-2 TLB entries do not have it. With BTM all VMIDs
  are shared between CPU and SMMU.

* So, if the SMMU driver allocates VMID privately and we enable BTM, then
  CPU invalidations will remove unrelated SMMU TLB entries. Instead, the
  SMMU driver needs to coordinate with KVM on VMID allocation.

* Private stage-2 address spaces in the SMMU would need to allocate VMIDs
  that aren't used by KVM, but that's not a use-case at the moment:

  - For assigning devices to a host process or to a VM, we use private
    stage-1 mappings. stage-2 will be used to enable nesting translation,
    and will typically mirror the KVM stage-2 since it pins the guest
    address space.

  - If the SMMU doesn't support stage-1, the driver falls back to stage-2
    for private address spaces. For such an implementation we disable BTM.

  - The old VFIO_TYPE1_NESTING_IOMMU lets userspace allocate a private
    stage-2, and has only been used for testing as far as I know. I don't
    think I ever found a program that used it in the wild, but haven't
    checked recently.

    The effects of using it with BTM enabled is performance degradation:
    TLB entries of that VFIO container get invalidated by unrelated KVM
    activity, and maybe that can be used in a side-channel attack. 
 
    It needs to be deprecated over a few releases (starting with a
    warning maybe?), and the replacement API shouldn't allow creating a
    stage-2 without a KVM context.

Thanks,
Jean


> 
> An earlier proposal sent out[1] a while back resulted in changing the
> ARM64/KVM VMID allocator similar to the ASID allocator to make it
> better suited for this.
> 
> This RFC adds,
>  -Support for pinned KVM VMID.
>  -Support associating KVM pointer and iommufd ctx.
>  -Changes to domain_alloc_user() to receive a kvm pointer.
>  -Configure SMMUV3 S2 using KVM VMID
>  -Finally enable BTM only if SMMUV3 supports S1 translation. This
>   is to make sure that PAGING domains always use S1 and S2 is only
>   used for nested domains with a valid KVM. The idea is to make sure
>   when BTM is enabled in Guest, we use KVM VMID for S2.
> 
> Not sure I miss any explicit TLB invalidations with any use case
> that may configure a S2 with a private VMID that matches a KVM
> one.
> 
> This is based on Jason's ongoing SMMUv3 refactor series[2].
> 
> Please take a look and let me know.
> 
> Thanks,
> Shameer
> 
> 1. https://lore.kernel.org/linux-arm-kernel/20210222155338.26132-1-shameerali.kolothum.thodi@huawei.com/
> 2. https://lore.kernel.org/linux-arm-kernel/0-v5-cd1be8dd9c71+3fa-smmuv3_newapi_p1_jgg@nvidia.com/
> 
> Jean-Philippe Brucker (1):
>   iommu/arm-smmu-v3: Enable broadcast TLB maintenance
> 
> Shameer Kolothum (6):
>   KVM: Add generic infrastructure to support pinned VMIDs
>   KVM: arm64: Introduce support to pin VMIDs
>   KVM: arm64: Add interfaces for pinned VMID support
>   iommufd: Associate kvm pointer to iommufd ctx
>   iommu: Pass in kvm pointer to domain_alloc_user
>   iommu/arm-smmu-v3: Use KVM VMID for s2 stage
> 
>  arch/arm64/include/asm/kvm_host.h           |  3 +
>  arch/arm64/kvm/Kconfig                      |  1 +
>  arch/arm64/kvm/arm.c                        | 14 ++++
>  arch/arm64/kvm/vmid.c                       | 84 ++++++++++++++++++++-
>  drivers/iommu/amd/iommu.c                   |  1 +
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 42 +++++++++--
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  3 +
>  drivers/iommu/intel/iommu.c                 |  1 +
>  drivers/iommu/iommufd/hw_pagetable.c        |  5 +-
>  drivers/iommu/iommufd/iommufd_private.h     |  3 +
>  drivers/iommu/iommufd/main.c                | 14 ++++
>  drivers/iommu/iommufd/selftest.c            |  1 +
>  drivers/vfio/device_cdev.c                  |  3 +
>  include/linux/iommu.h                       |  3 +-
>  include/linux/iommufd.h                     |  7 ++
>  include/linux/kvm_host.h                    | 18 +++++
>  virt/kvm/Kconfig                            |  3 +
>  virt/kvm/kvm_main.c                         | 23 ++++++
>  18 files changed, 218 insertions(+), 11 deletions(-)
> 
> -- 
> 2.34.1
>