[Design] iommu/arm-smmu-v3: Implementing set_dev_pasid for other domain types

[Michael Shavit]

Hi Jean-Philippe, all

We’d like to use the new iommu_attach_device_pasid API to control
mappings on a per SSID basis. To that end, we’ve started investigating
how to implement the set_dev_pasid callback in the arm-smmu-v3.c
driver for domain types other than SVA.

As noted in your smmuv3 aux-domain patch, the arm_smmu_domain
represents not only “the non-PASID address space but also (at least
for SMMUv3) the whole device context, PASID tables etc.” Multiple SVA
iommu_domains reach into a shared arm_smmu_domain to write their CDs
into a shared CD table.
Is moving to a model where each arm_smmu_master holds this previously
shared device context instead of the arm_smmu_domain a bad idea?

This would allow a pretty straightforward implementation of
set_dev_pasid for other domain types:
    * Attaching a domain with PASID to a device doesn’t have to
piggyback on any other domain’s CD table. The domain’s CD can directly
be written into the master’s CD table.
    * A domain attached with PASID is no different than one attached
without PASID, attach_dev becomes attach_dev_with_pasid(0).

At a high-level:
   1. The arm_smmu_domain would still own the arm_smmu_ctx_desc struct
that describes that domain (translation table, etc)
   2. Each arm_smmu_master would now own its own s1_cfg struct (and CD
table). Attaching a domain to a device would write the CD into that
device’s CD table instead of the previously shared CD table.

>From an implementation POV there’s a few other changes and
book-keeping that become required to handle edge-cases (e.g. We need
to find all the CD entries that a domain is written to when the SVA
driver tries to update a domain’s ASID value). Right now I’m more
concerned about whether this idea is sane at a design level.

As a first order consequence, when multiple devices share a domain:
    1. We’ll use more memory than we would have before.
    2. The SMMU will perform dma to distinct CD tables to find the
page table; however the page table itself would still be shared.

Is that acceptable considering some users might not need the
set_dev_pasid support this is geared towards?
Are there other consequences we've overlooked?

Thanks,
Michael