[PATCH RFCv1 07/14] iommufd: Add viommu set/unset_dev_id ops

[Tian, Kevin]

> From: Nicolin Chen <nicolinc at nvidia.com>
> Sent: Thursday, May 23, 2024 2:10 PM
> 
> On Thu, May 23, 2024 at 05:44:40AM +0000, Tian, Kevin wrote:
> > > From: Nicolin Chen <nicolinc at nvidia.com>
> > > Sent: Monday, May 13, 2024 12:39 PM
> > >
> > > On Sun, May 12, 2024 at 11:46:54AM -0300, Jason Gunthorpe wrote:
> > > > On Fri, Apr 12, 2024 at 08:47:04PM -0700, Nicolin Chen wrote:
> > > > > Add a pair of ops to set and unet device's virtual ID that belongs to
> > > > > a viommu object. They will be used, in the following patch, by
> iommufd
> > > > > to support some HW-acceleration feature from the host level.
> > > > >
> > > > > For instance, every device behind an ARM SMMU has a Stream ID. The
> ID
> > > > > is used by ATC invalidation commands so SMMU HW can direct
> > > invalidation
> > > > > requests to the corresponding PCI device where the ID belongs to. In a
> > > > > virtualization use case, a passthroughed device in the VM will have a
> > > > > virtuail Stream ID, used by the ATC invalidation commands in the
> guest
> > > > > system. NVIDIA's CMDQV extension for SMMUv3 provides a v-
> interface
> > > to
> > > > > execute the guest-level ATC invalidation commands directly, yet needs
> > > > > the HW to be aware of its virtual Stream ID so it can replace with its
> > > > > physical Stream ID.
> > > >
> > > > I imagine using this as well for the ATC invalidation commands. It
> > > > would be very easy and simplifying if the command fixup just extracted
> > > > the vSID from the ATC invalidation and used an xarray to turn it into
> > > > a pSID and then pushed the resulting command.
> > >
> > > You mean the nested SMMU series right? Actually the set_dev_id
> > > ioctl was a part of that until we wanted to try DEV_INVALIDATE.
> > >
> > > So again, yes, it makes sense to me that we move viommu and the
> > > set_dev_id to the nested series, and then drop DEV_INVALIDATE.
> >
> > I'm right about to ask how the nesting series is going. Per earlier
> > discussion iirc the nesting series will go in before VCMDQ?
> 
> Yes. It still should. Yet we ended up with adding VIOMMU to the
> nested SMMU series too. A shared S2 domain/hwpt isn't exclusive
> for VCMDQ use case but also for regular nesting on a multi-SMMU
> setup. So, VIOMMU turns out to be the best object that we have
> at this moment to hold individual VMIDs for different physical
> SMMUs sharing a single S2 domain. Its virtual device ID lookup
> feature can also allow us to forget about DEV_INVALIDATE ioctl
> for now.

Yeah, that makes sense. btw it's probably helpful to future reviews
if you can try to include more explanations about the design
tradeoff based on those discussions. Somehow it's not easy for me
to catch up with those discussions being lacking of many contexts
here. 😊

> 
> Jason listed all the tasks ahead in this thread too, using SMMU
> as an example:
> > So we have this stuff still open:
> >  - Identity STE with PASID (part 2b)
> >  - IOMMU_GET_HW_INFO (part 3)
> >  - IOMMU_HWPT_ALLOC_NEST_PARENT (part 3)
> >  - IOMMU_HWPT_DATA_ARM_SMMUV3 (part 3)
> >  - IOMMU_HWPT_INVALIDATE_DATA_ARM_SMMUV3
> >  - VIOMMU_ALLOC, VIOMMU_ATTACH
> >  - VIOMMU_INVALIDATE
> 
> By this series nesting setup is done. We need Baolu's solution
> or VQUEUE for fault reporting after that.
> 

this is a clear plan.