[PATCH 1/2] irqchip/gicv3-its: Support share device ID

[Stuart Yoder]

+Alex W

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Friday, April 24, 2015 1:19 PM
> To: 'Marc Zyngier'; Will Deacon
> Cc: Sethi Varun-B16395; Lian Minghuan-B31939; linux-pci at vger.kernel.org; Arnd Bergmann; Hu Mingkai-B21284;
> Zang Roy-R61911; Bjorn Helgaas; Wood Scott-B07421; linux-arm-kernel at lists.infradead.org
> Subject: RE: [PATCH 1/2] irqchip/gicv3-its: Support share device ID
> 
> 
> 
> > -----Original Message-----
> > From: Marc Zyngier [mailto:marc.zyngier at arm.com]
> > Sent: Friday, April 24, 2015 11:44 AM
> > To: Will Deacon; Yoder Stuart-B08248
> > Cc: Sethi Varun-B16395; Lian Minghuan-B31939; linux-pci at vger.kernel.org; Arnd Bergmann; Hu Mingkai-B21284;
> > Zang Roy-R61911; Bjorn Helgaas; Wood Scott-B07421; linux-arm-kernel at lists.infradead.org
> > Subject: Re: [PATCH 1/2] irqchip/gicv3-its: Support share device ID
> >
> > On 24/04/15 17:18, Will Deacon wrote:
> > > On Wed, Apr 22, 2015 at 08:41:02PM +0100, Stuart Yoder wrote:
> > >>>> However, there is an improvement we envision as possible due to
> > >>>> the limited number of SMMU contexts (i.e. 64).  If there are
> > >>>> 64 SMMU context registers it means that there is a max of
> > >>>> 64 software contexts where things can be isolated.  But, say I have
> > >>>> an SRIOV card with 64 VFs, and I want to assign 8 of the VFs
> > >>>> to a KVM VM.  Those 8 PCI devices could share the same
> > >>>> streamID/ITS-device-ID since they all share the same isolation
> > >>>> context.
> > >>>>
> > >>>> What would be nice is at the time the 8 VFS are being added
> > >>>> to the IOMMU domain is for the pcidevid -> streamID mapping
> > >>>> table to be updated dynamically.  It simply lets us make
> > >>>> more efficient use of the limited streamIDs we have.
> > >>>>
> > >>>> I think it is this improvement that Minghuan had in mind
> > >>>> in this patch.
> > >>>
> > >>> Ok, but in this case it should be possible to use a single context bank for
> > >>> all of the VF streamIDs by configuring the appropriate SMR, no?
> > >>
> > >> Yes, but there are limited SMRs.  In our case there are only
> > >> 128 SMRs in SMMU-500 and we have potentially way more masters than
> > >> that.
> > >
> > > Right, but you still only have 64 context banks at the end of the day, so do
> > > you really anticipate having more than 128 masters concurrently using the
> > > SMMU? If so, then we have devices sharing context banks so we could consider
> > > reusing SMRs across masters, but historically that's not been something that
> > > we've managed to solve.
> > >
> > >>> Wouldn't
> > >>> that sort of thing be preferable to dynamic StreamID assignment? It would
> > >>> certainly make life easier for the MSIs.
> > >>
> > >> It would be preferable, but given only 128 total stream IDS and
> > >> 64 context registers it's potentially an issue.  On our LS2085 SoC it is
> > >> PCI and the fsl-mc bus (see description here:
> > >> https://lkml.org/lkml/2015/3/5/795) that potentially have way
> > >> more masters than streamIDS. So, for those busses we would essentially
> > >> view a streamID as a "context ID"-- each SMR is associated with
> > >> 1 context bank register.
> > >>
> > >> For PCI we have a programmable "PCI req ID"-to-"stream ID"
> > >> mapping table in the PCI controller that is dynamically
> > >> programmable.
> > >>
> > >> Looking at it like that means that we could have
> > >> any number of masters but only 64 "contexts"
> > >> and since the masters all all programmable it's
> > >> seems feasbile to envision doing some bus/vendor
> > >> specific set up when a device is added to an
> > >> IOMMU domain.   One thing that would need to be conveyed
> > >> to the SMMU driver if doing dynamic streamID setup
> > >> is what streamIDs are available to be used.
> > >
> > > Ok, but this is going to make life difficult for the MSI people, I suspect.
> > >
> > > Marc...?
> >
> > We're really facing two conflicting requirements: in order to minimize
> > SMR usage, we want to alias multiple ReqIDs to a single StreamID
> 
> Another reason could be the isolation characteristics of the
> hardware...see comment below about PCI bridges.
> 
> > but in
> > order to efficiently deal with MSIs, we want to see discrete DeviceIDs
> > (the actual ReqIDs). I don't easily see how we reconcile the two.
> >
> > We can deal with the aliasing, provided that we extend the level of
> > quirkiness that pci_for_each_dma_alias can deal with. But that doesn't
> > solve any form of hotplug/SR-IOV behaviour.
> >
> > Somehow, we're going to end-up with grossly oversized ITTs, just to
> > accommodate for the fact that we have no idea how many MSIs we're going
> > to end-up needing. I'm not thrilled with that prospect.
> 
> How can we avoid that in the face of hotplug?
> 
> And what are we really worried about regarding over-sized ITTs...bytes
> of memory saved?
> 
> A fundamental thing built into the IOMMU subsystem in Linux is
> representing iommu groups that can represent things like
> multiple PCI devices that for hardware reasons cannot
> be isolated (and the example I've seen given relates to
> devices behind PCI bridges).
> 
> So, I think the thing we are facing here is that while the
> IOMMU subsystem has accounted for reprsenting the isolation
> characteristics of a system with iommu groups, there is
> no corresponding "msi group" concept.
> 
> In the SMMU/GIC-500-ITS world the iommu isolation
> ID (the stream ID) and the GIC-ITS device ID are in
> fact the same ID.
> 
> Is there some way we could sanely correlate IOMMU group creation
> (which establishes isolation granularity) with the creation
> of an ITT for the GIC-ITS?
> 
> (I don't have a good feel for how device IDs work on x86,
> I know there is an interrupt remapping table in the
> x86 IOMMUs that is distinct from the memory isolation
> page tables)

For reference see Alex Williamson's description of IOMMU
isolation scenarios in Documentation/vfio.txt:

   ...
   This isolation is not always at the granularity of a single device
   though.  Even when an IOMMU is capable of this, properties of devices,
   interconnects, and IOMMU topologies can each reduce this isolation.
   For instance, an individual device may be part of a larger multi-
   function enclosure.  While the IOMMU may be able to distinguish
   between devices within the enclosure, the enclosure may not require
   transactions between devices to reach the IOMMU.  Examples of this
   could be anything from a multi-function PCI device with backdoors
   between functions to a non-PCI-ACS (Access Control Services) capable
   bridge allowing redirection without reaching the IOMMU.  Topology
   can also play a factor in terms of hiding devices.  A PCIe-to-PCI
   bridge masks the devices behind it, making transaction appear as if
   from the bridge itself.  Obviously IOMMU design plays a major factor
   as well.

   Therefore, while for the most part an IOMMU may have device level
   granularity, any system is susceptible to reduced granularity.  The
   IOMMU API therefore supports a notion of IOMMU groups.  A group is
   a set of devices which is isolatable from all other devices in the
   system.  Groups are therefore the unit of ownership used by VFIO.
   ...

The isolation granularity issues he raises are going to impact how
we deal with MSIs with GIC ITS because again, isolation ID and 
GIC ITS device ID are the same.

Stuart