[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Varun Sethi]

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Tuesday, June 17, 2014 12:24 AM
> To: Will Deacon
> Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
> Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-
> kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar
> Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-
> kernel at lists.infradead.org
> Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
> 
> 
> 
> > -----Original Message-----
> > From: Will Deacon [mailto:will.deacon at arm.com]
> > Sent: Monday, June 16, 2014 12:04 PM
> > To: Yoder Stuart-B08248
> > Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> > devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
> > Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;
> > linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring;
> > Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin;
> > linux-arm- kernel at lists.infradead.org
> > Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> > bindings
> >
> > Hi Stuart,
> >
> > On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > > Do you have use-cases where you really need to change these
> > > > mappings dynamically?
> > >
> > > Yes.  In the case of a PCI bus-- you may not know in advance how many
> > > PCI devices there are until you probe the bus.   We have another FSL
> > > proprietary bus we call the "fsl-mc" bus that is similar.
> >
> > For that case, though, you could still describe an algorithmic
> > transformation from RequesterID to StreamID which corresponds to a
> > fixed mapping.
> >
> > > Another thing to consider-- starting with SMMUv2, as you know, there
> > > is a new distributed architecture with multiple TBUs and a
> > > centralized TCU that walks the SMMU page tables.  So instead of
> > > sprinkling multiple SMMUs all over an SoC you now have the option a
> > > 1 central TCU and
> > sprinkling
> > > multiple TBUs around.   However, this means that the stream ID
> > namespace
> > > is now global and can be pretty limited.  In the SMMU implementation
> > > we have there are only 64 stream ID total for our Soc.  But we have
> > > many
> > more
> > > masters than that.
> > >
> > > So we look at stream IDs as really corresponding to an 'isolation
> > context'
> > > and not to a bus master.  An isolation context is the domain you are
> > > trying to isolate with the SMMU.  Devices that all belong to the
> > > same 'isolation context' can share the same stream ID, since they
> > > share the same domain and page tables.
> >
> > Ok, this is more compelling.
> >
> > > So, perhaps by default some/most SMMU masters may have a default
> > > stream
> > ID
> > > of 0x0 that is used by the host...and that could be represented
> > > statically in the device tree.
> > >
> > > But, we absolutely will need to dynamically set new stream IDs into
> > > masters when a new IOMMU 'domain' is created and devices
> > > are added to it.   All the devices in a domain will share
> > > the same stream ID.
> > >
> > > So whatever we do, let's please have an architecture flexible enough
> > > to allow for this.
> >
> > What is the software interface to the logic that assigns the StreamIDs?
> > Is
> > it part of the SMMU, or a separate device (or set of devices)?
> 
> For us at the hardware level there are a few different ways that the
> streamIDs can be set.  It is not part of the SMMU.  In the cases where
> there is simply
> 1 device to 1 streamID (e.g. USB controller) there is an SoC register
> where
> you just program the stream ID.   In the case of PCI, our PCI controller
> has a RequesterID-to-streamID table that you set via some PCI controller
> registers.
> 
> The way we generally thought it would work was something like
> this:
>    -u-boot/bootloader makes any static streamID allocation if needed,
>     sets a default streamID  (e.g. 0x0) in device and expresses
>     that in the device tree
>    -device tree would express relationship between devices
>     (including bus controllers) and the SMMU through mmu-masters
>     property
>    -u-boot would express the range of unused (or used) streamIDs via a
> new
>     device tree property so the kernel SMMU driver knows what streamIDs
> are
>     free
>    -in the SMMU driver a different vendor specific 'add_device' callback
>     could be used to handle our special cases where we need to set/change
>     the stream ID for devices added to a domain

Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID.

-Varun