Master-aware devices and sideband ID data

[Mark Rutland]

On Fri, May 29, 2015 at 06:46:07PM +0100, Chalamarla, Tirumalesh wrote:
> 
> > On May 27, 2015, at 10:39 AM, Mark Rutland <mark.rutland at arm.com> wrote:
> > 
> > On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
> >> This is some thing we also like to see in ITS and SMMU drivers. 
> >>> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland at arm.com> wrote:
> >>> 
> >>> Hi all,
> >>> 
> >>> For some devices, identification of particular masters is critical to
> >>> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> >>> is determined from sideband signals on the bus, and it may or may not be
> >>> possible to probe and/or configure this information. Worse still, these
> >>> information may be rewritten by intermediate buses, so the
> >>> information for a given master may differ at different points in the bus
> >>> hierarchy.
> >>> 
> >>> We currently describe this master information for devices attached to
> >>> IOMMUs, with a master ID being encoded in the iommu-cells. However this
> >>> only covers the ID between the master and its IOMMU(s), and we don't
> >>> currently have a mechanism to describe the master IDs as they are seen
> >>> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
> >>> 
> >>> The following are example of master IDs:
> >>> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> >>> * SMMU Stream IDs (SIDs)
> >>> * GICv3 ITS Device IDs (DIDs)
> >>> 
> >>> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> >>> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> >>> 
> >>> +-------------+
> >>> | PCIe master |
> >>> +-------------+
> >>>   ||
> >>>   || Requester ID (RID)
> >>>   || Probeable from RC.
> >>>   \/
> >>> +-------------------+
> >>> | PCIe Root Complex |
> >>> +-------------------+
> >>>   ||
> >>>   || SMMU Stream ID (SID)
> >>>   || Derived from RID, static non-probeable mapping.
> >>>   \/
> >>> +--------------+
> >>> | SMMU (IOMMU) |
> >>> +--------------+
> >>>   ||
> >>>   || ITS Device ID (DID)
> >>>   || Derived from SID, static non-probeable mapping.
> >>>   \/
> >>> +----------------------------+
> >>> | GICv3 ITS (MSI controller) |
> >>> +----------------------------+
> >>> 
> >>> In simpler cases, you might have a simpler set of master ID
> >>> translations, e.g. just a DID:
> >>> 
> >>> +-----------------+
> >>> | Platform device |
> >>> +-----------------+
> >>>   ||
> >>>   || ITS Device ID (DID)
> >>>   || Hard-wired on the bus.
> >>>   \/
> >>> +----------------------------+
> >>> | GICv3 ITS (MSI controller) |
> >>> +----------------------------+
> >>> 
> >>> Ignoring current bindings for the moment, I can see how we can describe
> >>> this with a generic master id-binding, with master-id-map along the
> >>> lines of interrupt-map, with a tuple format like:
> >>> <child-id-base child-id-length parent parent-id-base>
> >>> 
> >>> For the PCIe example, this would look something like the following (with
> >>> properties omitted for brevity):
> >>> 
> >>> PCI: pci at af000000 {
> >>> 	...
> >>> 	
> >>> 	/* Requester ID of PCIe endpoints, implicit at runtime */
> >>> 	master-id-cells = <1>;
> >>> 
> >>> 	/* RIDS idmapped to SIDS @ SMMU */
> >>> 	master-id-map = <0 0x10000 &SMMU 0>;
> >>> }
> >>> 
> >>> SMMU: iommu at bf000000 {
> >>> 	...
> >>> 
> >>> 	/* SID, derived from RID */
> >>> 	master-id-cells = <1>;
> >>> 
> >>> 	/* 
> >>> 	 * For some reason the high bit of the ID was negated.
> >>> 	 */
> >>> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
> >>> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
> >>> 
> >>> };
> >>> 
> >>> ITS: its at cf000000 {
> >>> 	...
> >>> 
> >>> 	/* DID, derived from SID */
> >>> 	master-id-cells = <2>;
> >>> 
> >>> 	/* 
> >>> 	 * Master traffic not propagated beyond this point, so no
> >>> 	 * master-id-ranges
> >>> 	 */
> >>> };
> >> 
> >> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  
> > 
> > The set of IDs that we expect to transform (i.e. those masters use)
> > would be implicit in the first master-id-map from a master. In the PCI
> > example above, no master is expected to generate an ID outside of the
> > range 0-0x10000 inclusive (and that's all the SMMU would see).
> > 
> > For devices which are not hotpluggable, the nodes for those devices
> > would describe the specific set of IDs they use.
> > 
> > Generally, between a master and a slave there might not be one
> > contiguous set of valid IDs as these may be rewritten along the way (as
> > happens at the SMMU between the PCI root complex and the ITS in the
> > example above).
> > 
> > Which drivers do you think need this information? What exactly are they
> > trying to check and enforce?
> > 
> 
> I think, i miss understood by the names, so it is not base it is some thing like
> 
> x => f(x) => y           (and f(x) is bitwise | )
> 
> ok now i see it. 

It's subtraction and addition rather than bitwise ORing.

Consider a SID 0x8040 entering the SMMU. This falls in the range
described by the SMMU's second master-id-map entry (I've added brackets
to separate each entry:
 
<(0x8000) (0x8000) (&ITS) (0x0 0x0000)>

As 0x8040 falls in the range described by the first two elements, we kow
we can translate the ID to calculate the ID at the ITS. To do so, we
subtract the child-id-base (0x8000), and add the parent-id-base (0x0
0x0000). That give us 0x0 0x0040 as our ITS device ID.

> >>> For the simpler case, this would look something like:
> >>> 
> >>> DEV: device at af000000 {
> >>> 	master-id-cells = <1>;
> >>> 	master-ids = <0xf>, <0xb>;
> >>> 	master-id-map = <0 0xf &ITS 0 0>;
> >>> };
> >>> 
> >>> ITS: its at cf000000 {
> >>> 	...
> >>> 
> >>> 	/* DID */
> >>> 	master-id-cells = <2>;
> >>> };
> >>> 
> >> Is this is not depending heavily on discover order, how do drivers
> >> know which device to get which ID. is it implicitly assumed in
> >> discovery order? 
> > 
> > I'm not sure I follow the question. Could you elaborate?
> > 
> >> what happens to hot pluggable devices.  
> > 
> > It would be necessary to be able to discover the ID assigned to the
> > device by the hotpluggable bus. For example, this could depend on which
> > slot the device is plugged into.
> > 
> > If you can't discover the ID associated with a hotpluggable device from
> > the bus it is plugged into I can't see how hotplug could work.
> > 
> > From that point on I would expect the ID transformations to be static as
> > in the example.
> > 
> 
> what if i have an SMMU with two ITS masters and i want to decide which
> one to use based on runtime like which CPU i am serving.
> 
> i am thinking in terms go NUMA based systems, where my pci device is
> on node 0, if the interrupt is targeted to CPU on node 1, i would like
> to use ITS1, otherwise 
> ITS0. 

The binding can express that both these translation paths exist. For
example, on the SMMU node you could have:

master-id-map = <0x0000 0x8000 &ITS_1 0x0 0x0000>,
                <0x0000 0x8000 &ITS_2 0x0 0x0000>;

Which would mean that master IDs are idmapped to both ITS instances.

> is it possible to specify some thing like that or do you think its is
> best to leave to ITS driver. 

It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

Thanks,
Mark.