[PATCH 1/2] irqchip/gicv3-its: Support share device ID

[Marc Zyngier]

On 27/04/15 14:08, Varun Sethi wrote:
> Hi Marc,
> 
>> -----Original Message-----
>> From: Marc Zyngier [mailto:marc.zyngier at arm.com]
>> Sent: Monday, April 27, 2015 1:28 PM
>> To: Sethi Varun-B16395; Yoder Stuart-B08248
>> Cc: Will Deacon; Lian Minghuan-B31939; linux-pci at vger.kernel.org; Arnd
>> Bergmann; Hu Mingkai-B21284; Zang Roy-R61911; Bjorn Helgaas; Wood Scott-
>> B07421; linux-arm-kernel at lists.infradead.org
>> Subject: Re: [PATCH 1/2] irqchip/gicv3-its: Support share device ID
>>
>> Hi Varun,
>>
>> On 26/04/15 19:20, Varun Sethi wrote:
>>> Hi Marc,
>>>
>>>>>> We can deal with the aliasing, provided that we extend the level of
>>>>>> quirkiness that pci_for_each_dma_alias can deal with. But that
>>>>>> doesn't solve any form of hotplug/SR-IOV behaviour.
>>>>>>
>>> [varun] Can you please elaborate on "extending the quirkiness of
>>> pci_for_each_dma_alias". How do you see the case for transparent host
>>> bridege being handled? We would see a device ID corresponding to the
>>> host bridge for masters behind that bridge.
>>
>> The PCI code already has code to deal with aliases, and can deal with them in
>> a number of cases.
>>
>> At the moment, this aliasing code can only deal with aliases that belong to
>> the same PCI bus (or aliasing with the bus itself). Given the way the problem
>> has been described, I understand that you can have devices sitting on
>> different buses that will end up with the same DeviceID. This is where
>> expanding the "quirkiness" of pci_for_each_dma_alias comes into play. You
>> need to teach it about this kind of topology.
>>
> [varun] Agreed, in our case the PCIe controller maintains a stream ID to device ID  translation table. So, we can actually avoid this problem by setting up unique stream IDs across PCIe controllers. We would need a layer to allow translation from device id to stream ID.
> 
>>>>>> Somehow, we're going to end-up with grossly oversized ITTs, just to
>>>>>> accommodate for the fact that we have no idea how many MSIs we're
>>>>>> going to end-up needing. I'm not thrilled with that prospect.
>>>>>
>>>>> How can we avoid that in the face of hotplug?
>>>>
>>>> Fortunately, hotplug is not always synonymous of aliasing. The ITS is
>>>> built around the hypothesis that aliasing doesn't happen, and that
>>>> you know upfront how many LPIs the device will be allowed to generate.
>>>>
>>>>> And what are we really worried about regarding over-sized
>>>>> ITTs...bytes of memory saved?
>>>>
>>>> That's one thing, yes. But more fundamentally, how do you size your
>>>> MSI capacity for a single alias? Do you evenly split your LPI space
>>>> among all possible aliases? Assuming 64 aliases and 16 bits of
>>>> interrupt ID space, you end up with 10 bit per alias. Is that always
>>>> enough? Or do you need something more fine-grained?
>>>>
>>>>> A fundamental thing built into the IOMMU subsystem in Linux is
>>>>> representing iommu groups that can represent things like multiple
>>>>> PCI devices that for hardware reasons cannot be isolated (and the
>>>>> example I've seen given relates to devices behind PCI bridges).
>>>>>
>>>>> So, I think the thing we are facing here is that while the IOMMU
>>>>> subsystem has accounted for reprsenting the isolation
>>>>> characteristics of a system with iommu groups, there is no corresponding
>> "msi group"
>>>>> concept.
>>>>>
>>>>> In the SMMU/GIC-500-ITS world the iommu isolation ID (the stream ID)
>>>>> and the GIC-ITS device ID are in fact the same ID.
>>>>
>>>> The DeviceID is the "MSI group" you mention. This is what provides
>>>> isolation at the ITS level.
>>>>
>>> [varun] True, in case of a transparent host bridge device Id won't
>>> provide the necessary isolation.
>>
>> Well, it depends how you look at it. How necessary is this isolation, since
>> we've already established that you couldn't distinguish between these
>> devices at the IOMMU level?
>>
> [varun] Yes, the devices would fall in the same IOMMU group. So,
> devices would end up sharing the interrupt?

No, they would end-up sharing an Interrupt Translation Table (ITT),
Basically the equivalent of your page IOMMU page tables, but for
interrupts. Basically, all the devices in this group would have their
own set of interrupts, but would also be able to spoof interrupts for
each others. Probably not a big deal isolation wise, but problematic
from an allocation point of view (given the current way the ITS driver
is architected).

	M.
-- 
Jazz is not dead. It just smells funny...