[PATCH v9 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 3/3: vfio changes

[Pranav Sawargaonkar]

On Mon, Jun 20, 2016 at 9:12 PM, Pranav Sawargaonkar
<pranav.sawargaonkar at gmail.com> wrote:
> Hi Eric,
>
> Tested this series on APM X-Gene2 with E1000 and sata sil card.
> Tested-By: Pranavkumar Sawargaonkar <psawargaonkar at apm.com>
>
> Thanks,
> Pranav
>
>
> On Thu, Jun 9, 2016 at 1:25 PM, Auger Eric <eric.auger at redhat.com> wrote:
>> Alex,
>>> On Wed, 8 Jun 2016 10:29:35 +0200
>>> Auger Eric <eric.auger at linaro.org> wrote:
>>>
>>>> Dear all,
>>>> Le 20/05/2016 à 18:01, Eric Auger a écrit :
>>>>> Alex, Robin,
>>>>>
>>>>> While my 3 part series primarily addresses the problematic of mapping
>>>>> MSI doorbells into arm-smmu, it fails in :
>>>>>
>>>>> 1) determining whether the MSI controller is downstream or upstream to
>>>>> the IOMMU,
>>>>>     => indicates whether the MSI doorbell must be mapped
>>>>>     => participates in the decision about 2)
>>>>>
>>>>> 2) determining whether it is safe to assign a PCIe device.
>>>>>
>>>>> I think we share this understanding with Robin. All above of course
>>>>> stands for ARM.
>>>>>
>>>>> I get stuck with those 2 issues and I have few questions about iommu
>>>>> group setup, PCIe, iommu dt/ACPI description. I would be grateful to you
>>>>> if you could answer part of those questions and advise about the
>>>>> strategy to fix those.
>>>>
>>>> gentle reminder about the questions below; hope I did not miss any reply.
>>>> If anybody has some time to spent on this topic...
>>>>
>>>>>
>>>>> Best Regards
>>>>>
>>>>> Eric
>>>>>
>>>>> QUESTIONS:
>>>>>
>>>>> 1) Robin, you pointed some host controllers which also are MSI
>>>>> controllers
>>>>> (http://thread.gmane.org/gmane.linux.kernel.pci/47174/focus=47268). In
>>>>> that case MSIs never reach the IOMMU. I failed in finding anything about
>>>>> MSIs in PCIe ACS spec. What should be the iommu groups in that
>>>>> situation. Isn't the upstreamed code able to see some DMA transfers are
>>>>> not properly isolated and alias devices in the same group? According to
>>>>> your security warning, Alex, I would think the code does not recognize
>>>>> it, can you confirm please?
>>>> my current understanding is end points would be in separate groups (assuming
>>>> ACS support) although MSI controller frame is not properly protected.
>>>
>>> We don't currently consider MSI differently from other DMA and we don't
>>> currently have any sort of concept of a device within the intermediate
>>> fabric as being a DMA target.  We expect fabric devices to only be
>>> transaction routers.  We use ACS to determine whether there's any
>>> possibility of DMA being redirected before it reaches the IOMMU, but it
>>> seems that a DMA being consumed by an interrupt controller before it
>>> reaches the IOMMU would be another cause for an isolation breach.
>>>
>> OK thank you for the confirmation
>>>>> 2) can other PCIe components be MSI controllers?
>>>
>>> I'm not even entirely sure what this means.  Would a DMA write from an
>>> endpoint target the MMIO space of an intermediate, fabric device?
>> With the example provided by Robin we have a host controller acting as
>> an MSI controller. I wondered whether we could have some other fabric
>> devices (downstream to the host controller in PCIe terminology) also
>> likely to act as MSI controllers.
>>>
>>>>> 3) Am I obliged to consider arbitrary topologies where an MSI controller
>>>>> stands between the PCIe host and the iommu? in the PCIe space or
>>>>> platform space? If this only relates to PCIe couldn' I check if an MSI
>>>>> controller exists in the PCIe tree?
>>>> In my last series, I consider the assignment of platform device unsafe as
>>>> soon as there is a GICv2m. This is a change in the user experience compared to
>>>> what we have before.
>>>
>>> If the MSI controller is downstream of our DMA translation, it doesn't
>>> seem like we have much choice but to mark it unsafe.  The endpoint is
>>> fully able to attempt to exploit it.
>> OK the orginal question was related to non PCIe topologies:
>>
>> - we know some PCIe fabric topologies where the PCIe host controller
>> implements MSI controller.
>> - Shall we be prepared to address the same kind of issues with platform
>> MSI controllers. Are there some SOCs where we would put an unsafe MSI
>> platform controller before IOMMU translation. Or do we consider it is a
>> platform topology we don't support for assignment?
>>
>>>
>>>>> 4) Robin suggested in a private thread to enumerate through a list of
>>>>> "registered" doorbells and if any belongs to an unsafe MSI controller,
>>>>> consider the assignment is unsafe. This would be a first step before
>>>>> doing something more complex. Alex, would that be acceptable to you for
>>>>> issue #2?
>>>> I implemented this technique in my last series waiting for more discussion
>>>> on 4, 5.
>>>
>>> Seems sufficient.  I don't mind taking a broad swing versus all the
>>> extra complexity of defining which devices are safe vs unsafe.
>> OK
>>>
>>>>> 5) About issue #1: don't we miss tools in dt/ACPI to describe the
>>>>> location of the iommu on ARM? This is not needed on x86 because
>>>>> irq_remapping and IOMMU are at the same place but my understanding is
>>>>> that it is on ARM where
>>>>> - there is no connection between the MSI controller - which implements
>>>>> irq remapping - and the iommu
>>>>> - MSI are conveyed on the same address space as standard memory
>>>>> transactions.
>>>
>>> It seems pretty dubious to me to have fixed address, unprotected MSI
>>> controllers sitting in the DMA space of a device before IOMMU
>>> translation.
>> same for me ;-)
>>   Seems like you not only need to mark interrupts as
>>> unsafe, but exclude the address space of the MSI controller from the
>>> available IOVA space to the user.
>> I currently do not see how to achieve that. The guest can program the
>> assigned device DMA target address with the MSI frame PA. there is no
>> IOMMU to protect. How can we make it if we don't trap on DMA programming?
>>>
>>>>> 6)  can't we live with iommu/MSI controller respective location uncertainty?
>>>>>
>>>>> - in my current series, with the above Xilinx MSI controller, I would
>>>>> see there is an arm-smmu requiring mapping behind the PCI host, would
>>>>> query the characteristics of the MSI doorbell (not implemented by that
>>>>> controller), so no mapping would be done. So it would work I think.
>>>>> - However in case we have this topology: PCIe host -> MSI controller
>>>>> generally used behind an IOMMU (so registering a doorbell) -> IOMMU,
>>>>> this wouldn't work since the doorbell would be mapped.
>>>
>>> I'm a little confused which direction "behind" is here, but it seems
>>> like any time the MSI controller lives in the DMA address space of the
>>> endpoint, both interfering with the available IOVA space to the user
>>> and potentially an attack vector for the user, we need to call it out
>>> as unsafe.  Maybe some of them are for exclusive use of the device and
>>> the attack vector is relatively contained, but they still affect the
>>> IOVA space of the user.  Such a configuration might be safe, but as I
>>> said I'm not opposed to being pretty liberal in applying the unsafe
>>> requirement if the platform has done something unfriendly.  Thanks,
>> OK that's clear.
>>
>> Thank you for your feedbacks
>>
>> Best Regards
>>
>> Eric
>>>
>>> Alex
>>>

Oops sorry top posting in earlier reply,

Tested this series on APM X-Gene2 with E1000 and sata sil card.
Tested-By: Pranavkumar Sawargaonkar <psawargaonkar at apm.com>

Thanks,
Pranav