[PATCH] arm64: dts: rockchip: disable IOMMU when running rk3588 in PCIe endpoint mode

[Robin Murphy]

On 2025-02-11 7:44 pm, Niklas Cassel wrote:
> Hello Robin,
> 
> On Tue, Feb 11, 2025 at 05:49:29PM +0000, Robin Murphy wrote:
>> On 2025-02-07 2:39 pm, Niklas Cassel wrote:
>>> Commit da92d3dfc871 ("arm64: dts: rockchip: enable the mmu600_pcie IOMMU
>>> on the rk3588 SoC") enabled the mmu600_pcie IOMMU, both in the normal case
>>> (when all PCIe controllers are running in Root Complex mode) and in the
>>> case when running the pcie3x4 PCIe controller in Endpoint mode.
>>>
>>> There have been no issues detected when running the PCIe controllers in
>>> Root Complex mode. During PCI probe time, we will add a SID to the IOMMU
>>> for each PCI device enumerated on the bus, including the root port itself.
>>>
>>> However, when running the pcie3x4 PCIe controller in Endpoint mode, we
>>> will only add a single SID to the IOMMU (the SID specified in the iommus
>>> DT property).
>>>
>>> The enablement of IOMMU in endpoint mode was verified on setup with two
>>> Rock 5b:s, where the BDF of the Root Complex has BDF (00:00.0).
>>>
>>> A Root Complex sending a TLP to the Endpoint will have Requester ID set
>>> to the BDF of the initiator. On the EP side, the Requester ID will then
>>> be used as the SID. This works fine if the Root Complex has a BDF that
>>> matches the iommus DT property, however, if the Root Complex has any other
>>> BDF, we will see something like:
>>> arm-smmu-v3 fc900000.iommu: event: C_BAD_STREAMID client: (unassigned sid) sid: 0x1600 ssid: 0x0
>>> on the endpoint side.
>>>
>>> For PCIe controllers running in endpoint mode that always uses the
>>> incoming Requester ID as the SID, the iommus DT property simply isn't
>>> a viable solution. (Neither is iommu-map a viable solution, as there is
>>> no enumeration done on the endpoint side.)
>>
>> Well, strictly the controller should own all the StreamIDs it's capable of
>> emitting - if that's just one per possible bus number (as the iATU
>> FUNC_NUM/FUNC_BYPASS stuff appears to suggest?) then 256 "iommus" entries
>> isn't *entirely* unmanageable. If it were to support being arbitrary (or
>> multiple) devices/functions, though, then we probably would want to look for
>> a different solution, because nobody wants a 512KB DT property... :)
> 
> Well, FUNC_BYPASS and FUNC_NUM is in the IATU_REGION_CTRL_1_OFF_OUTBOUND_i
> register, so it is for outbound PCI TLPs, not inbound PCI TLPs.
> (Only inbound PCI TLPs will get sent to the IOMMU after passing the iATU).
> 
> There is FUNC_NUM in IATU_REGION_CTRL_1_OFF_INBOUND_i register, but it has
> a different meaning. (An inbound PCI TLP will only get translated if the
> FUNC_NUM matches the value in this register).
> (This check is only performed if the "Function Number Match Enable" bit
> of the "iATU Region Control 2 Register" is set.)

Sigh, the "i" in iATU doesn't stand for "inbound", does it... major 
brain fart there.

> The SMMU on the rock5b, when running the PCIe controller in endpoint mode,
> has printed the following:
> arm-smmu-v3 fc900000.iommu: event: C_BAD_STREAMID client: (unassigned sid) sid: 0x1600 ssid: 0x0
> but also:
> arm-smmu-v3 fc900000.iommu: event: C_BAD_STREAMID client: (unassigned sid) sid: 0x20 ssid: 0x0

Yeah, that one pretty much settles it - we can certainly expect host 
root ports with nonzero device numbers, so that's at least 13 bits of 
the StreamID space to cover, which isn't going to fly.

> depending on which host system we had connected to it.
> 
> So I'm a bit worried that 256 "iommus" entries will not be enough.
> 
> I don't have any idea on how to solve this, but right now I don't see any
> other option but to disable the IOMMU when running the PCIe controller in
> endpoint mode.

Agreed; FWIW, for the patch as it is:

Acked-by: Robin Murphy <robin.murphy at arm.com>

> (We have no issues when running with the iommu enabled when running the PCIe
> controller(s) in Root Complex mode.)
> 
> 
> Kind regards,
> Niklas