[PATCH 13/18] arm64: ioremap: use nGnRnE mappings on platforms that require it

[Arnd Bergmann]

On Tue, Feb 9, 2021 at 1:25 AM Hector Martin <marcan at marcan.st> wrote:
> On 09/02/2021 08.20, Mark Kettenis wrote:
> I probed writing to i<<28 for i = [0..255], using nGnRE. This reveals
> that nGnRE writes are allowed (i.e. either succeed or error out
> differently) in the following ranges:
>
> 0x400000000 - 0x4ffffffff (apciec0)
> 0x580000000 - 0x67fffffff (apciec1)
> 0x6a0000000 - 0x6ffffffff (apcie)
>
> Which matches the `ranges` properties of the respective apcie devices in
> the ADT.

Right, these are the same ranges that I found in the adt and that Mark
listed in his code snippet, so it seems we all see the same partitioning
of the address space. I also see them reflected in the
/defaults/pmap-io-ranges property in ADT, which seems to have an entry
for every register range that has some mmio registers, along with what
appears to be a bitmask of some attributes, and it clearly shows
the above ranges as having a distinct set of bits from the others
(in little-endian):

 00000000 04000000 00000080 00000000 27000080 65494350
 00000080 04000000 00000080 00000000 27000080 65494350
 00000080 05000000 00000080 00000000 27000080 65494350
 00000000 06000000 00000080 00000000 27000080 65494350
 000000a0 06000000 00000020 00000000 27000080 65494350
 000000c0 06000000 00000040 00000000 27000080 65494350
   ^64-bit address       ^64-bit length            ^ 64-bit flags?

As opposed to e.g.

 0000f002 05000000 00400000 00000000 07400000 54524144
 0000f802 05000000 00400000 00000000 07400000 54524144
 00800021 05000000 00400000 00000000 07400000 44495344
 0000a801 05000000 00400000 00000000 07400000 54524144
00000367 02000000 00400000 00000000 07400000 54524144
...

There is one more entry for the 0x700000000-0x7ffffffff range, which
has yet another distinct bitmask, but does not seem to correspond
to any registers listed in other nodes.

> The first two are obviously the TB3 ports, and have more
> features (three ranges instead of two, presumably IO port ranges are
> supported on those, there's some extra DMA stuff, etc).

The PCI ranges property identifies these as 64-bit prefetchable (0x43),
32-bit non-prefetchable (0x02), and 32-bit pre prefetchable (0x42)
respectively. The third bus only lacks the 32-bit prefetchable range,
that is normally ok. Is this the NVMe host or something else?

None of them have an I/O space ranges though, only memory space.

> So the hardware behavior is to block nGnRE everywhere except in those
> ranges (i.e. the nGnRnE fault takes precedence over other errors, like
> the address not existing at all).

Ok, so if we want this to get encoded in a 'struct resource' flag, the PCI
resources should work just fine as these resources come from the
PCI layer rather than of_address_to_resource(). I think it would be
reasonable here to add something to of_address_to_resource() to
set such a flag if we can find an unused one, and then require the
drivers for this platform to go through devm_ioremap_resource()
or similar.

       Arnd