RPi4 can't deal with 64 bit PCI accesses

[Florian Fainelli]

On 2/22/2021 8:56 AM, Robin Murphy wrote:
> On 2021-02-22 15:47, Nicolas Saenz Julienne wrote:
>> Hi everyone,
>> Raspberry Pi 4, a 64bit arm system on chip, contains a PCIe bus that
>> can't
>> handle 64bit accesses to its MMIO address space, in other words,
>> writeq() has
>> to be split into two distinct writel() operations. This isn't ideal,
>> as it
>> misrepresents PCI's promise of being able to treat device memory as
>> regular
>> memory, ultimately breaking a bunch of PCI device drivers[1].
>>
>> I'd like to have a go at fixing this in a way that can be distributed
>> in a
>> generic distro without prejudice to other users.
>>
>> AFAIK there is no way to detect this limitation through generic PCIe
>> capabilities, so one solution would be to expose it through firmware
>> (devicetree in this case), and pass the limitations through 'struct
>> device' so
>> as for the drivers to choose the right access method in a way that
>> doesn't
>> affect performance much[2]. All in all, most of this doesn't need to be
>> PCI-centric as the property could be applied to any MMIO bus.
> 
> It is indeed something that people can get wrong with internal buses as
> well - for example commit f2d9848aeb9f is such a workaround, also
> conveniently illustrating the case of significant functionality having
> to be disabled where the device *does* require 64-bit atomicity for
> correctness.
> 
> Working around kernel I/O accessors is all very well, but another
> concern for PCI in particular is when things like framebuffer memory can
> get mmap'ed into userspace (or even memremap'ed within the kernel). Even
> in AArch32, compiled code may result in 64-bit accesses being generated
> depending on how the CPU and interconnect handle LDRD/STRD/LDM/STM/etc.,
> so it's basically not safe to ever let that happen at all.

Agreed, this makes finding a generic solution a tiny bit harder. Do you
have something in mind Nicolas?
-- 
Florian