RPi4 can't deal with 64 bit PCI accesses

[Robin Murphy]

On 2021-02-25 10:29, Neil Armstrong wrote:
> On 24/02/2021 21:35, Florian Fainelli wrote:
>>
>>
>> On 2/24/2021 12:25 PM, Christoph Hellwig wrote:
>>> On Wed, Feb 24, 2021 at 08:55:10AM -0800, Florian Fainelli wrote:
>>>>> Working around kernel I/O accessors is all very well, but another
>>>>> concern for PCI in particular is when things like framebuffer memory can
>>>>> get mmap'ed into userspace (or even memremap'ed within the kernel). Even
>>>>> in AArch32, compiled code may result in 64-bit accesses being generated
>>>>> depending on how the CPU and interconnect handle LDRD/STRD/LDM/STM/etc.,
>>>>> so it's basically not safe to ever let that happen at all.
>>>>
>>>> Agreed, this makes finding a generic solution a tiny bit harder. Do you
>>>> have something in mind Nicolas?
>>>
>>> The only workable solution is a new
>>>
>>> bool 64bit_mmio_supported(void)

Note that to be sufficiently generic this would have to be a per-device 
property - a system could have an affected PCIe root complex but still 
have other devices elsewhere in the SoC that can, or even need to, use 
64-bit accesses.

>>> check that is used like:
>>>
>>> 	if (64bit_mmio_supported())
>>> 		readq(foodev->regs, REG_OFFSET);
>>> 	else
>>> 		lo_hi_readq(foodev->regs, REG_OFFSET);
>>>
>>> where 64bit_mmio_supported() return false for all 32-bit kernels,
>>> true for all non-broken 64-bit kernels and is an actual function
>>> for arm64 multiplatforms builds that include te RPi quirk.
>>>
>>> The above would then replace the existing magic from the
>>> <linux/io-64-nonatomic-lo-hi.h> and <linux/io-64-nonatomic-hi-lo.h>
>>> headers.
>>
>> That would work. The use case described by Robin is highly unlikely to
>> exist on the Pi4 given that you cannot easily access the PCIe bus and
>> plug an arbitrary GPU, so maybe there is nothing to do for framebuffer
>> memory.

Framebuffers are only the most obvious example - I don't feel the 
inclination to audit every driver/subsystem that can possibly make a 
non-iomem remapping or userspace mmap of a prefetchable BAR, but I'm 
sure there are more.

> Erf, not really, with the compute module ATX/ITX boards are being designed with a full PCIe connector like:
> https://www.indiegogo.com/projects/over-board-raspberry-pi-4-mini-itx-motherboard/#/

Right, this whole thread looks to have come about due to random 
endpoints getting connected to the exposed bus on compute modules. If it 
was an issue at all for the XHCI on standard Pi 4 boards I don't think 
people would just be starting to notice it now...

Robin.