RPi4 can't deal with 64 bit PCI accesses

[Nicolas Saenz Julienne]

On Thu, 2021-02-25 at 11:10 +0000, Robin Murphy wrote:
> On 2021-02-25 10:29, Neil Armstrong wrote:
> > On 24/02/2021 21:35, Florian Fainelli wrote:
> > > 
> > > 
> > > On 2/24/2021 12:25 PM, Christoph Hellwig wrote:
> > > > On Wed, Feb 24, 2021 at 08:55:10AM -0800, Florian Fainelli wrote:
> > > > > > Working around kernel I/O accessors is all very well, but another
> > > > > > concern for PCI in particular is when things like framebuffer memory can
> > > > > > get mmap'ed into userspace (or even memremap'ed within the kernel). Even
> > > > > > in AArch32, compiled code may result in 64-bit accesses being generated
> > > > > > depending on how the CPU and interconnect handle LDRD/STRD/LDM/STM/etc.,
> > > > > > so it's basically not safe to ever let that happen at all.
> > > > > 
> > > > > Agreed, this makes finding a generic solution a tiny bit harder. Do you
> > > > > have something in mind Nicolas?
> > > > 
> > > > The only workable solution is a new
> > > > 
> > > > bool 64bit_mmio_supported(void)
> 
> Note that to be sufficiently generic this would have to be a per-device 
> property - a system could have an affected PCIe root complex but still 
> have other devices elsewhere in the SoC that can, or even need to, use 
> 64-bit accesses.

Yes, that's what I had in mind myself. All in all, why penalize the rest of
busses in the system. What I'm planning is to introduce a '64bit-mmio-broken'
DT property that'll utimately live somwhere in 'struct device.'

WRT why not defaulting to 32-bit accesses for distro images if they support
RPi4. My *un-educated* guess is that, the performance penalty of checking for a
device flag is (way) lower than having to resort to two distinct write
operations with their assorted memory barriers. I'm sure you can
comment/correct me here.

> > > > check that is used like:
> > > > 
> > > > 	if (64bit_mmio_supported())
> > > > 		readq(foodev->regs, REG_OFFSET);
> > > > 	else
> > > > 		lo_hi_readq(foodev->regs, REG_OFFSET);
> > > > 
> > > > where 64bit_mmio_supported() return false for all 32-bit kernels,
> > > > true for all non-broken 64-bit kernels and is an actual function
> > > > for arm64 multiplatforms builds that include te RPi quirk.
> > > > 
> > > > The above would then replace the existing magic from the
> > > > <linux/io-64-nonatomic-lo-hi.h> and <linux/io-64-nonatomic-hi-lo.h>
> > > > headers.
> > > 
> > > That would work. The use case described by Robin is highly unlikely to
> > > exist on the Pi4 given that you cannot easily access the PCIe bus and
> > > plug an arbitrary GPU, so maybe there is nothing to do for framebuffer
> > > memory.
> 
> Framebuffers are only the most obvious example - I don't feel the 
> inclination to audit every driver/subsystem that can possibly make a 
> non-iomem remapping or userspace mmap of a prefetchable BAR, but I'm 
> sure there are more.

IIUC the only solution to the issue here is to disallow mmaping memory
belonging to a broken bus, right? In this case, the function above would do the
trick.

> > Erf, not really, with the compute module ATX/ITX boards are being designed
> > with a full PCIe connector like:
> > https://www.indiegogo.com/projects/over-board-raspberry-pi-4-mini-itx-motherboard/#/
> 
> Right, this whole thread looks to have come about due to random 
> endpoints getting connected to the exposed bus on compute modules. If it 
> was an issue at all for the XHCI on standard Pi 4 boards I don't think 
> people would just be starting to notice it now...

Indeed. For the record, here's the original complaint, although I'm sure others
exist: https://github.com/raspberrypi/linux/issues/4158

Regards,
Nicolas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20210225/d916647f/attachment.sig>