[PATCH 4/6] arm64/io: Provide a WC friendly __iowriteXX_copy()

[Catalin Marinas]

On Tue, Feb 20, 2024 at 09:17:08PM -0400, Jason Gunthorpe wrote:
> The kernel provides driver support for using write combining IO memory
> through the __iowriteXX_copy() API which is commonly used as an optional
> optimization to generate 16/32/64 byte MemWr TLPs in a PCIe environment.
> 
> iomap_copy.c provides a generic implementation as a simple 4/8 byte at a
> time copy loop that has worked well with past ARM64 CPUs, giving a high
> frequency of large TLPs being successfully formed.
> 
> However modern ARM64 CPUs are quite sensitive to how the write combining
> CPU HW is operated and a compiler generated loop with intermixed
> load/store is not sufficient to frequently generate a large TLP. The CPUs
> would like to see the entire TLP generated by consecutive store
> instructions from registers. Compilers like gcc tend to intermix loads and
> stores and have poor code generation, in part, due to the ARM64 situation
> that writeq() does not codegen anything other than "[xN]". However even
> with that resolved compilers like clang still do not have good code
> generation.
> 
> This means on modern ARM64 CPUs the rate at which __iowriteXX_copy()
> successfully generates large TLPs is very small (less than 1 in 10,000)
> tries), to the point that the use of WC is pointless.
> 
> Implement __iowrite32/64_copy() specifically for ARM64 and use inline
> assembly to build consecutive blocks of STR instructions. Provide direct
> support for 64/32/16 large TLP generation in this manner. Optimize for
> common constant lengths so that the compiler can directly inline the store
> blocks.
> 
> This brings the frequency of large TLP generation up to a high level that
> is comparable with older CPU generations.
> 
> As the __iowriteXX_copy() family of APIs is intended for use with WC
> incorporate the DGH hint directly into the function.
> 
> Cc: Arnd Bergmann <arnd at arndb.de>
> Cc: Catalin Marinas <catalin.marinas at arm.com>
> Cc: Will Deacon <will at kernel.org>
> Cc: Mark Rutland <mark.rutland at arm.com>
> Cc: linux-arch at vger.kernel.org
> Cc: linux-arm-kernel at lists.infradead.org
> Signed-off-by: Jason Gunthorpe <jgg at nvidia.com>

Apart from the slightly more complicated code, I don't expect it to make
things worse on any of the existing hardware.

So, with the typo fix that Will mentioned:

Reviewed-by: Catalin Marinas <catalin.marinas at arm.com>