ARM_LPAE + VMSPLIT_2G cause swiotlb warning on Raspberry Pi 4

[Arnd Bergmann]

On Tue, May 16, 2023, at 22:46, Stefan Wahren wrote:
> Hi,
> today i wanted to test the new arm/multi_v7_lpae_defconfig on my 
> Raspberry Pi 4 8GB with Linux 6.4-rc2, but it fails everytime to boot 
> properly from SD card:

> [    1.435024]  warn_slowpath_fmt from swiotlb_map+0x328/0x330
> [    1.435062]  swiotlb_map from dma_map_page_attrs+0x204/0x328
> [    1.435100]  dma_map_page_attrs from bcm2835_dma_probe+0x1d8/0x41c
> [    1.435141]  bcm2835_dma_probe from platform_probe+0x5c/0xbc
...
> So i went back to 6.3, 6.2, 6.1 & 5.15 (enabling the config options 
> manually on top of multi_v7_defconfig), but it shows the same broken 
> behavior.
>
> If i switch back to VMSPLIT_3G and keep ARM_LPAE, the system boots 
> properly from SD card.
>
> Any ideas what's the issue here or how to investigate further?

I haven't found the exact problem, but I looked at this and have some
observations that may help in narrowing it down further:

- the bcm2835_dma_probe() maps the zero page only to detect and
  optimize DMA from zero page, failing to map this would not need
  to be a fatal error here if we can just skip the optimization

- apparently the zero page here is above the 1GB boundary, which
  leads to the swiotlb-map call instead of just returning the
  direct-mapped DMA address. If the zero page is swiotlb translated,
  the optimization in the dma driver becomes pointless, and
  other bits are likely also slowed down

- I suspect that allocating the zero page from low memory would
  avoid the problem and generally help here, but we still need to
  understand the rest of the problem. Does this patch avoid the
  problem?
--- a/arch/arm/mm/mmu.c
+++ b/arch/arm/mm/mmu.c
 @@ -1783,7 +1783,7 @@ void __init paging_init(const struct machine_desc *mdesc)
        top_pmd = pmd_off_k(0xffff0000);
 
        /* allocate the zero page. */
-       zero_page = early_alloc(PAGE_SIZE);
+       zero_page = memblock_alloc_low(PAGE_SIZE, PAGE_SIZE);
 
        bootmem_init();
 
- The swiotlb_map function allocates a bounce buffer page and then
  does a DMA translation. From your log output, it looks like the
  allocation was successful but the map failed, which would indicate
  that the bounce buffer page is outside of the addressable range for
  this device, but I don't see how that can happen

- The swiotlb bounce buffers should get allocated from ZONE_DMA if that
  is enabled, and ZONE_DMA is supposed to be reachable from all DMA
  master devices in a system

- The bcm2835 platform sets the DMA limit to SZ_1G and selects
  CONFIG_ZONE_DMA, so the swiotlb buffers are meant to work here.

- Can you printk() the swiotlb_addr in swiotlb_map() and the
  "start" address in swiotlb_init_io_tlb_mem() to see if they
  are in range?

     Arnd