ARM_LPAE + VMSPLIT_2G cause swiotlb warning on Raspberry Pi 4
Stefan Wahren
stefan.wahren at i2se.com
Tue May 16 23:23:49 PDT 2023
Hi,
Am 17.05.23 um 00:17 schrieb Florian Fainelli:
> +Robin, Christoph,
>
> Hi Stephan,
>
> On 5/16/23 13:46, Stefan Wahren wrote:
>> Hi,
>> today i wanted to test the new arm/multi_v7_lpae_defconfig on my
>> Raspberry Pi 4 8GB with Linux 6.4-rc2, but it fails everytime to boot
>> properly from SD card:
>>
>> [ 0.000000] Booting Linux on physical CPU 0x0
>> [ 0.000000] Linux version 6.4.0-rc2 (stefanw at stefanw-SCHENKER)
>> (arm-linux-gnueabihf-gcc (GCC) 11.3.1 20220604 [releases/gcc-11
>> revision 591c0f4b92548e3ae2e8173f4f93984b1c7f62bb], GNU ld
>> (Linaro_Binutils-2022.06) 2.37.20220122) #2 SMP Tue May 16 18:57:48
>> CEST 2023
>> [ 0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7),
>> cr=30c5383d
>> [ 0.000000] CPU: div instructions available: patching division code
>> [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT
>> instruction cache
>> [ 0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B Rev 1.4
>> [ 0.000000] random: crng init done
>> [ 0.000000] Memory policy: Data cache writealloc
>> [ 0.000000] efi: UEFI not found.
>> [ 0.000000] Reserved memory: created CMA memory pool at
>> 0x0000000037000000, size 64 MiB
>> [ 0.000000] OF: reserved mem: initialized node linux,cma,
>> compatible id shared-dma-pool
>> [ 0.000000] OF: reserved mem:
>> 0x0000000037000000..0x000000003affffff (65536 KiB) map reusable linux,cma
>> [ 0.000000] OF: reserved mem:
>> 0x000000003ef625a0..0x000000003ef62681 (0 KiB) nomap non-reusable nvram at 0
>> [ 0.000000] Zone ranges:
>> [ 0.000000] DMA [mem 0x0000000000000000-0x000000003fffefff]
>> [ 0.000000] Normal [mem 0x000000003ffff000-0x000000006fffffff]
>> [ 0.000000] HighMem [mem 0x0000000070000000-0x00000001ffffffff]
>> [ 0.000000] Movable zone start for each node
>> [ 0.000000] Early memory node ranges
>> [ 0.000000] node 0: [mem 0x0000000000000000-0x000000003b3fffff]
>> [ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff]
>> [ 0.000000] node 0: [mem 0x0000000100000000-0x00000001ffffffff]
>> [ 0.000000] Initmem setup node 0 [mem
>> 0x0000000000000000-0x00000001ffffffff]
>> [ 0.000000] On node 0, zone Normal: 1024 pages in unavailable ranges
>> [ 0.000000] percpu: Embedded 16 pages/cpu s35988 r8192 d21356 u65536
>> [ 0.000000] Kernel command line: dma.dmachans=0x37f5
>> bcm2709.boardrev=0xd03114 bcm2709.serial=0x2e99bd7a
>> bcm2709.uart_clock=48000000 bcm2709.disk_led_gpio=42
>> bcm2709.disk_led_active_low=0 smsc95xx.macaddr=DC:A6:32:91:38:52
>> vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000
>> console=ttyS1,115200 console=tty1 root=PARTUUID=a5c5156c-02
>> rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
>> plymouth.ignore-serial-consoles
>>
>> ...
>>
>> [ 1.422481] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
>> [ 1.422621] pcieport 0000:00:00.0: PME: Signaling with IRQ 35
>> [ 1.434696] ------------[ cut here ]------------
>> [ 1.434720] WARNING: CPU: 2 PID: 1 at kernel/dma/swiotlb.c:891
>> swiotlb_map+0x328/0x330
>> [ 1.434759] bcm2835-dma fe007000.dma: swiotlb addr
>> 0xffffffffffffffff+4096 overflow (mask ffffffff, bus limit ffffffff).
>> [ 1.434790] Modules linked in:
>> [ 1.434813] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.4.0-rc2 #2
>> [ 1.434838] Hardware name: BCM2711
>> [ 1.434862] unwind_backtrace from show_stack+0x10/0x14
>> [ 1.434902] show_stack from dump_stack_lvl+0x40/0x4c
>> [ 1.434938] dump_stack_lvl from __warn+0x7c/0x124
>> [ 1.434980] __warn from warn_slowpath_fmt+0x118/0x174
>> [ 1.435024] warn_slowpath_fmt from swiotlb_map+0x328/0x330
>> [ 1.435062] swiotlb_map from dma_map_page_attrs+0x204/0x328
>> [ 1.435100] dma_map_page_attrs from bcm2835_dma_probe+0x1d8/0x41c
>> [ 1.435141] bcm2835_dma_probe from platform_probe+0x5c/0xbc
>> [ 1.435174] platform_probe from really_probe+0xc8/0x2dc
>> [ 1.435214] really_probe from __driver_probe_device+0x88/0x19c
>> [ 1.435262] __driver_probe_device from driver_probe_device+0x30/0x104
>> [ 1.435310] driver_probe_device from __driver_attach+0x90/0x174
>> [ 1.435357] __driver_attach from bus_for_each_dev+0x6c/0xb4
>> [ 1.435402] bus_for_each_dev from bus_add_driver+0xcc/0x1cc
>> [ 1.435445] bus_add_driver from driver_register+0x7c/0x118
>> [ 1.435479] driver_register from do_one_initcall+0x40/0x1e0
>> [ 1.435509] do_one_initcall from kernel_init_freeable+0x1b8/0x220
>> [ 1.435549] kernel_init_freeable from kernel_init+0x18/0x12c
>> [ 1.435589] kernel_init from ret_from_fork+0x14/0x1c
>> [ 1.435619] Exception stack(0xf081dfb0 to 0xf081dff8)
>> [ 1.435641] dfa0: 00000000
>> 00000000 00000000 00000000
>> [ 1.435667] dfc0: 00000000 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000
>> [ 1.435692] dfe0: 00000000 00000000 00000000 00000000 00000013
>> 00000000
>> [ 1.435711] ---[ end trace 0000000000000000 ]---
>> [ 1.435731] bcm2835-dma fe007000.dma: Failed to map zero page
>> [ 1.435750] bcm2835-dma: probe of fe007000.dma failed with error -12
>> [ 1.488044] Serial: 8250/16550 driver, 5 ports, IRQ sharing enabled
>> [ 1.490574] printk: console [ttyS1] disabled
>>
>> So i went back to 6.3, 6.2, 6.1 & 5.15 (enabling the config options
>> manually on top of multi_v7_defconfig), but it shows the same broken
>> behavior.
>>
>> If i switch back to VMSPLIT_3G and keep ARM_LPAE, the system boots
>> properly from SD card.
>>
>> Any ideas what's the issue here or how to investigate further?
>
> The "brcm,bcm2835-dma" Device Tree node is located within the "soc" bus
> node which has the following:
>
> /* Emulate a contiguous 30-bit address range for DMA */
> dma-ranges = <0xc0000000 0x0 0x00000000 0x40000000>;
>
> which is necessary in order to use the non-allocating VPU L2 cache
> aperture. Eventually we should trickle through translate_phys_to_dma and
> do:
>
> >static inline dma_addr_t translate_phys_to_dma(struct device *dev,
> >> phys_addr_t paddr)
> {
> const struct bus_dma_region *m;
>
> for (m = dev->dma_range_map; m->size; m++)
> if (paddr >= m->cpu_start && paddr - m->cpu_start <
> m->size)
> >> return (dma_addr_t)paddr - m->offset;
>
> /* make sure dma_capable fails when no translation is
> available */
> >> return DMA_MAPPING_ERROR;
> }
>
> and I suspect that we did return DMA_MAPPING_ERROR (~0 =
> 0xffff_ffff_ffff_ffff) here which would suggest that somehow the
> dma_range_map function pointer has not been set properly. Could you
> instrument drivers/of/device.c and see whether that is, or is not the case?
i applied this on top of Linux 6.3:
diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h
index 18aade195884..e337b26a7110 100644
--- a/include/linux/dma-direct.h
+++ b/include/linux/dma-direct.h
@@ -33,6 +33,8 @@ static inline dma_addr_t translate_phys_to_dma(struct
device *dev,
if (paddr >= m->cpu_start && paddr - m->cpu_start < m->size)
return (dma_addr_t)paddr - m->offset;
+ WARN_ONCE(1, "Unable to xlate: %pa\n", &paddr);
+
/* make sure dma_capable fails when no translation is available */
return DMA_MAPPING_ERROR;
}
and got this on my Raspberry Pi 4 (arm/multi_v7_lpae_defconfig):
...
[ 1.419785] ------------[ cut here ]------------
[ 1.419811] WARNING: CPU: 2 PID: 1 at include/linux/dma-direct.h:36
dma_map_page_attrs+0x360/0x368
[ 1.419862] Unable to xlate: 0x000000006ffd9000
[ 1.419879] Modules linked in:
[ 1.419902] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.3.0-dirty #11
[ 1.419926] Hardware name: BCM2711
[ 1.419950] unwind_backtrace from show_stack+0x10/0x14
[ 1.419990] show_stack from dump_stack_lvl+0x40/0x4c
[ 1.420031] dump_stack_lvl from __warn+0x7c/0x124
[ 1.420077] __warn from warn_slowpath_fmt+0x118/0x174
[ 1.420122] warn_slowpath_fmt from dma_map_page_attrs+0x360/0x368
[ 1.420169] dma_map_page_attrs from bcm2835_dma_probe+0x1d8/0x41c
[ 1.420215] bcm2835_dma_probe from platform_probe+0x5c/0xbc
[ 1.420262] platform_probe from really_probe+0xc8/0x2dc
[ 1.420300] really_probe from __driver_probe_device+0x84/0xe4
[ 1.420335] __driver_probe_device from driver_probe_device+0x30/0x104
[ 1.420369] driver_probe_device from __driver_attach+0x90/0x174
[ 1.420403] __driver_attach from bus_for_each_dev+0x6c/0xb4
[ 1.420435] bus_for_each_dev from bus_add_driver+0xcc/0x1cc
[ 1.420465] bus_add_driver from driver_register+0x7c/0x118
[ 1.420498] driver_register from do_one_initcall+0x40/0x1e0
[ 1.420534] do_one_initcall from kernel_init_freeable+0x1b8/0x220
[ 1.420578] kernel_init_freeable from kernel_init+0x18/0x12c
[ 1.420627] kernel_init from ret_from_fork+0x14/0x1c
[ 1.420662] Exception stack(0xf081dfb0 to 0xf081dff8)
[ 1.420684] dfa0: 00000000
00000000 00000000 00000000
[ 1.420710] dfc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[ 1.420734] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 1.420753] ---[ end trace 0000000000000000 ]---
[ 1.420771] ------------[ cut here ]------------
[ 1.420785] WARNING: CPU: 2 PID: 1 at include/linux/dma-direct.h:36
swiotlb_map+0x1dc/0x3e4
[ 1.420829] Unable to xlate: 0x000000006ffd9000
[ 1.420844] Modules linked in:
[ 1.420863] CPU: 2 PID: 1 Comm: swapper/0 Tainted: G W
6.3.0-dirty #11
[ 1.420890] Hardware name: BCM2711
[ 1.420906] unwind_backtrace from show_stack+0x10/0x14
[ 1.420942] show_stack from dump_stack_lvl+0x40/0x4c
[ 1.420978] dump_stack_lvl from __warn+0x7c/0x124
[ 1.421021] __warn from warn_slowpath_fmt+0x118/0x174
[ 1.421066] warn_slowpath_fmt from swiotlb_map+0x1dc/0x3e4
[ 1.421114] swiotlb_map from dma_map_page_attrs+0x210/0x368
[ 1.421159] dma_map_page_attrs from bcm2835_dma_probe+0x1d8/0x41c
[ 1.421201] bcm2835_dma_probe from platform_probe+0x5c/0xbc
[ 1.421244] platform_probe from really_probe+0xc8/0x2dc
[ 1.421281] really_probe from __driver_probe_device+0x84/0xe4
[ 1.421314] __driver_probe_device from driver_probe_device+0x30/0x104
[ 1.421348] driver_probe_device from __driver_attach+0x90/0x174
[ 1.421382] __driver_attach from bus_for_each_dev+0x6c/0xb4
[ 1.421413] bus_for_each_dev from bus_add_driver+0xcc/0x1cc
[ 1.421442] bus_add_driver from driver_register+0x7c/0x118
[ 1.421475] driver_register from do_one_initcall+0x40/0x1e0
[ 1.421509] do_one_initcall from kernel_init_freeable+0x1b8/0x220
[ 1.421551] kernel_init_freeable from kernel_init+0x18/0x12c
[ 1.421599] kernel_init from ret_from_fork+0x14/0x1c
[ 1.421634] Exception stack(0xf081dfb0 to 0xf081dff8)
[ 1.421654] dfa0: 00000000
00000000 00000000 00000000
[ 1.421679] dfc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[ 1.421704] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 1.421722] ---[ end trace 0000000000000000 ]---
[ 1.421747] ------------[ cut here ]------------
>
> Thanks!
More information about the linux-arm-kernel
mailing list