ARM_LPAE + VMSPLIT_2G cause swiotlb warning on Raspberry Pi 4
Robin Murphy
robin.murphy at arm.com
Wed May 17 03:06:52 PDT 2023
On 2023-05-17 07:23, Stefan Wahren wrote:
> Hi,
>
> Am 17.05.23 um 00:17 schrieb Florian Fainelli:
>> +Robin, Christoph,
>>
>> Hi Stephan,
>>
>> On 5/16/23 13:46, Stefan Wahren wrote:
>>> Hi,
>>> today i wanted to test the new arm/multi_v7_lpae_defconfig on my
>>> Raspberry Pi 4 8GB with Linux 6.4-rc2, but it fails everytime to boot
>>> properly from SD card:
>>>
>>> [ 0.000000] Booting Linux on physical CPU 0x0
>>> [ 0.000000] Linux version 6.4.0-rc2 (stefanw at stefanw-SCHENKER)
>>> (arm-linux-gnueabihf-gcc (GCC) 11.3.1 20220604 [releases/gcc-11
>>> revision 591c0f4b92548e3ae2e8173f4f93984b1c7f62bb], GNU ld
>>> (Linaro_Binutils-2022.06) 2.37.20220122) #2 SMP Tue May 16 18:57:48
>>> CEST 2023
>>> [ 0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7),
>>> cr=30c5383d
>>> [ 0.000000] CPU: div instructions available: patching division code
>>> [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT
>>> instruction cache
>>> [ 0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B Rev 1.4
>>> [ 0.000000] random: crng init done
>>> [ 0.000000] Memory policy: Data cache writealloc
>>> [ 0.000000] efi: UEFI not found.
>>> [ 0.000000] Reserved memory: created CMA memory pool at
>>> 0x0000000037000000, size 64 MiB
>>> [ 0.000000] OF: reserved mem: initialized node linux,cma,
>>> compatible id shared-dma-pool
>>> [ 0.000000] OF: reserved mem:
>>> 0x0000000037000000..0x000000003affffff (65536 KiB) map reusable
>>> linux,cma
>>> [ 0.000000] OF: reserved mem:
>>> 0x000000003ef625a0..0x000000003ef62681 (0 KiB) nomap non-reusable
>>> nvram at 0
>>> [ 0.000000] Zone ranges:
>>> [ 0.000000] DMA [mem 0x0000000000000000-0x000000003fffefff]
>>> [ 0.000000] Normal [mem 0x000000003ffff000-0x000000006fffffff]
>>> [ 0.000000] HighMem [mem 0x0000000070000000-0x00000001ffffffff]
>>> [ 0.000000] Movable zone start for each node
>>> [ 0.000000] Early memory node ranges
>>> [ 0.000000] node 0: [mem 0x0000000000000000-0x000000003b3fffff]
>>> [ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff]
>>> [ 0.000000] node 0: [mem 0x0000000100000000-0x00000001ffffffff]
>>> [ 0.000000] Initmem setup node 0 [mem
>>> 0x0000000000000000-0x00000001ffffffff]
>>> [ 0.000000] On node 0, zone Normal: 1024 pages in unavailable ranges
>>> [ 0.000000] percpu: Embedded 16 pages/cpu s35988 r8192 d21356 u65536
>>> [ 0.000000] Kernel command line: dma.dmachans=0x37f5
>>> bcm2709.boardrev=0xd03114 bcm2709.serial=0x2e99bd7a
>>> bcm2709.uart_clock=48000000 bcm2709.disk_led_gpio=42
>>> bcm2709.disk_led_active_low=0 smsc95xx.macaddr=DC:A6:32:91:38:52
>>> vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000
>>> console=ttyS1,115200 console=tty1 root=PARTUUID=a5c5156c-02
>>> rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
>>> plymouth.ignore-serial-consoles
>>>
>>> ...
>>>
>>> [ 1.422481] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
>>> [ 1.422621] pcieport 0000:00:00.0: PME: Signaling with IRQ 35
>>> [ 1.434696] ------------[ cut here ]------------
>>> [ 1.434720] WARNING: CPU: 2 PID: 1 at kernel/dma/swiotlb.c:891
>>> swiotlb_map+0x328/0x330
>>> [ 1.434759] bcm2835-dma fe007000.dma: swiotlb addr
>>> 0xffffffffffffffff+4096 overflow (mask ffffffff, bus limit ffffffff).
>>> [ 1.434790] Modules linked in:
>>> [ 1.434813] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.4.0-rc2 #2
>>> [ 1.434838] Hardware name: BCM2711
>>> [ 1.434862] unwind_backtrace from show_stack+0x10/0x14
>>> [ 1.434902] show_stack from dump_stack_lvl+0x40/0x4c
>>> [ 1.434938] dump_stack_lvl from __warn+0x7c/0x124
>>> [ 1.434980] __warn from warn_slowpath_fmt+0x118/0x174
>>> [ 1.435024] warn_slowpath_fmt from swiotlb_map+0x328/0x330
>>> [ 1.435062] swiotlb_map from dma_map_page_attrs+0x204/0x328
>>> [ 1.435100] dma_map_page_attrs from bcm2835_dma_probe+0x1d8/0x41c
>>> [ 1.435141] bcm2835_dma_probe from platform_probe+0x5c/0xbc
>>> [ 1.435174] platform_probe from really_probe+0xc8/0x2dc
>>> [ 1.435214] really_probe from __driver_probe_device+0x88/0x19c
>>> [ 1.435262] __driver_probe_device from
>>> driver_probe_device+0x30/0x104
>>> [ 1.435310] driver_probe_device from __driver_attach+0x90/0x174
>>> [ 1.435357] __driver_attach from bus_for_each_dev+0x6c/0xb4
>>> [ 1.435402] bus_for_each_dev from bus_add_driver+0xcc/0x1cc
>>> [ 1.435445] bus_add_driver from driver_register+0x7c/0x118
>>> [ 1.435479] driver_register from do_one_initcall+0x40/0x1e0
>>> [ 1.435509] do_one_initcall from kernel_init_freeable+0x1b8/0x220
>>> [ 1.435549] kernel_init_freeable from kernel_init+0x18/0x12c
>>> [ 1.435589] kernel_init from ret_from_fork+0x14/0x1c
>>> [ 1.435619] Exception stack(0xf081dfb0 to 0xf081dff8)
>>> [ 1.435641] dfa0: 00000000
>>> 00000000 00000000 00000000
>>> [ 1.435667] dfc0: 00000000 00000000 00000000 00000000 00000000
>>> 00000000 00000000 00000000
>>> [ 1.435692] dfe0: 00000000 00000000 00000000 00000000 00000013
>>> 00000000
>>> [ 1.435711] ---[ end trace 0000000000000000 ]---
>>> [ 1.435731] bcm2835-dma fe007000.dma: Failed to map zero page
>>> [ 1.435750] bcm2835-dma: probe of fe007000.dma failed with error -12
>>> [ 1.488044] Serial: 8250/16550 driver, 5 ports, IRQ sharing enabled
>>> [ 1.490574] printk: console [ttyS1] disabled
>>>
>>> So i went back to 6.3, 6.2, 6.1 & 5.15 (enabling the config options
>>> manually on top of multi_v7_defconfig), but it shows the same broken
>>> behavior.
>>>
>>> If i switch back to VMSPLIT_3G and keep ARM_LPAE, the system boots
>>> properly from SD card.
>>>
>>> Any ideas what's the issue here or how to investigate further?
>>
>> The "brcm,bcm2835-dma" Device Tree node is located within the "soc"
>> bus node which has the following:
>>
>> /* Emulate a contiguous 30-bit address range for DMA */
>> dma-ranges = <0xc0000000 0x0 0x00000000 0x40000000>;
>>
>> which is necessary in order to use the non-allocating VPU L2 cache
>> aperture. Eventually we should trickle through translate_phys_to_dma
>> and do:
>>
>> >static inline dma_addr_t translate_phys_to_dma(struct device *dev,
>> >> phys_addr_t paddr)
>> {
>> const struct bus_dma_region *m;
>>
>> for (m = dev->dma_range_map; m->size; m++)
>> if (paddr >= m->cpu_start && paddr - m->cpu_start <
>> m->size)
>> >> return (dma_addr_t)paddr - m->offset;
>>
>> /* make sure dma_capable fails when no translation is
>> available */
>> >> return DMA_MAPPING_ERROR;
>> }
>>
>> and I suspect that we did return DMA_MAPPING_ERROR (~0 =
>> 0xffff_ffff_ffff_ffff) here which would suggest that somehow the
>> dma_range_map function pointer has not been set properly. Could you
>> instrument drivers/of/device.c and see whether that is, or is not the
>> case?
>
> i applied this on top of Linux 6.3:
>
> diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h
> index 18aade195884..e337b26a7110 100644
> --- a/include/linux/dma-direct.h
> +++ b/include/linux/dma-direct.h
> @@ -33,6 +33,8 @@ static inline dma_addr_t translate_phys_to_dma(struct
> device *dev,
> if (paddr >= m->cpu_start && paddr - m->cpu_start < m->size)
> return (dma_addr_t)paddr - m->offset;
>
> + WARN_ONCE(1, "Unable to xlate: %pa\n", &paddr);
> +
> /* make sure dma_capable fails when no translation is available */
> return DMA_MAPPING_ERROR;
> }
>
> and got this on my Raspberry Pi 4 (arm/multi_v7_lpae_defconfig):
>
> ...
>
> [ 1.419785] ------------[ cut here ]------------
> [ 1.419811] WARNING: CPU: 2 PID: 1 at include/linux/dma-direct.h:36
> dma_map_page_attrs+0x360/0x368
> [ 1.419862] Unable to xlate: 0x000000006ffd9000
This suggests that the SWIOTLB buffer itself has somehow been allocated
from ZONE_NORMAL rather than ZONE_DMA (unfortunately you trimmed the log
line from swiotlb_print_info() saying exactly where it is). That can't
happen with VMSPLIT_3G where we have less than 1GB of lowmem anyway,
which explains that difference. I don't have an idea off-hand why LPAE
would affect it.
Thanks,
Robin.
> [ 1.419879] Modules linked in:
> [ 1.419902] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.3.0-dirty #11
> [ 1.419926] Hardware name: BCM2711
> [ 1.419950] unwind_backtrace from show_stack+0x10/0x14
> [ 1.419990] show_stack from dump_stack_lvl+0x40/0x4c
> [ 1.420031] dump_stack_lvl from __warn+0x7c/0x124
> [ 1.420077] __warn from warn_slowpath_fmt+0x118/0x174
> [ 1.420122] warn_slowpath_fmt from dma_map_page_attrs+0x360/0x368
> [ 1.420169] dma_map_page_attrs from bcm2835_dma_probe+0x1d8/0x41c
> [ 1.420215] bcm2835_dma_probe from platform_probe+0x5c/0xbc
> [ 1.420262] platform_probe from really_probe+0xc8/0x2dc
> [ 1.420300] really_probe from __driver_probe_device+0x84/0xe4
> [ 1.420335] __driver_probe_device from driver_probe_device+0x30/0x104
> [ 1.420369] driver_probe_device from __driver_attach+0x90/0x174
> [ 1.420403] __driver_attach from bus_for_each_dev+0x6c/0xb4
> [ 1.420435] bus_for_each_dev from bus_add_driver+0xcc/0x1cc
> [ 1.420465] bus_add_driver from driver_register+0x7c/0x118
> [ 1.420498] driver_register from do_one_initcall+0x40/0x1e0
> [ 1.420534] do_one_initcall from kernel_init_freeable+0x1b8/0x220
> [ 1.420578] kernel_init_freeable from kernel_init+0x18/0x12c
> [ 1.420627] kernel_init from ret_from_fork+0x14/0x1c
> [ 1.420662] Exception stack(0xf081dfb0 to 0xf081dff8)
> [ 1.420684] dfa0: 00000000
> 00000000 00000000 00000000
> [ 1.420710] dfc0: 00000000 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000
> [ 1.420734] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
> [ 1.420753] ---[ end trace 0000000000000000 ]---
> [ 1.420771] ------------[ cut here ]------------
> [ 1.420785] WARNING: CPU: 2 PID: 1 at include/linux/dma-direct.h:36
> swiotlb_map+0x1dc/0x3e4
> [ 1.420829] Unable to xlate: 0x000000006ffd9000
> [ 1.420844] Modules linked in:
> [ 1.420863] CPU: 2 PID: 1 Comm: swapper/0 Tainted: G W
> 6.3.0-dirty #11
> [ 1.420890] Hardware name: BCM2711
> [ 1.420906] unwind_backtrace from show_stack+0x10/0x14
> [ 1.420942] show_stack from dump_stack_lvl+0x40/0x4c
> [ 1.420978] dump_stack_lvl from __warn+0x7c/0x124
> [ 1.421021] __warn from warn_slowpath_fmt+0x118/0x174
> [ 1.421066] warn_slowpath_fmt from swiotlb_map+0x1dc/0x3e4
> [ 1.421114] swiotlb_map from dma_map_page_attrs+0x210/0x368
> [ 1.421159] dma_map_page_attrs from bcm2835_dma_probe+0x1d8/0x41c
> [ 1.421201] bcm2835_dma_probe from platform_probe+0x5c/0xbc
> [ 1.421244] platform_probe from really_probe+0xc8/0x2dc
> [ 1.421281] really_probe from __driver_probe_device+0x84/0xe4
> [ 1.421314] __driver_probe_device from driver_probe_device+0x30/0x104
> [ 1.421348] driver_probe_device from __driver_attach+0x90/0x174
> [ 1.421382] __driver_attach from bus_for_each_dev+0x6c/0xb4
> [ 1.421413] bus_for_each_dev from bus_add_driver+0xcc/0x1cc
> [ 1.421442] bus_add_driver from driver_register+0x7c/0x118
> [ 1.421475] driver_register from do_one_initcall+0x40/0x1e0
> [ 1.421509] do_one_initcall from kernel_init_freeable+0x1b8/0x220
> [ 1.421551] kernel_init_freeable from kernel_init+0x18/0x12c
> [ 1.421599] kernel_init from ret_from_fork+0x14/0x1c
> [ 1.421634] Exception stack(0xf081dfb0 to 0xf081dff8)
> [ 1.421654] dfa0: 00000000
> 00000000 00000000 00000000
> [ 1.421679] dfc0: 00000000 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000
> [ 1.421704] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
> [ 1.421722] ---[ end trace 0000000000000000 ]---
> [ 1.421747] ------------[ cut here ]------------
>
>>
>> Thanks!
More information about the linux-arm-kernel
mailing list