[PATCH v3 RESEND] riscv: mm: still create swiotlb buffer for kmalloc() bouncing if required

[Geert Uytterhoeven]

Hi Jisheng,

On Mon, Mar 25, 2024 at 12:15 PM Jisheng Zhang <jszhang at kernel.org> wrote:
> After commit f51f7a0fc2f4 ("riscv: enable DMA_BOUNCE_UNALIGNED_KMALLOC
> for !dma_coherent"), for non-coherent platforms with less than 4GB
> memory, we rely on users to pass "swiotlb=mmnn,force" kernel parameters
> to enable DMA bouncing for unaligned kmalloc() buffers. Now let's go
> further: If no bouncing needed for ZONE_DMA, let kernel automatically
> allocate 1MB swiotlb buffer per 1GB of RAM for kmalloc() bouncing on
> non-coherent platforms, so that no need to pass "swiotlb=mmnn,force"
> any more.
>
> The math of "1MB swiotlb buffer per 1GB of RAM for kmalloc() bouncing"
> is taken from arm64. Users can still force smaller swiotlb buffer by
> passing "swiotlb=mmnn".
>
> Signed-off-by: Jisheng Zhang <jszhang at kernel.org>
> Reviewed-by: Alexandre Ghiti <alexghiti at rivosinc.com>

Thanks for your patch, which is now commit fc7a50eed9860d4b ("riscv:
mm: still create swiotlb buffer for kmalloc() bouncing if required")
in riscv/for-next (next-20240429 and later).

On RZ/Five, which has 1 GiB of RAM, i.e. a bit less for free use:

     Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
     Built 1 zonelists, mobility grouping on.  Total pages: 225792
     mem auto-init: stack:off, heap alloc:off, heap free:off
    +software IO TLB: SWIOTLB bounce buffer size adjusted to 0MB
                                                             ^^^
    +software IO TLB: area num 1.
    +software IO TLB: mapped [mem 0x000000007ef56000-0x000000007f056000] (1MB)
                                                                         ^^^
     Virtual kernel memory layout:

I was a bit intrigued by the "0MB".  However, that seems to be correct:

     mem_init: memblock_phys_mem_size() = 939524096
     mem_init: size = 917504
     mem_init: swiotlb_size_or_default() = 67108864

and it must be rounded up to 1 MB later.

Apparently arm64 has the same discrepancy, which I never really noticed
before (and I have no arm64 platforms with 1 GiB RAM or less):

     mem_init: memblock_phys_mem_size() = 2013265920
     mem_init: size = 1966080
     mem_init: swiotlb_size_or_default() = 67108864
     software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB
                                                             ^^^
     software IO TLB: area num 2.
     software IO TLB: mapped [mem 0x00000000b9400000-0x00000000b9600000] (2MB)
                                                                         ^^^

> --- a/arch/riscv/mm/init.c
> +++ b/arch/riscv/mm/init.c
> @@ -161,11 +161,25 @@ static void print_vm_layout(void) { }
>
>  void __init mem_init(void)
>  {
> +       bool swiotlb = max_pfn > PFN_DOWN(dma32_phys_limit);
>  #ifdef CONFIG_FLATMEM
>         BUG_ON(!mem_map);
>  #endif /* CONFIG_FLATMEM */
>
> -       swiotlb_init(max_pfn > PFN_DOWN(dma32_phys_limit), SWIOTLB_VERBOSE);
> +       if (IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) && !swiotlb &&
> +           dma_cache_alignment != 1) {
> +               /*
> +                * If no bouncing needed for ZONE_DMA, allocate 1MB swiotlb
> +                * buffer per 1GB of RAM for kmalloc() bouncing on
> +                * non-coherent platforms.
> +                */
> +               unsigned long size =
> +                       DIV_ROUND_UP(memblock_phys_mem_size(), 1024);
> +               swiotlb_adjust_size(min(swiotlb_size_or_default(), size));
> +               swiotlb = true;
> +       }
> +
> +       swiotlb_init(swiotlb, SWIOTLB_VERBOSE);
>         memblock_free_all();
>
>         print_vm_layout();

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds