SPARSEMEM memory needs?

[Ard Biesheuvel]

On Mon, 6 Jun 2022 at 22:37, Joakim Tjernlund
<Joakim.Tjernlund at infinera.com> wrote:
>
> On Mon, 2022-06-06 at 21:10 +0200, Ard Biesheuvel wrote:
> > Hello Joakim,
> >
> > On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
> > <Joakim.Tjernlund at infinera.com> wrote:
> > >
> > > I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
> > >
> > > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > > [    0.000000] ------------[ cut here ]------------
> > > [    0.000000] Call trace:
> > > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > > [    0.000000]  vmemmap_populate+0xa4/0x180
> > > [    0.000000]  __populate_section_memmap+0x50/0x70
> > > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > > [    0.000000]  sparse_init+0xb0/0x224
> > > [    0.000000]  bootmem_init+0x40/0x80
> > > [    0.000000]  setup_arch+0x244/0x540
> > > [    0.000000]  start_kernel+0x60/0x804
> > > [    0.000000]  __primary_switched+0xa0/0xa8
> > > [    0.000000] ---[ end trace 0000000000000000 ]---
> > > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> >
> > I'm not sure which backtrace belongs with which memblock debug
> > message, but something looks wrong here.  vmemmap_pud_populate() does
> > an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
> > here.
>
> I have added this to get a BT, I trimmed the regs ways and just kept the BT
> diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
> index bdce883f9286..f8326e1295ed 100644
> --- a/mm/sparse-vmemmap.c
> +++ b/mm/sparse-vmemmap.c
> @@ -418,9 +418,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
>                         warned = true;
>                 }
>                 return NULL;
> -       } else
> +       } else {
> +               WARN_ON(1);
>                 return __earlyonly_bootmem_alloc(node, size, size,
>                                 __pa(MAX_DMA_ADDRESS));
> +       }
>  }
>
>
> I guess I may have trimmed the log a bit too much. How does this look?
>
> [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134

OK, so this one is unaccounted for.

> [    0.000000] Call trace:
> [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> [    0.000000]  vmemmap_populate+0xa4/0x180
> [    0.000000]  __populate_section_memmap+0x50/0x70
> [    0.000000]  sparse_init_nid+0x164/0x1d4
> [    0.000000]  sparse_init+0xb0/0x224
> [    0.000000]  bootmem_init+0x40/0x80
> [    0.000000]  setup_arch+0x244/0x540
> [    0.000000]  start_kernel+0x60/0x804
> [    0.000000]  __primary_switched+0xa0/0xa8
> [    0.000000] ---[ end trace 0000000000000000 ]---
> [    0.000000] memblock_alloc_try_nid_raw: 4096 bytes align=0x1000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> [    0.000000] memblock_reserve: [0x00000000703e7000-0x00000000703e7fff] memblock_alloc_range_nid+0xc8/0x134

This is a single page for the vmemmap PUD

> [    0.000000] Call trace:
> [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> [    0.000000]  vmemmap_populate+0xd0/0x180
> [    0.000000]  __populate_section_memmap+0x50/0x70
> [    0.000000]  sparse_init_nid+0x164/0x1d4
> [    0.000000]  sparse_init+0xb0/0x224
> [    0.000000]  bootmem_init+0x40/0x80
> [    0.000000]  setup_arch+0x244/0x540
> [    0.000000]  start_kernel+0x60/0x804
> [    0.000000]  __primary_switched+0xa0/0xa8
> [    0.000000] ---[ end trace f68728a0d3053b52 ]---
> [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134

This allocates 2 MB to cover the struct page[] array for the first
section of memory (32 MB), starting at address 0x6000_0000

> [    0.000000] Call trace:
> [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> [    0.000000]  vmemmap_populate+0xd0/0x180
> [    0.000000]  __populate_section_memmap+0x50/0x70
> [    0.000000]  sparse_init_nid+0x164/0x1d4
> [    0.000000]  sparse_init+0xb0/0x224
> [    0.000000]  bootmem_init+0x40/0x80
> [    0.000000]  setup_arch+0x244/0x540
> [    0.000000]  start_kernel+0x60/0x804
> [    0.000000]  __primary_switched+0xa0/0xa8
> [    0.000000] ---[ end trace f68728a0d3053b53 ]---
> [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> [    0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134

This allocates 2 MB to cover the struct page[] array for the second
section of memory (4 MB), starting at 0x7000_0000

So the problem is that you only have 36 MB of DRAM, with a large hole
in the middle. Sparsemem was actually designed for that (hence the
name), and flatmem would make things much worse.