SPARSEMEM memory needs?

[Joakim Tjernlund]

On Mon, 2022-06-06 at 23:50 +0200, Ard Biesheuvel wrote:
> On Mon, 6 Jun 2022 at 22:37, Joakim Tjernlund
> <Joakim.Tjernlund at infinera.com> wrote:
> > 
> > On Mon, 2022-06-06 at 21:10 +0200, Ard Biesheuvel wrote:
> > > Hello Joakim,
> > > 
> > > On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
> > > <Joakim.Tjernlund at infinera.com> wrote:
> > > > 
> > > > I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
> > > > 
> > > > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > > > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > > > [    0.000000] ------------[ cut here ]------------
> > > > [    0.000000] Call trace:
> > > > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > > > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > > > [    0.000000]  vmemmap_populate+0xa4/0x180
> > > > [    0.000000]  __populate_section_memmap+0x50/0x70
> > > > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > > > [    0.000000]  sparse_init+0xb0/0x224
> > > > [    0.000000]  bootmem_init+0x40/0x80
> > > > [    0.000000]  setup_arch+0x244/0x540
> > > > [    0.000000]  start_kernel+0x60/0x804
> > > > [    0.000000]  __primary_switched+0xa0/0xa8
> > > > [    0.000000] ---[ end trace 0000000000000000 ]---
> > > > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > > > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> > > 
> > > I'm not sure which backtrace belongs with which memblock debug
> > > message, but something looks wrong here.  vmemmap_pud_populate() does
> > > an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
> > > here.
> > 
> > I have added this to get a BT, I trimmed the regs ways and just kept the BT
> > diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
> > index bdce883f9286..f8326e1295ed 100644
> > --- a/mm/sparse-vmemmap.c
> > +++ b/mm/sparse-vmemmap.c
> > @@ -418,9 +418,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
> >                         warned = true;
> >                 }
> >                 return NULL;
> > -       } else
> > +       } else {
> > +               WARN_ON(1);
> >                 return __earlyonly_bootmem_alloc(node, size, size,
> >                                 __pa(MAX_DMA_ADDRESS));
> > +       }
> >  }
> > 
> > 
> > I guess I may have trimmed the log a bit too much. How does this look?
> > 
> > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> 
> OK, so this one is unaccounted for.

Need to find out where this is coming from then, thanks.

> 
> > [    0.000000] Call trace:
> > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > [    0.000000]  vmemmap_populate+0xa4/0x180
> > [    0.000000]  __populate_section_memmap+0x50/0x70
> > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > [    0.000000]  sparse_init+0xb0/0x224
> > [    0.000000]  bootmem_init+0x40/0x80
> > [    0.000000]  setup_arch+0x244/0x540
> > [    0.000000]  start_kernel+0x60/0x804
> > [    0.000000]  __primary_switched+0xa0/0xa8
> > [    0.000000] ---[ end trace 0000000000000000 ]---
> > [    0.000000] memblock_alloc_try_nid_raw: 4096 bytes align=0x1000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [    0.000000] memblock_reserve: [0x00000000703e7000-0x00000000703e7fff] memblock_alloc_range_nid+0xc8/0x134
> 
> This is a single page for the vmemmap PUD
> 
> > [    0.000000] Call trace:
> > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> > [    0.000000]  vmemmap_populate+0xd0/0x180
> > [    0.000000]  __populate_section_memmap+0x50/0x70
> > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > [    0.000000]  sparse_init+0xb0/0x224
> > [    0.000000]  bootmem_init+0x40/0x80
> > [    0.000000]  setup_arch+0x244/0x540
> > [    0.000000]  start_kernel+0x60/0x804
> > [    0.000000]  __primary_switched+0xa0/0xa8
> > [    0.000000] ---[ end trace f68728a0d3053b52 ]---
> > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> 
> This allocates 2 MB to cover the struct page[] array for the first
> section of memory (32 MB), starting at address 0x6000_0000
> 
> > [    0.000000] Call trace:
> > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> > [    0.000000]  vmemmap_populate+0xd0/0x180
> > [    0.000000]  __populate_section_memmap+0x50/0x70
> > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > [    0.000000]  sparse_init+0xb0/0x224
> > [    0.000000]  bootmem_init+0x40/0x80
> > [    0.000000]  setup_arch+0x244/0x540
> > [    0.000000]  start_kernel+0x60/0x804
> > [    0.000000]  __primary_switched+0xa0/0xa8
> > [    0.000000] ---[ end trace f68728a0d3053b53 ]---
> > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [    0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134
> 
> This allocates 2 MB to cover the struct page[] array for the second
> section of memory (4 MB), starting at 0x7000_0000

OK, so every section of RAM costs 2MB to administer. I guess there is nothing one can do about that?
The one think that his my mind is that we would be happy with ARM64_PA_BITS_32, we don't have any
addresses above that in this small system.

> 
> So the problem is that you only have 36 MB of DRAM, with a large hole
> in the middle. Sparsemem was actually designed for that (hence the
> name), and flatmem would make things much worse.

Yes, that hole is not ideal.