SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 11:38, Ard Biesheuvel <ardb at kernel.org> wrote:
>
> On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund
> <Joakim.Tjernlund at infinera.com> wrote:
> >
> > On Tue, 2022-06-07 at 11:02 +0200, Ard Biesheuvel wrote:
> > > On Tue, 7 Jun 2022 at 10:45, Joakim Tjernlund
> > > <Joakim.Tjernlund at infinera.com> wrote:
> > > >
> > > > On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> > > > > On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> > > > > <Joakim.Tjernlund at infinera.com> wrote:
> > > > > >
> > > > > > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> > > > > ..
> > > > > > >
> > > > > > > What you might try is changing the section size to 32 MB and mapping
> > > > > > > the vmemmap region down to pages. That way, the vmemmap region should
> > > > > > > only take up
> > > > > > > - 512 KiB for the struct page array[] itself
> > > > > > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > > > > >
> > > > > > > You could try the below and see if it makes any difference?
> > > > > > >
> > > > > > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > > > > > b/arch/arm64/include/asm/sparsemem.h
> > > > > > > index 4b73463423c3..a008f4342532 100644
> > > > > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > > > > @@ -23,7 +23,7 @@
> > > > > > >   * entries could not be created for vmemmap mappings.
> > > > > > >   * 16K follows 4K for simplicity.
> > > > > > >   */
> > > > > > > -#define SECTION_SIZE_BITS 27
> > > > > > > +#define SECTION_SIZE_BITS 25
> > > > > > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > > > > >
> > > > > > >  #endif
> > > > > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > > > > > index 5b1946f1805c..d25560a53a67 100644
> > > > > > > --- a/arch/arm64/mm/mmu.c
> > > > > > > +++ b/arch/arm64/mm/mmu.c
> > > > > > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > > > > > addr, unsigned long end,
> > > > > > >  }
> > > > > > >  #endif
> > > > > > >
> > > > > > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > > > > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > > > > > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > > > > > end, int node,
> > > > > > >                 struct vmem_altmap *altmap)
> > > > > > >  {
> > > > > >
> > > > > > That was a really good idea, now I have:
> > > > > >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > > > > >
> > > > > > Reserved dropped from 14+MB to 7+MB :)
> > > > > >
> > > > >
> > > > > Glad to hear that.
> > > > >
> > > > > > Should I look into something particular before testing this in a bigger scale?
> > > > > >
> > > > >
> > > > > A bit of stress testing would be useful, although I wouldn't be able
> > > > > to recommend anything that will be able to run with that little
> > > > > memory.
> > > > >
> > > > > Just be aware that this is not a change that is likely to be accepted
> > > > > by the arm64 maintainers - it is something you will have to carry in
> > > > > your downstream.
> > > >
> > > > In its current form, agreed.
> > > > If it could be generalized maybe, as is today aarch64 mandates lots of RAM
> > > > but your hack shows promise so maybe something will come of it.
> > > >
> > >
> > > I'm not sure - using PMD maps actually uses 4 KiB less for every 128
> > > MiB of memory, and it is the rounding up to multiples of 128 MiB that
> > > is biting you. IOW, on a system that has a multiple of 128 MiB which
> > > is laid out straight-forwardly, this change results in more memory to
> > > be used (the number of sections also increases, which means more
> > > per-section bookkeeping)
> >
> > hmm, I don't quite understand this but seems to me that there something more
> > going on as I recovered about 7MB with your hack.
>
> Yeah, I noticed that. I guess the rounding up to 128 MiB results in
> some overhead somewhere else as well.
>
> > In any case, I figure exact memory model should be a build option and one
> > will have to choose what fits your system best.
> >
>
> We used

'just' - no idea how I manage to typo that ...

> removed support for everything except SPARSEMEM_VMEMMAP,