SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 10:45, Joakim Tjernlund
<Joakim.Tjernlund at infinera.com> wrote:
>
> On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> > On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> > <Joakim.Tjernlund at infinera.com> wrote:
> > >
> > > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> > ..
> > > >
> > > > What you might try is changing the section size to 32 MB and mapping
> > > > the vmemmap region down to pages. That way, the vmemmap region should
> > > > only take up
> > > > - 512 KiB for the struct page array[] itself
> > > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > >
> > > > You could try the below and see if it makes any difference?
> > > >
> > > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > > b/arch/arm64/include/asm/sparsemem.h
> > > > index 4b73463423c3..a008f4342532 100644
> > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > @@ -23,7 +23,7 @@
> > > >   * entries could not be created for vmemmap mappings.
> > > >   * 16K follows 4K for simplicity.
> > > >   */
> > > > -#define SECTION_SIZE_BITS 27
> > > > +#define SECTION_SIZE_BITS 25
> > > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > >
> > > >  #endif
> > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > > index 5b1946f1805c..d25560a53a67 100644
> > > > --- a/arch/arm64/mm/mmu.c
> > > > +++ b/arch/arm64/mm/mmu.c
> > > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > > addr, unsigned long end,
> > > >  }
> > > >  #endif
> > > >
> > > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > > end, int node,
> > > >                 struct vmem_altmap *altmap)
> > > >  {
> > >
> > > That was a really good idea, now I have:
> > >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > >
> > > Reserved dropped from 14+MB to 7+MB :)
> > >
> >
> > Glad to hear that.
> >
> > > Should I look into something particular before testing this in a bigger scale?
> > >
> >
> > A bit of stress testing would be useful, although I wouldn't be able
> > to recommend anything that will be able to run with that little
> > memory.
> >
> > Just be aware that this is not a change that is likely to be accepted
> > by the arm64 maintainers - it is something you will have to carry in
> > your downstream.
>
> In its current form, agreed.
> If it could be generalized maybe, as is today aarch64 mandates lots of RAM
> but your hack shows promise so maybe something will come of it.
>

I'm not sure - using PMD maps actually uses 4 KiB less for every 128
MiB of memory, and it is the rounding up to multiples of 128 MiB that
is biting you. IOW, on a system that has a multiple of 128 MiB which
is laid out straight-forwardly, this change results in more memory to
be used (the number of sections also increases, which means more
per-section bookkeeping)

As Arnd explained in the other thread, 128 MiB of DRAM is already
considered less than a reasonable minimum even on 32-bit architectures
(if I am paraphrasing that correctly), and so getting anyone to care
about running the arm64 kernel on a device with 36 MB of DRAM is going
to be very difficult.

> >
> > Out of curiosity, what type of system is this? What user space are you using?
> This is a very fancy custom ASIC, user space is a custom Yocto in 32 bits musl, mainly busybox
> for utilities.
>

OK, so even if I am not a fan of this practice [0], running a 32-bit
kernel would likely be an option for you as well. Have you considered
this?

[0] https://security.googleblog.com/2022/02/mitigating-kernel-risks-on-32-bit-arm.html