SPARSEMEM memory needs?

[Joakim Tjernlund]

On Tue, 2022-06-07 at 11:02 +0200, Ard Biesheuvel wrote:
> On Tue, 7 Jun 2022 at 10:45, Joakim Tjernlund
> <Joakim.Tjernlund at infinera.com> wrote:
> > 
> > On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> > > On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> > > <Joakim.Tjernlund at infinera.com> wrote:
> > > > 
> > > > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> > > ..
> > > > > 
> > > > > What you might try is changing the section size to 32 MB and mapping
> > > > > the vmemmap region down to pages. That way, the vmemmap region should
> > > > > only take up
> > > > > - 512 KiB for the struct page array[] itself
> > > > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > > > 
> > > > > You could try the below and see if it makes any difference?
> > > > > 
> > > > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > > > b/arch/arm64/include/asm/sparsemem.h
> > > > > index 4b73463423c3..a008f4342532 100644
> > > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > > @@ -23,7 +23,7 @@
> > > > >   * entries could not be created for vmemmap mappings.
> > > > >   * 16K follows 4K for simplicity.
> > > > >   */
> > > > > -#define SECTION_SIZE_BITS 27
> > > > > +#define SECTION_SIZE_BITS 25
> > > > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > > > 
> > > > >  #endif
> > > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > > > index 5b1946f1805c..d25560a53a67 100644
> > > > > --- a/arch/arm64/mm/mmu.c
> > > > > +++ b/arch/arm64/mm/mmu.c
> > > > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > > > addr, unsigned long end,
> > > > >  }
> > > > >  #endif
> > > > > 
> > > > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > > > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > > > end, int node,
> > > > >                 struct vmem_altmap *altmap)
> > > > >  {
> > > > 
> > > > That was a really good idea, now I have:
> > > >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > > > 
> > > > Reserved dropped from 14+MB to 7+MB :)
> > > > 
> > > 
> > > Glad to hear that.
> > > 
> > > > Should I look into something particular before testing this in a bigger scale?
> > > > 
> > > 
> > > A bit of stress testing would be useful, although I wouldn't be able
> > > to recommend anything that will be able to run with that little
> > > memory.
> > > 
> > > Just be aware that this is not a change that is likely to be accepted
> > > by the arm64 maintainers - it is something you will have to carry in
> > > your downstream.
> > 
> > In its current form, agreed.
> > If it could be generalized maybe, as is today aarch64 mandates lots of RAM
> > but your hack shows promise so maybe something will come of it.
> > 
> 
> I'm not sure - using PMD maps actually uses 4 KiB less for every 128
> MiB of memory, and it is the rounding up to multiples of 128 MiB that
> is biting you. IOW, on a system that has a multiple of 128 MiB which
> is laid out straight-forwardly, this change results in more memory to
> be used (the number of sections also increases, which means more
> per-section bookkeeping)

hmm, I don't quite understand this but seems to me that there something more
going on as I recovered about 7MB with your hack.
In any case, I figure exact memory model should be a build option and one
will have to choose what fits your system best. 

> 
> As Arnd explained in the other thread, 128 MiB of DRAM is already
> considered less than a reasonable minimum even on 32-bit architectures
> (if I am paraphrasing that correctly), and so getting anyone to care
> about running the arm64 kernel on a device with 36 MB of DRAM is going
> to be very difficult.

That is a lot of RAM in the embedded space.

> 
> > > 
> > > Out of curiosity, what type of system is this? What user space are you using?
> > This is a very fancy custom ASIC, user space is a custom Yocto in 32 bits musl, mainly busybox
> > for utilities.
> > 
> 
> OK, so even if I am not a fan of this practice [0], running a 32-bit
> kernel would likely be an option for you as well. Have you considered
> this?
> 
> [0] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsecurity.googleblog.com%2F2022%2F02%2Fmitigating-kernel-risks-on-32-bit-arm.html&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7Ce1561709c32a446e957708da48647e8b%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637901893702410819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Mwus3JktdxLJ2znXumuVGI8k5hNTkniMgbr7CKwjvrE%3D&reserved=0

Briefly considered 32 bits kernel but as I understand it, it is even more unsupported so I have opted
for 64 bits kernel for now. I may have to revisit that though. ARM is still new to me.
Thanks for the link, good read.

     Jocke