SPARSEMEM memory needs?

Ard Biesheuvel ardb at kernel.org
Tue Jun 7 02:38:32 PDT 2022


On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund
<Joakim.Tjernlund at infinera.com> wrote:
>
> On Tue, 2022-06-07 at 11:02 +0200, Ard Biesheuvel wrote:
> > On Tue, 7 Jun 2022 at 10:45, Joakim Tjernlund
> > <Joakim.Tjernlund at infinera.com> wrote:
> > >
> > > On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> > > > On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> > > > <Joakim.Tjernlund at infinera.com> wrote:
> > > > >
> > > > > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> > > > ..
> > > > > >
> > > > > > What you might try is changing the section size to 32 MB and mapping
> > > > > > the vmemmap region down to pages. That way, the vmemmap region should
> > > > > > only take up
> > > > > > - 512 KiB for the struct page array[] itself
> > > > > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > > > >
> > > > > > You could try the below and see if it makes any difference?
> > > > > >
> > > > > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > > > > b/arch/arm64/include/asm/sparsemem.h
> > > > > > index 4b73463423c3..a008f4342532 100644
> > > > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > > > @@ -23,7 +23,7 @@
> > > > > >   * entries could not be created for vmemmap mappings.
> > > > > >   * 16K follows 4K for simplicity.
> > > > > >   */
> > > > > > -#define SECTION_SIZE_BITS 27
> > > > > > +#define SECTION_SIZE_BITS 25
> > > > > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > > > >
> > > > > >  #endif
> > > > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > > > > index 5b1946f1805c..d25560a53a67 100644
> > > > > > --- a/arch/arm64/mm/mmu.c
> > > > > > +++ b/arch/arm64/mm/mmu.c
> > > > > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > > > > addr, unsigned long end,
> > > > > >  }
> > > > > >  #endif
> > > > > >
> > > > > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > > > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > > > > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > > > > end, int node,
> > > > > >                 struct vmem_altmap *altmap)
> > > > > >  {
> > > > >
> > > > > That was a really good idea, now I have:
> > > > >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > > > >
> > > > > Reserved dropped from 14+MB to 7+MB :)
> > > > >
> > > >
> > > > Glad to hear that.
> > > >
> > > > > Should I look into something particular before testing this in a bigger scale?
> > > > >
> > > >
> > > > A bit of stress testing would be useful, although I wouldn't be able
> > > > to recommend anything that will be able to run with that little
> > > > memory.
> > > >
> > > > Just be aware that this is not a change that is likely to be accepted
> > > > by the arm64 maintainers - it is something you will have to carry in
> > > > your downstream.
> > >
> > > In its current form, agreed.
> > > If it could be generalized maybe, as is today aarch64 mandates lots of RAM
> > > but your hack shows promise so maybe something will come of it.
> > >
> >
> > I'm not sure - using PMD maps actually uses 4 KiB less for every 128
> > MiB of memory, and it is the rounding up to multiples of 128 MiB that
> > is biting you. IOW, on a system that has a multiple of 128 MiB which
> > is laid out straight-forwardly, this change results in more memory to
> > be used (the number of sections also increases, which means more
> > per-section bookkeeping)
>
> hmm, I don't quite understand this but seems to me that there something more
> going on as I recovered about 7MB with your hack.

Yeah, I noticed that. I guess the rounding up to 128 MiB results in
some overhead somewhere else as well.

> In any case, I figure exact memory model should be a build option and one
> will have to choose what fits your system best.
>

We used removed support for everything except SPARSEMEM_VMEMMAP, and I
don't think it will be coming back. Note that half of the overhead you
are observing is a result of the peculiar physical placement of DRAM,
and the current arrangement is fine except for extreme cases like
yours.

But of course, feel free to propose a change along these lines: I am
not a arm64 maintainer, and so it is not up to me ultimately.

> >
> > As Arnd explained in the other thread, 128 MiB of DRAM is already
> > considered less than a reasonable minimum even on 32-bit architectures
> > (if I am paraphrasing that correctly), and so getting anyone to care
> > about running the arm64 kernel on a device with 36 MB of DRAM is going
> > to be very difficult.
>
> That is a lot of RAM in the embedded space.
>

Of course. But this also means that you have no need for a kernel that
uses 64-bit pointers, each of which wastes 4 bytes of memory too.

> >
> > > >
> > > > Out of curiosity, what type of system is this? What user space are you using?
> > > This is a very fancy custom ASIC, user space is a custom Yocto in 32 bits musl, mainly busybox
> > > for utilities.
> > >
> >
> > OK, so even if I am not a fan of this practice [0], running a 32-bit
> > kernel would likely be an option for you as well. Have you considered
> > this?
> >
..
>
> Briefly considered 32 bits kernel but as I understand it, it is even more unsupported so I have opted
> for 64 bits kernel for now. I may have to revisit that though. ARM is still new to me.

It is a bad idea for many reasons, but you may not have a choice.



More information about the linux-arm-kernel mailing list