Pmemfs/guestmemfs discussion recap and open questions

[Frank van der Linden]

On Tue, Oct 29, 2024 at 9:39 AM Mike Rapoport <rppt at kernel.org> wrote:
>
> Hi David,
>
> On Fri, Oct 25, 2024 at 11:07:27PM -0700, David Rientjes wrote:
> > On Wed, 16 Oct 2024, David Rientjes wrote:
> >
> > > ----->o-----
> > > My takeaway: based on the feedback that was provided in the discussion:
> > >
> > >  - we need an allocator abstraction for persistent memory that can return
> > >    memory with various characteristics: 1GB or not, kernel direct map or
> > >    not, HVO or not, etc.
> > >
> > >  - built on top of that, we need the ability to carve out very large
> > >    ranges of memory (cloud provider use case) with NUMA awareness on the
> > >    kernel command line
> > >
> >
> > Following up on this, I think this physical memory allocator would also be
> > possible to use as a backend for hugetlb.  Hopefully this would be an
> > allocator that would be generally useful for multiple purposes, something
> > like a mm/phys_alloc.c.
>
> Can you elaborate on this? mm/page_alloc.c already allocates physical
> memory :)
>
> Or you mean an allocator that will deal with memory carved out from what page
> allocator manages?
>
> > Frank van der Linden may also have thoughts on the above?

Yeah 'physical allocator' is a bit of a misnomer. You're right, an
allocator that deals with memory not under page allocator control is a
better description.

To elaborate a bit: there are various scenarios where allocating
contiguous stretches of physical memory is useful. HugeTLB, VM guest
memory. Or where you are presented with an external range of VM_PFNMAP
memory and need to manage it in a simple way and hand it out for guest
memory support (see NVidia's github for nvgrace-egm). However, all of
these cases may come with slightly different requirements: is the
memory purely external? Does it have struct pages? If so, is it in the
direct map? Is the memmap for the memory optimized (HVO-style)? Does
it need to be persistent? When does it need to be zeroed out?

So that's why it seems like a good idea to come up with a slightly
more generalized version of pool allocator - something that manages,
usually larger, chunks of physically contiguous memory. A is
initialized with certain properties (persistence, etc). It has methods
to grow and shrink the pool if needed. It's in no way meant to be
anywhere near as sophisticated as the page allocator, that would not
be useful (and pointless code duplication). A simple fixed-size chunk
pool will satisfy a lot of these cases.

A number of the building blocks are already there: there's CMA,
there's ZONE_DEVICE which has tools to manipulate some of these
properties (by going through a hotremove / hotplug cycle). I created a
simple prototype that essentially uses CMA as a pool provider, and
uses some ZONE_DEVICE tools to initialize memory however you want it
when it's added to the pool. I also added some new init code to to
avoid things like unneeded memmap allocation at boot for hugetlbfs
pages. I put hugetlbfs on top of it - but in a restricted way for
prototyping purposes (no reservations, no demotion).

Anyway, this is the basic idea.

- Frank