[PATCH RFC v2 0/4] mm: Introduce MAP_BELOW_HINT

[Liam R. Howlett]

* Steven Price <steven.price at arm.com> [241023 05:31]:
> >>   * Box64 seems to have a custom allocator based on reading 
> >>     /proc/self/maps to allocate a block of VA space with a low enough 
> >>     address [1]
> >>
> >>   * PHP has code reading /proc/self/maps - I think this is to find a 
> >>     segment which is close enough to the text segment [2]
> >>
> >>   * FEX-Emu mmap()s the upper 128TB of VA on Arm to avoid full 48 bit
> >>     addresses [3][4]
> > 
> > Can't the limited number of applications that need to restrict the upper
> > bound use an LD_PRELOAD compatible library to do this?
> 
> I'm not entirely sure what point you are making here. Yes an LD_PRELOAD
> approach could be used instead of a personality type as a 'hack' to
> preallocate the upper address space. The obvious disadvantage is that
> you can't (easily) layer LD_PRELOAD so it won't work in the general case.

My point is that riscv could work around the limited number of
applications that requires this.  It's not really viable for you.

> 
> >>
> >>   * pmdk has some funky code to find the lowest address that meets 
> >>     certain requirements - this does look like an ALSR alternative and 
> >>     probably couldn't directly use MAP_BELOW_HINT, although maybe this 
> >>     suggests we need a mechanism to map without a VA-range? [5]
> >>
> >>   * MIT-Scheme parses /proc/self/maps to find the lowest mapping within 
> >>     a range [6]
> >>
> >>   * LuaJIT uses an approach to 'probe' to find a suitable low address 
> >>     for allocation [7]
> >>
> > 
> > Although I did not take a deep dive into each example above, there are
> > some very odd things being done, we will never cover all the use cases
> > with an exact API match.  What we have today can be made to work for
> > these users as they have figured ways to do it.
> > 
> > Are they pretty? no.  Are they common? no.  I'm not sure it's worth
> > plumbing in new MM code in for these users.
> 
> My issue with the existing 'solutions' is that they all seem to be fragile:
> 
>  * Using /proc/self/maps is inherently racy if there could be any other
> code running in the process at the same time.

Yes, it is not thread safe.  Parsing text is also undesirable.

> 
>  * Attempting to map the upper part of the address space only works if
> done early enough - once an allocation arrives there, there's very
> little you can robustly do (because the stray allocation might be freed).
> 
>  * LuaJIT's probing mechanism is probably robust, but it's inefficient -
> LuaJIT has a fallback of linear probing, following by no hint (ASLR),
> followed by pseudo-random probing. I don't know the history of the code
> but it looks like it's probably been tweaked to try to avoid performance
> issues.
> 
> >> The biggest benefit I see of MAP_BELOW_HINT is that it would allow a
> >> library to get low addresses without causing any problems for the rest
> >> of the application. The use case I'm looking at is in a library and 
> >> therefore a personality mode wouldn't be appropriate (because I don't 
> >> want to affect the rest of the application). Reading /proc/self/maps
> >> is also problematic because other threads could be allocating/freeing
> >> at the same time.
> > 
> > As long as you don't exhaust the lower limit you are trying to allocate
> > within - which is exactly the issue riscv is hitting.
> 
> Obviously if you actually exhaust the lower limit then any
> MAP_BELOW_HINT API would also fail - there's really not much that can be
> done in that case.

Today we reverse the search, so you end up in the higher address
(bottom-up vs top-down) - although the direction is arch dependent.

If the allocation is too high/low then you could detect, free, and
handle the failure.

> 
> > I understand that you are providing examples to prove that this is
> > needed, but I feel like you are better demonstrating the flexibility
> > exists to implement solutions in different ways using todays API.
> 
> My intention is to show that today's API doesn't provide a robust way of
> doing this. Although I'm quite happy if you can point me at a robust way
> with the current API. As I mentioned my goal is to be able to map memory
> in a (multithreaded) library with a (ideally configurable) number of VA
> bits. I don't particularly want to restrict the whole process, just
> specific allocations.

If you don't need to restrict everything, won't the hint work for your
usecase?  I must be missing something from your requirements.

> 
> I had typed up a series similar to this one as a MAP_BELOW flag would
> fit my use-case well.
> 
> > I think it would be best to use the existing methods and work around the
> > issue that was created in riscv while future changes could mirror amd64
> > and arm64.
> 
> The riscv issue is a different issue to the one I'm trying to solve. I
> agree MAP_BELOW_HINT isn't a great fix for that because we already have
> differences between amd64 and arm64 and obviously no software currently
> out there uses this new flag.
> 
> However, if we had introduced this flag in the past (e.g. if MAP_32BIT
> had been implemented more generically, across architectures and with a
> hint value, like this new flag) then we probably wouldn't be in this
> situation. Applications that want to restrict the VA space would be able
> to opt-in and be portable across architectures.

I don't think that's true.  Some of the applications want all of the
allocations below a certain threshold and by the time they are adding
flags to allocations, it's too late.  What you are looking for is a
counterpart to mmap_min_addr, but for higher addresses?  This would have
to be set before any of the allocations occur for a specific binary (ie:
existing libraries need to be below that threshold too), I think?

> 
> Another potential option is a mmap3() which actually allows the caller
> to place constraints on the VA space (e.g. minimum, maximum and
> alignment). There's plenty of code out there that has to over-allocate
> and munmap() the unneeded part for alignment reasons. But I don't have a
> specific need for that, and I'm guessing you wouldn't be in favour.

You'd probably want control of the direction of the search too.

I think mmap3() would be difficult to have accepted as well.

...

Thanks,
Liam