[RFC v8 00/20] Unifying LKL into UML

[Octavian Purdila]

On Tue, Mar 16, 2021 at 11:29 PM Johannes Berg
<johannes at sipsolutions.net> wrote:
>
> Hi,

Hi Johannes,

> > My interpretation of MMU/NOMMU is like this;
> >
> > With (emulated) MMU architecture you will have more smooth integration
> > with other subsystems of kernel tree, because some subsystems/features
> > are written with "#ifdef CONFIG_MMU".  While NOMMU doesn't, it will
> > bring a simplified design with better portability.
> >
> > LKL takes rather to benefit better portability.
>
> I don't think it *matters* so much for portability? I mean, every system
> under the sun is going to allow some kind of "mprotect", right? You
> don't really want to port LKL to systems that don't have even that?
>

One use case where this matters are non OS environments such as
bootloaders [1], running on bare-bone hardware or kernel drivers [2,
3]. IMO it would be nice to keep these properties.

[1] https://www.freelists.org/post/linux-kernel-library/UEFI-LKL-port
[2] https://github.com/lkl/lkl-win-fsd
[3] https://www.haiku-os.org/tags/lkl-haiku-fsd/

> > >  * Why pthreads and all? You already require jump_buf, so UML's
> > >    switch_threads() ought to be just fine for scheduling? It almost
> > >    seems like you're doing this just so you can serialize against "other
> > >    threads" (application threads), but wouldn't that trivially be
> > >    handled by the application? You could let it hook into switch_to() or
> > >    something, but why should a single "LKL" CPU ever require multiple
> > >    threads? Seems to me that the userspace could be required to
> > >    "lkl_run()" or so (vs. lkl_start()). Heck, you could even exit
> > >    lkl_run() every time you switch tasks in the kernel, and leave
> > >    scheduling the kernel vs. the application entirely up to the
> > >    application? (A trivial application would be simply doing something
> > >    like "while (1) { lkl_run(); pause(); }" mimicking the idle loop of
> > >    UML.
> >
> > There is a description about this design choice in the LKL paper (*1);
> >
> >   "implementations based on setjmp - longjmp require usage of a single
> >   stack space partitioned between all threads. As the Linux kernel
> >   uses deep stacks (especially in the VFS layer), in an environment
> >   with small stack sizes (e.g. inside another operating system's
> >   kernel) this will place a very low limit on the number of possible
> >   threads."
> >
> > (from page 2, Section II, 2) Thread Support)
> >
> > This is a reason of using pthread as a context primitive.
>
> That impliciation (setjmp doesnt do stacks, so must use pthread) really
> isn't true, you also have posix contexts or windows fibers. That would
> probably be much easier to understands, since real threads imply that
> you have actual concurrency, which _shouldn't_ be true in the case of
> Linux emulated as being on a single CPU.
>
> Perhaps that just means you chose the wrong abstraction.
>
> In usfstl (something I've been working on) for example, we have an
> abstraction called (execution) "contexts", and they can be implemented
> using pthreads, fibers, or posix contexts, and you switch between them.
>
> (see https://github.com/linux-test-project/usfstl/blob/main/src/ctx-common.c)
>
> Using real pthreads implies that you have real threading, but then you
> need access to real mutexes, etc.
>
> If your abstraction was instead "switch context" then you could still
> implement it using pthreads+mutexes, or you could implement it using
> fibers on windows, or posix contexts - but you'd have a significantly
> reduced API surface, since you'd only expose __switch_to() or similar,
> and maybe a new stack allocation etc.
>

You are right. When I started the implementation for ucontext it was
obvious that it would be much simpler to have abstractions closer to
what Linux has (alloc, free and switch threads). But I never got to
finish that and then things went into a different direction.

> Additionally, I do wonder how UML does this now, it *does* use setjmp,
> so are you saying it doesn't properly use the kernel stacks?
>

To clarify a bit the statement in the paper, the context there was
that we should push the thread implementation to the
application/environment we run rather than providing "LKL" threads.
This was particularly important for running LKL in other OSes kernel
drivers. But you are right, we can use the switch abstraction and
implement it with threads and mutexes for those environments where it
helps.

> > to design that with UML, what we need to do are;
> >
> > 1) change Makefile to output liblinux.a
>
> or liblinux.so, I guess, dynamic linking should be ok.
>
> > we faced linker script issue, which is related with generating
> > relocatable object in the middle.
> >
> > 2) make the linker-script clean with 2-stage build
> > we fix the linker issues of (1)
> >
> > 3) expose syscall as a function call
> > conflicts names (link-time and compile-time conflicts)
> >
> > 4) header rename, object localization
> > to fix the issue (3)
> >
> > This is a common set of modifications to a library of UML.
>
> All of this is just _build_ issues. It doesn't mean you couldn't take
> some minimal code + liblinux.a and link it to get a "linux" equivalent
> to the current UML?
>
> TBH, I started thinking that it might be _really_ nice to be able to
> write an application that's *not quite UML* but has all the properties
> of UML built into it, i.e. can run userspace etc.
>
> > Other parts are a choice of design, I believe.
> > Because a library is more _reusable_ than an executable (by it means), the
> > choice of LKL is to be portable, which the current UML doesn't pursue it
> > extensibly (focus on intel platforms).
> >
>
> I don't think this really conflicts.
>
> You could have a liblinux.a/liblinux.so and some code that links it all
> together to get "linux" (UML). Having userspace running inside the UML
> (liblinux) might only be supported on x86 for now, MMU vs. NOMMU might
> be something that's configurable at build time, and if you pick NOMMU
> you cannot run userspace either, etc.
>
> But conceptually, why wouldn't it be possible to have a liblinux.so that
> *does* build with MMU and userspace support, and UML is a wrapper around
> it?
>

This is an interesting idea. Conceptually I think it is possible.
There are lots of details to be figured out before we do this. I think
that having a NOMMU version could be a good step in the right
direction, especially since I think a liblinux.so has more NOMMU
usecases than MMU usecases - but I haven't given too much thought to
the MMU usecases.