[RFC PATCH 00/13] nommu UML

Sat Oct 26 03:19:08 PDT 2024

Hi,

On Thu, 2024-10-24 at 21:09 +0900, Hajime Tazaki wrote:
> This is a series of patches of nommu arch addition to UML.  It would
> be nice to ask comments/opinions on this.
> 
> There are several limitations/issues which we already found; here is
> the list of those issues.
> 
> - prompt configured with /etc/profile is broken (variables are not
>   expanded, ${HOSTNAME%%.*}:$PWD#)
> - there are no mechanism implemented to cache for mapped memory of
>   exec(2) thus, always read files from filesystem upon every exec,
>   which makes slow on some benchmark (lmbench).
> - a crash on userspace programs crashes a UML kernel, not signaling
>   with SIGSEGV to the program.
> - commit c27e618 (during v6.12-rc1 merge) introduces invalid access to
>   a vma structure for our case, which updates the internal procedure
>   of maple_tree subsystem.  We're trying to fix issue but still a
>   random process on exit(2) crashes.

Btw. are you handling FP register save/restore? If it is not there, it
probably would not be too hard to add (XSAVE, etc.), though it might
add a bit of additional overhead. Especially as UML always saves the FP
state rather than optimizing it like the x86 architectures.

I am a bit confused overall. I mean, zpoline seems kind of neat, but a
requirement on patching userspace code also seems like a lot.

To me, it seems much more natural to catch the userspace syscalls using
a SECCOMP filter[1]. While quite a lot slower, that should be much more
portable across architectures. For improved speed one could still do
architecture specific things inside the vDSO or by using zpoline. But
those would then "just" be optimizations and unpatched code would still
work correctly (e.g. JIT).

For me, a big argument in favour of such an approach is its simplicity.
I am mostly basing that on the fact that this patchset should properly
handle other signals like SIGFPE and SIGSEGV. And, once it does that,
you will already have all the infrastructure to do the correct register
save/restore using the host mcontex, which is what is needed in the
SIGSYS handler when using SECCOMP. The filter itself should be simple
as it just needs to catch all syscalls within valid userspace
executable memory[2] ranges.

Benjamin

[1] Maybe not surprising, as I have been working on a SECCOMP based UML
that does not require ptrace.
[2] I am assuming that userspace executable code is already confined to
a certain address space within the UML process. Obviously, the kernel
itself and loaded modules need to be free to do host syscalls and
should not be affected by the SECCOMP filter.

> 
> UML has been built with CONFIG_MMU since day 0.  The feature
> introduces the nommu mode in a different angle from what Linux Kernel
> Library tried.
> 
> 
> What is it for ?
> ================
> 
> - Alleviate syscall hook overhead implemented with ptrace(2)
> - To exercises nommu code over UML (and over KUnit)
> - Less dependency to host facilities
> 
> 
> How it works ?
> ==============
> 
> To illustrate how this feature works, the below shows how syscalls are
> called under nommu/UML environment.
> 
> - boot kernel, setup zpoline trampoline code (detailed later) at address 0x0
> - (userspace starts)
> - calls vfork/execve syscalls
> - during execve, more specifically during load_elf_fdpic_binary()
>   function, kernel translates `syscall/sysenter` instructions with `call
>   *%rax`, which usually point to address 0 to NR_syscalls (around
>   512), where trampoline code was installed during startup.
> - when syscalls are issued by userspace, it jumps to *%rax, slides
>   until `nop` instructions end, and jump to hooked function,
>   `__kernel_vsyscall`, which is an entrypoint for syscall under nommu
>   UML environment.
> - call handler function in sys_call_table[] and follow how UML syscall
>   works.
> - return to userspace
> 
> 
> What are the differences from MMU-full UML ?
> ============================================
> 
> The current nommu implementation adds 3 different functions which
> MMU-full UML doesn't have:
> 
> - kernel address space can directly be accessible from userspace
>   - so, uaccess() always returns 1
>   - generic implementation of memcpy/strcpy/futex is also used
> - alternate syscall entrypoint without ptrace
> - translation of syscall/sysenter instructions to a trampoline code
>   and syscall hooks
> 
> With those modifications, it allows us to use unmodified userspace
> binaries with nommu UML.
> 
> 
> History
> =======
> 
> This feature was originally introduced by Ricardo Koller at Open
> Source Summit NA 2020, then integrated with the syscall translation
> functionality with the clean up to the original code.
> 
> Building and run
> ================
> 
> ```
> % make ARCH=um x86_64_nommu_defconfig
> % make ARCH=um
> ```
> 
> will build UML with CONFIG_MMU=n applied.
> 
> Kunit tests can run with the following command:
> 
> ```
> % ./tools/testing/kunit/kunit.py run --kconfig_add CONFIG_MMU=n
> ```
> 
> To run a typical Linux distribution, we need nommu-aware userspace.
> We can use a stock version of Alpine Linux with nommu-built version of
> busybox and musl-libc.
> 
> 
> Preparing root filesystem
> =========================
> 
> nommu UML requires to use a specific standard library which is aware
> of nommu kernel.  We have tested custom-build musl-libc and busybox,
> both of which have built-in support for nommu kernels.
> 
> There are no available Linux distributions for nommu under x86_64
> architecture, so we need to prepare our own image for the root
> filesystem.  We use Alpine Linux as a base distribution and replace
> busybox and musl-libc on top of that.  The following are the step to
> prepare the filesystem for the quick start.
> 
> ```
>      container_id=$(docker create ghcr.io/thehajime/alpine:3.20.3-um-nommu)
>      docker start $container_id
>      docker wait $container_id
>      docker export $container_id > alpine.tar
>      docker rm $container_id
> 
>      mnt=$(mktemp -d)
>      dd if=/dev/zero of=alpine.ext4 bs=1 count=0 seek=1G
>      sudo chmod og+wr "alpine.ext4"
>      yes 2>/dev/null | mkfs.ext4 "alpine.ext4" || true
>      sudo mount "alpine.ext4" $mnt
>      sudo tar -xf alpine.tar -C $mnt
>      sudo umount $mnt
> ```
> 
> This will create a file image, `alpine.ext4`, which contains busybox
> and musl with nommu build on the Alpine Linux root filesystem.  The
> file can be specified to the argument `ubd0=` to the UML command line.
> 
> ```
>   ./vmlinux eth0=tuntap,tap100,0e:fd:0:0:0:1,172.17.0.1 ubd0=./alpine.ext4 rw mem=1024m loglevel=8 init=/sbin/init
> ```
> 
> We plan to upstream apk packages for busybox and musl so that we can
> follow the proper procedure to set up the root filesystem.
> 
> 
> Quick start with docker
> =======================
> 
> There is a docker image that you can quickly start with a simple step.
> 
> ```
>   docker run -it -v /dev/shm:/dev/shm --rm ghcr.io/thehajime/alpine:3.20.3-um-nommu
> ```
> 
> This will launch a UML instance with an pre-configured root filesystem.
> 
> Benchmark
> =========
> 
> The below shows an example of performance measurement conducted with
> lmbench and (self-crafted) getpid benchmark (with v6.12-rc3 linus tree).
> 
> ### lmbench (usec)
> 
> > > native|um|um-nommu|
> > --|--|--|--|
> > select-10    |0.5645|28.3738|0.2647|
> > select-100   |2.3872|28.8385|1.1021|
> > select-1000  |20.5527|37.6364|9.4264|
> > syscall      |0.1735|26.8711|0.1037|
> > read         |0.3442|28.5771|0.1370|
> > write        |0.2862|28.7340|0.1236|
> > stat         |1.9236|38.5928|0.4640|
> > open/close   |3.8308|66.8451|0.7789|
> > fork+sh      |1176.4444|8221.5000|21443.0000|
> > fork+execve  |533.1053|3034.5000|4894.3333|
> 
> ### do_getpid bench (nsec)
> 
> > > native|um|um-nommu|
> > --|--|--|--|
> > getpid | 180 | 31579 | 101|
> 
> 
> Limitations
> ===========
> 
> generic nommu limitations
> -------------------------
> Since this port is a kernel of nommu architecture so, the
> implementation inherits the characteristics of other nommu kernels
> (riscv, arm, etc), described below.
> 
> - vfork(2) should be used instead of fork(2)
> - ELF loader only loads PIE (position independent executable) binaries
> - processes share the address space among others
> - mmap(2) offers a subset of functionalities (e.g., unsupported
>   MMAP_FIXED)
> 
> Thus, we have limited options to userspace programs.  We have tested
> Alpine Linux with musl-libc, which has a support nommu kernel.
> 
> access to mmap_min_addr
> ----------------------
> As the mechanism of syscall translations relies on an ability to
> write/read memory address zero (0x0), we need to configure host kernel
> with the following command:
> 
> ```
> % sh -c "echo 0 > /proc/sys/vm/mmap_min_addr"
> ```
> 
> supported architecture
> ----------------------
> The current implementation of nommu UML only works on x86_64 SUBARCH.
> We have not tested with 32-bit environment.
> 
> target of syscall translation
> -----------------------------
> The syscall translation only applies to the executable and interpreter
> of ELF binary files which are processed by execve(2) syscall for the
> moment: other libraries such as linked library and dlopen-ed one
> aren't translated; we may be able to trigger the translation by
> LD_PRELOAD.
> 
> Note that with musl-libc in Alpine Linux which we've been tested, most
> of syscalls are implemented in the interpreter file
> (ld-musl-x86_64.so) and calling syscall/sysenter instructions from the
> linked/loaded libraries might be rare.  But it is definitely possible
> so, a workaround with LD_PRELOAD is effective.
> 
> 
> Further readings about NOMMU UML
> ================================
> 
> - NOMMU UML (original code by Ricardo Koller)
> https://static.sched.com/hosted_files/ossna2020/ec/kollerr_linux_um_nommu.pdf
> 
> - zpoline: syscall translation mechanism
> https://www.usenix.org/conference/atc23/presentation/yasukata
> Please review the following changes for suitability for inclusion. If you have
> any objections or suggestions for improvement, please respond to the patches. If
> you agree with the changes, please provide your Acked-by.
> 
> The following changes since commit c2ee9f594da826bea183ed14f2cc029c719bf4da:
> 
>   KVM: selftests: Fix build on on non-x86 architectures (2024-10-21 15:49:33 -0700)
> 
> are available in the Git repository at:
> 
>   https://github.com/thehajime/linux 82a7ee8b31c51edb47e144922581824a3b5e371d
>   https://github.com/thehajime/linux/tree/um-nommu-v6.12-rc4-rfc
> 
> Hajime Tazaki (13):
>   fs: binfmt_elf_efpic: add architecture hook elf_arch_finalize_exec
>   x86/um: nommu: elf loader for fdpic
>   um: nommu: memory handling
>   x86/um: nommu: syscall handling
>   x86/um: nommu: syscall translation by zpoline
>   x86/um: nommu: process/thread handling
>   um: nommu: configure fs register on host syscall invocation
>   x86/um/vdso: nommu: vdso memory update
>   x86/um: nommu: signal handling
>   x86/um: nommu: stack save/restore on vfork
>   um: change machine name for uname output
>   um: nommu: add documentation of nommu UML
>   um: nommu: plug nommu code into build system
> 
>  Documentation/virt/uml/nommu-uml.rst    | 219 +++++++++++++++++++++++
>  arch/um/Kconfig                         |  13 +-
>  arch/um/Makefile                        |   6 +
>  arch/um/configs/x86_64_nommu_defconfig  |  64 +++++++
>  arch/um/include/asm/futex.h             |   4 +
>  arch/um/include/asm/mmu.h               |   8 +
>  arch/um/include/asm/mmu_context.h       |  14 +-
>  arch/um/include/asm/ptrace-generic.h    |  17 ++
>  arch/um/include/asm/tlbflush.h          |  23 ++-
>  arch/um/include/asm/uaccess.h           |   7 +-
>  arch/um/include/shared/common-offsets.h |   3 +
>  arch/um/include/shared/os.h             |   9 +
>  arch/um/kernel/Makefile                 |   3 +-
>  arch/um/kernel/exec.c                   |   8 +
>  arch/um/kernel/mem.c                    |  13 ++
>  arch/um/kernel/physmem.c                |   6 +
>  arch/um/kernel/process.c                |  34 +++-
>  arch/um/kernel/skas/Makefile            |   3 +-
>  arch/um/kernel/trap.c                   |   4 +
>  arch/um/os-Linux/main.c                 |   5 +
>  arch/um/os-Linux/process.c              |  22 +++
>  arch/um/os-Linux/skas/process.c         |   4 +
>  arch/um/os-Linux/start_up.c             |  47 +++++
>  arch/um/os-Linux/time.c                 |   3 +-
>  arch/um/os-Linux/util.c                 |   3 +-
>  arch/x86/um/Makefile                    |  18 ++
>  arch/x86/um/asm/elf.h                   |  12 +-
>  arch/x86/um/asm/module.h                |  19 +-
>  arch/x86/um/asm/processor.h             |  12 ++
>  arch/x86/um/do_syscall_64.c             | 113 ++++++++++++
>  arch/x86/um/entry_64.S                  | 110 ++++++++++++
>  arch/x86/um/shared/sysdep/syscalls_64.h |   4 +
>  arch/x86/um/signal.c                    |  26 +++
>  arch/x86/um/syscalls_64.c               |  67 +++++++
>  arch/x86/um/vdso/um_vdso.c              |  20 +++
>  arch/x86/um/vdso/vma.c                  |  16 +-
>  arch/x86/um/zpoline.c                   | 228 ++++++++++++++++++++++++
>  fs/Kconfig.binfmt                       |   2 +-
>  fs/binfmt_elf_fdpic.c                   |  10 ++
>  39 files changed, 1164 insertions(+), 35 deletions(-)
>  create mode 100644 Documentation/virt/uml/nommu-uml.rst
>  create mode 100644 arch/um/configs/x86_64_nommu_defconfig
>  create mode 100644 arch/x86/um/do_syscall_64.c
>  create mode 100644 arch/x86/um/entry_64.S
>  create mode 100644 arch/x86/um/zpoline.c
> 