[PATCH 11/11] kernel/x86: enable x32 support for amd64

[Stefan Lippers-Hollmann]

Hi

On 2023-11-14, Elliott Mitchell wrote:
> On Wed, Nov 15, 2023 at 05:35:11AM +0100, Stefan Lippers-Hollmann wrote:
> > On 2023-11-14, Elliott Mitchell wrote:
[...]
> > What would be the reason for enabling x32?
> > There is very little upstream buy-in for x32, when this question came
> > up for Debian, it was rejected to be enabled for -among other reasons-
> > concerns about the the system call ABI and its security hardening, as
> > well as concerns about the long term ABI compatibility (the later of
> > which probably not that relevant for OpenWrt, the former however is).
>
> What I see seems like x32 is still making progress on Debian.  I note
> musl has a x32 target.  While it still isn't a release architecture for
> Debian, I could see it becoming one.

I wouldn't really consider this to be very likely, if at all, it has
become even less likely than it was over a decade ago.

> > I do understand that this is a pet peeve among the high-density
> > virtualization crowd, but anywhere else, x32 as a concept is dead (and
> > never took off in the first place).
>
> In my view x32 has some substantial use cases.  Have you ever seen `ls`,
> `rm`, `grep`, `find`, or `test` needing more than 4GB of memory?  Any
> of these needing that much memory would suggest something was wrong
> (or perhaps someone was trying to abuse them in an impressive way).  For
> quite a number of shell utilities being able to use more than 4GB of
> memory is pointless.

The counter question would be, does it hurt?
Yes, the x32 proponents (mostly from the high-density virtualization
crowd and, as Linus called it "extreme benchmarking" environments) will
point out that pure-x32 uses slightly less disk space and slightly less
RAM than pure64, but we are talking about very little in today's world.

> This is speculation, but I suspect there would be substantial value in
> having mixed systems where some utilities were x32 and some were amd64.

Please point out what exactly those would be, especially in the context
of OpenWrt.

If you really want to mix x32 and amd64 binaries, you will need a full
library set (binary packages - co-installable) for both x32 and x86_64
installed and in RAM - which is completely detrimental to any of the
gains promised above.

x32 only works, if you have (aside from the kernel itself) a pure-x32
userspace, or your losses in terms of additional disk space and RAM
usage outweigh your gains by an order of magnitude. In a general purpose
distribution you might ignore that by claiming that 'only' your big
database management system (and its dependencies) would have to be a 64
binary (or some other -very isolated, compared to the rest of the running
system- big component), but the situation for OpenWrt is different.

This kind of mixed userspace would be a nightmare for complexity on the
source side - and/or would require Debian-style multiarch support.
Neither of which is very likely to succeed for OpenWrt (let alone for
opkg).

> > *If* (and imho, again purely my own irrelevant opinion, that is a
> > big IF) x32 would really be deemed worthwhile for OpenWrt, it still
> > doesn't make sense to enable this whole userspace ABI for the x86_64
> > kernel now, before your desired additional patches are available
> > (and even then, you'd probably want a dedicated x32 subtarget instead
> > of  killing off the pure64 target for OpenWrt - something I'd
> > personally be even less in favour of).
>
> Enabling the kernel support is always the first step.  How does one test
> userspace if the kernel won't even load the executables?

In a general purpose distribution one might say that, however even there
one would expect quite a lot of testing having happened before enabling
a complete new/ additional ABI to userspace (duplicate to the native one),
something akin to the semi-official debian-ports infrastructure, used for
bootstrapping of new ports or retiring old ones that still have some form
of interest.

But in this case there is nothing to test in isolation, as there are
no x32 binaries compatible with OpenWrt (you could only test inside
a non-OpenWrt x32 chroot, but that's barely grounds for enabling x32
in OpenWrt's kernel, just in case; otherwise float emulation would be
enabled for mips, to facilitate Debian chroots - spoiler, it's not).

In contrast to the changes necessary (buildsystem, image generation,
kernel configs, etc.) to test an OpenWrt x32 subtarget, a lazy-man's
two-line patch of enabling X86_X32 && X86_X32_ABI would be trivial
in comparison to anything of the rest that would be needed.

Given that OpenWrt doesn't have any binary compatibility between
major versions and needs to get rebuilt in total for each new revision,
there is no need to stage- and phase in additions like this, add the
new sub-target in its entirety (or not).

[...]
> On Wed, Nov 15, 2023 at 05:41:54AM +0100, Stefan Lippers-Hollmann wrote:
> > On 2023-11-15, Stefan Lippers-Hollmann wrote:
> > > What would be the reason for enabling x32?
> > > There is very little upstream buy-in for x32, when this question came
> > > up for Debian, it was rejected to be enabled for -among other reasons-
> > > concerns about the the system call ABI and its security hardening, as
> > > well as concerns about the long term ABI compatibility (the later of
> > > which probably not that relevant for OpenWrt, the former however is).
> >
> > Just to add some references to this:
> > - Can we drop upstream Linux x32 support?
> >   https://lkml.org/lkml/2018/12/10/1145
> >   the whole thread is interesting and doesn't display too much
> >   sympathy for x32
>
> Yes, and did it get dropped?  No, it didn't.  At least some of the issues
> brought up there have been fixed.

It didn't get any more traction either, rather to the contrary (disk-
and RAM are growing in modern hardware, what would be the real gains
here).

> > - information about the never merged/ defunct x32 port proposed for
> >   Debian
> >   https://wiki.debian.org/X32Port
>
> There are pointers to many bugs which were found and fixed.  There don't
> appear to be that many unresolved ones.  Unfortunately I don't know
> whether it is likely to be part of the next formal release.

> I also note even the most extreme of the OpenWRT virtual machine
> suggestions doesn't suggest more than 4GB of memory.

and as show cased, a pure64 OpenWrt can get away with ~16 MB disk space
(using a sparse image, rather than the ~121 MB raw disk image) and easily
gets away with ~96 MB RAM with its default package set - how much do you
think x32 would improve that (we're talking about x86_64 hardware here,
not legacy i386 systems)?
Is it worth the effort depicted above?
(and yes, I've tested both with 23.05.0 on qemu; 12 MB qcow2 sparse
image, 96 MB RAM (64 MB throws OOM)).
Even the runtime performance improvements should be negligible, all the
routing happens in kernelspace (for the most part), which would remain
x86_64 - any speed improvements on the userspace would be (apart from
being minimal in isolation) completely irrelevant to routing performance.
dnsmasq, ls, cp, rm and friends being 5% faster or slower doesn't really
matter (and around 5% is the most even its proponents claim).

> I don't know the future, but enabling kernel support is the first step.

If you wanted to add it to Debian (with working a multi-arch
implementation and organically grown repositories (which aren't
rebuild aside from sourceful uploads or targeted binNMUs), you would
be right - for OpenWrt, no, there really is no need to enable it,
before you have the rest of the x32 subtarget ready to be merged (and
I can't imagine any reason to enable it for the current OpenWrt x86_64
subtarget at all).

For OpenWrt, I could imagine only two approaches to this:
- make the x86_64 subtarget x32-only
- add a new x32-only subtarget and leave x86_64 as it is
Neither would really be 'sensible', but the only workable approach
would imho be the later, even if the intention was (I hope not to
witness that) to kill off the old x86_64 target.

Yes, as laid out before, there isn't any strict need for OpenWrt to
remain binary compatible between releases, however this would seriously
mess up base vs. ports for (at least) a few weeks (broken packages,
package relations, implicit expectations by build recipes throughout
the tree) - and anyone doing this in one step (archive wide), without
prior testing, would be insane, so the later step of adding a new
subtarget in parallel to the old one would be necessary for multiple
months (if not at least a stable release cycle shipping both) either
way.

So I really don't see any reason to enable x32 for the x86_64 subtarget,
there's nothing to be gained, just major disadvantages.
While I'm not a proponent for a pure-x32 subtarget either (at all), this
would be the only workable approach to introduce it.

Regards
	Stefan Lippers-Hollmann