OpenWrt Next Generation Ideas

[Elliott Mitchell]

On Fri, Mar 31, 2023 at 10:26:37PM +0100, Daniel Golle wrote:
> On Fri, Mar 31, 2023 at 01:34:03PM -0700, Elliott Mitchell wrote:
> > On Fri, Mar 31, 2023 at 02:47:23PM +0100, Daniel Golle wrote:
> > > 
> > > Another example is selection of the rootfs. Kernel folks argue that
> > > we should use an initramfs for that, however, we try to avoid the
> > > overhead of using an initramfs with it's own userland just for that.
> > 
> > There are two options here.  First, 10 mildly different images can be
> > built for each of the 10 variants of hardware.  Second, 1 image can be
> > built which loads appropriate drivers during boot.
> > 
> > Issue is OpenWRT aims for the first option for ARM/MIPS/RISC-V.  For x86
> > though there are 2 candidate images to cover all configurations, which
> > for the first approach should have 7-12 images.
> > 
> > x86 has the "generic" image and the "64" image.  For the first approach
> > there should be a different build for:
> > 
> > 1> Running on raw hardware.
> > 
> > 2> Running on HyperV.
> > 
> > 3> Running on KVM.
> > 
> > 4> Running on QEMU.
> > 
> > 5> Running on Xen.
> > 
> > So should OpenWRT's x86 builds be distinct from ARM/MIPS/RISC-V in using
> > an initramfs in order to combine these?
> 
> x86 is generally not as space constraint as other platforms and most
> importantly uses a boot method (IBM PC bootsector or UEFI) which is
> available on 99.9% of x86 hardware. The reason why we have individual
> images for every ARM/MIPS/RISC-V device is simply because a generic
> boot method is not (yet) implemented on those devices. We **have to**
> provide per-devices images for OpenWrt to be able to boot using the
> vendor-provided device firmware. Using an initramfs would **not** solve
> that.

A key reason to choose OpenWRT on x86 is small image size.  If memory
footprint wasn't a concern most Linux distributions handle this situation
fine.  Small VMs can burn memory just as effectively as large VMs.
Simply the difference between being killed by a tiger versus army ants.

> Sidenote: some ARM64 devices do support UEFI and there are other
> vendor-specific "generic" methods such as ONIE -- however, that's a
> very small minority of high-end devices and literally NO off-the-shelf
> consumer hardware supports any of that as of today.

I had been given the impression UEFI/Tianocore was attempting to take
over booting for higher-end ARM device.  I'm not in that market so I
don't know, but I find this believable given how awful device-trees are.


> > Should OpenWRT have 12 distinct x86 images?
> 
> Why would that be needed? The whole point of x86 is to be compatible,
> and usually has plenty of RAM and storage, so shipping all drivers
> needed is also not a problem.

Lose track of memory consumption and you'll exhaust 2TB on a high-end
server very quickly.  The information on OpenWRT in VMs seemed to suggest
256MB as a minimum, which is already larger than several in-use VMs I
have (and those are using ordinary x86 Linux distributions).


> > On Fri, Mar 31, 2023 at 03:52:47PM +0300, Arınç ÜNAL wrote:
> > > On 31.03.2023 14:33, Daniel Golle wrote:
> > > > 
> > > > On Fri, Mar 31, 2023 at 12:44:12PM +0300, Arınç ÜNAL wrote:
> > > >>
> > > >> Some Modest Virtualization Observations
> > > > 
> > > > How is this related? Virtualization (with OpenWrt being the guest)
> > > > matters on x86 which is usually not that space-constraint.
> > > > And maybe armvirt. If space is a problem for older x86 boards, let's
> > > > disable guest support in x86/legacy.
> > > 
> > > It was a broad example without any explanation. What caught my attention 
> > > there is the configuration of the kernel that causes problems, if I 
> > > understand correctly.
> > 
> > The real issue is everything I've looked at makes it seem x86 hasn't
> > received the attention it needs.  Looks like it is purely a development
> > tool, not really providing the functionality it could.
> 
> Just like any other platform the level of support depends on voluntary
> contributions by developers and users.
> If you see a job, it's yours :)
> 
> > 
> > Notable kernel options:
> > # CONFIG_XEN_BACKEND is not set
> > Most Xen backend drivers aren't well suited for OpenWRT, but the
> > ethernet backend is.
> 
> Please send a patch or open a pull-request adding the needed kernel
> options to support Xen networking to target/linux/x64/64/config-5.15.

That is likely to happen, though right now I'm still gathering
information for my goal.

> > # CONFIG_CFG80211 is not set
> > # CONFIG_LIB80211 is not set
> > If 802.11 devices could be fully passed into a VM you could have a
> > proper access point in a VM.
> 
> Wireless drivers are supported -- just like on any other platform --
> via backported drivers. See package/kernel/mac80211. We generally do
> not use cfg80211 and lib80211 at the version level of the kernel, but
> use more recent backported drivers instead. So this is intentional
> and will provide you with **more recent drivers**.

Ah, okay.  I hadn't realized this was the case.

> > Then we have the architectures chosen.  "64" => x86_64-linux-musl,
> > "generic" => i386-linux-musl/Pentium4
> > 
> > Could be CONFIG_MPENTIUM4 merely uses options ideal for a P4 and could be
> > reasonable for most of AMD's 64-bit processors, but otherwise that is
> > about as non-generic as possible.
> 
> The point here is not optimization, but finding a good compromise to
> not having to compile **all** packages for **all** CPU architectures.
> The same is btw also true for MIPS and ARM targets. We pick a good
> compromise in terms of optimization.
> Example:
>  * All MIPS32 targets use either mips4k or mips24k optimization --
>    in reality there are plenty of mips34k and mips1004k SoCs around,
>    all of them happily run either of the two variants, but one could
>    of course produce an optimized build for e.g. mips1004kc or even
>    mips24kec...
>  * Some applies for ARM64 where we optmize only for Cortex-A53 and
>    Cortex-A72 -- the variety of actual CPU cores is much bigger.
> 
> In terms of x86/generic being optimized for pentium4 goes along the
> lines that most commercial software did the same thing. That also
> doesn't mean that AMD's CPU extensions are not supported, for cases
> where this is actually relevant (e.g. RAID code in kernel or FFmpeg
> in userland) the CPU type is probed at run-time and optimized code
> for the actual CPU is being used.

This is the first time I've seen an open-source project of significance
having labeled "generic" be optimized for P4.  Everyone targets i686
(since that gets you SMP, and was widely copied) and amd64 (since that
gets you >4GB).

Quite a number of projects could build optimized for P4, but nothing ever
built that way by default.  (I could believe some commercial software
opting for P4 though)

> > Having done a bit more looking recently, could be "generic" should point
> > at x86_64-linux-muslx32, not x86_64-linux-musl.  (x32 is running ILP32 on
> > an amd64 processor)
> 
> x86/generic ships a 32-bit kernel, and that's on purpose and in order
> to support actual 32-bit CPUs.

find target/linux/x86 -name config-\* -print0 | xargs -0 grep -e^CONFIG_MPENTIUM4 -e^CONFIG_M586MMX -e^CONFIG_AGP
target/linux/x86/legacy/config-5.15:CONFIG_AGP=y
target/linux/x86/legacy/config-5.15:CONFIG_AGP_INTEL=y
target/linux/x86/legacy/config-5.15:CONFIG_M586MMX=y
target/linux/x86/legacy/config-5.10:CONFIG_AGP=y
target/linux/x86/legacy/config-5.10:CONFIG_AGP_INTEL=y
target/linux/x86/legacy/config-5.10:CONFIG_M586MMX=y
target/linux/x86/generic/config-5.15:CONFIG_AGP=y
target/linux/x86/generic/config-5.15:CONFIG_AGP_INTEL=y
target/linux/x86/generic/config-5.15:CONFIG_MPENTIUM4=y
target/linux/x86/generic/config-5.10:CONFIG_AGP=y
target/linux/x86/generic/config-5.10:CONFIG_AGP_INTEL=y
target/linux/x86/generic/config-5.10:CONFIG_MPENTIUM4=y
target/linux/x86/64/config-5.15:CONFIG_AGP=y
target/linux/x86/64/config-5.15:CONFIG_AGP_INTEL=y
target/linux/x86/64/config-5.10:CONFIG_AGP=y
target/linux/x86/64/config-5.10:CONFIG_AGP_INTEL=y

Could be things are leaking through the cracks (Intel encouraging
defaulting their stuff on), but that leaves a *distinct* impression
someone likes Intel processors.  True, this isn't anything like the
degree `icc` goes to, but this still seems oddly favorable towards
Intel...

If my recollection is correct, "generic" should run okay on most
post-2010 AMD processors, but still.


> x86_64-linux-muslx32 would be a 64-bit running a 32-bit userland.

Please read carefully and do your research before replying.  32-bit
userland on 64-bit kernel would be i386-linux-musl w/amd64 kernel.

x86_64-linux-muslx32 is amd64 userland, but built ILP32.  The benefits
are you get 16 GPRs, rIP-relative addressing and 64-bit math.  The
exchange is 32-bit pointers means 4GB maximum memory.

According to sources substantial performance increase and smaller memory
use (these two are interrelated).  If you're okay with limiting memory to
4GB (which is overkill for many programs and most OpenWRT targets are
smaller).

I'm unsure whether the X32 ABI is on its way in or out.  Though
apparently the suggestion to end it in 2018 was severely shot down.
Given OpenWRT's targets, this seems near ideal.  Issue is musl (likely
other software too) considers its support experimental.


> > All of the suggested VM setups are massively larger than any other
> > OpenWRT supported device.  The smallest was the Xen setup which merely
> > suggested 256MB of memory.
> 
> I do run OpenWrt x86_64 VMs with as little as 64 MiB of RAM for tests
> and it works fine (using QEMU KVM, but it should be similar for other
> virtualization platforms). For production use when actually performing
> routing and maybe also a DNS cache, a bit more RAM is needed, 128 MiB
> should be ok for that. If you have actual Wi-Fi hardware as well, then
> 256 MiB would be needed (but that's hardly ever the case inside a VM).

Rather a lot of motherboards are shipping with 802.11 support.  Only
issue being whether the chipsets used work for APs.

That is double what most APs have, so either your setups are large or
else something is wrong with OpenWRT's x86 support.

> > Should also be noted, some higher-end ARM boards have enough hardware
> > to potentially be worth having an OpenWRT AP in a VM.
> 
> Sure, hardware virtualization is available on all recent ARM64 CPUs.
> The Linux kernel recently dropped support for virtualization on 32-bit
> ARM, that's sad, but there nothing we can do about it.
> 
> Virtualizing an actual Wi-Fi access point is a bit more tricky, as
> you would either need something like virtio-cfg80211 (ie. a way to
> have the guest access the wifi driver of the host) or a CPU with an
> IOMMU, so we can pass-through the PCIe device to the guest (router
> SoCs don't usually support that).

Router SoCs might not, but many ARM-server boards will.  Simply turns to
an issue of whether the board has built-in good 802.11, or an add-on card
can be used.  Just an issue of whether someone with such hardware wishes
to use it for that purpose.

> > The bottom line is, except for the Geode support, all of the x86
> > configurations looked like purely meant as development tools.  Whereas
> > with modern hardware it is entirely reasonable to have a proper OpenWRT
> > AP in a VM.
> 
> On x86/64 machines which come with an IOMMU which allows you to
> pass-through PCIe Wi-Fi hardware you can perfectly do that and there is
> nothing missing. I've tried that using libvirt (QEMU/KVM) and it worked
> fine with ath10k PCIe hardware.

Mostly it is choices which look better for development and poor for an
AP in a VM.  ISA support?!  AGP support?!  I would complain about
CONFIG_FB, but that appears to have leaked into all the configurations.

Unfortunately the tendancy to remove older configurations and then copy
means one needs to use `git blame --find-copies-harder` to trace things.
That is rather distinctly slower.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg at m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445