OpenWrt Next Generation Ideas

[Daniel Golle]

On Fri, Mar 31, 2023 at 01:34:03PM -0700, Elliott Mitchell wrote:
> On Fri, Mar 31, 2023 at 02:47:23PM +0100, Daniel Golle wrote:
> > 
> > Another example is selection of the rootfs. Kernel folks argue that
> > we should use an initramfs for that, however, we try to avoid the
> > overhead of using an initramfs with it's own userland just for that.
> 
> There are two options here.  First, 10 mildly different images can be
> built for each of the 10 variants of hardware.  Second, 1 image can be
> built which loads appropriate drivers during boot.
> 
> Issue is OpenWRT aims for the first option for ARM/MIPS/RISC-V.  For x86
> though there are 2 candidate images to cover all configurations, which
> for the first approach should have 7-12 images.
> 
> x86 has the "generic" image and the "64" image.  For the first approach
> there should be a different build for:
> 
> 1> Running on raw hardware.
> 
> 2> Running on HyperV.
> 
> 3> Running on KVM.
> 
> 4> Running on QEMU.
> 
> 5> Running on Xen.
> 
> So should OpenWRT's x86 builds be distinct from ARM/MIPS/RISC-V in using
> an initramfs in order to combine these?

x86 is generally not as space constraint as other platforms and most
importantly uses a boot method (IBM PC bootsector or UEFI) which is
available on 99.9% of x86 hardware. The reason why we have individual
images for every ARM/MIPS/RISC-V device is simply because a generic
boot method is not (yet) implemented on those devices. We **have to**
provide per-devices images for OpenWrt to be able to boot using the
vendor-provided device firmware. Using an initramfs would **not** solve
that.

Sidenote: some ARM64 devices do support UEFI and there are other
vendor-specific "generic" methods such as ONIE -- however, that's a
very small minority of high-end devices and literally NO off-the-shelf
consumer hardware supports any of that as of today.

> Should OpenWRT have 12 distinct x86 images?

Why would that be needed? The whole point of x86 is to be compatible,
and usually has plenty of RAM and storage, so shipping all drivers
needed is also not a problem.

> 
> Both approaches are nominally viable, problem is *one* approach needs to
> be chosen for x86 on modern hardware.
> 
> 
> On Fri, Mar 31, 2023 at 03:52:47PM +0300, Arınç ÜNAL wrote:
> > On 31.03.2023 14:33, Daniel Golle wrote:
> > > 
> > > On Fri, Mar 31, 2023 at 12:44:12PM +0300, Arınç ÜNAL wrote:
> > >>
> > >> Some Modest Virtualization Observations
> > > 
> > > How is this related? Virtualization (with OpenWrt being the guest)
> > > matters on x86 which is usually not that space-constraint.
> > > And maybe armvirt. If space is a problem for older x86 boards, let's
> > > disable guest support in x86/legacy.
> > 
> > It was a broad example without any explanation. What caught my attention 
> > there is the configuration of the kernel that causes problems, if I 
> > understand correctly.
> 
> The real issue is everything I've looked at makes it seem x86 hasn't
> received the attention it needs.  Looks like it is purely a development
> tool, not really providing the functionality it could.

Just like any other platform the level of support depends on voluntary
contributions by developers and users.
If you see a job, it's yours :)

> 
> Notable kernel options:
> # CONFIG_XEN_BACKEND is not set
> Most Xen backend drivers aren't well suited for OpenWRT, but the
> ethernet backend is.

Please send a patch or open a pull-request adding the needed kernel
options to support Xen networking to target/linux/x64/64/config-5.15.

> 
> # CONFIG_CFG80211 is not set
> # CONFIG_LIB80211 is not set
> If 802.11 devices could be fully passed into a VM you could have a
> proper access point in a VM.

Wireless drivers are supported -- just like on any other platform --
via backported drivers. See package/kernel/mac80211. We generally do
not use cfg80211 and lib80211 at the version level of the kernel, but
use more recent backported drivers instead. So this is intentional
and will provide you with **more recent drivers**.

> 
> Then we have the architectures chosen.  "64" => x86_64-linux-musl,
> "generic" => i386-linux-musl/Pentium4
> 
> Could be CONFIG_MPENTIUM4 merely uses options ideal for a P4 and could be
> reasonable for most of AMD's 64-bit processors, but otherwise that is
> about as non-generic as possible.

The point here is not optimization, but finding a good compromise to
not having to compile **all** packages for **all** CPU architectures.
The same is btw also true for MIPS and ARM targets. We pick a good
compromise in terms of optimization.
Example:
 * All MIPS32 targets use either mips4k or mips24k optimization --
   in reality there are plenty of mips34k and mips1004k SoCs around,
   all of them happily run either of the two variants, but one could
   of course produce an optimized build for e.g. mips1004kc or even
   mips24kec...
 * Some applies for ARM64 where we optmize only for Cortex-A53 and
   Cortex-A72 -- the variety of actual CPU cores is much bigger.

In terms of x86/generic being optimized for pentium4 goes along the
lines that most commercial software did the same thing. That also
doesn't mean that AMD's CPU extensions are not supported, for cases
where this is actually relevant (e.g. RAID code in kernel or FFmpeg
in userland) the CPU type is probed at run-time and optimized code
for the actual CPU is being used.

> 
> Having done a bit more looking recently, could be "generic" should point
> at x86_64-linux-muslx32, not x86_64-linux-musl.  (x32 is running ILP32 on
> an amd64 processor)

x86/generic ships a 32-bit kernel, and that's on purpose and in order
to support actual 32-bit CPUs.
x86_64-linux-muslx32 would be a 64-bit running a 32-bit userland.

> 
> 
> All of the suggested VM setups are massively larger than any other
> OpenWRT supported device.  The smallest was the Xen setup which merely
> suggested 256MB of memory.

I do run OpenWrt x86_64 VMs with as little as 64 MiB of RAM for tests
and it works fine (using QEMU KVM, but it should be similar for other
virtualization platforms). For production use when actually performing
routing and maybe also a DNS cache, a bit more RAM is needed, 128 MiB
should be ok for that. If you have actual Wi-Fi hardware as well, then
256 MiB would be needed (but that's hardly ever the case inside a VM).

> 
> 
> Should also be noted, some higher-end ARM boards have enough hardware
> to potentially be worth having an OpenWRT AP in a VM.

Sure, hardware virtualization is available on all recent ARM64 CPUs.
The Linux kernel recently dropped support for virtualization on 32-bit
ARM, that's sad, but there nothing we can do about it.

Virtualizing an actual Wi-Fi access point is a bit more tricky, as
you would either need something like virtio-cfg80211 (ie. a way to
have the guest access the wifi driver of the host) or a CPU with an
IOMMU, so we can pass-through the PCIe device to the guest (router
SoCs don't usually support that).

> 
> The bottom line is, except for the Geode support, all of the x86
> configurations looked like purely meant as development tools.  Whereas
> with modern hardware it is entirely reasonable to have a proper OpenWRT
> AP in a VM.

On x86/64 machines which come with an IOMMU which allows you to
pass-through PCIe Wi-Fi hardware you can perfectly do that and there is
nothing missing. I've tried that using libvirt (QEMU/KVM) and it worked
fine with ath10k PCIe hardware.

However, should you miss anything, feel free to suggest changes.