OpenWrt Next Generation Ideas

[Arınç ÜNAL]

On 31.03.2023 18:36, Daniel Golle wrote:
> On Fri, Mar 31, 2023 at 05:35:22PM +0300, Arınç ÜNAL wrote:
>> On 31.03.2023 16:47, Daniel Golle wrote:
>>> On Fri, Mar 31, 2023 at 03:52:47PM +0300, Arınç ÜNAL wrote:
>>>> On 31.03.2023 14:33, Daniel Golle wrote:
>>>>> Hi!
>>>>>
>>>>> On Fri, Mar 31, 2023 at 12:44:12PM +0300, Arınç ÜNAL wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> These are the ideas I've been thinking about for the future of OpenWrt for a
>>>>>> while. It looks complete enough to share it with all of you.
>>>>>>
>>>>>> I'm willing to put a great deal of effort to get as much out-of-tree patches
>>>>>> on mainline Linux as possible.
>>>>>>
>>>>>> You can make a comment on Notion or discuss it here, I'm wondering if the
>>>>>> ideas are feasible and how well it would benefit the people maintaining
>>>>>> OpenWrt.
>>>>>>
>>>>>> https://arinc9.notion.site/OpenWrt-next-gen-ideas-6db745e7584b4823950291c96f2326bb
>>>>>
>>>>> I will comment here, I don't have an account on Notion and it seems
>>>>> to be required to be able to comment there.
>>>>>
>>>>>> defconfig for each device instead of config for each (sub)target.
>>>>>
>>>>> Given that we support thousands of devices this will not only increase
>>>>> the time needed to build a release or snapshot by several magnitudes,
>>>>> but also make debugging **much** harder. As of now, all devices of a
>>>>> subtarget are using the same kernel, hence e.g. symbol offsets in a
>>>>> kernel stack dump match for all of them. To reproduce or investigate
>>>>> a problem it's hence enough to have similar hardware, not necessarily
>>>>> the exact same board. As we are already lacking testers and maintainers
>>>>> for the relatively small amount of targets/subtargets, have a build for
>>>>> each board would make things much worse...
>>>>>
>>>>> Per-device builds would also be an invitation to downstream users to
>>>>> introduce device-specific (kernel-)hacks. If you want that, better go
>>>>> for OpenEmbedded.
>>>>>
>>>>> We can modularize things more or even have more sub-targets if it's
>>>>> really needed to save space.
>>>>>
>>>>> The disadvantages outweight the advantages imho when it comes to having
>>>>> a complete kernel build for each device.
>>>>
>>>> Hmm, what about we enable the bare minimum of kernel options for a target,
>>>> which is already how it is, then select the rest as kernel modules (like on
>>>> the makefile of a target for each device) on the defconfig for each device?
>>>> So, in the end, it wouldn't be any different than selecting a kernel module
>>>> package from the OpenWrt SDK which, I believe, does not change the symbol
>>>> offsets in the kernel stack.
>>>>
>>>> My reason for pushing for the use defconfigs is that anyone can build the
>>>> Linux kernel for their device, without needing OpenWrt. So the work for
>>>> adding support for a device would benefit far more people.
>>>
>>> This is pretty much what we are currently doing.
>>> The exception are network drivers to allow for failsafe mode to work
>>> and provide SSH access **before** any modules are loaded.
>>
>> Got it, network drivers should also be built into the kernel on the
>> defconfigs then.
>>
>>>
>>>>
>>>>>
>>>>>> Some Modest Virtualization Observations
>>>>>
>>>>> How is this related? Virtualization (with OpenWrt being the guest)
>>>>> matters on x86 which is usually not that space-constraint.
>>>>> And maybe armvirt. If space is a problem for older x86 boards, let's
>>>>> disable guest support in x86/legacy.
>>>>
>>>> It was a broad example without any explanation. What caught my attention
>>>> there is the configuration of the kernel that causes problems, if I
>>>> understand correctly.
>>>>
>>>>>
>>>>>> Contribute defconfigs and all the devicetrees on OpenWrt to Linux.
>>>>>
>>>>> For devicetrees this would of course be desirable, but also implies a
>>>>> lot of work and discussions. If you are up to get it started (ie. setup
>>>>> a tree to collect cleaned-up and ready to submit dts), I think it would
>>>>> be worth the effort, at least for more recent targets/SoCs.
>>>>
>>>> I've been meaning to do this for the mt7621 SoC devices for months. The main
>>>> roadblock is that some drivers are out-of-tree, like the NAND flash so it
>>>> makes no sense to have them defined on the devicetree. Getting the
>>>> out-of-tree patches on mainline Linux is another step so it'll happen
>>>> eventually.
>>>
>>> Hm, I thought that Weijie had sent the mt7621-nand driver also upstream,
>>> but I haven't been following the process...
>>
>> I haven't checked for a while too, maybe it did get in. Then there's the mac
>> incrementing on the devicetree which doesn't exist on mainline Linux.
> 
> I think Rafal has been taking care of that lately, but I might be
> wrong.
> 
> label-mac also is a downstream use of DTS specific to OpenWrt which
> didn't yet make it upstream.
> 
>>
>>>
>>>>
>>>> I'll get this started with my Linux fork on GitHub.
>>>
>>> Very nice, I will join in there.
>>
>> I'm leaving the link here for future reference.
>>
>> https://github.com/arinc9/linux
> 
> Thank you!
> 
>>
>>>
>>>>
>>>>>
>>>>> Regarding defconfigs I don't think we need an individual defconfig for
>>>>> each board. The problem here is also that OpenWrt currently has a layered
>>>>> approach (generic->target->subtarget) approach while Linux itself has
>>>>> a flat approach, and using that would result in a lot of duplication,
>>>>> which would in turn make keeping all those defconfigs up-to-date quite
>>>>> a lot of work...
>>>>
>>>> Not really, once you make a defconfig for a device, the only case for it to
>>>> change in the future is if a kernel option was renamed, which is very rare.
>>>> Anyway, even in that case, there are known ways to update them by bulk.
>>>>
>>>> https://lore.kernel.org/kernel-janitors/20220929090645.1389-1-lukas.bulwahn@gmail.com/
>>>
>>> I agree for "established" platforms for which drivers are already
>>> upstream and not many changes are still expected to happen.
>>> Yet, a defconfig **for each board** seems like overkill to me, esp.
>>> because 90%+ will not be able to boot the resulting image generated
>>> by a vanilla kernel build in any meaningful way.
>>> Having defconfig for each sub-target would be agreeable, though I must
>>> say that I do like our layerd approach to defconfig.
>>> Maybe we should pick single representative **and resourceful** boards
>>> and try upstreaming defconfig for those (in case it didn't already happen).
>>> However, not using the generic defconfig anyway is only relevant for
>>> rather resource-constraint boards (as otherwise you'd just use, let's say
>>> multi_v7_defconfig or the like). Are you sure upstream is interested in
>>> being flooded with hundreds of defconfigs, most of them for very similar
>>> devices?
>>
>> I'm not sure, there's Thomas who maintains MIPS. There's a lot of ARM
>> maintainers. I'll have to talk it through with them.
>>
>> The main issue I see that forces me to make defconfig per device is because
>> every device will slightly need different kernel modules.
>>
>> It's basically taking the kernel modules from here and put it to the
>> defconfig for the device.
>>
>> https://github.com/openwrt/openwrt/blob/master/target/linux/ramips/image/mt7621.mk#L137
> 
> Taking MT7621 as the example I believe that the main differences are
> mostly the selection of wifi drivers, USB support and occasionally AHCI.
> 
> You really think it makes sense to provide 200+ defconfigs just to
> cover the (limited) number of combinations?
> Wouldn't a handful or even a single defconfig be sufficient?

Yeah, I agree. A defconfig with the most basic stuff and network drivers 
as built-in, the rest as kernel modules per (sub)target should be 
enough. I'll get started with the mt7621 subtarget.

The "the most basic stuff and network drivers as built-in" part is 
already covered with current configs on OpenWrt, if I understand 
correctly. For the kernel modules, I guess I'll just collect them from 
the OpenWrt makefile.

> 
>>
>>>
>>>>
>>>>>
>>>>>> Either submit all out-of-tree patches on OpenWrt to Linux or get rid
>>>>>> of them and find a better solution for what the unacceptable patch
>>>>>> does.
>>>>>
>>>>> This would of course be great, but especially for legacy devices it may
>>>>> not be possible in many cases. Think of all the devices stuck on
>>>>> swconfig, just to name one example... Think of all the completely
>>>>> broken vendor bootloaders which require hacks (mangeling kernel cmdline
>>>>> and such) and cannot easily be replaced...
>>>>
>>>> Those can stay until eventually the support for them will be dropped on
>>>> newer OpenWrt versions. I believe there are a lot of out-of-tree patches
>>>> that are not for old devices and can be on mainline Linux.
>>>
>>> In terms of hardware support you are probably right.
>>> When it comes to vendor bootloaders which require mangeling kernel
>>> bootargs the kernel folks have always argued that users should just
>>> change the bootloader environment -- however, this requires a serial
>>> console attached and makes reverting to stock firmware harder.
>>
>> Makes sense. It makes sense to maintain this part on OpenWrt.
>>
>>>
>>> Another example is selection of the rootfs. Kernel folks argue that
>>> we should use an initramfs for that, however, we try to avoid the
>>> overhead of using an initramfs with it's own userland just for that.
>>>
>>> To resolve this situation at least for future boards (ie. with
>>> post-2020 versions of U-Boot) I've put quite some effort into adding
>>> native support for mapping the filesystem sub-image(s) of a uImage.FIT
>>> image as a block device in Linux:
>>>
>>> https://github.com/dangowrt/linux/commit/8a70f52fddd518d1d7093a12874ca3a5de86fab1
>>>
>>> I've already had quite a lot of debate about that, currently I'm waiting
>>> for add/remove/size-change notifications for in-kernel block device users
>>> being added to the kernel, as that would be required for a really clean
>>> implementation.
>>
>> This is very nice. Please CC me when there's anything new. This was one of
>> the things I was planning to figure out. I think the current way, even with
>> a FIT image, is to put the location of the filesystem device on the bootargs
>> so the kernel can find it once the SPI/NAND driver probes the flash?
> 
> Yes, the current way is very platform-specific and storage-type specific.
> With recent devboards supporting several boot devices on the same board
> I felt the need to have one unified method which can work on all of them,
> hence I came up with the uImage.FIT partition parser. The partition parser,
> however, has been rejected upstream and I was told to instead implement
> this as a tiny block driver.
> At least for devices with more or less recent U-Boot this is imho a
> very good option.

This will be very useful for companies that use Buildroot, mainline 
U-Boot and Linux on their own embedded devices. This will be THE 
solution for where I work. I was already in the middle of writing 
somewhat of a guide for this whole process.

> 
>>
>>>
>>> Most other patches in target/linux/generic/pending-5.* should be sutiable
>>> for upstream inclusion. Patches in target/linux/generic/hack-5.* are more
>>> tragic cases...
>>>
>>> Apart from the technical debate the issue here is of course also of
>>> organizational nature. While not accepting new patches unless they are
>>> already accepted upstream is a matter of policy, upstreaming all the
>>> existing patches is a lot of work, and the question arises *who* is
>>> going to volunteer do that...
>>
>> This is what I'm planning to spend my time on for the next couple of years.
> 
> That's very good to hear! Let me know if you have any questions
> regarding existing patches, for most I should be able to provide you
> with some background information.

Thanks a lot. I'm planning to present this whole plan on Battlemesh v15. 
This conversation helped shape it in my mind a lot.

Arınç