[PATCH v2] rcu: Use an intermediate irq_work to start process_srcu()

Tue Jul 7 23:58:08 PDT 2026

Kontakt Thorsten Leemhuis (<regressions at leemhuis.info>) kirjutas
kuupäeval E, 6. juuli 2026 kell 14:33:
>
> On 6/24/26 10:55, Emil Renner Berthing wrote:
> > Quoting Samuel Holland (2026-06-23 17:53:19)
> >> On 2026-06-23 4:34 AM, Emil Renner Berthing wrote:
> >>> Quoting Boqun Feng (2026-06-22 17:38:10)
> >>>> On Sat, Jun 20, 2026 at 05:25:25AM -0400, Emil Renner Berthing wrote:
> >>>>> Quoting Boqun Feng (2026-06-18 17:30:10)
> >>>>>> On Thu, Jun 18, 2026 at 08:11:36AM -0500, Emil Renner Berthing wrote:
> >>>>>>> Quoting Boqun Feng (2026-03-20 23:29:16)
> >>>>>>>> Since commit c27cea4416a3 ("rcu: Re-implement RCU Tasks Trace in terms
> >>>>>>>> of SRCU-fast") we switched to SRCU in BPF. However as BPF instrument can
> >>>>>>>> happen basically everywhere (including where a scheduler lock is held),
> >>>>>>>> call_srcu() now needs to avoid acquiring scheduler lock because
> >>>>>>>> otherwise it could cause deadlock [1]. Fix this by following what the
> >>>>>>>> previous RCU Tasks Trace did: using an irq_work to delay the queuing of
> >>>>>>>> the work to start process_srcu().
> >>>>>>>>
> >>>>>>>> [boqun: Apply Joel's feedback]
> >>>>>>>> [boqun: Apply Andrea's test feedback]
> >>>>>>>>
> >>>>>>>> Reported-by: Andrea Righi <arighi at nvidia.com>
> >>>>>>>> Closes: https://lore.kernel.org/all/abjzvz_tL_siV17s@gpd4/
> >>>>>>>> Fixes: commit c27cea4416a3 ("rcu: Re-implement RCU Tasks Trace in terms of SRCU-fast")
> >>>>>>>> Link: https://lore.kernel.org/rcu/3c4c5a29-24ea-492d-aeee-e0d9605b4183@nvidia.com/ [1]
> >>>>>>>> Suggested-by: Zqiang <qiang.zhang at linux.dev>
> >>>>>>>> Tested-by: Andrea Righi <arighi at nvidia.com>
> >>>>>>>> Signed-off-by: Boqun Feng <boqun at kernel.org>
> >>>>>>>
> >>>>>>> Thanks for the patch, which is now upstream as
> >>>>>>> 7c405fb3279b ("rcu: Use an intermediate irq_work to start process_srcu()")
> >>>>>>>
> >>>>>>> Unfortunately it seems to halt booting on the Allwinner D1 single-core RISC-V
> >>>>>>> SoC both on the 7.0 and 7.1 kernels:
> >>>>>>>
> >>>>>>
> >>>>>> By "single-core", is it only one CPU in that system?
> >>>>>>
> >>>>>>> https://esmil.dk/d1-7.0.txt
> >>>>>>> https://esmil.dk/d1-7.1.txt
> >>>>>>>
> >>>>>
> >>>>> Yes, the SoC only has one CPU core capable of running Linux at least.
> >>>>>
> >>>>>> From the callstack,
> >>>>>>
> >>>>>> [  243.002818] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> >>>>>> [  243.010681] task:kworker/0:0     state:D stack:0     pid:9     tgid:9     ppid:2      task_flags:0x4208060 flags:0x00000000
> >>>>>> [  243.021858] Workqueue: device_link_wq device_link_release_fn
> >>>>>> [  243.027587] Call Trace:
> >>>>>> [  243.030079] [<ffffffff80745b66>] __schedule+0x232/0x530
> >>>>>> [  243.035365] [<ffffffff80745e82>] schedule+0x1e/0x94
> >>>>>> [  243.040298] [<ffffffff8074aa7a>] schedule_timeout+0x6e/0xa8
> >>>>>> [  243.045929] [<ffffffff8004d6a0>] wakeup_preempt+0x8c/0x98
> >>>>>> [  243.051380] [<ffffffff80746aee>] wait_for_completion+0x3a/0xb0
> >>>>>> [  243.057267] [<ffffffff80087226>] __synchronize_srcu.part.0+0x4e/0x60
> >>>>>> [  243.063712] [<ffffffff80084414>] rcu_tasks_get_gp_data+0xc/0x10
> >>>>>> [  243.069699] [<ffffffff80490898>] device_link_release_fn+0x14/0x80
> >>>>>> [  243.075850] [<ffffffff8003dbd0>] process_one_work+0xf8/0x1c8
> >>>>>> [  243.081583] [<ffffffff8003e282>] worker_thread+0x11a/0x254
> >>>>>> [  243.087127] [<ffffffff8003e164>] rescuer_thread+0x3a0/0x3a4
> >>>>>> [  243.092755] [<ffffffff8003e164>] rescuer_thread+0x3a0/0x3a4
> >>>>>> [  243.098381] [<ffffffff8004499e>] kthread+0xbe/0xe4
> >>>>>> [  243.103231] [<ffffffff8001349e>] ret_from_fork_kernel+0x6/0xd0
> >>>>>> [  243.109127] [<ffffffff800517a8>] schedule_tail+0x8/0xac
> >>>>>> [  243.114410] [<ffffffff800448dc>] kthreads_online_cpu+0x0/0x4
> >>>>>> [  243.120121] [<ffffffff8074be0a>] ret_from_fork_kernel_asm+0x12/0x18
> >>>>>>
> >>>>>> I will guess for some reasons the irq_work was missed or the IPI never
> >>>>>> happened. Is it possible for you to enable trace points ipi_send_cpu and
> >>>>>> ipi_entry to get more information?
> >>>>>
> >>>>> I have to admit I've not used tracepoints before, but it seems like only
> >>>>> ipi_send_cpu and ipi_send_cpumask is available on RISC-V.
> >>>>> I've now added trace_event=ipi:* and tp_printk to the command line, and this
> >>>>
> >>>> Thank you! Yeah, this is what I meant ;-)
> >>>>
> >>>>> is what I get:
> >>>>> https://esmil.dk/d1-7.0-2.txt
> >>>>>
> >>>>
> >>>>
> >>>> [    0.071132] rcu:     Max phase no-delay instances is 1000.
> >>>> [    0.077837] ipi_send_cpu: cpu=0 callsite=irq_work_queue+0x22/0x4c callback=srcu_irq_work+0x0/0x58
> >>>>
> >>>> Is this the last line you can see from your (serial?) console, or we are
> >>>> missing a few more lines? Maybe it'll should the same hung task
> >>>> messages?
> >>>
> >>> Yes, sorry for not being clear about that. This is the exact same binary booted
> >>> without the two extra command line parametrs:
> >>> https://esmil.dk/d1-7.0-3.txt
> >>>
> >>> But with the two parameters added the board reboots before getting to the hung
> >>> task messages. The bootloader enables the watchdog, so I think what happens is
> >>> that the kernel halts earlier with the tracepoints enabled, and doesn't get to
> >>> load the watchdog driver to keep the machine from rebooting.
> >>>
> >>> The same happens with just trace_event=ipi:* added, just without the last
> >>> message above.
> >>>
> >>>>
> >>>> The ipi_send_cpu message means RCU was able to send an IPI, now if there
> >>>> is no ipi_handler between the send and the hung task, that would mean
> >>>> the IPI was missed.
> >>
> >> Assuming that your OpenSBI version is exactly v1.3 per the boot log, and not
> >> "v1.3 + patches", then you would indeed not be receiving IPIs. You need at least
> >> this commit: https://github.com/riscv-software-src/opensbi/commit/446fa65eb579
> >> (included in v1.4) or alternatively to add the CLINT to the M-mode devicetree.
> >
> > So then we're in a situation similar to [1], where upgrading the kernel from
> > 6.19 to 7.0 requires you to update the firmware first.
>
> Which normally should never happen to ensure users feel save to update
> the kernel. But I wonder if that is relevant, as Indrek Kruusa reported
> problems with that board, too -- and there is a fix that fixes it:
> https://lore.kernel.org/linux-riscv/20260624220434.4183732-1-felixonmars@archlinux.org/
>
> Does that maybe fix your problem, too (even if it was bisected to a
> different commit, it might be related somehow, that's why I'm asking)?
>
> Ciao, Thorsten
>

I tend to think we have two different situations here. I am using
newer OpenSBI (v1.8) and an experimental u-boot based on newer
v2026.07-rc1 [1].
So, my firmware is different, also the context Emil is pointing to is different.

Best regards,
Indrek

[1] https://github.com/ikruusa/u-boot/tree/v2026-07-rc1-thead-base-b1

> > To me and from the thread
> > that seems like a really obvious regression. So the question is, how can we make
> > the current kernel work like < 7.0 kernels worked?
> >
> > [1]: https://lore.kernel.org/all/87jzgjnh9z.ffs@tglx/#t
>
>