Lockdep-RCU splat in ARM CPU hotplug

[Stefan Wiehler]

Hi,

With CONFIG_PROVE_RCU_LIST=y and by executing

  $ echo 0 > /sys/devices/system/cpu/cpu1/online

one can trigger the following Lockdep-RCU splat on ARM (reproducible on an Orange Pi PC in QEMU):

  =============================
  WARNING: suspicious RCU usage
  6.8.0-rc7-00001-g0db1d0ed8958 #10 Not tainted
  -----------------------------
  kernel/locking/lockdep.c:3762 RCU-list traversed in non-reader section!!
 
  other info that might help us debug this:
 
 
  RCU used illegally from offline CPU!
  rcu_scheduler_active = 2, debug_locks = 1
  no locks held by swapper/1/0.
 
  stack backtrace:
  CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.8.0-rc7-00001-g0db1d0ed8958 #10
  Hardware name: Allwinner sun8i Family
   unwind_backtrace from show_stack+0x10/0x14
   show_stack from dump_stack_lvl+0x60/0x90
   dump_stack_lvl from lockdep_rcu_suspicious+0x150/0x1a0
   lockdep_rcu_suspicious from __lock_acquire+0x11fc/0x29f8
   __lock_acquire from lock_acquire+0x10c/0x348
   lock_acquire from _raw_spin_lock_irqsave+0x50/0x6c
   _raw_spin_lock_irqsave from check_and_switch_context+0x7c/0x4a8
   check_and_switch_context from arch_cpu_idle_dead+0x10/0x7c
   arch_cpu_idle_dead from do_idle+0xbc/0x138
   do_idle from cpu_startup_entry+0x28/0x2c
   cpu_startup_entry from secondary_start_kernel+0x11c/0x124
   secondary_start_kernel from 0x401018a0

Originally the splat was found on an AXM5516 with v5.15, so the issue presumably exists for quite some time already on all ARM boards.

Lockdep-RCU is triggered by this call of raw_spin_lock_irqsave() in check_and_switch_context() while the CPU is already marked offline: https://elixir.bootlin.com/linux/v6.8-rc7/source/arch/arm/mm/context.c#L257

On ARM64, we have cpu_die_early() calling rcutree_report_cpu_dead() which presumably prevents such a splat from occurring: https://elixir.bootlin.com/linux/v6.8-rc7/source/arch/arm64/kernel/smp.c#L412

Simply calling rcutree_report_cpu_dead() in arch_cpu_idle_dead() on ARM seems to have no effect though. As my understanding of the CPU hotplugging subsystem on ARM is a bit limited, I would appreciate some help here.

Thanks and regards,

Stefan