arm64 torture test hotplug failures (offlining causes -EBUSY)

[Joel Fernandes]

Hi Zhouyi,

On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
>
[..]
> On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> >
> > Hello,
> > I am seeing -EBUSY returned a lot during torture_onoff() when running
> > rcutorture on arm64. This causes hotplug failure 30% of the time. I am
> > also seeing this in 6.1-rc kernels. I believe see this only for CPU0.
> >
> > This causes warnings in torture tests:
> > [  217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > [  221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> >
> > Full kernel log here:
> > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log
> >
> > Any ideas on why this is happening and only for CPU 0 (presumably the
> > boot CPU)? I'd personally need these warnings to go away for my tests
> > as this causes rcutorture's tests to not cleanly pass for me. It
> > appears remove_cpu() -> device_offline() is what returns the error.
> >
> I guess this probably because CPU 0 is the tick_do_timer_cpu in
> nohz_full mode, which prevent that cpu from
> going offline [1]. We have discussed this topic, but there is no
> agreement on how to solve it yet.

But I am seeing the issue in TRACE02 config which is:
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set

So that is not NO_HZ_FULL:
http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/
However, I can't seem to find the full kernel logs for that.

Also, other than the TRACE02 fail, I only see the issue with configs
with CONFIG_NO_HZ_FULL=y

Can you try TRACE02 specifically, and see if you can reproduce the
same issue on your setup? Meanwhile, I'll try to trace what is
returning the -EBUSY.

Thanks!

 - Joel

>
> [1] https://lore.kernel.org/lkml/20221127175317.GF4001@paulmck-ThinkPad-P17-Gen-1/T/
>
> Thanks
> Zhouyi
> > You can browse through all the torture test artifacts here which
> > contains everything (build logs, vmlinux, etc).
> > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/
> >
> > Thanks for your help!
> >
> >  - Joel