arm64 torture test hotplug failures (offlining causes -EBUSY)

[Joel Fernandes]

On Tue, Jan 17, 2023 at 4:54 AM Paul E. McKenney <paulmck at kernel.org> wrote:
>
> On Mon, Jan 16, 2023 at 11:36:57PM -0500, Joel Fernandes wrote:
> > > On Jan 16, 2023, at 11:30 PM, Paul E. McKenney <paulmck at kernel.org> wrote:
> > > On Tue, Jan 17, 2023 at 12:15:07AM +0000, Joel Fernandes wrote:
> > >>> On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote:
> > >>> Hi Zhouyi,
> > >>>
> > >>> On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
> > >>>>
> > >>> [..]
> > >>>> On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > >>>>>
> > >>>>> Hello,
> > >>>>> I am seeing -EBUSY returned a lot during torture_onoff() when running
> > >>>>> rcutorture on arm64. This causes hotplug failure 30% of the time. I am
> > >>>>> also seeing this in 6.1-rc kernels. I believe see this only for CPU0.
> > >>>>>
> > >>>>> This causes warnings in torture tests:
> > >>>>> [  217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > >>>>> [  221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > >>>>>
> > >>>>> Full kernel log here:
> > >>>>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log
> > >>>>>
> > >>>>> Any ideas on why this is happening and only for CPU 0 (presumably the
> > >>>>> boot CPU)? I'd personally need these warnings to go away for my tests
> > >>>>> as this causes rcutorture's tests to not cleanly pass for me. It
> > >>>>> appears remove_cpu() -> device_offline() is what returns the error.
> > >>>>>
> > >>>> I guess this probably because CPU 0 is the tick_do_timer_cpu in
> > >>>> nohz_full mode, which prevent that cpu from
> > >>>> going offline [1]. We have discussed this topic, but there is no
> > >>>> agreement on how to solve it yet.
> > >>>
> > >>> But I am seeing the issue in TRACE02 config which is:
> > >>> CONFIG_NO_HZ_IDLE=y
> > >>> # CONFIG_NO_HZ_FULL is not set
> > >>>
> > >>> So that is not NO_HZ_FULL:
> > >>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/
> > >>> However, I can't seem to find the full kernel logs for that.
> > >>>
> > >>> Also, other than the TRACE02 fail, I only see the issue with configs
> > >>> with CONFIG_NO_HZ_FULL=y
> > >>>
> > >>> Can you try TRACE02 specifically, and see if you can reproduce the
> > >>> same issue on your setup? Meanwhile, I'll try to trace what is
> > >>> returning the -EBUSY.
> > >>
> > >> How about something simple like the following? (untested)
> > >>
> > >> ---8<-----------------------
> > >>
> > >> diff --git a/kernel/torture.c b/kernel/torture.c
> > >> index bc8fb361efc0..cd64110694c0 100644
> > >> --- a/kernel/torture.c
> > >> +++ b/kernel/torture.c
> > >> @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes,
> > >>            // PCI probe frequently disables hotplug during boot.
> > >>            (*n_offl_attempts)--;
> > >>            s = " (-EBUSY forgiven during boot)";
> > >> +        } else if (tick_nohz_full_running && ret == -EBUSY) {
> > >> +            (*n_offl_attempts)--;
> > >> +            s = " (-EBUSY forgiven if nohz_full is running)";
> > >
> > > But this should be forgiven for the timekeeping CPU, not everyone,
> > > correct?
> > >
> > > Yes, I know that CPU-hotplug operations can fail, but in my testing
> > > they almost never do.  This means that a new failure might well be a
> > > real bug somewhere that needs attention.
> >
> > Sure. We may need to expose some API to reveal that.
> >
> > It appeared though that Thomas in the other thread related to patch
> > from Zhouyi, was suggesting that rcutorture tolerate hotplug failure
> > though, because they are not abnormal, right?
>
> Based on my rcutorture testing experience on x86, they are not at all
> normal.  The only time I have seen rcutorture CPU-hotplug failures has
> been due to some bug that needed fixing.

I see, ok. I need to debug what is returning -EBUSY for !NO_HZ_FULL on
arm64, I will report back once I do.

Meanwhile, Marc I am wondering if you are able to reproduce the issue
on your side on TRACE02 config, like I am?

Here is the TRACE02 config fragment:
http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/ConfigFragment/*view*/

Here are instructions on how to run it (torture test parameters etc)
if you are loading the module yourself:
http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/bare-metal/*view*/

> Is there a plan to make CPU hotplug failures more frequent?

I am not aware of such a plan but I was going by "There are quite some
reasons why a CPU-hotplug or a hot-unplug operation can fail, which is
not a fatal problem, really." in [1].

What about an rcutorture to skip hotplug for a certain cpu id,
rcutorture.skip_hotplug_cpus="0". Can be a last resort. But we/I
should debug this issue more before getting to that.

Thanks,

- Joel
[1] https://lkml.org/lkml/2022/11/27/182