arm64 torture test hotplug failures (offlining causes -EBUSY)

[Zhouyi Zhou]

On Tue, Jan 17, 2023 at 8:15 AM Joel Fernandes <joel at joelfernandes.org> wrote:
>
> On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote:
> > Hi Zhouyi,
> >
> > On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
> > >
> > [..]
> > > On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > > >
> > > > Hello,
> > > > I am seeing -EBUSY returned a lot during torture_onoff() when running
> > > > rcutorture on arm64. This causes hotplug failure 30% of the time. I am
> > > > also seeing this in 6.1-rc kernels. I believe see this only for CPU0.
> > > >
> > > > This causes warnings in torture tests:
> > > > [  217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > > > [  221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > > >
> > > > Full kernel log here:
> > > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log
> > > >
> > > > Any ideas on why this is happening and only for CPU 0 (presumably the
> > > > boot CPU)? I'd personally need these warnings to go away for my tests
> > > > as this causes rcutorture's tests to not cleanly pass for me. It
> > > > appears remove_cpu() -> device_offline() is what returns the error.
> > > >
> > > I guess this probably because CPU 0 is the tick_do_timer_cpu in
> > > nohz_full mode, which prevent that cpu from
> > > going offline [1]. We have discussed this topic, but there is no
> > > agreement on how to solve it yet.
> >
> > But I am seeing the issue in TRACE02 config which is:
> > CONFIG_NO_HZ_IDLE=y
> > # CONFIG_NO_HZ_FULL is not set
> >
> > So that is not NO_HZ_FULL:
> > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/
> > However, I can't seem to find the full kernel logs for that.
> >
> > Also, other than the TRACE02 fail, I only see the issue with configs
> > with CONFIG_NO_HZ_FULL=y
> >
> > Can you try TRACE02 specifically, and see if you can reproduce the
> > same issue on your setup? Meanwhile, I'll try to trace what is
> > returning the -EBUSY.
I am trying TRACE02 on my X86_64 machine using cross compile and
qemu-system-aarch64 now, my equipment is limited, but hope I can be of
beneficial to the community ;-)
>
> How about something simple like the following? (untested)
>
> ---8<-----------------------
>
> diff --git a/kernel/torture.c b/kernel/torture.c
> index bc8fb361efc0..cd64110694c0 100644
> --- a/kernel/torture.c
> +++ b/kernel/torture.c
> @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes,
>                         // PCI probe frequently disables hotplug during boot.
>                         (*n_offl_attempts)--;
>                         s = " (-EBUSY forgiven during boot)";
> +               } else if (tick_nohz_full_running && ret == -EBUSY) {
> +                       (*n_offl_attempts)--;
> +                       s = " (-EBUSY forgiven if nohz_full is running)";
 Fantastic fix!! thus we can fix the time keeper cpu torture problem
without touch the time keeper code.

Thanks
Zhouyi
>                 }
>                 if (verbose)
>                         pr_alert("%s" TORTURE_FLAG