arm64 torture test hotplug failures (offlining causes -EBUSY)

[Joel Fernandes]

On Wed, Jan 18, 2023 at 10:15 AM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
>
> On Tue, Jan 17, 2023 at 7:42 PM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
> >
> > On Tue, Jan 17, 2023 at 12:34 PM Joel Fernandes <joel at joelfernandes.org> wrote:
> > >
> > >
> > >
> > > > On Jan 16, 2023, at 10:15 PM, Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
> > > >
> > > > On Tue, Jan 17, 2023 at 9:45 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > > >>
> > > >>> On Tue, Jan 17, 2023 at 08:37:16AM +0800, Zhouyi Zhou wrote:
> > > >>> On Tue, Jan 17, 2023 at 8:15 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > > >>>>
> > > >>>> On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote:
> > > >>>>> Hi Zhouyi,
> > > >>>>>
> > > >>>>> On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
> > > >>>>>>
> > > >>>>> [..]
> > > >>>>>> On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > > >>>>>>>
> > > >>>>>>> Hello,
> > > >>>>>>> I am seeing -EBUSY returned a lot during torture_onoff() when running
> > > >>>>>>> rcutorture on arm64. This causes hotplug failure 30% of the time. I am
> > > >>>>>>> also seeing this in 6.1-rc kernels. I believe see this only for CPU0.
> > > >>>>>>>
> > > >>>>>>> This causes warnings in torture tests:
> > > >>>>>>> [  217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > > >>>>>>> [  221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > > >>>>>>>
> > > >>>>>>> Full kernel log here:
> > > >>>>>>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log
> > > >>>>>>>
> > > >>>>>>> Any ideas on why this is happening and only for CPU 0 (presumably the
> > > >>>>>>> boot CPU)? I'd personally need these warnings to go away for my tests
> > > >>>>>>> as this causes rcutorture's tests to not cleanly pass for me. It
> > > >>>>>>> appears remove_cpu() -> device_offline() is what returns the error.
> > > >>>>>>>
> > > >>>>>> I guess this probably because CPU 0 is the tick_do_timer_cpu in
> > > >>>>>> nohz_full mode, which prevent that cpu from
> > > >>>>>> going offline [1]. We have discussed this topic, but there is no
> > > >>>>>> agreement on how to solve it yet.
> > > >>>>>
> > > >>>>> But I am seeing the issue in TRACE02 config which is:
> > > >>>>> CONFIG_NO_HZ_IDLE=y
> > > >>>>> # CONFIG_NO_HZ_FULL is not set
> > > >>>>>
> > > >>>>> So that is not NO_HZ_FULL:
> > > >>>>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/
> > > >>>>> However, I can't seem to find the full kernel logs for that.
> > > >>>>>
> > > >>>>> Also, other than the TRACE02 fail, I only see the issue with configs
> > > >>>>> with CONFIG_NO_HZ_FULL=y
> > > >>>>>
> > > >>>>> Can you try TRACE02 specifically, and see if you can reproduce the
> > > >>>>> same issue on your setup? Meanwhile, I'll try to trace what is
> > > >>>>> returning the -EBUSY.
> > > >>> I am trying TRACE02 on my X86_64 machine using cross compile and
> > > >>> qemu-system-aarch64 now, my equipment is limited, but hope I can be of
> > > >>> beneficial to the community ;-)
> > > >>
> > > >> Cool, I am assuming you are trying the patch you shared which you wrote in
> > > >> November. I bet you will still see the issue.
> > > > yes, I still see the issue with no hz full.
> > > >>
> > > >>>>
> > > >>>> How about something simple like the following? (untested)
> > > >>>>
> > > >>>> ---8<-----------------------
> > > >>>>
> > > >>>> diff --git a/kernel/torture.c b/kernel/torture.c
> > > >>>> index bc8fb361efc0..cd64110694c0 100644
> > > >>>> --- a/kernel/torture.c
> > > >>>> +++ b/kernel/torture.c
> > > >>>> @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes,
> > > >>>>                        // PCI probe frequently disables hotplug during boot.
> > > >>>>                        (*n_offl_attempts)--;
> > > >>>>                        s = " (-EBUSY forgiven during boot)";
> > > >>>> +               } else if (tick_nohz_full_running && ret == -EBUSY) {
> > > >>>> +                       (*n_offl_attempts)--;
> > > >>>> +                       s = " (-EBUSY forgiven if nohz_full is running)";
> > > >>> Fantastic fix!! thus we can fix the time keeper cpu torture problem
> > > >>> without touch the time keeper code.
> > > >>
> > > >> Thanks. Unfortunately this does not fix the issue for TRACE02 and the patch
> > > >> you shared does not fix it either -- because TRACE02 is not a no-hz-full
> > > >> test. :-(
> > > >>
> > > >> We will need to do a bit of tracing to figure out where the -EBUSY is coming
> > > >> from for TRACE02.
> > > > agree TRACE02 is another issue, unfortunately I can't reproduce the
> > > > bug neither with your original Image [1]
> > > > nor with my cross compiled kernel using [2].
> > > >
> > > > I guess there may be two reasons:
> > > > 1) my testbed is X86_64 based.
> > > > 2) the command that I invoke qemu is not right:
> > > > 2-1) the newly compiled linux-5.15.89-rc1
> > > > qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic -smp 4
> > >
> > > Does 8 CPUs make any difference? That is my setup.
> > 8 CPUs make no difference ;-(
> > >
> > > Not sure what else is different. It could be a CPU model specific issue, or something. But why donot you just use the same setup you used in November and check TRACE02? That is actually what I was requesting you to rest, since you saw the same issue on that setup.
> > I guess it may be a CPU model specific issue, while I can't invoke
> > qemu-system-aarch64  with  "-machine virt,gic-version=host -cpu host"
> > because I didn't have an aarch64 bare metal host.
> >
> > OK, I am doing the same setup on linux-5.15.y as I did last November
> > in the PPC VM of Open Source Lab of Oregon State University, this will
> > take about 20 hours, and report what I found after the test finishes.
> There are some problems in launching linux-5.15.y in the PPC VM of
> Open Source Lab of Oregon State University, I am digging out why, so I
> can't report the test result today, I'm sorry ;-(. I will report to
> you once I have any progress as soon as possible.
>

No problem! Looking forward to seeing what you see.

On my side I am not able to reproduce the TRACE02 issue any more, so
it was a one-off I think. However, I see arm64 hotplug failures as
expected on all other CONFIG_RCU_NO_HZ_FULL configs.

Will debug / work on a solution for that a bit later today hopefully.

Thanks,

 - Joel