arm64 torture test hotplug failures (offlining causes -EBUSY)
Zhouyi Zhou
zhouzhouyi at gmail.com
Mon Jan 16 19:15:18 PST 2023
On Tue, Jan 17, 2023 at 9:45 AM Joel Fernandes <joel at joelfernandes.org> wrote:
>
> On Tue, Jan 17, 2023 at 08:37:16AM +0800, Zhouyi Zhou wrote:
> > On Tue, Jan 17, 2023 at 8:15 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > >
> > > On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote:
> > > > Hi Zhouyi,
> > > >
> > > > On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
> > > > >
> > > > [..]
> > > > > On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > > > > >
> > > > > > Hello,
> > > > > > I am seeing -EBUSY returned a lot during torture_onoff() when running
> > > > > > rcutorture on arm64. This causes hotplug failure 30% of the time. I am
> > > > > > also seeing this in 6.1-rc kernels. I believe see this only for CPU0.
> > > > > >
> > > > > > This causes warnings in torture tests:
> > > > > > [ 217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > > > > > [ 221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > > > > >
> > > > > > Full kernel log here:
> > > > > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log
> > > > > >
> > > > > > Any ideas on why this is happening and only for CPU 0 (presumably the
> > > > > > boot CPU)? I'd personally need these warnings to go away for my tests
> > > > > > as this causes rcutorture's tests to not cleanly pass for me. It
> > > > > > appears remove_cpu() -> device_offline() is what returns the error.
> > > > > >
> > > > > I guess this probably because CPU 0 is the tick_do_timer_cpu in
> > > > > nohz_full mode, which prevent that cpu from
> > > > > going offline [1]. We have discussed this topic, but there is no
> > > > > agreement on how to solve it yet.
> > > >
> > > > But I am seeing the issue in TRACE02 config which is:
> > > > CONFIG_NO_HZ_IDLE=y
> > > > # CONFIG_NO_HZ_FULL is not set
> > > >
> > > > So that is not NO_HZ_FULL:
> > > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/
> > > > However, I can't seem to find the full kernel logs for that.
> > > >
> > > > Also, other than the TRACE02 fail, I only see the issue with configs
> > > > with CONFIG_NO_HZ_FULL=y
> > > >
> > > > Can you try TRACE02 specifically, and see if you can reproduce the
> > > > same issue on your setup? Meanwhile, I'll try to trace what is
> > > > returning the -EBUSY.
> > I am trying TRACE02 on my X86_64 machine using cross compile and
> > qemu-system-aarch64 now, my equipment is limited, but hope I can be of
> > beneficial to the community ;-)
>
> Cool, I am assuming you are trying the patch you shared which you wrote in
> November. I bet you will still see the issue.
yes, I still see the issue with no hz full.
>
> > >
> > > How about something simple like the following? (untested)
> > >
> > > ---8<-----------------------
> > >
> > > diff --git a/kernel/torture.c b/kernel/torture.c
> > > index bc8fb361efc0..cd64110694c0 100644
> > > --- a/kernel/torture.c
> > > +++ b/kernel/torture.c
> > > @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes,
> > > // PCI probe frequently disables hotplug during boot.
> > > (*n_offl_attempts)--;
> > > s = " (-EBUSY forgiven during boot)";
> > > + } else if (tick_nohz_full_running && ret == -EBUSY) {
> > > + (*n_offl_attempts)--;
> > > + s = " (-EBUSY forgiven if nohz_full is running)";
> > Fantastic fix!! thus we can fix the time keeper cpu torture problem
> > without touch the time keeper code.
>
> Thanks. Unfortunately this does not fix the issue for TRACE02 and the patch
> you shared does not fix it either -- because TRACE02 is not a no-hz-full
> test. :-(
>
> We will need to do a bit of tracing to figure out where the -EBUSY is coming
> from for TRACE02.
agree TRACE02 is another issue, unfortunately I can't reproduce the
bug neither with your original Image [1]
nor with my cross compiled kernel using [2].
I guess there may be two reasons:
1) my testbed is X86_64 based.
2) the command that I invoke qemu is not right:
2-1) the newly compiled linux-5.15.89-rc1
qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic -smp 4
-serial file:/tmp/consoleJan1702.log -kernel arch/arm64/boot/Image
-append "console=ttyAMA0 oops=panic panic_on_warn=1 panic=-1
ftrace_dump_on_oops=orig_cpu debug earlyprintk=serial slub_debug=UZ
rcutorture.torture_type=tasks-tracing rcutorture.onoff_interval=1000
rcutorture.onoff_holdoff=1000 rcutorture.n_barrier_cbs=4
rcutorture.stat_interval=15 rcutorture.shutdown_secs=1200
test_no_idle_hz=1 verbose=1" -m 2048 -net user,hostfwd=tcp::10024-:22
-net nic
2-2) original Image [1]
qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic -smp
4 -serial file:/tmp/consoleJan1701.log -kernel /home/zzy/Image
-append "console=ttyAMA0 oops=panic panic_on_warn=1 panic=-1
ftrace_dump_on_oops=orig_cpu debug earlyprintk=serial slub_debug=UZ
rcutorture.torture_type=tasks-tracing rcutorture.onoff_interval=1000
rcutorture.onoff_holdoff=30 n_barrier_cbs=4
rcutorture.stat_interval=15 rcutorture.shutdown_secs=1200
test_no_idle_hz=1 verbose=1" -m 2048 -net
user,hostfwd=tcp::10023-:22 -net nic
As Mark can reproduce the issue using [1], there must be something
wrong with my x86_64 based environment.
Sorry not to be of help this time.
I am very happy and interested to perform further tests whenever there
are further instructions ;-)
[1] http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/Image
[2] http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/.config
>
> I wonder if we should ignore -EBUSY altogether, since as Thomas mentioned,
> hotplug failure is "normal". Thoughts?
This decision is too important for a beginner like me, however may
thanks for your trust in me ;-) What does Paul think about it ;-)
Thanks
Zhouyi
>
> thanks,
>
> - Joel
>
More information about the linux-arm-kernel
mailing list