arm64 torture test hotplug failures (offlining causes -EBUSY)

[Joel Fernandes]

On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote:
> Hi Zhouyi,
> 
> On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
> >
> [..]
> > On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > >
> > > Hello,
> > > I am seeing -EBUSY returned a lot during torture_onoff() when running
> > > rcutorture on arm64. This causes hotplug failure 30% of the time. I am
> > > also seeing this in 6.1-rc kernels. I believe see this only for CPU0.
> > >
> > > This causes warnings in torture tests:
> > > [  217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > > [  221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16
> > >
> > > Full kernel log here:
> > > http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log
> > >
> > > Any ideas on why this is happening and only for CPU 0 (presumably the
> > > boot CPU)? I'd personally need these warnings to go away for my tests
> > > as this causes rcutorture's tests to not cleanly pass for me. It
> > > appears remove_cpu() -> device_offline() is what returns the error.
> > >
> > I guess this probably because CPU 0 is the tick_do_timer_cpu in
> > nohz_full mode, which prevent that cpu from
> > going offline [1]. We have discussed this topic, but there is no
> > agreement on how to solve it yet.
> 
> But I am seeing the issue in TRACE02 config which is:
> CONFIG_NO_HZ_IDLE=y
> # CONFIG_NO_HZ_FULL is not set
> 
> So that is not NO_HZ_FULL:
> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/
> However, I can't seem to find the full kernel logs for that.
> 
> Also, other than the TRACE02 fail, I only see the issue with configs
> with CONFIG_NO_HZ_FULL=y
> 
> Can you try TRACE02 specifically, and see if you can reproduce the
> same issue on your setup? Meanwhile, I'll try to trace what is
> returning the -EBUSY.

How about something simple like the following? (untested)

---8<-----------------------

diff --git a/kernel/torture.c b/kernel/torture.c
index bc8fb361efc0..cd64110694c0 100644
--- a/kernel/torture.c
+++ b/kernel/torture.c
@@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes,
 			// PCI probe frequently disables hotplug during boot.
 			(*n_offl_attempts)--;
 			s = " (-EBUSY forgiven during boot)";
+		} else if (tick_nohz_full_running && ret == -EBUSY) {
+			(*n_offl_attempts)--;
+			s = " (-EBUSY forgiven if nohz_full is running)";
 		}
 		if (verbose)
 			pr_alert("%s" TORTURE_FLAG