arm64 torture test hotplug failures (offlining causes -EBUSY)
Joel Fernandes
joel at joelfernandes.org
Mon Jan 16 20:34:32 PST 2023
> On Jan 16, 2023, at 10:15 PM, Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
>
> On Tue, Jan 17, 2023 at 9:45 AM Joel Fernandes <joel at joelfernandes.org> wrote:
>>
>>> On Tue, Jan 17, 2023 at 08:37:16AM +0800, Zhouyi Zhou wrote:
>>> On Tue, Jan 17, 2023 at 8:15 AM Joel Fernandes <joel at joelfernandes.org> wrote:
>>>>
>>>> On Mon, Jan 16, 2023 at 05:38:00PM -0500, Joel Fernandes wrote:
>>>>> Hi Zhouyi,
>>>>>
>>>>> On Mon, Jan 16, 2023 at 1:33 PM Zhouyi Zhou <zhouzhouyi at gmail.com> wrote:
>>>>>>
>>>>> [..]
>>>>>> On Tue, Jan 17, 2023 at 1:27 AM Joel Fernandes <joel at joelfernandes.org> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>> I am seeing -EBUSY returned a lot during torture_onoff() when running
>>>>>>> rcutorture on arm64. This causes hotplug failure 30% of the time. I am
>>>>>>> also seeing this in 6.1-rc kernels. I believe see this only for CPU0.
>>>>>>>
>>>>>>> This causes warnings in torture tests:
>>>>>>> [ 217.582290] rcu-torture:torture_onoff task: offline 0 failed: errno -16
>>>>>>> [ 221.866362] rcu-torture:torture_onoff task: offline 0 failed: errno -16
>>>>>>>
>>>>>>> Full kernel log here:
>>>>>>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TREE04/console.log
>>>>>>>
>>>>>>> Any ideas on why this is happening and only for CPU 0 (presumably the
>>>>>>> boot CPU)? I'd personally need these warnings to go away for my tests
>>>>>>> as this causes rcutorture's tests to not cleanly pass for me. It
>>>>>>> appears remove_cpu() -> device_offline() is what returns the error.
>>>>>>>
>>>>>> I guess this probably because CPU 0 is the tick_do_timer_cpu in
>>>>>> nohz_full mode, which prevent that cpu from
>>>>>> going offline [1]. We have discussed this topic, but there is no
>>>>>> agreement on how to solve it yet.
>>>>>
>>>>> But I am seeing the issue in TRACE02 config which is:
>>>>> CONFIG_NO_HZ_IDLE=y
>>>>> # CONFIG_NO_HZ_FULL is not set
>>>>>
>>>>> So that is not NO_HZ_FULL:
>>>>> http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/console.log.diags/
>>>>> However, I can't seem to find the full kernel logs for that.
>>>>>
>>>>> Also, other than the TRACE02 fail, I only see the issue with configs
>>>>> with CONFIG_NO_HZ_FULL=y
>>>>>
>>>>> Can you try TRACE02 specifically, and see if you can reproduce the
>>>>> same issue on your setup? Meanwhile, I'll try to trace what is
>>>>> returning the -EBUSY.
>>> I am trying TRACE02 on my X86_64 machine using cross compile and
>>> qemu-system-aarch64 now, my equipment is limited, but hope I can be of
>>> beneficial to the community ;-)
>>
>> Cool, I am assuming you are trying the patch you shared which you wrote in
>> November. I bet you will still see the issue.
> yes, I still see the issue with no hz full.
>>
>>>>
>>>> How about something simple like the following? (untested)
>>>>
>>>> ---8<-----------------------
>>>>
>>>> diff --git a/kernel/torture.c b/kernel/torture.c
>>>> index bc8fb361efc0..cd64110694c0 100644
>>>> --- a/kernel/torture.c
>>>> +++ b/kernel/torture.c
>>>> @@ -220,6 +220,9 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes,
>>>> // PCI probe frequently disables hotplug during boot.
>>>> (*n_offl_attempts)--;
>>>> s = " (-EBUSY forgiven during boot)";
>>>> + } else if (tick_nohz_full_running && ret == -EBUSY) {
>>>> + (*n_offl_attempts)--;
>>>> + s = " (-EBUSY forgiven if nohz_full is running)";
>>> Fantastic fix!! thus we can fix the time keeper cpu torture problem
>>> without touch the time keeper code.
>>
>> Thanks. Unfortunately this does not fix the issue for TRACE02 and the patch
>> you shared does not fix it either -- because TRACE02 is not a no-hz-full
>> test. :-(
>>
>> We will need to do a bit of tracing to figure out where the -EBUSY is coming
>> from for TRACE02.
> agree TRACE02 is another issue, unfortunately I can't reproduce the
> bug neither with your original Image [1]
> nor with my cross compiled kernel using [2].
>
> I guess there may be two reasons:
> 1) my testbed is X86_64 based.
> 2) the command that I invoke qemu is not right:
> 2-1) the newly compiled linux-5.15.89-rc1
> qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic -smp 4
Does 8 CPUs make any difference? That is my setup.
Not sure what else is different. It could be a CPU model specific issue, or something. But why donot you just use the same setup you used in November and check TRACE02? That is actually what I was requesting you to rest, since you saw the same issue on that setup.
Thanks,
Joel
> -serial file:/tmp/consoleJan1702.log -kernel arch/arm64/boot/Image
> -append "console=ttyAMA0 oops=panic panic_on_warn=1 panic=-1
> ftrace_dump_on_oops=orig_cpu debug earlyprintk=serial slub_debug=UZ
> rcutorture.torture_type=tasks-tracing rcutorture.onoff_interval=1000
> rcutorture.onoff_holdoff=1000 rcutorture.n_barrier_cbs=4
> rcutorture.stat_interval=15 rcutorture.shutdown_secs=1200
> test_no_idle_hz=1 verbose=1" -m 2048 -net user,hostfwd=tcp::10024-:22
> -net nic
> 2-2) original Image [1]
> qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic -smp
> 4 -serial file:/tmp/consoleJan1701.log -kernel /home/zzy/Image
> -append "console=ttyAMA0 oops=panic panic_on_warn=1 panic=-1
> ftrace_dump_on_oops=orig_cpu debug earlyprintk=serial slub_debug=UZ
> rcutorture.torture_type=tasks-tracing rcutorture.onoff_interval=1000
> rcutorture.onoff_holdoff=30 n_barrier_cbs=4
> rcutorture.stat_interval=15 rcutorture.shutdown_secs=1200
> test_no_idle_hz=1 verbose=1" -m 2048 -net
> user,hostfwd=tcp::10023-:22 -net nic
>
> As Mark can reproduce the issue using [1], there must be something
> wrong with my x86_64 based environment.
>
> Sorry not to be of help this time.
>
> I am very happy and interested to perform further tests whenever there
> are further instructions ;-)
>
> [1] http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/Image
> [2] http://box.joelfernandes.org:9080/job/rcutorture_stable_arm/job/linux-5.15.y/7/artifact/tools/testing/selftests/rcutorture/res/2023.01.15-14.51.11/TRACE02/.config
>>
>> I wonder if we should ignore -EBUSY altogether, since as Thomas mentioned,
>> hotplug failure is "normal". Thoughts?
> This decision is too important for a beginner like me, however may
> thanks for your trust in me ;-) What does Paul think about it ;-)
>
> Thanks
> Zhouyi
>>
>> thanks,
>>
>> - Joel
>>
More information about the linux-arm-kernel
mailing list