'rcu_preempt detected stalls on CPUs/tasks...' issue of cyclictest on rt-linux

[Neeraj upadhyay]

Hi,

On Fri, Jul 26, 2024 at 12:46 PM richard clark
<richard.xnu.clark at gmail.com> wrote:
>
> Hi，
>
> On Wed, Jul 10, 2024 at 7:22 PM Neeraj upadhyay <neeraj.iitr10 at gmail.com> wrote:
> >
> > Hello Richard,
> >
> > On Wed, Jul 10, 2024 at 1:56 PM richard clark
> > <richard.xnu.clark at gmail.com> wrote:
> > >
> > > Hi,
> > > I am running a Ubuntu 20.04.5 LTS on Nvidia Jetson AGX Orin platform
> > > with 12-cores as a guestOS, the kernel version is - 6.1.83-rt28.
> > > Kernel cmdline is:
> > > 'root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4
> > > console=ttyTCU0,115200 console=tty0 firmware_class.path=/etc/firmware
> > > fbcon=map:0 net.ifnames=0'
> > >
> > > The cyclictest command 'cyclictest -Smp99 -H 3000
> > > --histfile=orin_idle_hyp_4h.hist -D 4h' will hang randomly during the
> > > test, then the minicom console will show below messages:
> > > ...
> > >
> > > [97619.450889] [CPU11-E] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> > > [97619.450894] [CPU11-E] rcu:   1-...!: (0 ticks this GP)
> > > idle=dc88/0/0x0 softirq=0/0 fqs=2 (false positive?)
> > > [97619.450914] [ CPU1-E] NMI backtrace for cpu 1
> > > [97619.451912] [CPU11-E] rcu: rcu_preempt kthread timer wakeup didn't
> > > happen for 5251 jiffies! g6029253 f0x0 RCU_GP_WAIT_FQS(5)
> > > ->state=0x402
> > > [97619.451916] [CPU11-E] rcu:   Possible timer handling issue on cpu=1
> > > timer-softirq=342864
> >
> > This log indicates that jiffies timers are not getting handled on CPU1, due to
> > which GP kthread was not woken up. Can you check irq, softirq and timer traces
> > on CPU1, to see if the softirqs/timers are getting served on this CPU?
> >
> Please ignore this issue. As narrow down this is an issue from
> homemade hypervisor which unmasks the vtimer ctrl.IMASK bit by
> accident, so the vcpu can't get further vtimer interrupt, thus the
> issue happens...
> Thanks
>

Great! Thanks for providing details on the root cause!


- Neeraj

> >
> > - Neeraj
> >