[Query] ARM64: Softlockup when using PSCI for secondary CPU_ON

[Sharma Bhupesh]

Hi Mark,

> On Behalf Of Sharma Bhupesh
> Sent: Monday, August 03, 2015 3:47 PM
> 
> Hi Mark,
> 
> Thanks for the reply.
> 
> > From: Mark Rutland [mailto:mark.rutland at arm.com]
> > Sent: Monday, August 03, 2015 3:33 PM
> >
> > On Mon, Aug 03, 2015 at 05:12:00AM +0100, Sharma Bhupesh wrote:
> > > Hi ARM64 experts,
> > >
> > > I am working on a Freescale ARM64 based SoC, which has 4 Cortex-A53
> > > cores and I am using the PSCI conducts to turn on the secondary CPUs
> > > via a custom implementation of a secure runtime firmware (something
> > similar to the ARM Trusted Firmware).
> > >
> > > Following is my boot-flow:
> > >
> > > - BootROM (runs in EL3, All secondaries are held up in reset) ->
> > >
> > > - UEFI bootloader- PEI Phase (Runs only on Primary, starts up in
> > > EL3)
> > > ->
> > >
> > > - Secure Firmware (runs in EL3, transitions primary core to EL2) ->
> > >
> > > - UEFI bootloader- Rest of the Phases (Runs only on Primary, in EL2)
> > > ->
> > >
> > > - UEFI Linux loader launches linux (Runs only on Primary,
> > > transitions core to EL1) ->
> > >
> > > - Linux runs and calls PSCI conducts to turn on secondary cores via
> > > smc calls to secure run-time firmware ->
> >
> > Which version of Linux are you using?
> 
> I am using 3.19.3
> 
> > Is Linux itself booted at EL1 or EL2?
> 
> UEFI leaves the execution level of the primary core in EL2.
> KVM is turned-off in my defconfig, so I am not loading any hypervisor at
> EL2 as of now.
> 
> > EL2 is very strongly preferable unless you're already running a
> > hypervisor at EL2. That way Linux can ensure that things are
> > initialised correctly at EL2 (e.g. CNTVOFF), regardless of whether you
> > want to use KVM.
> 
> I notice that in the kernel primary core successfully switches to EL1 and
> starts executing the relevant 'start_kernel' routine.
> 
> > > - Now I can see that the ARM64 linux is able to recognize that the
> > secondaries are boot'ed properly:
> > >
> > > PSCI Logs:
> > > ----------
> > > psci: probing for conduit method from DT.
> > > psci: PSCIv0.2 detected in firmware.
> > > psci: Using standard PSCI v0.2 function IDs ..
> > >
> > > Architected cp15 timer(s) running at 25.00MHz (phys).
> > > sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every
> > > 2748779069440ns
> > >
> > > ..
> > >
> > > CPU1: Booted secondary processor
> > > Detected VIPT I-cache on CPU1
> > > CPU2: Booted secondary processor
> > > Detected VIPT I-cache on CPU2
> > > CPU3: Booted secondary processor
> > > Detected VIPT I-cache on CPU3
> > > Brought up 4 CPUs
> > > SMP: Total of 4 processors activated.
> >
> > Ok. This all looks fine.
> >
> > > - However, when the Linux boot proceeds further, I see softlockups
> > > which usually show that one of the secondaries was stuck in
> > 'secondary_start_kernel' (detailed logs below).
> > >
> > > - When I turn of CONFIG_SMP from the .config, I can see that the
> > > softlockups disappear. So, I am correlating these softlockups to SMP
> > and PSCI conducts.
> >
> > Ok, that likely means it's in the idle loop. Given that the CPUs made
> > it into the kernel this sounds like a generic SMP issue rather than
> > something specific to PSCI.
> 
> Ok.
> 
> > > preempt self-detected stall on CPU { 3}  (t=17194 jiffies g=-267
> > > c=-268 q=1)
> >
> > > INFO: rcu_preempt detected stalls on CPUs/tasks: { 1 2} (detected by
> > > 0, t=287104493908 jiffies, g=-267, c=-268, q=2)
> >
> > It looks like your timekeeping has gone wrong. As mentioned above, I'd
> > suspect that CNTVOFF_EL2 is not programmed consistently across all CPUs.
> >
> > Are you able to boot Linux at EL2 instead? Or can you verify that
> > CNTVOFF_EL2 is initialised?
> >
> 
> Ok, let me check the CNTVOFF_EL2 part then.
> I will get back with details soon.
> 

Indeed. Using the following code in the run-time EL3 firmware to correctly set the
CNTVOFF_EL2 to zero for secondaries solves the soft-lockup. I took a reference from your u-boot
patch (see [1]).

+	/* Initialize Generic Timers */
+	msr	cntvoff_el2, xzr

[1] http://marc.info/?l=u-boot&m=140230240621878&w=2

Many thanks for the quick pointers.

Regards,
Bhupesh