[Query] ARM64: Softlockup when using PSCI for secondary CPU_ON

[Sharma Bhupesh]

Hi Mark,

Thanks for the reply.

> From: Mark Rutland [mailto:mark.rutland at arm.com]
> Sent: Monday, August 03, 2015 3:33 PM
>
> On Mon, Aug 03, 2015 at 05:12:00AM +0100, Sharma Bhupesh wrote:
> > Hi ARM64 experts,
> >
> > I am working on a Freescale ARM64 based SoC, which has 4 Cortex-A53
> > cores and I am using the PSCI conducts to turn on the secondary CPUs
> > via a custom implementation of a secure runtime firmware (something
> similar to the ARM Trusted Firmware).
> >
> > Following is my boot-flow:
> >
> > - BootROM (runs in EL3, All secondaries are held up in reset) ->
> >
> > - UEFI bootloader- PEI Phase (Runs only on Primary, starts up in EL3)
> > ->
> >
> > - Secure Firmware (runs in EL3, transitions primary core to EL2) ->
> >
> > - UEFI bootloader- Rest of the Phases (Runs only on Primary, in EL2)
> > ->
> >
> > - UEFI Linux loader launches linux (Runs only on Primary, transitions
> > core to EL1) ->
> >
> > - Linux runs and calls PSCI conducts to turn on secondary cores via
> > smc calls to secure run-time firmware ->
> 
> Which version of Linux are you using?

I am using 3.19.3
 
> Is Linux itself booted at EL1 or EL2?

UEFI leaves the execution level of the primary core in EL2.
KVM is turned-off in my defconfig, so I am not loading any hypervisor at EL2 as of now.

> EL2 is very strongly preferable unless you're already running a
> hypervisor at EL2. That way Linux can ensure that things are initialised
> correctly at EL2 (e.g. CNTVOFF), regardless of whether you want to use
> KVM.

I notice that in the kernel primary core successfully switches to EL1 and starts executing the
relevant 'start_kernel' routine.
 
> > - Now I can see that the ARM64 linux is able to recognize that the
> secondaries are boot'ed properly:
> >
> > PSCI Logs:
> > ----------
> > psci: probing for conduit method from DT.
> > psci: PSCIv0.2 detected in firmware.
> > psci: Using standard PSCI v0.2 function IDs ..
> >
> > Architected cp15 timer(s) running at 25.00MHz (phys).
> > sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every
> > 2748779069440ns
> >
> > ..
> >
> > CPU1: Booted secondary processor
> > Detected VIPT I-cache on CPU1
> > CPU2: Booted secondary processor
> > Detected VIPT I-cache on CPU2
> > CPU3: Booted secondary processor
> > Detected VIPT I-cache on CPU3
> > Brought up 4 CPUs
> > SMP: Total of 4 processors activated.
> 
> Ok. This all looks fine.
> 
> > - However, when the Linux boot proceeds further, I see softlockups
> > which usually show that one of the secondaries was stuck in
> 'secondary_start_kernel' (detailed logs below).
> >
> > - When I turn of CONFIG_SMP from the .config, I can see that the
> > softlockups disappear. So, I am correlating these softlockups to SMP
> and PSCI conducts.
> 
> Ok, that likely means it's in the idle loop. Given that the CPUs made it
> into the kernel this sounds like a generic SMP issue rather than
> something specific to PSCI.

Ok.
 
> > preempt self-detected stall on CPU { 3}  (t=17194 jiffies g=-267
> > c=-268 q=1)
> 
> > INFO: rcu_preempt detected stalls on CPUs/tasks: { 1 2} (detected by
> > 0, t=287104493908 jiffies, g=-267, c=-268, q=2)
> 
> It looks like your timekeeping has gone wrong. As mentioned above, I'd
> suspect that CNTVOFF_EL2 is not programmed consistently across all CPUs.
> 
> Are you able to boot Linux at EL2 instead? Or can you verify that
> CNTVOFF_EL2 is initialised?
> 

Ok, let me check the CNTVOFF_EL2 part then.
I will get back with details soon.

Regards,
Bhupesh