SError Interrupt on CPU0, code 0xbf000000 makes kernel panic

[Joakim Tjernlund]

On Thu, 2022-03-24 at 15:05 +0000, Robin Murphy wrote:
> On 2022-03-24 14:50, Joakim Tjernlund wrote:
> > On Thu, 2022-03-24 at 14:17 +0000, Marc Zyngier wrote:
> > > On Thu, 24 Mar 2022 14:01:53 +0000,
> > > Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
> > > > 
> > > > On Thu, 2022-03-24 at 13:16 +0000, Robin Murphy wrote:
> > > > > On 2022-03-24 12:10, Joakim Tjernlund wrote:
> > > > > > We have a custom SOC, CPU A53, that when an app accesses non existing address space reports:
> > > > > > # > devmem 0x20000000 w 0x1000 #this will open /dev/mem and write
> > > > > >    
> > > > > > [   37.570886] SError Interrupt on CPU0, code 0xbf000000 -- SError
> > > > > > [   37.571974] CPU: 0 PID: 72 Comm: devmem Not tainted 5.15.26-g18447c6fff6f-dirty #26
> > > > > > [   37.573150] Hardware name: infinera,xr (DT)
> > > > > > [   37.573599] pstate: 60000010 (nZCv q A32 LE aif -DIT -SSBS)
> > > > > > [   37.574705] pc : 000000000098775c
> > > > > > [   37.575063] lr : 0000000000986918
> > > > > > [   37.575392] sp : 00000000ffd140a8
> > > > > > [   37.575725] x12: 0000000000a36c10
> > > > > > [   37.576443] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000020
> > > > > > [   37.577872] x8 : 00000000ffd141c0 x7 : 00000000ffd14104 x6 : 0000000000986c9c
> > > > > > [   37.579278] x5 : 000000000000001f x4 : 0000000000000004 x3 : 0000000000a37020
> > > > > > [   37.580635] x2 : 0000000000000003 x1 : 0000000000001000 x0 : 0000000000000000
> > > > > > [   37.582164] Kernel panic - not syncing: Asynchronous SError Interrupt
> > > > > > [   37.582685] Kernel Offset: disabled
> > > > > > [   37.582932] CPU features: 0x00001001,20000842
> > > > > > [   37.583509] Memory Limit: none
> > > > > > [   37.630058] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---
> > > > > > 
> > > > > > and the kernel panics. This is a surprise as I expected the app to just be killed bus a SIGBUS.
> > > > > > Is this what to expect?
> > > > > > I see that kernel looks for the RAS extension but we don't have that.
> > > > > > 
> > > > > > Can anything be done not to panic the kernel for such accesses?
> > > > > 
> > > > > No. The error comes back to the CPU in an unattributable manner, so all
> > > > > it knows is that *something*, at some point in the past, went
> > > > > catastrophically wrong. Saying "this is fine..." and carrying on
> > > > > regardless isn't really viable. IIRC the RAS extension places
> > > > > constraints on the delivery of async SError such that it's slightly more
> > > > > possible to do something with, but without that all bets are off.
> > > > 
> > > > And this is because we don't have RAS? If we did have RAS
> > > > would/could kernel  sort out the error and the app would get an
> > > > SIGBUS or similar?
> > > 
> > > With RAS, the error would be containable, and attributed to the
> > > userspace task by the kernel on the next exception. Without RAS, panic
> > > is the only option, as we have no idea what the damage is. The machine
> > > is on fire, for all we know.
> > 
> > Thanks, now I know.
> > 
> > > 
> > > > 
> > > > > 
> > > > > > Can one build a som sort of blacklisted address spaces which the MMU will block?
> > > > > 
> > > > > Sure, just configure the kernel with CONFIG_DEVMEM=n and it should never
> > > > > access anything invalid.
> > > > > I'm not even entirely joking there - even for address ranges that the
> > > > > kernel *does* know about, you can still SError or deadlock by poking at
> > > > > something that's currently clock-gated or powered off, or lose coherency
> > > > > and cause corruption by accessing memory with the wrong attributes; at
> > > > > worst writing the wrong thing to the wrong place may even physically
> > > > > damage the hardware.
> > > > > 
> > > > I know /dev/mem is bad and it was an example but such SW errors can
> > > > happen elsewhere to, we got one from a badly configured UIO device
> > > > as well.  HW errors we just have to live with but I hoped we could
> > > > handle some SW errors better.
> > > 
> > > I think you have the wrong end of the stick here. This *is* a HW
> > > error, and the HW tells you so in no uncertain terms that something is
> > > really bad.
> > 
> > Yes, SW induced HW error is a better description.
> > 
> > > 
> > > If the device is supposed to be assignable to userspace, it either
> > > must be designed not to respond with a SError no matter what userspace
> > > is throwing at it (because let's face it, userspace will eventually do
> > > something really bad), or the whole system must be designed in a way
> > > that such error can be contained and attributed to the offending
> > > party.
> > > 
> > > Just giving userspace any odd device and hoping that it will all be
> > > fine is unfortunately wishful thinking.
> > 
> > Sure, just want to limit the damage where I can. A ptr access to non existing space is not really harmful
> 
> Well, except when it is... try that on a Qualcomm SoC where the EL2 
> firmware will trap you and reset the system before you even know you've 
> done anything wrong. If you know enough to know that an error triggered 
> by accessing some address is truly benign, you know enough to avoid 
> making that access in the first place.

of course the error will be dealt with but why make bug finding harder than it has to be?

 Jocke