SError Interrupt on CPU0, code 0xbf000000 makes kernel panic

[Robin Murphy]

On 2022-03-24 14:50, Joakim Tjernlund wrote:
> On Thu, 2022-03-24 at 14:17 +0000, Marc Zyngier wrote:
>> On Thu, 24 Mar 2022 14:01:53 +0000,
>> Joakim Tjernlund <Joakim.Tjernlund at infinera.com> wrote:
>>>
>>> On Thu, 2022-03-24 at 13:16 +0000, Robin Murphy wrote:
>>>> On 2022-03-24 12:10, Joakim Tjernlund wrote:
>>>>> We have a custom SOC, CPU A53, that when an app accesses non existing address space reports:
>>>>> # > devmem 0x20000000 w 0x1000 #this will open /dev/mem and write
>>>>>    
>>>>> [   37.570886] SError Interrupt on CPU0, code 0xbf000000 -- SError
>>>>> [   37.571974] CPU: 0 PID: 72 Comm: devmem Not tainted 5.15.26-g18447c6fff6f-dirty #26
>>>>> [   37.573150] Hardware name: infinera,xr (DT)
>>>>> [   37.573599] pstate: 60000010 (nZCv q A32 LE aif -DIT -SSBS)
>>>>> [   37.574705] pc : 000000000098775c
>>>>> [   37.575063] lr : 0000000000986918
>>>>> [   37.575392] sp : 00000000ffd140a8
>>>>> [   37.575725] x12: 0000000000a36c10
>>>>> [   37.576443] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000020
>>>>> [   37.577872] x8 : 00000000ffd141c0 x7 : 00000000ffd14104 x6 : 0000000000986c9c
>>>>> [   37.579278] x5 : 000000000000001f x4 : 0000000000000004 x3 : 0000000000a37020
>>>>> [   37.580635] x2 : 0000000000000003 x1 : 0000000000001000 x0 : 0000000000000000
>>>>> [   37.582164] Kernel panic - not syncing: Asynchronous SError Interrupt
>>>>> [   37.582685] Kernel Offset: disabled
>>>>> [   37.582932] CPU features: 0x00001001,20000842
>>>>> [   37.583509] Memory Limit: none
>>>>> [   37.630058] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---
>>>>>
>>>>> and the kernel panics. This is a surprise as I expected the app to just be killed bus a SIGBUS.
>>>>> Is this what to expect?
>>>>> I see that kernel looks for the RAS extension but we don't have that.
>>>>>
>>>>> Can anything be done not to panic the kernel for such accesses?
>>>>
>>>> No. The error comes back to the CPU in an unattributable manner, so all
>>>> it knows is that *something*, at some point in the past, went
>>>> catastrophically wrong. Saying "this is fine..." and carrying on
>>>> regardless isn't really viable. IIRC the RAS extension places
>>>> constraints on the delivery of async SError such that it's slightly more
>>>> possible to do something with, but without that all bets are off.
>>>
>>> And this is because we don't have RAS? If we did have RAS
>>> would/could kernel  sort out the error and the app would get an
>>> SIGBUS or similar?
>>
>> With RAS, the error would be containable, and attributed to the
>> userspace task by the kernel on the next exception. Without RAS, panic
>> is the only option, as we have no idea what the damage is. The machine
>> is on fire, for all we know.
> 
> Thanks, now I know.
> 
>>
>>>
>>>>
>>>>> Can one build a som sort of blacklisted address spaces which the MMU will block?
>>>>
>>>> Sure, just configure the kernel with CONFIG_DEVMEM=n and it should never
>>>> access anything invalid.
>>>> I'm not even entirely joking there - even for address ranges that the
>>>> kernel *does* know about, you can still SError or deadlock by poking at
>>>> something that's currently clock-gated or powered off, or lose coherency
>>>> and cause corruption by accessing memory with the wrong attributes; at
>>>> worst writing the wrong thing to the wrong place may even physically
>>>> damage the hardware.
>>>>
>>> I know /dev/mem is bad and it was an example but such SW errors can
>>> happen elsewhere to, we got one from a badly configured UIO device
>>> as well.  HW errors we just have to live with but I hoped we could
>>> handle some SW errors better.
>>
>> I think you have the wrong end of the stick here. This *is* a HW
>> error, and the HW tells you so in no uncertain terms that something is
>> really bad.
> 
> Yes, SW induced HW error is a better description.
> 
>>
>> If the device is supposed to be assignable to userspace, it either
>> must be designed not to respond with a SError no matter what userspace
>> is throwing at it (because let's face it, userspace will eventually do
>> something really bad), or the whole system must be designed in a way
>> that such error can be contained and attributed to the offending
>> party.
>>
>> Just giving userspace any odd device and hoping that it will all be
>> fine is unfortunately wishful thinking.
> 
> Sure, just want to limit the damage where I can. A ptr access to non existing space is not really harmful

Well, except when it is... try that on a Qualcomm SoC where the EL2 
firmware will trap you and reset the system before you even know you've 
done anything wrong. If you know enough to know that an error triggered 
by accessing some address is truly benign, you know enough to avoid 
making that access in the first place.

Robin.

> and I want the app to take the hit for it. At least then you can log/trouble shoot easier.
> 
>   Jocke