[PATCH] arm64/entry: Mask DAIF in cpu_switch_to(), call_on_irq_stack()

[Will Deacon]

On Fri, Jul 18, 2025 at 03:28:14PM +0100, Ada Couprie Diaz wrote:
> `cpu_switch_to()` and `call_on_irq_stack()` manipulate SP to change
> to different stacks along with the Shadow Call Stack if it is enabled.
> Those two stack changes cannot be done atomically and both functions
> can be interrupted by SErrors or Debug Exceptions which, though unlikely,
> is very much broken : if interrupted, we can end up with mismatched stacks
> and Shadow Call Stack leading to clobbered stacks.
> 
> In `cpu_switch_to()`, it can happen when SP_EL0 points to the new task,
> but x18 stills points to the old task's SCS. When the interrupt handler
> tries to save the task's SCS pointer, it will save the old task
> SCS pointer (x18) into the new task struct (pointed to by SP_EL0),
> clobbering it.
> 
> In `call_on_irq_stack()`, it can happen when switching from the task stack
> to the IRQ stack and when switching back. In both cases, we can be
> interrupted when the SCS pointer points to the IRQ SCS, but SP points to
> the task stack. The nested interrupt handler pushes its return addresses
> on the IRQ SCS. It then detects that SP points to the task stack,
> calls `call_on_irq_stack()` and clobbers the task SCS pointer with
> the IRQ SCS pointer, which it will also use !
> 
> This leads to tasks returning to addresses on the wrong SCS,
> or even on the IRQ SCS, triggering kernel panics via CONFIG_VMAP_STACK
> or FPAC if enabled.
> 
> This is possible on a default config, but unlikely.
> However, when enabling CONFIG_ARM64_PSEUDO_NMI, DAIF is unmasked and
> instead the GIC is responsible for filtering what interrupts the CPU
> should receive based on priority.
> Given the goal of emulating NMIs, pseudo-NMIs can be received by the CPU
> even in `cpu_switch_to()` and `call_on_irq_stack()`, possibly *very*
> frequently depending on the system configuration and workload, leading
> to unpredictable kernel panics.
> 
> Completely mask DAIF in `cpu_switch_to()` and restore it when returning.
> Do the same in `call_on_irq_stack()`, but restore and mask around
> the branch.
> Mask DAIF even if CONFIG_SHADOW_CALL_STACK is not enabled for consistency
> of behaviour between all configurations.
> 
> Introduce and use an assembly macro for saving and masking DAIF,
> as the existing one saves but only masks IF.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz at arm.com>
> Reported-by: Cristian Prundeanu <cpru at amazon.com>
> Fixes: 59b37fe52f49955791a460752c37145f1afdcad1 ("arm64: Stash shadow stack pointer in the task struct on interrupt")
> ---
> Hi,
> I spent some time evaluating the performance impact of this change
> to make sure that it would be OK to mask in those functions.
> They have very few instructions so have few chances to be interrupted
> to begin with so the impact should be minimal.

It's definitely worthwhile doing the benchmarking, so thanks for that,
but realistically I don't think we have an awful lot of choice but to
fix this, do we?

The code looks good to me:

Acked-by: Will Deacon <will at kernel.org>

Will