[PATCH] arm64/entry: Fix arm64-specific rseq brokenness

[Mark Rutland]

On Tue, Apr 28, 2026 at 09:39:56AM +0800, Jinjie Ruan wrote:
> On 4/25/2026 12:45 AM, Mark Rutland wrote:
> > From 79b65cbbfa20aa2cb0bc248591fab5459cdc101b Mon Sep 17 00:00:00 2001
> > From: Mark Rutland <mark.rutland at arm.com>
> > Date: Thu, 23 Apr 2026 16:51:12 +0100
> > Subject: [PATCH] arm64/entry: Fix arm64-specific rseq brokenness
> > 
> > Mathias Stearn reports that since v6.19, there are two big issues
> > affecting rseq:
> > 
> > (1) On arm64 specifically, rseq critical sections aren't aborted when
> >     they should be.
> > 
> > (2) The 'cpu_id_start' field is no longer written by the kernel in all
> >     cases it used to be, including some cases where TCMalloc depends on
> >     the kernel clobbering the field.
> > 
> > This patch fixes issue #1. This patch DOES NOT fix issue #2, which will
> > need to be addressed by other patches.
> > 
> > The arm64-specific brokenness is a result of commits:
> > 
> >   2fc0e4b4126c ("rseq: Record interrupt from user space")
> >   39a167560a61 ("rseq: Optimize event setting")
> > 
> > The first commit failed to add a call to rseq_note_user_irq_entry() on
> > arm64. Thus arm64 never sets rseq_event::user_irq to record that it may
> > be necessary to abort an active rseq critical section upon return to
> > userspace. On its own, this commit had no functional impact as the value
> > of rseq_event::user_irq was not consumed.
> > 
> > The second commit relied upon rseq_event::user_irq to determine whether
> > or not to bother to perform rseq work when returning to userspace. As
> > rseq_event::user_irq wasn't set on arm64, this work would be skipped,
> > and consequently an active rseq critical section would not be aborted.
> > 
> > Fix this by giving arm64 syscall-specific entry/exit paths, and
> > performing the relevant logic in syscall and non-syscall paths,
> > including calling rseq_note_user_irq_entry() for non-syscall entry.
> > 
> > Currently arm64 cannot use syscall_enter_from_user_mode(),
> > syscall_exit_to_user_mode(), and irqentry_exit_to_user_mode(), due to
> > ordering constraints with exception masking, and risk of ABI breakage
> > for syscall tracing/audit/etc. For the moment the entry/exit logic is
> > left as arm64-specific, but mirroring the generic code.
> > 
> > I intend to follow up with refactoring/cleanup, as we did for kernel
> > mode entry paths in commit:
> > 
> >   041aa7a85390 ("entry: Split preemption from irqentry_exit_to_kernel_mode()")
> > 
> > ... which will allow arm64 to use the GENERIC_IRQ_ENTRY functions directly.
> > 
> > Fixes: 39a167560a61 ("rseq: Optimize event setting")
> > Reported-by: Mathias Stearn <mathias at mongodb.com>
> > Link: https://lore.kernel.org/regressions/CAHnCjA25b+nO2n5CeifknSKHssJpPrjnf+dtr7UgzRw4Zgu=oA@mail.gmail.com/
> > Signed-off-by: Mark Rutland <mark.rutland at arm.com>
> > Cc: Catalin Marinas <catalin.marinas at arm.com>
> > Cc: Chris Kennelly <ckennelly at google.com>
> > Cc: Dmitry Vyukov <dvyukov at google.com>
> > Cc: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> > Cc: Peter Zijlstra <peterz at infradead.org>
> > Cc: Thomas Gleixner <tglx at linutronix.de>
> > Cc: Will Deacon <will at kernel.org>
> > ---
> >  arch/arm64/kernel/entry-common.c | 29 ++++++++++++++++++++++-------
> >  include/linux/irq-entry-common.h |  8 --------
> >  include/linux/rseq_entry.h       | 19 -------------------
> >  3 files changed, 22 insertions(+), 34 deletions(-)
> > 
> > diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> > index cb54335465f66..65ade1f1544f6 100644
> > --- a/arch/arm64/kernel/entry-common.c
> > +++ b/arch/arm64/kernel/entry-common.c
> > @@ -62,6 +62,12 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
> >  	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
> >  }
> >  
> > +static __always_inline void arm64_syscall_enter_from_user_mode(struct pt_regs *regs)
> > +{
> > +	enter_from_user_mode(regs);
> > +	mte_disable_tco_entry(current);
> 
> Did we skip sme_enter/exit_from_user_mode() on the syscall path on
> purpose? Not very familiar with ARM64 SME.
> > +}

That was by accident. I originally wrote the fix on a kernel that lacked
those functions, and I missed them when rebasing the fix.

I'll go fix that up for v2.

> > +
> >  /*
> >   * Handle IRQ/context state management when entering from user mode.
> >   * Before this function is called it is not safe to call regular kernel code,
> > @@ -70,20 +76,29 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
> >  static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
> >  {
> >  	enter_from_user_mode(regs);
> > +	rseq_note_user_irq_entry();
> 
> Can we just use irqentry_enter_from_user_mode() instead?

I've deliberately used enter_from_user_mode() here to keep things
balanced (i.e. enter_from_user_mode() pairs directly with
exit_to_user_mode()). We cannot use irqentry_exit_to_user_mode() as
explained in the commit message.

I'll update the commit message to make that a bit clearer.

[...]

> Otherwise, looks fine to me.

Great; thanks for taking a look.

Mark.