[PATCH v2] arm64/irqflags: __always_inline the arch_local_irq_*() helpers

[Breno Leitao]

On Tue, Apr 21, 2026 at 08:58:57AM -0700, Breno Leitao wrote:
> The arch_local_irq_*() wrappers in <asm/irqflags.h> dispatch between two
> underlying primitives: the __daif_* path on most systems, and the
> __pmr_* path on builds that use GIC PMR-based masking (Pseudo-NMI). The
> leaf primitives are already __always_inline, but the wrappers themselves
> are plain "static inline".
> 
> That is unsafe for noinstr callers: nothing prevents the compiler from
> emitting an out-of-line copy of e.g. arch_local_irq_disable(), and an
> out-of-line copy can be instrumented (ftrace, kcov, sanitizers), which
> breaks the noinstr contract on the entry/idle paths that rely on these
> helpers.
> 
> x86 hit and fixed exactly this class of bug in commit 7a745be1cc90
> ("x86/entry: __always_inline irqflags for noinstr").
> 
> Force-inline all of the arch_local_irq_*() wrappers so they cannot be
> emitted out-of-line:
> 
>   - arch_local_irq_enable()
>   - arch_local_irq_disable()
>   - arch_local_save_flags()
>   - arch_irqs_disabled_flags()
>   - arch_irqs_disabled()
>   - arch_local_irq_save()
>   - arch_local_irq_restore()
> 
> The primary motivation is noinstr safety. There is a useful side effect
> for fleet-wide profiling: when the wrapper is emitted out-of-line,
> samples taken inside it during the post-WFI IRQ unmask in
> default_idle_call() are attributed to arch_local_irq_enable rather than
> default_idle_call(), and the FP-unwinder loses default_idle_call() from
> the chain.

FWIW I run scripts/bloat-o-meter on the kernel with and without the
patch, and the the code size is mostly the same. here is the result:

	add/remove: 4/12 grow/shrink: 40/0 up/down: 1684/-652 (1032)
	Function                                     old     new   delta
	__schedule                                  8892    9024    +132
	irqentry_exit                                816     892     +76
	lockdep_hardirqs_off                         396     452     +56
	lock_is_held_type                            412     468     +56
	ct_idle_exit                                  76     132     +56
	cpu_idle_poll                                304     360     +56
	arch_stack_walk_reliable                    1152    1196     +44
	arch_stack_walk                             1184    1228     +44
	arch_bpf_stack_walk                          996    1040     +44
	lockdep_hardirqs_on                          464     504     +40
	el0_watchpt                                  576     616     +40
	el0_undef                                    560     600     +40
	el0_sys                                      560     600     +40
	el0_sve_acc                                  560     600     +40
	el0_svc                                      600     640     +40
	el0_sp                                       564     604     +40
	el0_softstp                                  728     768     +40
	el0_sme_acc                                  560     600     +40
	el0_pc                                       740     780     +40
	el0_mops                                     560     600     +40
	el0_inv                                      564     604     +40
	el0_interrupt                                656     696     +40
	el0_ia                                       716     756     +40
	el0_gcs                                      560     600     +40
	el0_fpsimd_exc                               560     600     +40
	el0_fpsimd_acc                               560     600     +40
	el0_fpac                                     560     600     +40
	el0_da                                       568     608     +40
	el0_bti                                      552     592     +40
	el0_brk64                                    560     600     +40
	el0_breakpt                                  720     760     +40
	asm_exit_to_user_mode                        416     456     +40
	__el0_error_handler_common                   592     632     +40
	cpuidle_enter_state                         1220    1248     +28
	check_preemption_disabled                    228     252     +24
	default_idle_call                            252     272     +20
	ct_kernel_enter                              388     404     +16
	ct_idle_enter                                 52      68     +16
	look_up_lock_class                           364     376     +12
	check_flags                                  492     504     +12
	__CortexA53843419_FFFF800081146000             -       8      +8
	__CortexA53843419_FFFF8000809C3004             -       8      +8
	__CortexA53843419_FFFF8000809AE000             -       8      +8
	__CortexA53843419_FFFF800080248004             -       8      +8
	__CortexA53843419_FFFF80008100C000             8       -      -8
	__CortexA53843419_FFFF8000809A9000             8       -      -8
	__CortexA53843419_FFFF8000809A8004             8       -      -8
	__CortexA53843419_FFFF800080448008             8       -      -8
	__CortexA53843419_FFFF8000801EE000             8       -      -8
	arch_local_irq_restore                        48       -     -48
	arch_local_save_flags                         80       -     -80
	arch_local_irq_save                           80       -     -80
	arch_local_irq_enable                         84       -     -84
	arch_local_irq_disable                        96       -     -96
	arch_irqs_disabled_flags                      96       -     -96
	arch_irqs_disabled                           128       -    -128
	Total: Before=163062863, After=163063895, chg +0.00%