[PATCH bpf-next v2 0/2] bpf: Optimize recursion detection on arm64

[Puranjay Mohan]

On Wed, Dec 17, 2025 at 11:36 PM Puranjay Mohan <puranjay at kernel.org> wrote:
>
> V1: https://lore.kernel.org/all/20251217162830.2597286-1-puranjay@kernel.org/
> Changes in V1->V2:
> - Patch 2:
>         - Put preempt_enable()/disable() around RMW accesses to mitigate
>           race conditions. Because on CONFIG_PREEMPT_RCU and sleepable
>           bpf programs, preemption can cause no prog to execute.
>
> BPF programs detect recursion using a per-CPU 'active' flag in struct
> bpf_prog. The trampoline currently sets/clears this flag with atomic
> operations.
>
> On some arm64 platforms (e.g., Neoverse V2 with LSE), per-CPU atomic
> operations are relatively slow. Unlike x86_64 - where per-CPU updates
> can avoid cross-core atomicity, arm64 LSE atomics are always atomic
> across all cores, which is unnecessary overhead for strictly per-CPU
> state.
>
> This patch removes atomics from the recursion detection path on arm64.
>
> It was discovered in [1] that per-CPU atomics that don't return a value
> were extremely slow on some arm64 platforms, Catalin added a fix in
> commit 535fdfc5a228 ("arm64: Use load LSE atomics for the non-return
> per-CPU atomic operations") to solve this issue, but it seems to have
> caused a regression on the fentry benchmark.
>
> Using the fentry benchmark from the bpf selftests shows the following:
>
>   ./tools/testing/selftests/bpf/bench trig-fentry
>
>  +---------------------------------------------+------------------------+
>  |               Configuration                 | Total Operations (M/s) |
>  +---------------------------------------------+------------------------+
>  | bpf-next/master with Catalin’s fix reverted |         51.862         |
>  |---------------------------------------------|------------------------|
>  | bpf-next/master                             |         43.067         |
>  | bpf-next/master with this change            |         53.856         |
>  +---------------------------------------------+------------------------+
>
> All benchmarks were run on a KVM based vm with Neoverse-V2 and 8 cpus.
>


Here is some more data about other attach types:

+-----------------+-----------+-----------+----------+
|     Metric      |  Before   |   After   | % Diff   |
+-----------------+-----------+-----------+----------+
| fentry          |   43.149  |   53.948  | +25.03%  |
| fentry.s        |   41.831  |   50.937  | +21.76%  |
| rawtp           |   50.834  |   58.731  | +15.53%  |
| fexit           |   31.118  |   34.360  | +10.42%  |
| tp              |   39.536  |   41.632  |  +5.30%  |
| syscall-count   |    8.053  |    8.305  |  +3.13%  |
| fmodret         |   33.940  |   34.769  |  +2.44%  |
| kprobe          |    9.970  |    9.998  |  +0.28%  |
| usermode-count  |  224.886  |  224.839  |  -0.02%  |
| kernel-count    |  154.229  |  153.043  |  -0.77%  |
+-----------------+-----------+-----------+----------+