[PATCH v10 0/9] riscv: optimize percpu access

[Jisheng Zhang]

On Thu, Nov 12, 2026 at 06:41:40PM +0800, Yunhui Cui wrote:
> RISC-V has no asm/percpu.h, so this_cpu_*() read-modify-write
> operations use generic fallbacks that disable local interrupts. Per-CPU
> addressing also derives the current offset from the CPU number and
> __per_cpu_offset[].
> 
> Cache the per-CPU offset in thread_info and implement preemptible this_cpu
> operations using recoverable GPR critical sections. Each operation records
> the registers used to form its per-CPU address so exception return can
> repair them if the task resumes on a different CPU. Use AMOs for
> native-width operations. Use Zabha for 8-bit and 16-bit operations when
> available, or a 32-bit LR/SC fallback otherwise. Finally, select
> HAVE_CMPXCHG_LOCAL so common per-CPU counter users can use
> this_cpu_try_cmpxchg().
> 
> The recoverable-GPR scheme follows the arm64 design. Only the protocol is
> common, however: RISC-V uses a cached offset in thread_info, its own GPR
> encoding and exception entry code, and AMO/LR/SC operation sequences.
> Factoring these pieces into generic code would share little beyond the
> field layout while coupling architecture entry paths, so keep the
> implementation under arch/riscv.
> 
> As with arm64's SDEI path, independent NMI-like entry paths must
> participate in the entry-state protocol directly. Keep the save, clear and
> restore operations in common RISC-V assembly macros so SBI Supervisor
> Software Events (SSE) can preserve an interrupted per-CPU operation. SBI
> SSE support is developed separately and is not a dependency of this
> series.
> 
> With CONFIG_PREEMPT_LAZY=y, GCC 15, the same linux-next base and
> configuration, an in-kernel microbenchmark pinned to one CPU ran each
> operation 2,000,000 times for 15 rounds. The table reports the median
> empty-loop-adjusted time; lower is better.
> 
>   operation                 v9 (ns/op)   v10 (ns/op)   change
>   this_cpu_read(u64)            5.668         0.333     -94.1%
>   this_cpu_write(u64)           5.668         0.395     -93.0%
>   this_cpu_add(u64)             8.002         8.004      +0.0%
>   this_cpu_add_return(u64)      8.335         8.004      -4.0%
>   this_cpu_xchg(u64)            8.278         8.004      -3.3%
>   this_cpu_cmpxchg(u64)        18.713        16.133     -13.8%

Hi Yunhui,

Thanks for this series. Could you plz run some benchmark to compare the
performance between v9 and v10? We can use some benchmarks used in arm64
version:
https://lore.kernel.org/linux-arm-kernel/28978498-348e-4cfd-af72-1146a106e96b@arm.com/

Something in my mind: hackbench, kernbench, sysbench-cpu, sysbench-mutex,
sysbench-thread, perf/futex , perf/sched and perf/syscall.

I will read the code during this weekend.

Thanks

> 
> With the same configuration, vmlinux .text decreased from 14,369,618 to
> 14,212,882 bytes (-156,736 bytes, -1.09%). percpu_counter_add_batch()
> decreased from 330 to 228 bytes.
> 
> Changes since v9:
> 
> - Rework the this_cpu operations to remain preemptible using recoverable
>   GPR critical sections, as suggested by David Laight and Jisheng Zhang.
> - Put the cached per-CPU offset first, then split the exception/fixup
>   infrastructure and operation classes into independently reviewable
>   patches.
> - Keep GPR-state save and restore in reusable assembly macros for NMI-like
>   entry paths such as SBI SSE.
> - Keep masked LR/SC cmpxchg operands XLEN-wide so RV64 high-byte and
>   high-halfword comparisons are not affected by LR.W sign extension.
> - Force both halves of the AMOCAS.Q new value into fixed registers so the
>   implicit register pair remains valid when either half is zero.
> - Use TYPEOF_UNQUAL() in wrappers and fully parenthesize this_cpu_write()
>   values.
> 
> Changes since v8:
> 
> - Drop the PIO helper patch, which has already been merged.
> - Rename the first patch and update its changelog as suggested by
>   Davidlohr Bueso.
> - Use value-returning AMOs instead of rd=x0 for CPU-local operations.
> - Remove unnecessary memory clobbers from the new relaxed AMO and LR/SC
>   code.
> - Use .option arch,+zabha for the Zabha inline asm, matching the
>   existing RISC-V cmpxchg implementation.
> - Isolate arch_cmpxchg_relaxed() behind typed always-inline helpers to
>   avoid macro variable capture while preserving atomic instrumentation.
> - Make the cmpxchg128 fallback use the already resolved per-CPU pointer,
>   avoiding the same macro variable capture.
> - Copy the cached per-CPU offset from prev to next during context switch,
>   as suggested by Thomas Gleixner.
> - Initialize secondary idle tasks in the common startup path, covering
>   parallel bringup and CPU hotplug, and remove the duplicate initialization
>   from smp_callin(), as reported by Sashiko.
> - Select HAVE_CMPXCHG_LOCAL in a separate patch.
> 
> Yunhui Cui (9):
>   riscv: store percpu offset into thread_info
>   riscv: add infrastructure for preemptible this_cpu operations
>   riscv: implement preemptible this_cpu read and write operations
>   riscv: implement preemptible this_cpu void RMW operations
>   riscv: implement preemptible this_cpu add_return operations
>   riscv: implement preemptible this_cpu xchg operations
>   riscv: implement preemptible this_cpu cmpxchg operations
>   riscv: implement preemptible this_cpu cmpxchg128
>   riscv: select HAVE_CMPXCHG_LOCAL
> 
>  arch/riscv/Kconfig                   |   1 +
>  arch/riscv/include/asm/asm.h         |  25 +-
>  arch/riscv/include/asm/percpu.h      | 701 +++++++++++++++++++++++++++
>  arch/riscv/include/asm/ptrace.h      |  15 +
>  arch/riscv/include/asm/switch_to.h   |   9 +
>  arch/riscv/include/asm/thread_info.h |   4 +-
>  arch/riscv/kernel/Makefile           |   1 +
>  arch/riscv/kernel/asm-offsets.c      |   4 +-
>  arch/riscv/kernel/entry-common.c     |  32 ++
>  arch/riscv/kernel/entry.S            |  10 +
>  arch/riscv/kernel/smpboot.c          |  10 +-
>  arch/riscv/net/bpf_jit_comp64.c      |   9 +-
>  12 files changed, 805 insertions(+), 16 deletions(-)
>  create mode 100644 arch/riscv/include/asm/percpu.h
>  create mode 100644 arch/riscv/kernel/entry-common.c
> 
> 
> base-commit: ddfe51b0a0f5dc9e415cc2c8ad8db17532dd426b
> -- 
> 2.39.5
>