[RFC v2 PATCH 0/16] Optimize this_cpu_*() ops for non-x86 (ARM64 for this series)
Usama Anjum
usama.anjum at arm.com
Wed Sep 2 04:51:41 PDT 2026
On 15/07/2026 7:04 pm, Yang Shi wrote:
>
> Hi,
>
> This is v2 RFC. In v2 a lot problems found out by Sashiko were fixed and more
> feature gaps were closed (please see the below changelog for the details).
> Although there are still some open issues, for example, it just can support
> 48 bits VA (for 4K and 64K) and 47 bits VA (for 16K), KPTI support has not
> been solved yet, etc, but I think the delta should be big enough and worth
> a new RFC to gather comments in order to make sure I'm on the right track.
>
> Some more benchmarks were done, for example, some latency related benchmarks
> that I mentioned at LSFMM because I thought responsiveness should be improved
> due to the removal of preempt_disable. Collected more PMU counters as well.
> Please refer to the benchmark section for more details.
>
> Look forward to comments.
>
>
> Changelog
> v2: * Added 3-level and 2-level page table support.
> * Tested with 16K and 64K page size. But we just support 48 bits VA (4K
> and 64K) and 47 bits VA with 16K for now. Please refer to the below
> "known issue" section for the detail reason.
> * Added support for memory hotplug.
> * Added support for KASAN (generic).
> * Treated percpu and local percpu area address as vmalloc address.
> * Fixed build failure for x86.
> * Fixed build failure for !CONFIG_NUMA.
> * Added KASAN support for local percpu area.
> * Some other misc bug fixes found out by Sashiko.
> * More code refactor and cleanup.
> * Regorganized the patches.
> * More benchmarks, refer to benchmark section for more details.
> * Rebased to v7.2-rc1.
>
>
> Introduction
> ============
> This patch series implemented the LSFMM 2026 proposal for optimizing
> this_cpu_*() ops on ARM64. For the details of the proposal, Please refer to:
> https://lore.kernel.org/linux-mm/CAHbLzkpcN-T8MH6=W3jCxcFj1gVZp8fRqe231yzZT-rV_E_org@mail.gmail.com/
> I didn't repeat it in the cover letter because there is no change to the
> proposal.
>
> The series is based on 7.1-rc1. It is basically minimum viable patches.
> There are still a few hacks in this series and it may break something,
> for example, KPTI, SMT machines which shared TLB, etc. But it shoule be
> good enough for now to demonstrate the core idea. The main purpose of the
> RFC is to gather feedback, figure out missing parts and risks, and make sure
> we are on the right track, as well as hopefully it can help the discussion
> for the upcoming LSFMM.
>
> I broke the patches down to arch-dependent and arch-independent parts so that
> hopefully the interested persons can do experiments on other architectures,
> for example, S390, easier.
>
> A new kernel config is introduced, HAVE_LOCAL_PER_CPU_MAP. The architectures
> which can support this feature will select it. Allocating and freeing percpu
> local mapping is protected by this config so that others won't pay the cost.
>
>
> Known Issues
> ============
> 1. KPTI
> -------
> We need determine what CPU we are on, then switch to the right page table.
> Currently arm64 kernel fetches tramp_pg_dir via swapper_pg_dir - fixed_offset,
> and fetches swapper_pg_dir from ttbr1. But ttbr1 may not hold swapper_pg_dir
> anymore except CPU #0. So we need to figure out the other way to handle it.
> Switching to tramp_pg_dir should be easy, but the reverse seems harder because
> tramp_pg_dir just maps the trampoline vectors.
> Maybe we can do two steps switch. Switch to swapper_pg_dir at the first step,
> then switch to per cpu page table (for entry) or tramp page table (for exit).
> Nobody should call this_cpu_*() at either userspace -> kernel entry stage or
> kernel -> userspace exit stage.
>
> 2. SW PAN
> ---------
> Has the similar issue as KPTI. It installs reserved_pg_dir to TTBR0 when running
> in kernel space, but fetching reserved_pg_dir via swapper_pg_dir - fixed_offset.
> Maybe we can save the physical address of swapper_pg_dir in a variable, then load
> it from that variable instead of ttbr1.
>
> 3. Shared TLB machines
> ----------------------
> Some machines may share TLB between CPUs, for example, SMT machines may share
> TLB between the two hardware threads in one core.
> The per cpu page table just can't work with it. Maybe we need a new
> cpufeature to indicate whether per cpu page table is allowed or not. Then
> just enable it for not-shared-TLB machines.
>
> 4. Don't support all VA bits
> ----------------------------
> We just support 48 bits VA (4K and 64K) and 47 bits VA (16K) for now. For 4K
> and 64K, supporting other VA bits is not hard, we just need to determine the
> size for percpu and local percpu area.
> But it is harder for supporting 48 bits VA + 16K page size. We just have two
> top level kernel page table entries with this configuration, but we assume we
> just need to sync up kernel page table at the top level for now. We need to
> sync up kernel page table at the second level in order to support it. I'm not
> sure whether it is worth it or not.
>
>
> Benchmark
> =========
> The benchmarks are done on 160 core AmpereOne machine. The baseline is
> v7.2-rc1 kernel.
>
> 1. Reduction of kernel text size
> --------------------------------
> The patchset can reduce at least 11 instructions for this_cpu_*() ops. Both
> preempt_disable() and preempt_enable() need 4 instructions to manipulate
> the preempt count, and preempt_enable() needs more instructions (compare +
> READ + compare) to determine whether reschedule is needed or not.
> Because this_cpu_*() ops are inlined and called in a lot of places so we
> can save a lot of instructions.
>
> The size of kernel text is reduced by ~184KB with default Fedora kernel
> config. This also helps reduce kernel icache miss rate and stalled frontend
> cycles as kernel build benchmark result showed.
>
> 2. Kernel Build
> ---------------
> Run kernel build (make -j160) with the default Fedora kernel config in a
> memcg.
> 13% - 18% sys time improvment
> 3% - 7% wall time improvement
>
> 5% fewer kernel icache miss, 5% fewer executed kernel instructions and
> 15% fewer stalled frontend cycles for kernel.
>
> 3. stress-ng vm ops
> -------------------
> stress-ng --vm 160 --vm-bytes 128M --vm-ops 100000000
> 8.5% improvement
>
> 4. stress-ng vm ops + fork
> --------------------------
> stress-ng --mmapfork 160 --mmapfork-bytes 128M --mmapfork-ops 500
> 15% improvement
>
> 5. Specjbb
> ----------
> The specjbb test latency curves showed the patched kernel has consistently
> lower p99 latency (the lower the better) than the baseline.
>
> 2.5% improvement on max-jOPS and 4% - 5% improvement on critical-jOPS.
> The specjbb benchmark is quite sensitive to latency and responsiveness,
> particularly critical-jOPS result. The patches are supposed to improve the
> responsiveness due to the reduction of preempt-disabled critical sections.
>
> 6. MySQL
> --------
> 1% - 2% gains on read-only test, 2% - 4% gains on write-only test. Also see
> 15% decrease on frontend cache stall.
tl;dr
Comparing GPR approach [1] with this series gives 8 improvements and 3 regressions.
Fastpath is a Linux kernel performance benchmarking service. The table
compares the both patched kernels: positive values are faster, negative
values are slower, and (I)/(R) indicate statistically significant
improvements/regressions. Unmarked differences are not significant after
accounting for confidence and noise thresholds.
+---------------------------------+--------------------+-----------------+-----------------------+
| Benchmark | percpu-pgtable [2] | gpr-fixup [3] | gpr-fixup [3] |
| | vs baseline [1] | vs baseline [1] | vs percpu-pgtable [2] |
+=================================+====================+=================+=======================+
| lmbench/lat-mem-rd | -0.26% | (I) 1.24% | (I) 1.50% |
| micromm/fork | (I) 1.10% | (I) 2.46% | (I) 1.35% |
| micromm/munmap | (I) 18.10% | (I) 19.78% | (I) 1.43% |
| micromm/vmalloc | (I) 11.92% | (I) 14.68% | (I) 2.47% |
| mmtests/hackbench | 0.27% | (I) 1.09% | 0.81% |
| mmtests/kernbench | (I) 1.19% | (I) 1.18% | -0.01% |
| mmtests/sysbench-cpu | 0.01% | 0.06% | 0.05% |
| mmtests/sysbench-mutex | -0.85% | -0.14% | 0.72% |
| mmtests/sysbench-thread | (R) -4.49% | (I) 3.02% | (I) 7.86% |
| perf/futex | 0.55% | (R) -2.80% | (R) -3.34% |
| perf/sched | (I) 1.61% | -0.39% | (R) -1.97% |
| perf/syscall | (I) 1.54% | 0.98% | -0.55% |
| pts/memtier-benchmark | (I) 2.37% | (I) 2.51% | 0.14% |
| pts/nginx | 0.42% | 0.96% | 0.54% |
| pts/perl-benchmark | (I) 1.83% | (I) 1.63% | -0.20% |
| pts/pgbench | -0.09% | 0.55% | 0.65% |
| pts/pybench | -0.05% | -0.07% | -0.02% |
| pts/redis | -0.00% | -0.04% | -0.04% |
| pts/sqlite-speedtest | 0.15% | 0.44% | 0.28% |
| repro-collection/mysql-workload | 0.73% | 0.25% | -0.48% |
| schbench/thread-contention | 0.42% | (I) 1.04% | 0.61% |
| sockperf/echo-lat-tcp | (I) 2.88% | (I) 1.69% | (R) -1.16% |
| sockperf/echo-lat-udp | (I) 3.93% | (I) 5.11% | (I) 1.14% |
| sockperf/packet-tp-tcp | -0.41% | (I) 1.37% | (I) 1.79% |
| sockperf/packet-tp-udp | (I) 1.45% | (I) 1.49% | 0.04% |
| specjbb/composite | (I) 1.39% | (I) 1.65% | 0.26% |
| speedometer/v2.0 | (R) -1.04% | (R) -1.04% | 0.00% |
| speedometer/v2.1 | -0.10% | 0.10% | 0.20% |
| syscall/getpid | -0.98% | 0.52% | (I) 1.51% |
| syscall/getppid | -0.43% | 0.14% | 0.57% |
| syscall/invalid | (I) 3.06% | (I) 3.56% | 0.48% |
+---------------------------------+--------------------+-----------------+-----------------------+
[1] v7.2
[2] v7.2-percpu-pgtable (this series)
[3] v7.2-percpu-gpr [1] (Based on email review, we fixed SDEI to restore x26
instead of corrupting x22, and adjusted this_cpu_write() helpers to
avoid GCC overflow warnings.)
The constraints/workarounds for the per-CPU page-table series were supplied as
this Kconfig fragment for all the different runs.
CONFIG_EXPERT=y
CONFIG_ARM64_4K_PAGES=y
CONFIG_ARM64_VA_BITS_48=y
CONFIG_ARM64_VA_BITS_39=n
CONFIG_ARM64_VA_BITS_52=n
CONFIG_ARM64_PA_BITS_48=y
CONFIG_ARM64_PA_BITS_52=n
CONFIG_UNMAP_KERNEL_AT_EL0=n
CONFIG_ARM64_SW_TTBR0_PAN=n
CONFIG_KASAN=n
[1] https://lore.kernel.org/all/20260804170503.3513916-19-mark.rutland@arm.com/
>
>
> Regression test
> ===============
> 1. memcg creation
> -----------------
> Create 10K memcgs. Each memcg creation needs to allocate multiple percpu
> variables, for example, percpu refcnt, rstat and objcg percpu refcnt.
>
> Consumed 2112K more virtual memory for percpu “local mapping” and a few
> more mega bytes consumed by per cpu page tables.
> No noticeable regression was found for elapsed time.
>
> 2. fork test
> ------------
> stress-ng --fork 160 --fork-ops 10000000
> fork() needs to allocate multiple percpu variables, for example, rss
> counters and mm_cid_cpu.
>
> Roughly 1% regression was found. However stress-ng fork test has quites
> small address space, the real life workloads typically have much larger
> address space and do more complicated works. The stress-ng mmapfork
> benchmark saw 15% improvement.
>
>
> The organization of patches
> ===========================
> The refactor and prepatory patches (patch 1 - patch 4)
> Percpu page table support patches (patch 5 - patch 8)
> Local percpu area support patches (patch 7 - patch 15)
> Use local percpu area for this_cpu ops (patch 16)
>
>
> Yang Shi (16):
> drivers: arch_numa: move percpu set up code to arch
> arm64: kconfig: make percpu related configs not depend on NUMA
> mm: pgalloc: introduce {pud|pmd}_populate_sync()
> vmalloc: pass in pgd pointer for vmap{__vunmap}_range_noflush()
> arm64: mm: enable percpu kernel page table
> arm64: mm: defined {pud|pmd}_populate_sync()
> arm64: mm: sync percpu page table for memory hotplug/unplug
> arm64: kasan: sync up kasan shadow area page table
> arm64: mm: define percpu virtual space area
> mm: percpu: prepare to use dedicated percpu area
> arm64: mm: map local percpu first chunk
> mm: percpu: set up first chunk and reserve chunk
> arm64: mm: introduce __per_cpu_local_off
> mm: percpu: allocate and free local percpu vm area
> arm64: kconfig: select HAVE_LOCAL_PER_CPU_MAP
> arm64: percpu: use local percpu for this_cpu_*() APIs
>
> arch/arm64/Kconfig | 12 +++++++---
> arch/arm64/include/asm/mmu.h | 5 ++++
> arch/arm64/include/asm/mmu_context.h | 9 +++++++-
> arch/arm64/include/asm/percpu.h | 37 ++++++++++++++++++++++++++++-
> arch/arm64/include/asm/pgalloc.h | 24 +++++++++++++++++++
> arch/arm64/include/asm/pgtable.h | 37 ++++++++++++++++++++++++++---
> arch/arm64/kernel/setup.c | 3 +++
> arch/arm64/kernel/smp.c | 44 +++++++++++++++++++++++++++++++++++
> arch/arm64/mm/kasan_init.c | 47 +++++++++++++++++++++++--------------
> arch/arm64/mm/mmu.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------
> arch/arm64/mm/ptdump.c | 4 ++++
> arch/riscv/kernel/smp.c | 51 ++++++++++++++++++++++++++++++++++++++++
> drivers/base/arch_numa.c | 51 +---------------------------------------
> include/linux/mm.h | 11 +++++++++
> include/linux/percpu.h | 4 +++-
> include/linux/pgalloc.h | 13 +++++++++++
> include/linux/vmalloc.h | 3 +++
> mm/Kconfig | 9 ++++++++
> mm/internal.h | 5 +++-
> mm/kmsan/hooks.c | 14 +++++------
> mm/percpu-internal.h | 14 +++++++++++
> mm/percpu-vm.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> mm/percpu.c | 58 +++++++++++++++++++++++++++++++++++++---------
> mm/sparse-vmemmap.c | 4 ++--
> mm/vmalloc.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------
> 25 files changed, 712 insertions(+), 144 deletions(-)
>
>
> Thanks,
> Yang
>
>
More information about the linux-arm-kernel
mailing list