[RFC v2 PATCH 0/16] Optimize this_cpu_*() ops for non-x86 (ARM64 for this series)

[Yang Shi]

On 7/22/26 2:36 AM, Mark Rutland wrote:
> On Tue, Jul 21, 2026 at 04:20:50PM -0700, Yang Shi wrote:
>> On 7/16/26 6:23 AM, Ryan Roberts wrote:
>>> On 15/07/2026 19:04, Yang Shi wrote:
>>> But I think that there are other issues with this concept that likely block it
>>> from being accepted upstream; see below...
>>> We have observed a few performance regressions recently, for which the root
>>> cause is increased use of this_cpu_*. We have somebody at Arm about to start an
>>> investigation into whether in-kernel rseq can solve the problem.
>> It is good to know someone is exploring the alternative path. But I doubt
>> restart sequence can really boost the performance. It needs check whether
>> preemption (or cpu migration) happens in the middle of percpu operations
>> then manipulate the ip to restart the operation.
> With the rseq approach, that check happens out-of-line in exception
> handling code, *only* when preemption occurs, so in the fast path there
> is *no code whatsoever* to check for preemption/migration, and no
> branches to cause a restart.
>
> Please note that with the approach I've prototyped, the exception
> handling code doesn't even adjust the PC; we just adjust a GPR
> containing the address. The forward progress guarantees (and behaviour
> under contention) should be the same as a single instruction. The fixup
> logic itself is trivial.
>
> Please see my message where I outlined that approach in detail:
>
>    https://lore.kernel.org/linux-arm-kernel/al_DpFJFcmVhxpvW@J2N7QTR9R3/
>
> I expect that should come with a reasonable benefit, but I don't have
> benchmark figures yet as I haven't finished converting the xchg and
> cmpxchg implementations.

Yes, but it still needs prologue and epilogue which consumes 4 
instructions in every inlined this_cpu ops if I read it correctly. I 
agree it is still a very good improvement. I just tried to compare it 
with my approach. My approach doesn't need prologue and epilogue, and 
the fixup in exception handling code as well.

>
>> It sounds like it just moved the cost from one place to the other
>> place and it also seems hacky TBH.
> The existing (high) cost is removed from the fast path. A new (low) cost
> is added to the rare path where we perform an exception return into the
> middle of a critical section.
>
> I don't think this is any more hacky than percpu page tables. It's far
> more contained to the relevant pieces of code (only the this_cpu_*() ops
> and exception entry/return).
>
>> It may reduce the preempt_disable/preempt_enable cost somehow for some
>> less contended cases, but it may be worse for high contended cases.
>> The percpu page table can work well for both.
> It entirely removes the preempt_disable/preempt_enable cost.
>
> I don't follow your concern with contention. Any contention on the
> memory location is going to be independent of preemption, and if a
> thread is preempted mid-operation, the impact is going to be the same
> with either the adress fixup or percpu page tables -- I don't think you
> avoid that.

Contention means this_cpu ops is preempted and migrated to other CPUs 
quite often. If so the fix up cost may be high because you need to 
reload percpu offset and recalculate the pointer every time, at least 
extra two instructions. This case may be rare, but the percpu page table 
approach doesn't need to worry about the worst case scenario at all. 
Manipulating the local percpu counter is guaranteed by page table no 
matter how often preemption and migration happen.

>
>> S390 implemented restart sequence. Heiko didn't share too much benchmark
>> data, just confirmed there is no regression and I had a lengthy discussion
>> with him
>> (https://lore.kernel.org/lkml/20260520092243.264847-1-hca@linux.ibm.com/).
> As above, for s390 Heiko didn't implement a restart sequence; he
> implemented an address fixup sequence. There is no restart.
>
> The discussion you had there seemed to focus on the s390 implementation
> details (e.g. why he used an address fixup rather than a restart), and
> the benefits of removing the preempt_count manipulation (which all the
> approaches, including mine, do).
>
> Am I missing some key detail?

I was trying to understand S390's implementation so that I can compare 
to the percpu page table approach. As we discussed above, both S390 and 
your proposal does reduce the cost, but not remove the cost. This is the 
key detail.

I also admit fixup sequence approach is more simple and self-contained 
than percpu page table. So we were also seeking other usecases for 
percpu page table so that it can be worth the complexity.

>
>> Improving this_cpu performance is just one of the usecase of percpu page
>> table. The other potential usecase is kernel text replication I mentioned at
>> LSFMM. The idea is not new, someone else has explored it, see https://lore.kernel.org/linux-arm-kernel/ZMKNYEkM7YnrDtOt@shell.armlinux.org.uk/.
> That was looked at in the past, but people weren't keen given the
> complexity it would lead to, which is why it never saw much traction.

Some our customers are interested in this. Russell's 2023 RFC said "the 
performance results from kernel text replication are workload specific, 
but appear to show a gain of between 6% and 17% for database-centric 
like workloads. When combined with userspace awareness of NUMA, this can 
result in a gain of over 50%.". The performance gain sounds impressive.

Brendan Jackman also mentioned percpu page table could make his "address 
space isolation" work better at LSFMM, but I have not explored it too 
much yet.

Multiple usercases may make percpu page table more convincing?

>
>> Percpu page table can make kernel support it more naturally.
> Perhaps, but the only difference is that you've moving the support code
> into the core kernel mm code. I think the general concerns people have
> with percpu page tables still apply.
>
> [...]
>
>>>> Known Issues
>>>> ============
>>>> 3. Shared TLB machines
>>>> ----------------------
>>>> Some machines may share TLB between CPUs, for example, SMT machines may share
>>>> TLB between the two hardware threads in one core.
>>>> The per cpu page table just can't work with it. Maybe we need a new
>>>> cpufeature to indicate whether per cpu page table is allowed or not. Then
>>>> just enable it for not-shared-TLB machines.
>>> The architectural feature you're referring to here is FEAT_TTCNP (common not
>>> private). There are many CPUs out there that use this feature (not just those
>>> that support SMT).
>>>
>>> FEAT_TTCNP is mandatory from Armv8.2 and the presence of the feature only tells
>>> you whether you can legally set the CnP bit - it doesn't actually tell you if
>>> the HW does any sharing - there is no way to detect this. So making per-cpu
>>> pgtables mutually exclusive with CNP is a non-starter as all CPUs from v8.2
>>> onwards advertise CNP.
>>>
>>> Even ignoring the CNP problem, you'll end up installing the per-cpu TLB entries
>>> tagged with whatever user ASID happens to be installed at the time. Which means
>>> you could end up with lots of duplicated TLB entries all differing only by ASID.
>>> It's probably not the end of the world, but it seems... inefficient.
>> Really? I thought ASID will be ignored if nG == 0.
> The nG bit only applies to leaf entries (Page or Block descriptors). All
> intermediate entries (Table descriptors) are ASID tagged.

OK, good to know it. But I still don't quite understand how come percpu 
page table can cause duplicated TLB entries with different ASIDs. 
Doesn't the current kernel (one single kernel page table) have ASID 
tagged intermediate entries too?

>
>>> The only way I can see to make this work fully is to rely on FEAT_ASID2, which
>>> allows specifying separate ASIDs for TTBR0 and TTBR1. Then each CPU can have
>>> it's own (permanently installed) ASID for local mappings in TTBR1. That would
>>> solve the CNP issue. FEAT_ASID2 is optional from v9.4 and mandatory from v9.5.
>> I agree it is safer to handle CNP problem by depending on FEAT_ASID2.
> I think that depending on FEAT_ASID2 is a non-starter.
>
> There is no extant hardware with FEAT_ASID2, and I don't think we want
> to maintain distinct versions of hte percpu code accross architecture
> versions.

I agree it is not perfect to have different version of percpu code by 
depending on architecture versions. But AFAICT percpu is not the first 
one. We already have KPTI, SW PAN, bbml2_noabort, etc.

Thanks,
Yang

>
> Mark.