Overhead of arm64 LSE per-CPU atomics?

[Catalin Marinas]

On Wed, Nov 05, 2025 at 09:40:32AM -0800, Paul E. McKenney wrote:
> On Wed, Nov 05, 2025 at 05:15:51PM +0000, Catalin Marinas wrote:
> > On Wed, Nov 05, 2025 at 08:25:51AM -0800, Paul E. McKenney wrote:
> > > On Wed, Nov 05, 2025 at 03:34:21PM +0000, Catalin Marinas wrote:
> > > > Given that this_cpu_*() are meant for the local CPU, there's less risk
> > > > of cache line bouncing between CPUs, so I'm happy to change them to
> > > > either use PRFM or LDADD (I think I prefer the latter). This would not
> > > > be a generic change for the other atomics, only the per-CPU ones.
> > > 
> > > I have easy access to only the one type of ARM system, and of course
> > > the choice must be driven by a wide range of systems.  But yes, it
> > > would be much better if we can just use this_cpu_inc().  I will use the
> > > non-atomics protected by interrupt disabling in the meantime, but look
> > > forward to being able to switch back.
> > 
> > BTW, did you find a problem with this_cpu_inc() in normal use with SRCU
> > or just in a microbenchmark hammering them? From what I understand from
> > the hardware folk, doing STADD in a loop saturates some queues in the
> > interconnect and slows down eventually. In normal use, it's just a
> > posted operation not affecting the subsequent instructions (or at least
> > that's the theory).
> 
> Only in a microbenchmark, and Breno did not find any issues in larger
> benchmarks, so good to hear!
> 
> Now, some non-arm64 systems deal with it just fine, but perhaps I owe
> everyone an apology for the firedrill.

That was a useful exercise, I learnt more things about the arm atomics.

> But let me put it this way...  Would you ack an SRCU patch that resulted
> in 100ns microbenchmark numbers on arm64 compared to <2ns numbers on
> other systems?

Only if it's backed by other microbenchmarks showing significant
improvements ;).

I think we should change the percpu atomics, it makes more sense to do
them near, but I'll keep the others as they are. Planning to post a
proper patch tomorrow and see if Will NAKs it ;) (I've been in meetings
all day). Something like below but with more comments and a commit log:

------------------------8<--------------------------
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 9abcc8ef3087..d4dff4b0cf50 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -77,7 +77,7 @@ __percpu_##name##_case_##sz(void *ptr, unsigned long val)		\
 	"	stxr" #sfx "\t%w[loop], %" #w "[tmp], %[ptr]\n"		\
 	"	cbnz	%w[loop], 1b",					\
 	/* LSE atomics */						\
-		#op_lse "\t%" #w "[val], %[ptr]\n"			\
+		#op_lse "\t%" #w "[val], %" #w "[tmp], %[ptr]\n"	\
 		__nops(3))						\
 	: [loop] "=&r" (loop), [tmp] "=&r" (tmp),			\
 	  [ptr] "+Q"(*(u##sz *)ptr)					\
@@ -124,9 +124,9 @@ PERCPU_RW_OPS(8)
 PERCPU_RW_OPS(16)
 PERCPU_RW_OPS(32)
 PERCPU_RW_OPS(64)
-PERCPU_OP(add, add, stadd)
-PERCPU_OP(andnot, bic, stclr)
-PERCPU_OP(or, orr, stset)
+PERCPU_OP(add, add, ldadd)
+PERCPU_OP(andnot, bic, ldclr)
+PERCPU_OP(or, orr, ldset)
 PERCPU_RET_OP(add, add, ldadd)
 
 #undef PERCPU_RW_OPS