Overhead of arm64 LSE per-CPU atomics?

[Paul E. McKenney]

On Wed, Nov 05, 2025 at 07:16:42PM +0000, Catalin Marinas wrote:
> On Wed, Nov 05, 2025 at 09:40:32AM -0800, Paul E. McKenney wrote:
> > On Wed, Nov 05, 2025 at 05:15:51PM +0000, Catalin Marinas wrote:
> > > On Wed, Nov 05, 2025 at 08:25:51AM -0800, Paul E. McKenney wrote:
> > > > On Wed, Nov 05, 2025 at 03:34:21PM +0000, Catalin Marinas wrote:
> > > > > Given that this_cpu_*() are meant for the local CPU, there's less risk
> > > > > of cache line bouncing between CPUs, so I'm happy to change them to
> > > > > either use PRFM or LDADD (I think I prefer the latter). This would not
> > > > > be a generic change for the other atomics, only the per-CPU ones.
> > > > 
> > > > I have easy access to only the one type of ARM system, and of course
> > > > the choice must be driven by a wide range of systems.  But yes, it
> > > > would be much better if we can just use this_cpu_inc().  I will use the
> > > > non-atomics protected by interrupt disabling in the meantime, but look
> > > > forward to being able to switch back.
> > > 
> > > BTW, did you find a problem with this_cpu_inc() in normal use with SRCU
> > > or just in a microbenchmark hammering them? From what I understand from
> > > the hardware folk, doing STADD in a loop saturates some queues in the
> > > interconnect and slows down eventually. In normal use, it's just a
> > > posted operation not affecting the subsequent instructions (or at least
> > > that's the theory).
> > 
> > Only in a microbenchmark, and Breno did not find any issues in larger
> > benchmarks, so good to hear!
> > 
> > Now, some non-arm64 systems deal with it just fine, but perhaps I owe
> > everyone an apology for the firedrill.
> 
> That was a useful exercise, I learnt more things about the arm atomics.

I am glad that it had some good effect.  ;-)

> > But let me put it this way...  Would you ack an SRCU patch that resulted
> > in 100ns microbenchmark numbers on arm64 compared to <2ns numbers on
> > other systems?
> 
> Only if it's backed by other microbenchmarks showing significant
> improvements ;).

Well, it did reduce from about 140ns with SRCU to about 100ns with
SRCU-fast-updown due to removing the full memory barrier, so there
is that.

> I think we should change the percpu atomics, it makes more sense to do
> them near, but I'll keep the others as they are. Planning to post a
> proper patch tomorrow and see if Will NAKs it ;) (I've been in meetings
> all day). Something like below but with more comments and a commit log:

I do like fixing this in the arm64 percpu atomics.  However, this gets
me assembler errors, perhaps because I am using an old version of gcc
or perhaps because I am still based off of v6.17-rc1:

	/tmp/ccYlMkU1.s: Assembler messages:
	/tmp/ccYlMkU1.s:9292: Error: invalid addressing mode at operand 2 -- `stadd x2,x4,[x0]'
	/tmp/ccYlMkU1.s:9428: Error: invalid addressing mode at operand 2 -- `stadd x3,x6,[x1]'
	/tmp/ccYlMkU1.s:9299: Error: attempt to move .org backwards
	/tmp/ccYlMkU1.s:9435: Error: attempt to move .org backwards

							Thanx, Paul

> ------------------------8<--------------------------
> diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
> index 9abcc8ef3087..d4dff4b0cf50 100644
> --- a/arch/arm64/include/asm/percpu.h
> +++ b/arch/arm64/include/asm/percpu.h
> @@ -77,7 +77,7 @@ __percpu_##name##_case_##sz(void *ptr, unsigned long val)		\
>  	"	stxr" #sfx "\t%w[loop], %" #w "[tmp], %[ptr]\n"		\
>  	"	cbnz	%w[loop], 1b",					\
>  	/* LSE atomics */						\
> -		#op_lse "\t%" #w "[val], %[ptr]\n"			\
> +		#op_lse "\t%" #w "[val], %" #w "[tmp], %[ptr]\n"	\
>  		__nops(3))						\
>  	: [loop] "=&r" (loop), [tmp] "=&r" (tmp),			\
>  	  [ptr] "+Q"(*(u##sz *)ptr)					\
> @@ -124,9 +124,9 @@ PERCPU_RW_OPS(8)
>  PERCPU_RW_OPS(16)
>  PERCPU_RW_OPS(32)
>  PERCPU_RW_OPS(64)
> -PERCPU_OP(add, add, stadd)
> -PERCPU_OP(andnot, bic, stclr)
> -PERCPU_OP(or, orr, stset)
> +PERCPU_OP(add, add, ldadd)
> +PERCPU_OP(andnot, bic, ldclr)
> +PERCPU_OP(or, orr, ldset)
>  PERCPU_RET_OP(add, add, ldadd)
>  
>  #undef PERCPU_RW_OPS
>