[PATCH v5 3/5] riscv: Optimize smp_cond_load_acquire() with Zalasr

[Qingwei Hu]

From: Yunhui Cui <cuiyunhui at bytedance.com>

The generic smp_cond_load_acquire() implementation spins with relaxed
loads and applies acquire ordering after the condition is satisfied.
With Zalasr, each iteration can use an acquire load instead. The final
satisfying load then provides the required ordering.

Use the Zalasr path only when the extension is available. Keep the
generic relaxed loop followed by smp_acquire__after_ctrl_dep() as the
fallback, and retain Zawrs waiting when configured. Evaluate the
original pointer once and avoid shadowing the helper's private variable.

Signed-off-by: Yunhui Cui <cuiyunhui at bytedance.com>
Co-developed-by: Qingwei Hu <qingwei.hu at bytedance.com>
Signed-off-by: Qingwei Hu <qingwei.hu at bytedance.com>
---
 arch/riscv/include/asm/barrier.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/arch/riscv/include/asm/barrier.h b/arch/riscv/include/asm/barrier.h
index d2200e826ec00..b37ae9544d08e 100644
--- a/arch/riscv/include/asm/barrier.h
+++ b/arch/riscv/include/asm/barrier.h
@@ -142,6 +142,29 @@ do {									\
 	}								\
 	(typeof(*ptr))VAL;						\
 })
+#define __smp_cond_load_wait(ptr, val)	__cmpwait_relaxed(ptr, val)
+#else
+#define __smp_cond_load_wait(...)	cpu_relax()
+#endif
+
+#ifdef CONFIG_SMP
+#define smp_cond_load_acquire(ptr, cond_expr) ({			\
+	__auto_type __zalasr_ptr = (ptr);				\
+	__unqual_scalar_typeof(*__zalasr_ptr) VAL;			\
+	if (riscv_has_extension_unlikely(RISCV_ISA_EXT_ZALASR)) {	\
+		for (;;) {						\
+			/* The satisfying load provides acquire ordering. */ \
+			VAL = smp_load_acquire(__zalasr_ptr);		\
+			if (cond_expr)					\
+				break;					\
+			__smp_cond_load_wait(__zalasr_ptr, VAL);		\
+		}							\
+	} else {							\
+		VAL = smp_cond_load_relaxed(__zalasr_ptr, cond_expr);	\
+		smp_acquire__after_ctrl_dep();				\
+	}								\
+	(typeof(*__zalasr_ptr))VAL;					\
+})
 #endif
 
 #include <asm-generic/barrier.h>
-- 
2.39.5