[PATCH v15 06/16] asm-generic: barrier: Add smp_cond_load_acquire_timeout()

[Ankur Arora]

Add the acquire variant of smp_cond_load_relaxed_timeout().

smp_cond_load_acquire_timeout() reuses the relaxed variant for
the actual wait. This has two paths out:

 C1. "if (cond_expr)": loop condition evaluates to true
 C2. "if (__scl_time_now <= 0 || __scl_timeout <= 0)": timeout case
     a. cond_expr evaluates to false
     b. cond_expr evaluates to true

C1 already provides LOAD->STORE order via the control-dependency.
C2b does not. So re-evaluate the "if (cond_expr)" branch in
smp_cond_load_acquire_timeout() to provide that, and follow that
with smp_acquire__after_ctrl_dep() for the additional LOAD->LOAD
order, together providing the full load-acquire order.

For the pure timeout case (C2a), we have neither, so just go
with a straight smp_load_acquire().

Cc: Kumar Kartikeya Dwivedi <memxor at gmail.com>
Cc: Alexei Starovoitov <ast at kernel.org>
Cc: Arnd Bergmann <arnd at arndb.de>
Cc: Will Deacon <will at kernel.org>
Cc: Catalin Marinas <catalin.marinas at arm.com>
Cc: Peter Zijlstra <peterz at infradead.org>
Cc: linux-arch at vger.kernel.org
Cc: bpf at vger.kernel.org
Reviewed-by: Catalin Marinas <catalin.marinas at arm.com>
Reviewed-by: Haris Okanovic <harisokn at amazon.com>
Tested-by: Haris Okanovic <harisokn at amazon.com>
Signed-off-by: Ankur Arora <ankur.a.arora at oracle.com>
---
Notes:

   In earlier revisions sashiko comments pointed out:
    "Does this standalone if block fail to extend a control dependency to
     the macro caller's subsequent code?"

   This was a problem for the timeout case where the "if (cond_expr())"
   was missing.

   My solution in earlier versions was to not provide acquire ordering
   on timeout -- that looked okay for BPF and we have similar semantics
   in other places as well (ex. try_page_mte_tagging()).

   On second thoughts, those semantics were unnecessary special for a
   non-performance path. So this revision just provides a full
   smp_load_acquire() in the failure path.

   Also update the comment in smp_cond_load_acquire_timeout() to describe
   the barrier semantics for the three cases.

   Catalin, Haris: I've retained both your R-bys. Hope that's okay.

 include/asm-generic/barrier.h | 48 +++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
index 4437d27c46b9..81db4da12f5b 100644
--- a/include/asm-generic/barrier.h
+++ b/include/asm-generic/barrier.h
@@ -379,6 +379,54 @@ do {									\
 })
 #endif
 
+/**
+ * smp_cond_load_acquire_timeout() - (Spin) wait for cond with ACQUIRE ordering
+ * until a timeout expires.
+ * @ptr: pointer to the variable to wait on.
+ * @cond_expr: boolean expression to wait for.
+ * @time_expr_ns: monotonic expression that evaluates to time in ns or,
+ *  on failure, returns a negative value.
+ * @timeout_ns: timeout value in ns
+ * (Both of the above are assumed to be compatible with s64.)
+ *
+ * Equivalent to using smp_cond_load_acquire() on the condition variable with
+ * a timeout.
+ */
+#ifndef smp_cond_load_acquire_timeout
+#define smp_cond_load_acquire_timeout(ptr, cond_expr,			\
+				      time_expr_ns, timeout_ns)		\
+({									\
+	__unqual_scalar_typeof(*(ptr)) VAL;				\
+	VAL = smp_cond_load_relaxed_timeout(ptr, cond_expr,		\
+					     time_expr_ns,		\
+					     timeout_ns);		\
+	/*								\
+	 * We arrive here once the loop condition is hit, on timeout,	\
+	 * or, if we hit both the timeout and the loop condition.	\
+	 *								\
+	 * For the first case, we come here having already evaluated	\
+	 * the control dependency.					\
+	 * In the last case -- low probability, possible in the last	\
+	 * iteration, especially on architectures with waiting		\
+	 * cpu_poll_relax() -- the control dependency has not been	\
+	 * evaluated.							\
+	 *								\
+	 * So, force it to be re-evaluated before			\
+	 * smp_acquire__after_ctrl_dep() to provide ACQUIRE ordering	\
+	 * for both.							\
+	 *								\
+	 * The other case is of pure timeout, where again we don't have \
+	 * the advantage of having the control dependency. Given that	\
+	 * this is the slowpath, we go with a full smp_load_acquire().	\
+	 */								\
+	if (cond_expr)							\
+		smp_acquire__after_ctrl_dep();				\
+	else								\
+		VAL = smp_load_acquire(ptr);				\
+	(typeof(*(ptr)))VAL;						\
+})
+#endif
+
 /*
  * pmem_wmb() ensures that all stores for which the modification
  * are written to persistent storage by preceding instructions have
-- 
2.43.7