[PATCH v11 01/14] asm-generic: barrier: Add smp_cond_load_relaxed_timeout()

[Ankur Arora]

Ankur Arora <ankur.a.arora at oracle.com> writes:

> Add smp_cond_load_relaxed_timeout(), which extends
> smp_cond_load_relaxed() to allow waiting for a duration.
>
> We loop around waiting for the condition variable to change while
> peridically doing a time-check. The loop uses cpu_poll_relax() to slow
> down the busy-wait, which, unless overridden by the architecture
> code, amounts to a cpu_relax().
>
> Note that there are two ways for the time-check to fail: the timeout
> case or, @time_expr_ns returning an invalid value (negative or zero).
> The second failure mode allows for clocks attached to the clock-domain
> of @cond_expr --  which might cease to operate meaningfully once some
> state internal to @cond_expr has changed -- to fail.
>
> Evaluation of @time_expr_ns: in the fastpath we want to keep the
> performance close to smp_cond_load_relaxed(). So defer evaluation
> of the potentially costly @time_expr_ns to the slowpath.
>
> This also means that there will always be some hardware dependent
> duration that has passed in cpu_poll_relax() iterations at the time
> of first evaluation. Additionally cpu_poll_relax() is not guaranteed
> to return at timeout boundary. In sum, expect timeout overshoot when
> we exit due to expiration of the timeout.
>
> The number of spin iterations before time-check, SMP_TIMEOUT_POLL_COUNT
> is chosen to be 200 by default. With a cpu_poll_relax() iteration
> taking ~20-30 cycles (measured on a variety of x86 platforms), we
> expect a time-check every ~4000-6000 cycles.
>
> The outer limit of the overshoot is double that when working with the
> parameters above. This might be higher or lower depending on the
> implementation of cpu_poll_relax() across architectures.
>
> Lastly, config option ARCH_HAS_CPU_RELAX indicates availability of a
> cpu_poll_relax() that is cheaper than polling. This might be relevant
> for cases with a long timeout.
>
> Cc: Arnd Bergmann <arnd at arndb.de>
> Cc: Will Deacon <will at kernel.org>
> Cc: Catalin Marinas <catalin.marinas at arm.com>
> Cc: Peter Zijlstra <peterz at infradead.org>
> Cc: linux-arch at vger.kernel.org
> Reviewed-by: Catalin Marinas <catalin.marinas at arm.com>
> Signed-off-by: Ankur Arora <ankur.a.arora at oracle.com>
> ---
> Notes:
>    - add a comment mentioning that smp_cond_load_relaxed_timeout() might
>      be using architectural primitives that don't support MMIO.
>      (David Laight, Catalin Marinas)
>
>  include/asm-generic/barrier.h | 69 +++++++++++++++++++++++++++++++++++
>  1 file changed, 69 insertions(+)
>
> diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
> index d4f581c1e21d..e5a6a1c04649 100644
> --- a/include/asm-generic/barrier.h
> +++ b/include/asm-generic/barrier.h
> @@ -273,6 +273,75 @@ do {									\
>  })
>  #endif
>
> +/*
> + * Number of times we iterate in the loop before doing the time check.
> + * Note that the iteration count assumes that the loop condition is
> + * relatively cheap.
> + */
> +#ifndef SMP_TIMEOUT_POLL_COUNT
> +#define SMP_TIMEOUT_POLL_COUNT		200
> +#endif
> +
> +/*
> + * Platforms with ARCH_HAS_CPU_RELAX have a cpu_poll_relax() implementation
> + * that is expected to be cheaper (lower power) than pure polling.
> + */
> +#ifndef cpu_poll_relax
> +#define cpu_poll_relax(ptr, val, timeout_ns)	cpu_relax()
> +#endif
> +
> +/**
> + * smp_cond_load_relaxed_timeout() - (Spin) wait for cond with no ordering
> + * guarantees until a timeout expires.
> + * @ptr: pointer to the variable to wait on.
> + * @cond_expr: boolean expression to wait for.
> + * @time_expr_ns: expression that evaluates to monotonic time (in ns) or,
> + *  on failure, returns a negative value.
> + * @timeout_ns: timeout value in ns
> + * Both of the above are assumed to be compatible with s64; the signed
> + * value is used to handle the failure case in @time_expr_ns.
> + *
> + * Equivalent to using READ_ONCE() on the condition variable.
> + *
> + * Callers that expect to wait for prolonged durations might want
> + * to take into account the availability of ARCH_HAS_CPU_RELAX.
> + *
> + * Note that @ptr is expected to point to a memory address. Using this
> + * interface with MMIO will be slower (since SMP_TIMEOUT_POLL_COUNT is
> + * tuned for memory) and might also break in interesting architecture
> + * dependent ways.
> + */
> +#ifndef smp_cond_load_relaxed_timeout
> +#define smp_cond_load_relaxed_timeout(ptr, cond_expr,			\
> +				      time_expr_ns, timeout_ns)		\
> +({									\
> +	typeof(ptr) __PTR = (ptr);					\
> +	__unqual_scalar_typeof(*ptr) VAL;				\
> +	u32 __n = 0, __spin = SMP_TIMEOUT_POLL_COUNT;			\
> +	s64 __timeout = (s64)timeout_ns;				\
> +	s64 __time_now, __time_end = 0;					\
> +									\
> +	for (;;) {							\
> +		VAL = READ_ONCE(*__PTR);				\
> +		if (cond_expr)						\
> +			break;						\
> +		cpu_poll_relax(__PTR, VAL, (u64)__timeout);		\
> +		if (++__n < __spin)					\
> +			continue;					\
> +		__time_now = (s64)(time_expr_ns);			\
> +		if (unlikely(__time_end == 0))				\
> +			__time_end = __time_now + __timeout;		\
> +		__timeout = __time_end - __time_now;			\
> +		if (__time_now <= 0 || __timeout <= 0) {		\
> +			VAL = READ_ONCE(*__PTR);			\
> +			break;						\
> +		}							\
> +		__n = 0;						\
> +	}								\
> +	(typeof(*ptr))VAL;						\
> +})
> +#endif
> +

A cluster of issues that got flagged by sashiko was around timeout_ns
being specified as s64 and a bunch of potential edge cases around
that.

These were mostly caused by an implicit assumption in the code that
the timeout specified by the caller is generally reasonable. So, way
below S64_MAX, not 0 etc.

I think this is worth cleaning up a bit. The change is mostly around
introducing a u32 __itertime and explicitly computing the waiting time.
And adding a check to ensure that we start with a valid value.

This does make the implementation a little more involved. So just wanted
to see if people have any opinions on this?

+#ifndef smp_cond_load_relaxed_timeout
+#define smp_cond_load_relaxed_timeout(ptr, cond_expr,          \
+                                     time_expr_ns, timeout_ns) \
+({                                                             \
+       typeof(ptr) __PTR = (ptr);                              \
+       __unqual_scalar_typeof(*(ptr)) VAL;                     \
+       u32 __count = 0, __spin = SMP_TIMEOUT_POLL_COUNT;       \
+       s64 __timeout = (s64)(timeout_ns);                      \
+       s64 __time_now, __time_end = 0;                         \
+       u32 __maybe_unused __itertime;                          \
+                                                               \
+       for (__itertime = NSEC_PER_USEC;                        \
+               VAL = READ_ONCE(*__PTR), __timeout > 0; ) {     \
+               if (cond_expr)                                  \
+                       break;                                  \
+               cpu_poll_relax(__PTR, VAL, __itertime);         \
+               if (++__count < __spin)                         \
+                       continue;                               \
+               __time_now = (s64)(time_expr_ns);               \
+               if (unlikely(__time_end == 0))                  \
+                       __time_end = __time_now + __timeout;    \
+               __timeout = __time_end - __time_now;            \
+               if (__time_now <= 0 || __timeout <= 0) {        \
+                       VAL = READ_ONCE(*__PTR);                \
+                       break;                                  \
+               }                                               \
+               __itertime = __timeout % NSEC_PER_MSEC +        \
+                               NSEC_PER_USEC;                  \
+               __count = 0;                                    \
+       }                                                       \
+       (typeof(*(ptr)))VAL;                                    \
+})
+#endif

Thanks

--
ankur