[RFC PATCH 08/16] LoongArch: membarrier: Add full memory barrier in switch_mm()

[Mathieu Desnoyers]

The expedited membarrier commands need a full memory barrier after
the scheduler stores to rq->curr, before the CPU returns to
user-space. membarrier() skips the IPI to a CPU which it sees running
another mm. This barrier orders the user-space accesses of the
incoming thread after that check. See scenario (A) in
kernel/sched/membarrier.c.

LoongArch has no dedicated barrier. It used to get it from
spin_unlock() in finish_lock_switch(), which was a full barrier
("dbar 0").

This is no longer true. Commit e031a5f3f1ed ("LoongArch: Support dbar
with different hints") made smp_store_release() an ordering barrier
("dbar 0b10010"). It does not order a previous store with a following
load. That commit kept a full barrier in queued_spin_unlock(). Commit
9c68ece8b2a5 ("LoongArch: Define the __io_aw() hook as mmiowb()") then
removed it.

Nothing else is a full barrier on a switch between two user
processes. switch_mm_irqs_off() writes CSRs and sets a bit in the mm
cpumask with a plain "amor". finish_task() uses smp_store_release().

The LoongArch manual lists the instructions which order memory
accesses: DBAR, IBAR, the AM*_DB instructions and LL/SC pairs. A CSR
write and ERTN are not among them.

This was found by reading the code and the architecture manual. On
Loongson-3A6000 or newer, a thread which is scheduled in would run
user-space code without the barrier which membarrier promised to the
caller. User-space would see a memory ordering failure: the caller of
an expedited membarrier command would miss a recent store of that
thread. Older processors treat every dbar hint as "dbar 0" and are not
affected.

Do as powerpc and riscv: add membarrier_arch_switch_mm() and call it
from switch_mm_irqs_off() when the mm changes. It issues smp_mb()
only when the next mm registered for an expedited command.

Compile-tested only.

Fixes: 9c68ece8b2a5 ("LoongArch: Define the __io_aw() hook as mmiowb()")
Cc: stable at vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
---
 arch/loongarch/Kconfig                   |  1 +
 arch/loongarch/include/asm/membarrier.h  | 33 ++++++++++++++++++++++++
 arch/loongarch/include/asm/mmu_context.h |  4 +++
 3 files changed, 38 insertions(+)
 create mode 100644 arch/loongarch/include/asm/membarrier.h

diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig
index b5ed4881d3c0..be6e7545d646 100644
--- a/arch/loongarch/Kconfig
+++ b/arch/loongarch/Kconfig
@@ -22,6 +22,7 @@ config LOONGARCH
 	select ARCH_HAS_FORTIFY_SOURCE
 	select ARCH_HAS_KCOV
 	select ARCH_HAS_KERNEL_FPU_SUPPORT if 64BIT && CPU_HAS_FPU
+	select ARCH_HAS_MEMBARRIER_CALLBACKS
 	select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
 	select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
 	select ARCH_HAS_PREEMPT_LAZY
diff --git a/arch/loongarch/include/asm/membarrier.h b/arch/loongarch/include/asm/membarrier.h
new file mode 100644
index 000000000000..1789f878821f
--- /dev/null
+++ b/arch/loongarch/include/asm/membarrier.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_LOONGARCH_MEMBARRIER_H
+#define _ASM_LOONGARCH_MEMBARRIER_H
+
+static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
+					     struct mm_struct *next,
+					     struct task_struct *tsk)
+{
+	/*
+	 * Only need the full barrier when switching between processes.
+	 * Barrier when switching from kernel to userspace is not
+	 * required here, given that it is implied by mmdrop(). Barrier
+	 * when switching from userspace to kernel is not needed after
+	 * store to rq->curr.
+	 */
+	if (IS_ENABLED(CONFIG_SMP) &&
+	    likely(!(atomic_read(&next->membarrier_state) &
+		     (MEMBARRIER_STATE_PRIVATE_EXPEDITED |
+		      MEMBARRIER_STATE_GLOBAL_EXPEDITED))))
+		return;
+
+	/*
+	 * The membarrier system call requires a full memory barrier
+	 * after storing to rq->curr, before going back to user-space.
+	 *
+	 * Nothing else provides it on this path. spin_unlock() and
+	 * smp_store_release() are ordering barriers ("dbar 0b10010"):
+	 * they do not order a previous store with a following load.
+	 */
+	smp_mb();
+}
+
+#endif /* _ASM_LOONGARCH_MEMBARRIER_H */
diff --git a/arch/loongarch/include/asm/mmu_context.h b/arch/loongarch/include/asm/mmu_context.h
index 304363bd3935..dba62dd7a131 100644
--- a/arch/loongarch/include/asm/mmu_context.h
+++ b/arch/loongarch/include/asm/mmu_context.h
@@ -9,6 +9,7 @@
 
 #include <linux/errno.h>
 #include <linux/sched.h>
+#include <linux/sched/mm.h>
 #include <linux/mm_types.h>
 #include <linux/smp.h>
 #include <linux/slab.h>
@@ -91,6 +92,9 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *
 	bool need_flush = false;
 	unsigned int cpu = smp_processor_id();
 
+	if (prev != next)
+		membarrier_arch_switch_mm(prev, next, tsk);
+
 	/* Check if our ASID is of an older version and thus invalid */
 	if (!asid_valid(next, cpu))
 		get_new_mmu_context(next, cpu, &need_flush);
-- 
2.47.3