[RFC PATCH 09/16] parisc: membarrier: Add full memory barrier in switch_mm()

[Mathieu Desnoyers]

The expedited membarrier commands need a full memory barrier after
the scheduler stores to rq->curr, before the CPU returns to
user-space. membarrier() skips the IPI to a CPU which it sees running
another mm. This barrier orders the user-space accesses of the
incoming thread after that check. See scenario (A) in
kernel/sched/membarrier.c.

parisc has no dedicated barrier. It used to get it from
smp_store_release() in finish_task() and from spin_unlock() in
finish_lock_switch(). The first one issued a SYNC instruction. The
second one issued an LDCW instruction, which is strongly ordered.

Both are ordered stores since v5.9:

 - commit 157e9afcc4fa ("Revert "parisc: Revert "Release spinlocks
   using ordered store"""),
 - commit e96ebd589deb ("parisc: Implement __smp_store_release and
   __smp_load_acquire barriers").

Linux runs with the PSW O bit clear. The PA-RISC 2.0 architecture
then allows plain loads and stores to be weakly ordered. It orders an
ordered store before a following ordered load, not before a following
plain load. So an ordered store is not a full barrier.

Nothing else is a full barrier on a switch between two user
processes. switch_mm_irqs_off() only writes control and space
registers.

The PA-RISC 2.0 manual lists the strongly ordered accesses: I/O
space, pages with the TLB U-bit set, LDCW, LDCD, the TLB purges and
SYNC. MTCTL, MTSP and RFI are not among them.

This was found by reading the code and the architecture manual. On a
PA-RISC 2.0 SMP machine, a thread which is scheduled in would run
user-space code without the promised barrier. User-space would see a
memory ordering failure: the caller of an expedited membarrier command
would miss a recent store of that thread. PA-RISC 1.1 is strongly
ordered and not affected.

Do as powerpc and riscv: add membarrier_arch_switch_mm() and call it
from switch_mm_irqs_off() when the mm changes. It issues smp_mb()
only when the next mm registered for an expedited command.

Compile-tested only.

Fixes: 157e9afcc4fa ("Revert "parisc: Revert "Release spinlocks using ordered store""")
Fixes: e96ebd589deb ("parisc: Implement __smp_store_release and __smp_load_acquire barriers")
Cc: stable at vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
---
 arch/parisc/Kconfig                   |  1 +
 arch/parisc/include/asm/membarrier.h  | 33 +++++++++++++++++++++++++++
 arch/parisc/include/asm/mmu_context.h |  2 ++
 3 files changed, 36 insertions(+)
 create mode 100644 arch/parisc/include/asm/membarrier.h

diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig
index 77f67028ad89..3d6b4cc8d6a5 100644
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -13,6 +13,7 @@ config PARISC
 	select ARCH_HAS_DMA_ALLOC if PA11
 	select ARCH_HAS_DMA_OPS
 	select ARCH_HAS_ELF_RANDOMIZE
+	select ARCH_HAS_MEMBARRIER_CALLBACKS
 	select ARCH_HAS_STRICT_KERNEL_RWX
 	select ARCH_HAS_STRICT_MODULE_RWX
 	select ARCH_HAS_UBSAN
diff --git a/arch/parisc/include/asm/membarrier.h b/arch/parisc/include/asm/membarrier.h
new file mode 100644
index 000000000000..822dcadaf9bb
--- /dev/null
+++ b/arch/parisc/include/asm/membarrier.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_PARISC_MEMBARRIER_H
+#define _ASM_PARISC_MEMBARRIER_H
+
+static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
+					     struct mm_struct *next,
+					     struct task_struct *tsk)
+{
+	/*
+	 * Only need the full barrier when switching between processes.
+	 * Barrier when switching from kernel to userspace is not
+	 * required here, given that it is implied by mmdrop(). Barrier
+	 * when switching from userspace to kernel is not needed after
+	 * store to rq->curr.
+	 */
+	if (IS_ENABLED(CONFIG_SMP) &&
+	    likely(!(atomic_read(&next->membarrier_state) &
+		     (MEMBARRIER_STATE_PRIVATE_EXPEDITED |
+		      MEMBARRIER_STATE_GLOBAL_EXPEDITED))))
+		return;
+
+	/*
+	 * The membarrier system call requires a full memory barrier
+	 * after storing to rq->curr, before going back to user-space.
+	 *
+	 * Nothing else provides it on this path. spin_unlock() and
+	 * smp_store_release() are ordered stores. PA-RISC 2.0 does not
+	 * order an ordered store with a following weakly ordered load.
+	 */
+	smp_mb();
+}
+
+#endif /* _ASM_PARISC_MEMBARRIER_H */
diff --git a/arch/parisc/include/asm/mmu_context.h b/arch/parisc/include/asm/mmu_context.h
index c9187fe836a3..42ff1b9444f1 100644
--- a/arch/parisc/include/asm/mmu_context.h
+++ b/arch/parisc/include/asm/mmu_context.h
@@ -4,6 +4,7 @@
 
 #include <linux/mm.h>
 #include <linux/sched.h>
+#include <linux/sched/mm.h>
 #include <linux/atomic.h>
 #include <linux/spinlock.h>
 #include <asm-generic/mm_hooks.h>
@@ -51,6 +52,7 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev,
 		struct mm_struct *next, struct task_struct *tsk)
 {
 	if (prev != next) {
+		membarrier_arch_switch_mm(prev, next, tsk);
 #ifdef CONFIG_TLB_PTLOCK
 		/* put physical address of page_table_lock in cr28 (tr4)
 		   for TLB faults */
-- 
2.47.3