[RFC PATCH 01/16] sched/membarrier: Fix stale runqueue state after a lazy mm switch

[Mathieu Desnoyers]

MEMBARRIER_CMD_GLOBAL_EXPEDITED sends an IPI to each CPU which runs
a registered process. It finds those CPUs with rq->membarrier_state,
a per-runqueue copy of the membarrier state of the mm.

Commit 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state
racy load") added the copy. The scheduler updates it when it switches
between two mm. A registration sends an IPI to update the runqueues
which keep running the mm.

membarrier_switch_mm() compares the previous active_mm with the next
mm. So it also skips the update when the previous task only borrowed
the mm. This is a kernel thread in lazy TLB mode, or a task past
exit_mm(). The copy can be stale in two such cases.

Case 1: a process registers while a lazy TLB kernel thread borrows
its mm on another CPU. T is a thread of a process with mm M, which is
not registered yet. S is a thread of another process.

  CPU 0                                        CPU 1

  T runs in user-space
    rq->membarrier_state == 0
  T migrates to CPU 1
  context_switch(T -> idle)
    idle borrows M as its active_mm
                                               context_switch(-> T)
                                               T: MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED
                                                 sets the state of M
                                                 sets rq->membarrier_state of CPU 1
                                                 CPU 0 runs idle: no IPI
                                               T migrates to CPU 0
  context_switch(idle -> T)
    membarrier_switch_mm(rq, idle->active_mm, T->mm)
      prev_mm == next_mm: no update
  T runs in user-space
    rq->membarrier_state == 0 (stale)
                                               S: MEMBARRIER_CMD_GLOBAL_EXPEDITED
                                                 CPU 0: state not set, no IPI

Case 2: a thread exits and the CPU then runs another thread of the
same process. A and B are threads of a registered process with mm M.
S is a thread of another process.

  CPU 0                                        CPU 1

  B exits
  exit_mm()
    B->mm = NULL
    membarrier_update_current_mm(NULL)
      rq->membarrier_state = 0
  context_switch(B -> A)
    membarrier_switch_mm(rq, B->active_mm, A->mm)
      prev_mm == next_mm: no update
  A runs in user-space
    rq->membarrier_state == 0 (stale)
                                               S: MEMBARRIER_CMD_GLOBAL_EXPEDITED
                                                 CPU 0: state not set, no IPI

kthread_unuse_mm() clears the copy in the same way as exit_mm().

In both cases, the registered process runs without the promised
memory barrier. This lasts until the CPU switches to the process from
another mm.

User-space sees this as a memory ordering failure. The registered
processes omit memory barriers from their fast paths. The caller
issues MEMBARRIER_CMD_GLOBAL_EXPEDITED where it needs those barriers.
An example is RCU-like synchronization over shared memory. Without the
IPI, the caller would miss a recent store of a registered process. It
would then, for example, free memory which that process still uses.
A thread exit in a registered multi-threaded process is enough to
trigger case 2.

Both cases were reproduced on v7.4-rc2: 1000 commands send no IPI to
the registered process.

Fix this by passing prev->mm rather than prev->active_mm. It is NULL
for a kernel thread and for an exited task, which forces the update.
The update is still skipped between two user tasks of the same mm.

Fixes: 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state racy load")
Cc: stable at vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
---
 kernel/sched/core.c  |  6 +++++-
 kernel/sched/sched.h | 15 +++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a088d4576f5..df6d15cf7109 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5551,7 +5551,11 @@ context_switch(struct rq *rq, struct task_struct *prev,
 		else
 			prev->active_mm = NULL;
 	} else {					// to user
-		membarrier_switch_mm(rq, prev->active_mm, next->mm);
+		/*
+		 * Pass prev->mm, not prev->active_mm: see
+		 * membarrier_switch_mm().
+		 */
+		membarrier_switch_mm(rq, prev->mm, next->mm);
 		/*
 		 * sys_membarrier() requires an smp_mb() between setting
 		 * rq->curr / membarrier_switch_mm() and returning to userspace.
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1d76950dbb0a..68353796a7f9 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3907,6 +3907,21 @@ static inline bool sched_energy_enabled(void) { return false; }
  * - prior user-space memory accesses and store to rq->membarrier_state,
  * - store to rq->membarrier_state and following user-space memory accesses.
  * In the same way it provides those guarantees around store to rq->curr.
+ *
+ * @prev_mm is the mm of the previous task. It is NULL for a kernel
+ * thread and for a task past exit_mm().
+ *
+ * The update is skipped only between two user tasks of the same mm.
+ * rq->membarrier_state is up to date in that case. It was set when
+ * the first of those tasks was scheduled in, or later by the
+ * registration IPI.
+ *
+ * It must not be skipped when the previous task only borrowed the mm
+ * as its active_mm. rq->membarrier_state can be stale then:
+ *
+ * - exit_mm() and kthread_unuse_mm() clear it,
+ * - a registration does not send an IPI to a CPU which runs a lazy
+ *   TLB kernel thread.
  */
 static inline void membarrier_switch_mm(struct rq *rq,
 					struct mm_struct *prev_mm,
-- 
2.47.3