[PATCH v1 5/5] KVM: riscv: Age G-stage PTEs locklessly

[SeungJu Cheon]

Aging G-stage PTEs currently runs with mmu_lock held for write, taken
by the common MMU notifier code. When MGLRU or kswapd ages a large
range, every vCPU taking a G-stage fault blocks on the lock, and the
dirty-logging read-side fast path blocks behind aging as well.

The preceding patches prepare G-stage page-table walks for lockless
aging by using consistent PTE snapshots, preserving concurrent
Accessed-bit updates, and deferring page-table frees with RCU.

Select KVM_MMU_LOCKLESS_AGING so the common code no longer takes
mmu_lock for aging, and protect the G-stage walk with an RCU read-side
critical section.

Read the root inside the RCU read-side critical section and check the
resulting snapshot instead of checking kvm->arch.pgd separately before
initializing the G-stage context. This ensures that the root used by
the walk remains protected until the walk completes.

On QEMU TCG with 4 vCPUs, running dirty_log_perf_test -v 3 -b 256M -i 3
with MGLRU aging of the VM's cgroup forced every 100ms (86 passes):

                              before      after
  mmu_lock write wait, total  44.4 s      0.15 s
  mmu_lock write contentions  1,002,092   30,092
  mmu_lock read contentions   351,937     0
  guest dirty-memory time     18.6 s      16.7 s

Before, 93% of write-lock waiters were kvm_mmu_notifier_clear_young().
With no aging, no meaningful difference was observed between the two
kernels.

Signed-off-by: SeungJu Cheon <suunj1331 at gmail.com>
---
 arch/riscv/kvm/Kconfig |  1 +
 arch/riscv/kvm/mmu.c   | 26 +++++++++++++-------------
 2 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/arch/riscv/kvm/Kconfig b/arch/riscv/kvm/Kconfig
index ec2cee0a39e0..77898d58ff9a 100644
--- a/arch/riscv/kvm/Kconfig
+++ b/arch/riscv/kvm/Kconfig
@@ -32,6 +32,7 @@ config KVM
 	select VIRT_XFER_TO_GUEST_WORK
 	select SCHED_INFO
 	select GUEST_PERF_EVENTS if PERF_EVENTS
+	select KVM_MMU_LOCKLESS_AGING
 	help
 	  Support hosting virtualized guest machines.
 
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 8aed69abf814..ff282bdbe492 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -349,30 +349,30 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	return false;
 }
 
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+static bool kvm_riscv_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range,
+			      bool test_only)
 {
 	struct kvm_gstage gstage;
 
-	if (!kvm->arch.pgd)
-		return false;
+	guard(rcu)();
+	lockdep_assert_not_held(&kvm->mmu_lock);
 
 	kvm_riscv_gstage_init(&gstage, kvm);
+	if (!gstage.pgd)
+		return false;
 
 	return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT,
-					  range->end << PAGE_SHIFT, false);
+					  range->end << PAGE_SHIFT, test_only);
 }
 
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
-	struct kvm_gstage gstage;
-
-	if (!kvm->arch.pgd)
-		return false;
-
-	kvm_riscv_gstage_init(&gstage, kvm);
+	return kvm_riscv_age_gfn(kvm, range, false);
+}
 
-	return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT,
-					  range->end << PAGE_SHIFT, true);
+bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+	return kvm_riscv_age_gfn(kvm, range, true);
 }
 
 static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
-- 
2.52.0