[RFC PATCH 1/1] KVM: arm: Optimize cache flush by only flushing on vcpu0

[Jiayuan Liang]

    When toggling cache state in a multi-vCPU guest, we currently flush the VM's
    stage2 page tables on every vCPU that transitions cache state. This leads to
    redundant cache flushes during guest boot, as each vCPU performs the same
    flush operation.

    In a typical guest boot sequence, vcpu0 is the first to enable caches, and
    other vCPUs follow afterward. By the time secondary vCPUs enable their caches,
    the flush performed by vcpu0 has already ensured cache coherency for the
    entire VM.

    Optimize this by only performing the stage2_flush_vm() operation on vcpu0,
    which is sufficient to maintain cache coherency while eliminating redundant
    flushes on other vCPUs. This can improve performance during guest boot in
    multi-vCPU configurations.

    Testing with a 64-core VM with 128GB memory using hugepages shows dramatic
    performance improvements, reducing busybox boot time from 33s to 5s.

    Test command:
    qemu-kvm \
        -nographic \
        -m 128G \
        -mem-path /dev/hugepages \
        -mem-prealloc \
        -cpu host -M virt \
        -smp 64 \
        -kernel ./Image \
        -append "root=/dev/ram earlycon=pl011,0x9000000 console=ttyAMA0 init=/linuxrc systemd.unified_cgroup_hierarchy=1 psi=1"

    Signed-off-by: Jiayuan Liang <ljykernel at 163.com>
---
 arch/arm64/kvm/mmu.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 754f2fe0cc67..fbc736657666 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2300,8 +2300,10 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)
 	 * If switching it off, need to clean the caches.
 	 * Clean + invalidate does the trick always.
 	 */
-	if (now_enabled != was_enabled)
-		stage2_flush_vm(vcpu->kvm);
+	if (now_enabled != was_enabled) {
+		if (vcpu->vcpu_id == 0)
+			stage2_flush_vm(vcpu->kvm);
+	}
 
 	/* Caches are now on, stop trapping VM ops (until a S/W op) */
 	if (now_enabled)
-- 
2.43.0