[PATCH] RISC-V: KVM: Flush VS-stage TLB before reusing a host CPU
guoyaxing
guoyaxing at bosc.ac.cn
Mon Oct 26 03:18:57 PDT 2026
在 2026/10/24 23:49, Anup Patel 写道:
> On Sat, Oct 10, 2026 at 2:36 PM guoyaxing <guoyaxing at bosc.ac.cn> wrote:
>>
>>
>>
>> 在 2026/10/10 15:50, Anup Patel 写道:
>>> On Sat, Oct 10, 2026 at 4:12 AM guoyaxing <guoyaxing at bosc.ac.cn> wrote:
>>>>
>>>>
>>>>
>>>> 在 2026/10/10 02:51, Anup Patel 写道:
>>>>> On Mon, Aug 17, 2026 at 12:25 PM Yaxing Guo <guoyaxing at bosc.ac.cn> wrote:
>>>>>>
>>>>>> Guest Linux uses mm_cpumask to decide whether an sfence.vma needs to be
>>>>>> local or remote. Under KVM that mask tracks guest CPUs, not the host CPUs
>>>>>> that previously backed a vCPU.
>>>>>>
>>>>>> This can miss stale VS-stage TLB entries when a vCPU migrates across host
>>>>>> CPUs. For example:
>>>>>>
>>>>>> - vcpu0 runs on host CPU1 and fills a VS-stage TLB entry.
>>>>>> - vcpu0 migrates to host CPU0.
>>>>>> - the guest updates the mapping on vcpu0 and issues a local sfence.vma.
>>>>>> - later, the guest task runs on vcpu1 while vcpu1 is backed by host CPU1.
>>>>>
>>>>> When the vcpu0 is brought back to host CPU1, the kvm_riscv_local_tlb_sanitize()
>>>>> will ensure TLB entries associated with Guest/VM are nuked.
>>>>>
>>>>> By default, kvm_riscv_local_hfence_gvma_vmid_all() will only do HFENCE.GVMA
>>>>> for the associated VMID because most RISC-V H-extension implemenation will
>>>>> tag VS-stage TLB entries with VMID and HFENCE.GVMA must remove both
>>>>> G-stage and VS-stage TLB entries for given VMID.
>>>>>
>>>>> If your HW does not tag VS-stage TLB enteries with VMID then it buggy
>>>>> / inefficient
>>>>> so you have to set kvm_riscv_vsstage_tlb_no_gpa in
>>>>> kvm_riscv_setup_vendor_features()
>>>>> as work-around.
>>>>>
>>>>> NACK to this patch from myside.
>>>>>
>>>>> Regards,
>>>>> Anup
>>>>>
>>>> Hi Anup,
>>>> Thanks for the review and clarification.
>>>>
>>>> Regarding “When the vcpu0 is brought back to host CPU1”: the failure
>>>> we hit is not caused by vcpu0 being brought back onto host CPU1. In our
>>>> case, after vcpu0 has already migrated away, neither vCPU migrates
>>>> again. The guest process is later scheduled inside the multi-vCPU guest
>>>> onto vcpu1, which is still running on host CPU1. So the stale TLB is
>>>> re-hit without any further KVM vCPU migration, and
>>>> kvm_riscv_local_tlb_sanitize() is not involved on that path.
>>>>
>>>> My commit message was a bit too terse on this point. The concrete
>>>> sequence we observed was described earlier in the RFC discussion:
>>>>
>>>>
>>>> https://lore.kernel.org/kvm/0d15b098-b8f7-4080-9f33-e8c49c2331e1@bosc.ac.cn/
>>>>
>>>> In short, the issue is: because vcpu0 migrated away, cpu1 still has
>>>> stale TLB entries while cpu0 has the new mapping. After that, neither
>>>> vcpu migrated again, but the process was scheduled from vcpu0 to vcpu1
>>>> (which is on cpu1), causing it to hit the stale TLB. Consider the
>>>> following scenario:
>>>>
>>>> 1. vcpu0 and vcpu1 are both running on cpu1. At this point, vcpu0 is
>>>> on cpu1, and a process on it reads a page, which gets cached in the TLB
>>>> (let's call this the "stale TLB entry").
>>>>
>>>> 2. vcpu0 migrates to cpu0, and then the process (on vcpu0, now on
>>>> cpu0) does a CoW (copy-on-write) and establishes a new mapping.
>>>>
>>>> 3. Neither vcpu0 nor vcpu1 migrates again. At this point, the process
>>>> gets scheduled inside the VM onto vcpu1 (which is on cpu1), reads the
>>>> same page, and hits the stale TLB entry.
>>>
>>> This cannot happen because in step#2 when mapping was updated the
>>> Guest OS will do TLB flush for all its VCPUs which means vcpu1 on cpu0
>>> will also receive TLB flush and the stale TLB will be removed. This means
>>> when process is moved back to vcpu1 on cpu0, HW will fetch updated
>>> PTE and new TLB entry will be populated.
>>>
>>
>> Hi Anup,
>>
>> Thanks for the follow-up. I may be missing something, so I wanted to
>> double-check this against the guest TLB shootdown path.
>>
>> If I follow the Linux guest CoW path correctly, when the mapping is
>> updated in step#2 it goes through:
>>
>> wp_page_copy()
>> -> ptep_clear_flush()
>> -> flush_tlb_page()
>> -> __flush_tlb_range(mm, mm_cpumask(mm), ...)
>>
>> So the remote SFENCE.VMA appears to be targeted only at CPUs in
>> mm_cpumask(mm), rather than at all VCPUs of the guest. mm_cpumask is
>> updated in switch_mm()/set_mm() for guest CPUs that have actually run
>> that mm.
>>
>> In our case, before the CoW the process had only ever run on vcpu0. From
>> the guest's point of view it did not migrate: the host migration of
>> vcpu0 from cpu1 to cpu0 is not visible to guest Linux, which only sees
>> vCPUs. So at CoW time mm_cpumask still contains only vcpu0 (or at least
>> does not contain vcpu1), and __flush_tlb_range() would not IPI vcpu1.
>> The local SFENCE on vcpu0 would only cover host cpu0, where vcpu0 is
>> currently running. The stale VS-stage TLB entry left on host cpu1 from
>> when vcpu0 previously ran there would then remain.
>>
>> Later, still without any further vCPU migration, the process is
>> scheduled inside the guest onto vcpu1, which remains on host cpu1, and
>> can hit that stale entry.
>>
>> This also seems more likely when the ASID allocator is enabled, because
>> set_mm_asid() does not always local_flush_tlb_all() on switch_mm(). So
>> even the first time the process lands on vcpu1, a stale ASID-tagged
>> entry on host cpu1 may still be present.
>>
>> In other words, I am not sure this case is covered by a full guest TLB
>> flush to all VCPUs on the mapping update. Guest shootdown looks
>> mm_cpumask-based, and host pCPU migration of a vCPU does not by itself
>> add other vCPUs that never ran the mm into that mask.
>
> This makes sense now considering ASID allocator in Guest Linux is global
> and not local. In any case, KVM RISC-V must not assume that the Guest OS
> has local ASID allocator hence the kvm_riscv_local_tlb_sanitize() must also
> santize TLB for other VCPUs of the same Guest which can potentially run
> on the previous host CPU where current VCPU ran.
>
> Having said the above, there are multiple issues with the current patch:
> 1) The kvm_riscv_local_hfence_gvma_vmid_all() is sufficient for most
> implementations to sanitize both G-stage and VS-stage TLB entries
> because both G-stage and VS-stage TLB entries are generally tagged
> with VMID. The kvm_riscv_vsstage_tlb_no_gpa work-around is for
> implmentations which do not tag VS-stage TLB entries with VMID
> and this patch remove this work-around.
> 2) The other VCPUs on the previous host CPU may be already running
> and may still hit stale TLB entry before kvm_riscv_local_tlb_sanitize()
> gets a chance to sanitize TLB on previous host CPU.
> 3) The per-CPU pointer in struct kvm_arch is racy because we
> may have two VCPUs on same host CPU which are migrated to
> other host CPUs. Instead, it is much simpler to have an atomic
> cpumask in each VCPU where any part of KVM can mark a host
> CPU in the cpumask of the VCPU to force TLB sanitization whenever
> the VCPU runs on that particular host CPU.
>
> How about the following change ?
>
> diff --git a/arch/riscv/include/asm/kvm_host.h
> b/arch/riscv/include/asm/kvm_host.h
> index a30600579231..c975eee7240c 100644
> --- a/arch/riscv/include/asm/kvm_host.h
> +++ b/arch/riscv/include/asm/kvm_host.h
> @@ -9,6 +9,7 @@
> #ifndef __RISCV_KVM_HOST_H__
> #define __RISCV_KVM_HOST_H__
>
> +#include <linux/cpumask.h>
> #include <linux/types.h>
> #include <linux/kvm.h>
> #include <linux/kvm_types.h>
> @@ -184,6 +185,9 @@ struct kvm_vcpu_arch {
> /* ISA feature bits (similar to MISA) */
> DECLARE_BITMAP(isa, RISCV_ISA_EXT_MAX);
>
> + /* Sanitize TLB host CPU mask */
> + cpumask_t tlb_sanitize_mask;
> +
> /* Vendor, Arch, and Implementation details */
> unsigned long mvendorid;
> unsigned long marchid;
> diff --git a/arch/riscv/kvm/tlb.c b/arch/riscv/kvm/tlb.c
> index 2ae34632cdcb..b1387a00c8b0 100644
> --- a/arch/riscv/kvm/tlb.c
> +++ b/arch/riscv/kvm/tlb.c
> @@ -207,11 +207,31 @@ void kvm_riscv_local_hfence_vvma_all(unsigned long vmid)
>
> void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu)
> {
> - unsigned long vmid;
> + struct kvm_vcpu *tvcpu;
> + unsigned long i, vmid;
>
> - if (!kvm_riscv_gstage_vmid_bits() ||
> - vcpu->arch.last_exit_cpu == vcpu->cpu)
> - return;
> + if (!cpumask_test_cpu(vcpu->cpu, &vcpu->arch.tlb_sanitize_mask)) {
> + if (!kvm_riscv_gstage_vmid_bits() ||
> + vcpu->arch.last_exit_cpu == vcpu->cpu)
> + return;
> + }
> +
> + /*
> + * The VCPU might have left stale TLBs on the previous host CPU so
> + * mark previous host CPU in the tlb_sanitize_mask of other VCPUs.
> + */
> + if (vcpu->arch.last_exit_cpu > -1 && vcpu->arch.last_exit_cpu !=
> vcpu->cpu) {
> + kvm_for_each_vcpu(i, tvcpu, vcpu->kvm) {
> + if (tvcpu == vcpu)
> + continue;
> + cpumask_set_cpu(vcpu->arch.last_exit_cpu,
> + &tvcpu->arch.tlb_sanitize_mask);
> + kvm_vcpu_kick(tvcpu);
> + }
> + }
> +
> + /* Clear current host CPU from the tlb_sanitize_mask */
> + cpumask_clear_cpu(vcpu->cpu, &vcpu->arch.tlb_sanitize_mask);
>
> /*
> * On RISC-V platforms with hardware VMID support, we share same
> diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
> index dd786f758eb7..c00dc0b191dc 100644
> --- a/arch/riscv/kvm/vcpu.c
> +++ b/arch/riscv/kvm/vcpu.c
> @@ -107,6 +107,8 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu
> *vcpu, bool kvm_sbi_reset)
>
> vcpu->arch.last_exit_cpu = -1;
>
> + cpumask_clear(&vcpu->arch.tlb_sanitize_mask);
> +
> kvm_riscv_vcpu_context_reset(vcpu, kvm_sbi_reset);
>
> kvm_riscv_vcpu_fp_reset(vcpu);
>
> Regards,
> Anup
Hi Anup,
Thanks for the explanation and for suggesting the per-vCPU sanitization
mask. Building on your proposal, I would like to suggest two changes:
1. Separate the invalidation strategy for vCPU migration and a mask hit.
Keep HFENCE.GVMA for vCPU migration, identified by
last_exit_cpu != vcpu->cpu, when hardware VMIDs are available. Use
HFENCE.VVMA when the current host CPU is marked in tlb_sanitize_mask.
A mask hit caused by another vCPU of the same VM running on this CPU
requires VS-stage sanitization; it does not by itself require G-stage
invalidation.
For example, if vCPU0 and vCPU1 both last ran on CPU0, switching from
vCPU1 back to vCPU0 only needs the mask-triggered VVMA. This avoids
requesting G-stage invalidation when the shared G-stage mappings have
not changed. If both migration and a mask hit occur, both fences run.
2. Move mask updates to kvm_arch_vcpu_put().
Mark the outgoing host CPU in the masks of the other vCPUs of the
same VM when a vCPU is put. This also covers a switch between vCPUs
on one host CPU without either vCPU migrating. Marking only when
migration is detected in kvm_riscv_local_tlb_sanitize() would miss
that case.
The marking now happens on the outgoing CPU before another vCPU can
enter guest mode there. The migration-time marking and kicks are
therefore removed. The trade-off is an O(nr_vcpus) walk on each
qualifying put().
How about the following change ?
diff --git a/arch/riscv/kvm/tlb.c b/arch/riscv/kvm/tlb.c
index 2ac3c0a20fce..ae8a40b3429c 100644
--- a/arch/riscv/kvm/tlb.c
+++ b/arch/riscv/kvm/tlb.c
@@ -160,32 +160,15 @@
void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu)
{
- struct kvm_vcpu *tvcpu;
- unsigned long i, vmid;
-
- if (!cpumask_test_cpu(vcpu->cpu, &vcpu->arch.tlb_sanitize_mask)) {
- if (!kvm_riscv_gstage_vmid_bits() ||
- vcpu->arch.last_exit_cpu == vcpu->cpu)
- return;
- }
-
- /*
- * The VCPU might have left stale TLBs on the previous host CPU so
- * mark previous host CPU in the tlb_sanitize_mask of other VCPUs.
- */
- if (vcpu->arch.last_exit_cpu > -1 &&
- vcpu->arch.last_exit_cpu != vcpu->cpu) {
- kvm_for_each_vcpu(i, tvcpu, vcpu->kvm) {
- if (tvcpu == vcpu)
- continue;
- cpumask_set_cpu(vcpu->arch.last_exit_cpu,
- &tvcpu->arch.tlb_sanitize_mask);
- kvm_vcpu_kick(tvcpu);
- }
- }
-
- /* Clear current host CPU from the tlb_sanitize_mask */
- cpumask_clear_cpu(vcpu->cpu, &vcpu->arch.tlb_sanitize_mask);
+ unsigned long vmid;
+ bool vcpu_migrated, sanitize_vsstage;
+
+ vcpu_migrated = vcpu->arch.last_exit_cpu != vcpu->cpu;
+ sanitize_vsstage = cpumask_test_and_clear_cpu(vcpu->cpu,
+ &vcpu->arch.tlb_sanitize_mask);
+ if (!sanitize_vsstage &&
+ (!kvm_riscv_gstage_vmid_bits() || !vcpu_migrated))
+ return;
/*
* On RISC-V platforms with hardware VMID support, we share same
@@ -199,13 +182,16 @@
*/
vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
- kvm_riscv_local_hfence_gvma_vmid_all(vmid);
+ if (vcpu_migrated && kvm_riscv_gstage_vmid_bits())
+ kvm_riscv_local_hfence_gvma_vmid_all(vmid);
/*
- * Flush VS-stage TLB entries for implementation where VS-stage
- * TLB does not cahce guest physical address and VMID.
+ * Flush VS-stage entries marked when another VCPU left this CPU, or when
+ * required by the split-TLB vendor workaround on this VCPU's migration.
*/
- if (static_branch_unlikely(&kvm_riscv_vsstage_tlb_no_gpa))
+ if (sanitize_vsstage ||
+ (vcpu_migrated &&
+ static_branch_unlikely(&kvm_riscv_vsstage_tlb_no_gpa)))
kvm_riscv_local_hfence_vvma_all(vmid);
}
diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
index f2e52f3b0786..6209c66b24e1 100644
--- a/arch/riscv/kvm/vcpu.c
+++ b/arch/riscv/kvm/vcpu.c
@@ -644,8 +644,18 @@
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
{
+ struct kvm_vcpu *tvcpu;
+ unsigned long i;
+ int cpu = vcpu->cpu;
void *nsh;
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
+
+ if (cpu >= 0 && vcpu->arch.last_exit_cpu == cpu) {
+ kvm_for_each_vcpu(i, tvcpu, vcpu->kvm) {
+ if (tvcpu != vcpu)
+ cpumask_set_cpu(cpu, &tvcpu->arch.tlb_sanitize_mask);
+ }
+ }
vcpu->cpu = -1;
Would these changes make sense to you, or is there anythingI might
be overlooking?
Regards,
Yaxing
More information about the kvm-riscv
mailing list