[PATCH] KVM: arm64: Don't livelock when AT can't resolve a fault IPA

[Will Deacon]

On Mon, Nov 02, 2026 at 10:47:49AM +0100, Ard Biesheuvel wrote:
> From: Ard Biesheuvel <ardb at kernel.org>
> 
> When a stage-2 permission fault doesn't report its IPA in HPFAR_EL2,
> __get_fault_info() recovers it with an AT S1E1R/S1E1A on FAR_EL2. If that
> fails, kvm_hyp_handle_memory_fault() goes straight back to the guest, on
> the assumption that the replayed access will either succeed or take a
> fault that the slow path can handle.
> 
> That assumption doesn't hold when the AT fails because its stage-1 walk
> takes a stage-2 fault on one of the guest's page-table pages, while the
> guest still holds a cached translation for the faulting VA. The guest
> doesn't walk its page tables, so it never takes the S1PTW fault that
> would let the slow path fix things up. It takes the same permission fault
> again, the AT fails again, and the vCPU livelocks in the world switch.
> 
> This was observed on Apple M2, which has no hardware Access flag
> management at stage 2, and no DIC, so execute permission is granted
> lazily and the first instruction fetch from a page takes a permission
> fault. With DAMON_STAT enabled (CONFIG_DAMON_STAT_ENABLED_DEFAULT=y),
> kdamond keeps sampling guest memory, and damon_pmdp_mkold() ages a whole
> PMD at a time through mmu_notifier_clear_young(). With THP-backed guest
> memory, kvm_age_gfn() therefore clears the stage-2 AF of 32MB of guest
> pages at once, without TLB invalidation. When that covers both freshly
> loaded code and the page-table page that maps it, the first instruction
> fetch from that code livelocks, and the vCPU only ever recovers when its
> thread happens to be moved to another CPU. In all 52 instrumented
> occurrences, the AT reported a stage-2 Access flag fault on a stage-1
> table walk (PAR_EL1.{S,PTW} = {1,1}, FST = Access flag fault, level 3),
> while an AT S12E1R on the same VA succeeded. With DAMON_STAT disabled,
> the AT didn't fail once in 210 boots. Nothing in this is specific to
> nested virtualisation, nor to instruction fetches: a write to a
> read-only memslot hit it too.
> 
> A stage-1 fault can lead to the same livelock. A guest that ages its own
> page tables without invalidating the TLB, as MGLRU does through
> ptep_test_and_clear_young(), can keep using a cached translation while
> the descriptor in memory has its Access flag clear. If such an access
> takes a stage-2 permission fault (lazy execute permission, or dirty
> logging), the AT reports a stage-1 Access flag fault (PAR_EL1.S = 0),
> which the guest never takes either, as it doesn't walk its page tables.
> This hasn't been observed, but it is why the invalidation below is done
> whenever the AT fails, rather than only when PAR_EL1.PTW reports a
> stage-2 fault on the walk. A guest that merely races with its own
> page-table updates isn't harmed by it: at worst, it takes a translation
> fault that it was allowed to take anyway.
> 
> Invalidate this VMID's TLB entries on the local CPU before replaying the
> access, so that the guest has to walk its page tables and takes the
> fault that the AT ran into: either a stage-2 fault on the page-table
> page, with a valid HPFAR_EL2, which the slow path resolves, or a stage-1
> fault, which the guest resolves itself. In all instrumented cases, the
> next exit was an S1PTW Access flag fault on the page-table page. This
> only happens when the AT fails, so the cost is negligible.
> 
> With this change, an EL2 UEFI guest that livelocked in 27 out of 210
> boots didn't hang once in 420 boots.
> 
> Signed-off-by: Ard Biesheuvel <ardb at kernel.org>
> ---
> Cc: Marc Zyngier <maz at kernel.org>
> Cc: Oliver Upton <oupton at kernel.org>
> Cc: Fuad Tabba <fuad.tabba at linux.dev>
> Cc: Joey Gouly <joey.gouly at arm.com>
> Cc: Steffen Eiden <seiden at linux.ibm.com>
> Cc: Suzuki K Poulose <suzuki.poulose at arm.com>
> Cc: Zenghui Yu <yuzenghui at huawei.com>
> 
>  arch/arm64/kvm/hyp/include/hyp/switch.h | 15 ++++++++++++++-
>  1 file changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> index e0c55644224c..4ba61b71d511 100644
> --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> @@ -843,8 +843,21 @@ static inline bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code)
>  static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu,
>  					       u64 *exit_code)
>  {
> -	if (!__populate_fault_info(vcpu))
> +	if (!__populate_fault_info(vcpu)) {

Presumably we need a barrier (DSB?) here, to ensure that the TLBI happens
after the AT instruction?

> +		/*
> +		 * The AT couldn't resolve the IPA, typically because its
> +		 * stage-1 walk took a stage-2 fault, such as an Access flag
> +		 * fault on an aged page-table page. If the guest still has a
> +		 * cached translation for the VA, it won't take that fault
> +		 * itself, and would replay the same permission fault forever.
> +		 * Drop this VMID's TLB entries on this CPU, so that the replay
> +		 * walks the page tables and faults with a valid HPFAR_EL2.
> +		 */
> +		__tlbi(vmalls12e1);

Can we get away with an ALLE1 here, or do we really need to blow away the
whole lot?

Will