[PATCH v2 29/45] KVM: arm64: GICv3: Set ICH_HCR_EL2.TDIR when interrupts overflow LR capacity

[Fuad Tabba]

Hi Marc,

On Fri, 14 Nov 2025 at 15:02, Marc Zyngier <maz at kernel.org> wrote:
>
> On Fri, 14 Nov 2025 14:20:46 +0000,
> Fuad Tabba <tabba at google.com> wrote:
> >
> > Hi Marc,
> >
> > On Sun, 9 Nov 2025 at 17:17, Marc Zyngier <maz at kernel.org> wrote:
> > >
> > > Now that we are ready to handle deactivation through ICV_DIR_EL1,
> > > set the trap bit if we have active interrupts outside of the LRs.
> > >
> > > Signed-off-by: Marc Zyngier <maz at kernel.org>
> > > ---
> > >  arch/arm64/kvm/vgic/vgic-v3.c | 7 +++++++
> > >  1 file changed, 7 insertions(+)
> > >
> > > diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
> > > index 1026031f22ff9..26e17ed057f00 100644
> > > --- a/arch/arm64/kvm/vgic/vgic-v3.c
> > > +++ b/arch/arm64/kvm/vgic/vgic-v3.c
> > > @@ -42,6 +42,13 @@ void vgic_v3_configure_hcr(struct kvm_vcpu *vcpu,
> > >                 ICH_HCR_EL2_VGrp0DIE : ICH_HCR_EL2_VGrp0EIE;
> > >         cpuif->vgic_hcr |= (cpuif->vgic_vmcr & ICH_VMCR_ENG1_MASK) ?
> > >                 ICH_HCR_EL2_VGrp1DIE : ICH_HCR_EL2_VGrp1EIE;
> > > +
> > > +       /*
> > > +        * Note that we set the trap irrespective of EOIMode, as that
> > > +        * can change behind our back without any warning...
> > > +        */
> > > +       if (irqs_active_outside_lrs(als))
> > > +               cpuif->vgic_hcr |= ICH_HCR_EL2_TDIR;
> > >  }
> >
> > I just tested these patches as they are on kvmarm/next
> > 2ea7215187c5759fc5d277280e3095b350ca6a50 ("Merge branch
> > 'kvm-arm64/vgic-lr-overflow' into kvmarm/next"), without any
> > additional pKVM patches. I tried running it with pKVM (non-protected)
> > and with just plain nVHE. In both cases, I get a trap to EL2 (0x18)
> > when booting a non-protected guest, which triggers a bug in
> > handle_trap() arch/arm64/kvm/hyp/nvhe/hyp-main.c:706
> >
> > This trap is happening because of setting this particular trap (TDIR).
> > Just removing this trap from vgic_v3_configure_hcr() from the ToT on
> > kvmarm/next boots fine.
>
> This is surprising, as I'm not hitting this on actual HW. Are you
> getting a 0x18 trap? If so, is it coming from the host? Can you
> correlate the PC with what the host is doing?

I should have given you that earlier, sorry.

Yes, it's an 0x18 trap from the host (although it happens when I boot
a guest). Here is the relevant part of the backtrace addr2lined and
the full one below.

handle_percpu_devid_irq+0x90/0x120 (kernel/irq/chip.c:930)
generic_handle_domain_irq+0x40/0x64 (include/linux/irqdesc.h:?)
gic_handle_irq+0x4c/0x110 (include/linux/irqdesc.h:?)
call_on_irq_stack+0x30/0x48 (arch/arm64/kernel/entry.S:893)

[   28.454804] Code: d65f03c0 92800008 f9000008 17fffffa (d4210000)
[   28.454873] kvm [266]: Hyp Offset: 0xfff1205c3fe00000
[   28.455157] Kernel panic - not syncing: HYP panic:
[   28.455157] PS:204023c9 PC:000e5fa4413e39bc ESR:00000000f2000800
[   28.455157] FAR:ffff800082733d3c HPFAR:0000000000500000 PAR:0000000000000000
[   28.455157] VCPU:0000000000000000
[   28.459703] CPU: 5 UID: 0 PID: 266 Comm: kvm-vcpu-0 Not tainted
6.18.0-rc3-g2ea7215187c5 #8 PREEMPT
[   28.460247] Hardware name: linux,dummy-virt (DT)
[   28.460615] Call trace:
[   28.460900]  show_stack+0x18/0x24 (C)
[   28.461234]  dump_stack_lvl+0x40/0x84
[   28.461421]  dump_stack+0x18/0x24
[   28.461566]  vpanic+0x11c/0x364
[   28.461698]  vpanic+0x0/0x364
[   28.461838]  nvhe_hyp_panic_handler+0x118/0x190
[   28.462056]  handle_percpu_devid_irq+0x90/0x120
[   28.462248]  handle_percpu_devid_irq+0x90/0x120
[   28.462439]  generic_handle_domain_irq+0x40/0x64
[   28.462643]  gic_handle_irq+0x4c/0x110
[   28.462814]  call_on_irq_stack+0x30/0x48
[   28.463003]  do_interrupt_handler+0x4c/0x6c
[   28.463184]  el1_interrupt+0x3c/0x60
[   28.463348]  el1h_64_irq_handler+0x18/0x24
[   28.463525]  el1h_64_irq+0x6c/0x70
[   28.463799]  local_daif_restore+0x8/0xc (P)
[   28.463980]  el0t_64_sync_handler+0x84/0x12c
[   28.464164]  el0t_64_sync+0x198/0x19c

> It would indicate that we are leaking trap bits on exit, and that QEMU
> is trapping ICC_DIR_EL1 on top of ICV_DIR_EL1 (which the HW I have
> access to doesn't seem to do).
>
> > I'm running this on QEMU with '-machine virt,gic-version=3 -cpu max'
> > and the kernel with 'kvm-arm.mode=protected' and with
> > 'kvm-arm.mode=nvhe'.
> >
> > Let me know if you need any more info or help testing.
>
> On top of the above, could you give the hack below a go? I haven't
> tested it at all (I'm in the middle of a bisect from hell...)

With the hack it boots, both nvhe and protected mode.

Cheers,
/fuad

> Thanks,
>
>         M.
>
> diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c
> index e950efa22547..71199e1a9294 100644
> --- a/arch/arm64/kvm/hyp/vgic-v3-sr.c
> +++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c
> @@ -243,7 +243,7 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if)
>                 cpu_if->vgic_hcr |= val & ICH_HCR_EL2_EOIcount;
>         }
>
> -       write_gicreg(compute_ich_hcr(cpu_if) & ~ICH_HCR_EL2_En, ICH_HCR_EL2);
> +       write_gicreg(0, ICH_HCR_EL2);
>  }
>
>  void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if)
>
> --
> Without deviation from the norm, progress is not possible.