[PATCH v2 29/45] KVM: arm64: GICv3: Set ICH_HCR_EL2.TDIR when interrupts overflow LR capacity

[Fuad Tabba]

Hi Marc,

On Fri, 14 Nov 2025 at 17:41, Marc Zyngier <maz at kernel.org> wrote:
>
> On Fri, 14 Nov 2025 15:53:33 +0000,
> Fuad Tabba <tabba at google.com> wrote:
> >
> > Hi Marc,
> >
> > On Fri, 14 Nov 2025 at 15:02, Marc Zyngier <maz at kernel.org> wrote:
> > >
> > > On Fri, 14 Nov 2025 14:20:46 +0000,
> > > Fuad Tabba <tabba at google.com> wrote:
> > > >
> > > > Hi Marc,
> > > >
> > > > On Sun, 9 Nov 2025 at 17:17, Marc Zyngier <maz at kernel.org> wrote:
> > > > >
> > > > > Now that we are ready to handle deactivation through ICV_DIR_EL1,
> > > > > set the trap bit if we have active interrupts outside of the LRs.
> > > > >
> > > > > Signed-off-by: Marc Zyngier <maz at kernel.org>
> > > > > ---
> > > > >  arch/arm64/kvm/vgic/vgic-v3.c | 7 +++++++
> > > > >  1 file changed, 7 insertions(+)
> > > > >
> > > > > diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
> > > > > index 1026031f22ff9..26e17ed057f00 100644
> > > > > --- a/arch/arm64/kvm/vgic/vgic-v3.c
> > > > > +++ b/arch/arm64/kvm/vgic/vgic-v3.c
> > > > > @@ -42,6 +42,13 @@ void vgic_v3_configure_hcr(struct kvm_vcpu *vcpu,
> > > > >                 ICH_HCR_EL2_VGrp0DIE : ICH_HCR_EL2_VGrp0EIE;
> > > > >         cpuif->vgic_hcr |= (cpuif->vgic_vmcr & ICH_VMCR_ENG1_MASK) ?
> > > > >                 ICH_HCR_EL2_VGrp1DIE : ICH_HCR_EL2_VGrp1EIE;
> > > > > +
> > > > > +       /*
> > > > > +        * Note that we set the trap irrespective of EOIMode, as that
> > > > > +        * can change behind our back without any warning...
> > > > > +        */
> > > > > +       if (irqs_active_outside_lrs(als))
> > > > > +               cpuif->vgic_hcr |= ICH_HCR_EL2_TDIR;
> > > > >  }
> > > >
> > > > I just tested these patches as they are on kvmarm/next
> > > > 2ea7215187c5759fc5d277280e3095b350ca6a50 ("Merge branch
> > > > 'kvm-arm64/vgic-lr-overflow' into kvmarm/next"), without any
> > > > additional pKVM patches. I tried running it with pKVM (non-protected)
> > > > and with just plain nVHE. In both cases, I get a trap to EL2 (0x18)
> > > > when booting a non-protected guest, which triggers a bug in
> > > > handle_trap() arch/arm64/kvm/hyp/nvhe/hyp-main.c:706
> > > >
> > > > This trap is happening because of setting this particular trap (TDIR).
> > > > Just removing this trap from vgic_v3_configure_hcr() from the ToT on
> > > > kvmarm/next boots fine.
> > >
> > > This is surprising, as I'm not hitting this on actual HW. Are you
> > > getting a 0x18 trap? If so, is it coming from the host? Can you
> > > correlate the PC with what the host is doing?
> >
> > I should have given you that earlier, sorry.
> >
> > Yes, it's an 0x18 trap from the host (although it happens when I boot
> > a guest). Here is the relevant part of the backtrace addr2lined and
> > the full one below.
> >
> > handle_percpu_devid_irq+0x90/0x120 (kernel/irq/chip.c:930)
> > generic_handle_domain_irq+0x40/0x64 (include/linux/irqdesc.h:?)
> > gic_handle_irq+0x4c/0x110 (include/linux/irqdesc.h:?)
> > call_on_irq_stack+0x30/0x48 (arch/arm64/kernel/entry.S:893)
> >
> > [   28.454804] Code: d65f03c0 92800008 f9000008 17fffffa (d4210000)
> > [   28.454873] kvm [266]: Hyp Offset: 0xfff1205c3fe00000
> > [   28.455157] Kernel panic - not syncing: HYP panic:
> > [   28.455157] PS:204023c9 PC:000e5fa4413e39bc ESR:00000000f2000800
> > [   28.455157] FAR:ffff800082733d3c HPFAR:0000000000500000 PAR:0000000000000000
>
> I expect you have a write to ICC_DIR_EL1 at this address?

It almost surely must be, but tracking it down hasn't been that easy.
That said, I think it's ending up in gic_eoimode1_eoi_irq(), which
calls gic_write_dir() if !gic_arm64_erratum_2941627_needed(d).

I wonder if your hardware needs that erratum.

> > [   28.455157] VCPU:0000000000000000
> > [   28.459703] CPU: 5 UID: 0 PID: 266 Comm: kvm-vcpu-0 Not tainted
> > 6.18.0-rc3-g2ea7215187c5 #8 PREEMPT
> > [   28.460247] Hardware name: linux,dummy-virt (DT)
> > [   28.460615] Call trace:
> > [   28.460900]  show_stack+0x18/0x24 (C)
> > [   28.461234]  dump_stack_lvl+0x40/0x84
> > [   28.461421]  dump_stack+0x18/0x24
> > [   28.461566]  vpanic+0x11c/0x364
> > [   28.461698]  vpanic+0x0/0x364
> > [   28.461838]  nvhe_hyp_panic_handler+0x118/0x190
> > [   28.462056]  handle_percpu_devid_irq+0x90/0x120
> > [   28.462248]  handle_percpu_devid_irq+0x90/0x120
> > [   28.462439]  generic_handle_domain_irq+0x40/0x64
> > [   28.462643]  gic_handle_irq+0x4c/0x110
> > [   28.462814]  call_on_irq_stack+0x30/0x48
> > [   28.463003]  do_interrupt_handler+0x4c/0x6c
> > [   28.463184]  el1_interrupt+0x3c/0x60
> > [   28.463348]  el1h_64_irq_handler+0x18/0x24
> > [   28.463525]  el1h_64_irq+0x6c/0x70
> > [   28.463799]  local_daif_restore+0x8/0xc (P)
> > [   28.463980]  el0t_64_sync_handler+0x84/0x12c
> > [   28.464164]  el0t_64_sync+0x198/0x19c
> >
> > > It would indicate that we are leaking trap bits on exit, and that QEMU
> > > is trapping ICC_DIR_EL1 on top of ICV_DIR_EL1 (which the HW I have
> > > access to doesn't seem to do).
> > >
> > > > I'm running this on QEMU with '-machine virt,gic-version=3 -cpu max'
> > > > and the kernel with 'kvm-arm.mode=protected' and with
> > > > 'kvm-arm.mode=nvhe'.
> > > >
> > > > Let me know if you need any more info or help testing.
> > >
> > > On top of the above, could you give the hack below a go? I haven't
> > > tested it at all (I'm in the middle of a bisect from hell...)
> >
> > With the hack it boots, both nvhe and protected mode.
>
> OK. At least we know what the issue is, and it shouldn't be too hard
> to fix. I guess there is an opportunity for cleanup here, and I'll
> look into it shortly (probably not before Monday though).

No hurry on my part. I'm just here for the reviews :)

Cheers,
/fuad

> Thanks again,
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.