KVM: Nested VGIC emulation leads to infinite IRQ exceptions

[Marc Zyngier]

Please use the kvmarm mailing list for KVM related discussions (added
for your convenience).

On Tue, 30 Sep 2025 22:11:54 +0100,
Volodymyr Babchuk <Volodymyr_Babchuk at epam.com> wrote:
> 
> 
> Hi all,
> 
> We are trying to run Xen as KVM nested hypervisor (again!) and we have
> encountered strange issue with GIC nested emulation. I am certain that
> we'll dig to the root cause, but probably someone on the ML will save us
> a couple of days of debugging by providing with some insights.
> 
> So, setup is following: QEMU 9.2 is running Xen 4.20 with KVM (latest
> Linux master branch) as accelerator.

9.2 is an odd choice, specially as it doesn't have any NV support.
ISTR that 10.1 is the first version to have some NV support, although
without E2H0 enablement which I expect Xen requires.

Anyway, if you're already running something, then I expect you're
patched QEMU to death to get there.

> QEMU provides a couple of virtio
> devices to the VM and some of these device are passed thought to DomU
> (we had to hook these devices to vSMMU, but this is another
> story). Sometimes we observe the following sequence of events:
> 
> 1. DomU gets IRQ from a virtio device
> 2. DomU acknowledges the IRQ by reading IAR1 register
> 3. DomU is unable to deactivate the IRQ (there is no write to the
>    EOI register)
> 
> We are not sure why this happens, but our current theory is that DomU's
> vvcpu0 is interrupted during handling of the IRQ by Xen's timer
> interrupt. Also, we are not able to catch this specific moment in KVM
> trace because of lots of lost events. Anyways, after this we are seeing
> the following loop:
> 
> 4. vCPU switches to Xen via IRQ Exception
> 5. Xen reads IAR1 to get IRQ nr, but gets 1023 (aka no IRQs)
> 6. Xen issues ERET to return back to guest
> 7. GOTO 4.

What is the configuration of ICH_HCR_EL2 in Xen at the point of
reading IAR? My hunch is that you are taking a maintenance interrupt,
disable the virtual CPU interface on taking that interrupt, which of
course results in no interrupt to acknowledge.

Reading ICH_MISR_EL2 at the point of entering Xen should give you a
clue of the reason why this is happening -- assuming my hunch is
correct.

> This basically renders the whole vCPU stuck. Also we noticed that DomU's
> vvCPU is stuck right after accessing virtio mmio register. So looks like
> this is what happens:
> 
> 1. QEMU sends virtio IRQ to the VM
> 2. Xen handles the IRQ and injects it into DomU
> 3. DomU tries to handle it and accesses a virtio mmio register
> 4. This produces a memory fault that leads to switch back to KVM (and
> then to QEMU of course) so QEMU can handle MMIO access
> 5. When QEMU continues vCPU thread, it immediately gets switched back to
> vEL2 (probably due to timer IRQ, but this is my speculation)
> 6. the vCPU is spinning in the aforementioned loop
> 
> Looks like this happens because of empty LRs, but we still didn't
> confirmed this because the issue is not 100% reproducible. I'll be glad
> to hear any suggestions.

I don't think this is likely. If the guest hasn't done an EOI, then
the interrupt should still be in the LR in the context of DomU, with
at least an Active state. You want to try and look at what Xen sees
there.

> This is a part of the KVM trace, where you can see that vCPU in question
> tries to perform ERET to Linux in DomU but is being brought back to
> vEL2. In this particular case this is vCPU1 / vvCPU0. I filtered out
> other vCPUs to reduce clutter.
> 
>  qemu-system-aar-41290   [000] d.... 12023.695620: kvm_entry: PC: 0x00000a0000267c80
>  qemu-system-aar-41290   [000] d.... 12023.695620: vgic_update_irq_pending: VCPU: 1, IRQ 25, level: 0
>  qemu-system-aar-41290   [000] d.... 12023.695621: kvm_get_timer_map: VCPU: 1, dv: 2, dp: 3, ev: 1, ep: 0
>  qemu-system-aar-41290   [000] d.... 12023.695621: kvm_timer_emulate: arch_timer_ctx_index: 1 (should_fire: 1)
>  qemu-system-aar-41290   [000] d.... 12023.695621: kvm_timer_emulate: arch_timer_ctx_index: 0 (should_fire: 0)
>  qemu-system-aar-41290   [000] ..... 12023.695621: kvm_exit: TRAP: HSR_EC: 0x001a (ERET), PC: 0x00000a00002674e0

[...]

There isn't much to go on here, as we mostly see the timers, which do
not help at all. I'd suggest you look at the maintenance interrupt,
and how Xen manipulates ICH_HCR_EL2, but that's the extent of what I
can do.

To help you further, I'd need a reproducer. I've asked you more than
once to provide a way to reproduce your setup, but got no answer. The
Debian package doesn't boot (it just messes up grub), and I don't have
the time to learn how to deal with Xen from scratch.

Until then, you'll have to apply some debugging by yourself.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.