[PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

[Miguel Luis]

Hi Marc,

> On 29 Jun 2023, at 07:03, Marc Zyngier <maz at kernel.org> wrote:
> 
> Hi Ganapatrao,
> 
> On Wed, 28 Jun 2023 07:45:55 +0100,
> Ganapatrao Kulkarni <gankulkarni at os.amperecomputing.com> wrote:
>> 
>> 
>> Hi Marc,
>> 
>> 
>> On 15-05-2023 11:00 pm, Marc Zyngier wrote:
>>> This is the 4th drop of NV support on arm64 for this year.
>>> 
>>> For the previous episodes, see [1].
>>> 
>>> What's changed:
>>> 
>>> - New framework to track system register traps that are reinjected in
>>>   guest EL2. It is expected to replace the discrete handling we have
>>>   enjoyed so far, which didn't scale at all. This has already fixed a
>>>   number of bugs that were hidden (a bunch of traps were never
>>>   forwarded...). Still a work in progress, but this is going in the
>>>   right direction.
>>> 
>>> - Allow the L1 hypervisor to have a S2 that has an input larger than
>>>   the L0 IPA space. This fixes a number of subtle issues, depending on
>>>   how the initial guest was created.
>>> 
>>> - Consequently, the patch series has gone longer again. Boo. But
>>>   hopefully some of it is easier to review...
>>> 
>> 
>> I am facing issue in booting NestedVM with V9 as well with 10 patchset.
>> 
>> I have tried V9/V10 on Ampere platform using kvmtool and I could boot
>> Guest-Hypervisor and then NestedVM without any issue.
>> However when I try to boot using QEMU(not using EDK2/EFI),
>> Guest-Hypervisor is booted with Fedora 37 using virtio disk. From
>> Guest-Hypervisor console(or ssh shell), If I try to boot NestedVM,
>> boot hangs very early stage of the boot.
>> 
>> I did some debug using ftrace and it seems the Guest-Hypervisor is
>> getting very high rate of arch-timer interrupts,
>> due to that all CPU time is going on in serving the Guest-Hypervisor
>> and it is never going back to NestedVM.
>> 
>> I am using QEMU vanilla version v7.2.0 with top-up patches for NV [1]
> 
> So I went ahead and gave QEMU a go. On my systems, *nothing* works (I
> cannot even boot a L1 with 'virtualization=on" (the guest is stuck at
> the point where virtio gets probed and waits for its first interrupt).
> 
> Worse, booting a hVHE guest results in QEMU generating an assert as it
> tries to inject an interrupt using the QEMU GICv3 model, something
> that should *NEVER* be in use with KVM.
> 
> With help from Eric, I got to a point where the hVHE guest could boot
> as long as I kept injecting console interrupts, which is again a
> symptom of the vGIC not being used.
> 
> So something is *majorly* wrong with the QEMU patches. I don't know
> what makes it possible for you to even boot the L1 - if the GIC is
> external, injecting an interrupt in the L2 is simply impossible.
> 
> Miguel, can you please investigate this?

Yes, I will investigate it. Sorry for the delay in replying as I took a break
short after KVM forum and I’ve just started to sync up.

Thanks,
Miguel

> 
> In the meantime, I'll add some code to the kernel side to refuse the
> external interrupt controller configuration with NV. Hopefully that
> will lead to some clues about what is going on.
> 
> Thanks,
> 
> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.