[PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

[Ganapatrao Kulkarni]

On 07-07-2023 03:16 pm, Ganapatrao Kulkarni wrote:
> 
> 
> On 04-07-2023 06:01 pm, Ganapatrao Kulkarni wrote:
>>
>> Hi Marc,
>>
>> On 29-06-2023 12:33 pm, Marc Zyngier wrote:
>>> Hi Ganapatrao,
>>>
>>> On Wed, 28 Jun 2023 07:45:55 +0100,
>>> Ganapatrao Kulkarni <gankulkarni at os.amperecomputing.com> wrote:
>>>>
>>>>
>>>> Hi Marc,
>>>>
>>>>
>>>> On 15-05-2023 11:00 pm, Marc Zyngier wrote:
>>>>> This is the 4th drop of NV support on arm64 for this year.
>>>>>
>>>>> For the previous episodes, see [1].
>>>>>
>>>>> What's changed:
>>>>>
>>>>> - New framework to track system register traps that are reinjected in
>>>>>     guest EL2. It is expected to replace the discrete handling we have
>>>>>     enjoyed so far, which didn't scale at all. This has already 
>>>>> fixed a
>>>>>     number of bugs that were hidden (a bunch of traps were never
>>>>>     forwarded...). Still a work in progress, but this is going in the
>>>>>     right direction.
>>>>>
>>>>> - Allow the L1 hypervisor to have a S2 that has an input larger than
>>>>>     the L0 IPA space. This fixes a number of subtle issues, 
>>>>> depending on
>>>>>     how the initial guest was created.
>>>>>
>>>>> - Consequently, the patch series has gone longer again. Boo. But
>>>>>     hopefully some of it is easier to review...
>>>>>
>>>>
>>>> I am facing issue in booting NestedVM with V9 as well with 10 patchset.
>>>>
>>>> I have tried V9/V10 on Ampere platform using kvmtool and I could boot
>>>> Guest-Hypervisor and then NestedVM without any issue.
>>>> However when I try to boot using QEMU(not using EDK2/EFI),
>>>> Guest-Hypervisor is booted with Fedora 37 using virtio disk. From
>>>> Guest-Hypervisor console(or ssh shell), If I try to boot NestedVM,
>>>> boot hangs very early stage of the boot.
>>>>
>>>> I did some debug using ftrace and it seems the Guest-Hypervisor is
>>>> getting very high rate of arch-timer interrupts,
>>>> due to that all CPU time is going on in serving the Guest-Hypervisor
>>>> and it is never going back to NestedVM.
>>>>
>>>> I am using QEMU vanilla version v7.2.0 with top-up patches for NV [1]
>>>
>>> So I went ahead and gave QEMU a go. On my systems, *nothing* works (I
>>> cannot even boot a L1 with 'virtualization=on" (the guest is stuck at
>>> the point where virtio gets probed and waits for its first interrupt).
>>>
>>> Worse, booting a hVHE guest results in QEMU generating an assert as it
>>> tries to inject an interrupt using the QEMU GICv3 model, something
>>> that should *NEVER* be in use with KVM.
>>>
>>> With help from Eric, I got to a point where the hVHE guest could boot
>>> as long as I kept injecting console interrupts, which is again a
>>> symptom of the vGIC not being used.
>>>
>>> So something is *majorly* wrong with the QEMU patches. I don't know
>>> what makes it possible for you to even boot the L1 - if the GIC is
>>> external, injecting an interrupt in the L2 is simply impossible.
>>>
>>> Miguel, can you please investigate this?
>>>
>>> In the meantime, I'll add some code to the kernel side to refuse the
>>> external interrupt controller configuration with NV. Hopefully that
>>> will lead to some clues about what is going on.
>>
>> Continued debugging of the issue and it seems the endless ptimer 
>> interrupts on Ampere platform is due to some mess up of CVAL of 
>> ptimer, resulting in interrupt triggered always when it is enabled.
>>
>> I see function "timer_set_offset" called from kvm_arm_timer_set_reg in 
>> QEMU case but there is no such calls in kvmtool boot.
>>
>> If I comment the timer_set_offset calls in kvm_arm_timer_set_reg 
>> function, then I could boot the Guest-Hypervisor then NestedVM from 
>> GH/L1.
>>
>> I also observed in QEMU case, kvm_arm_timer_set_reg is called to set 
>> CNT, CVAL and CTL of both vtimer and ptimer.
>> Not sure why QEMU is setting these registers explicitly? need to dig.
>>
> 
> I don't see any direct ioctl calls to change any timer registers. Looks 
> like it is happening from the emulation code(target/arm/helper.c)?

function "write_list_to_kvmstate(target/arm/kvm.c)" is issuing ioctl to 
write timer registers. PTIMER_CNT and TIMER_CNT writes from this 
function is resulting in offsets change.

> 
>>>
>>> Thanks,
>>>
>>>     M.
>>>
>>
>> Thanks,
>> Ganapat
>>
> 
> Thanks,
> Ganapat

Thanks,
Ganapat