[PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

[Miguel Luis]

Hi Eric,

> On 7 Jun 2023, at 16:40, Eric Auger <eauger at redhat.com> wrote:
> 
> Hi Miguel,
> 
> On 6/6/23 19:52, Miguel Luis wrote:
>> Hello Eric, Marc,
>> 
>>> On 6 Jun 2023, at 09:33, Eric Auger <eauger at redhat.com> wrote:
>>> 
>>> Hi Marc,
>>> 
>>> On 5/17/23 16:12, Marc Zyngier wrote:
>>>> On Wed, 17 May 2023 09:59:45 +0100,
>>>> Eric Auger <eauger at redhat.com> wrote:
>>>>> 
>>>>> Hi Marc,
>>>>> Hi Marc,
>>>>> On 5/16/23 22:28, Marc Zyngier wrote:
>>>>>> On Tue, 16 May 2023 17:53:14 +0100,
>>>>>> Eric Auger <eauger at redhat.com> wrote:
>>>>>>> 
>>>>>>> Hi Marc,
>>>>>>> 
>>>>>>> On 5/15/23 19:30, Marc Zyngier wrote:
>>>>>>>> This is the 4th drop of NV support on arm64 for this year.
>>>>>>>> 
>>>>>>>> For the previous episodes, see [1].
>>>>>>>> 
>>>>>>>> What's changed:
>>>>>>>> 
>>>>>>>> - New framework to track system register traps that are reinjected in
>>>>>>>> guest EL2. It is expected to replace the discrete handling we have
>>>>>>>> enjoyed so far, which didn't scale at all. This has already fixed a
>>>>>>>> number of bugs that were hidden (a bunch of traps were never
>>>>>>>> forwarded...). Still a work in progress, but this is going in the
>>>>>>>> right direction.
>>>>>>>> 
>>>>>>>> - Allow the L1 hypervisor to have a S2 that has an input larger than
>>>>>>>> the L0 IPA space. This fixes a number of subtle issues, depending on
>>>>>>>> how the initial guest was created.
>>>>>>>> 
>>>>>>>> - Consequently, the patch series has gone longer again. Boo. But
>>>>>>>> hopefully some of it is easier to review...
>>>>>>>> 
>>>>>>>> [1] https://lore.kernel.org/r/20230405154008.3552854-1-maz@kernel.org
>>>>>>> 
>>>>>>> I have started testing this and when booting my fedora guest I get
>>>>>>> 
>>>>>>> [  151.796544] kvm [7617]: Unsupported guest sys_reg access at:
>>>>>>> 23f425fd0 [80000209]
>>>>>>> [  151.796544]  { Op0( 3), Op1( 3), CRn(14), CRm( 3), Op2( 1), func_write },
>>>>>>> 
>>>>>>> as soon as the host has kvm-arm.mode=nested
>>>>>>> 
>>>>>>> This seems to be triggered very early by EDK2
>>>>>>> (ArmPkg/Drivers/TimerDxe/TimerDxe.c).
>>>>>>> 
>>>>>>> If I am not wrong this CNTV_CTL_EL0. Do you have any idea?
>>>>>> 
>>>>>> So here's my current analysis:
>>>>>> 
>>>>>> I assume you are running EDK2 as the L1 guest in a nested
>>>>>> configuration. I also assume that you are not running on an Apple
>>>>>> CPU. If these assumptions are correct, then EDK2 runs at vEL2, and is
>>>>>> in nVHE mode.
>>>>>> 
>>>>>> Finally, I'm going to assume that your implementation has FEAT_ECV and
>>>>>> FEAT_NV2, because I can't see how it could fail otherwise.
>>>>> all the above is correct.
>>>>>> 
>>>>>> In these precise conditions, KVM sets the CNTHCTL_EL2.EL1TVT bit so
>>>>>> that we can trap the EL0 virtual timer and faithfully emulate it (it
>>>>>> is otherwise written to memory, which isn't very helpful).
>>>>> 
>>>>> indeed
>>>>>> 
>>>>>> As it turns out, we don't handle these traps. I didn't spot it because
>>>>>> my test machines are all Apple boxes that don't have a nVHE mode, so
>>>>>> nothing on the nVHE path is getting *ANY* coverage. Hint: having
>>>>>> access to such a machine would help (shipping address on request!).
>>>>>> Otherwise, I'll eventually kill the nVHE support altogether.
>>>>>> 
>>>>>> I have written the following patch, which compiles, but that I cannot
>>>>>> test with my current setup. Could you please give it a go?
>>>>> 
>>>>> with the patch below, my guest boots nicely. You did it great on the 1st
>>>>> shot!!! So this fixes my issue. I will continue testing the v10.
>>>> 
>>>> Thanks a lot for reporting the issue and testing my hacks. I'll
>>>> eventually fold it into the rest of the series.
>>>> 
>>>> By the way, what are you using as your VMM? I'd really like to
>>>> reproduce your setup.
>>> Sorry I missed your reply. I am using libvirt + qemu (feat Miguel's RFC)
>>> and fedora L1 guest.
>>> 
>> 
>> Following this subject, I’ve forward ported Alexandru’s KUT patches
>> ( and I encourage others to do it also =) ) which expose an EL2 test that
> 
> Do you have a branch available with Alexandru's rebased kut series?

Now I do :)

https://github.com/mluis/kvm-unit-tests/tree/nv-WIP

Thanks,
Miguel

> 
> Thanks
> 
> Eric
>> does three checks:
>> 
>> - whether VHE is supported and enabled
>> - disable VHE
>> - re-enable VHE  
>> 
>> I’m running qemu with virtualization=on as well to run this test and it is passing although
>> problems seem to happen when running with virtualization=off, which I’m still looking into it.
>> 
>> Thanks
>> Miguel
>> 
>>> Thanks to your fix, this boots fine. But at the moment it does not
>>> reboot and hangs in edk2 I think. Unfortunately this time I have no
>>> trace on host :-( While looking at your series I will add some traces.
>>> 
>>> Eric
>>>> 
>>>> Cheers,
>>>> 
>>>> M.