[PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

[Miguel Luis]

Hello Eric, Marc,

> On 6 Jun 2023, at 09:33, Eric Auger <eauger at redhat.com> wrote:
> 
> Hi Marc,
> 
> On 5/17/23 16:12, Marc Zyngier wrote:
>> On Wed, 17 May 2023 09:59:45 +0100,
>> Eric Auger <eauger at redhat.com> wrote:
>>> 
>>> Hi Marc,
>>> Hi Marc,
>>> On 5/16/23 22:28, Marc Zyngier wrote:
>>>> On Tue, 16 May 2023 17:53:14 +0100,
>>>> Eric Auger <eauger at redhat.com> wrote:
>>>>> 
>>>>> Hi Marc,
>>>>> 
>>>>> On 5/15/23 19:30, Marc Zyngier wrote:
>>>>>> This is the 4th drop of NV support on arm64 for this year.
>>>>>> 
>>>>>> For the previous episodes, see [1].
>>>>>> 
>>>>>> What's changed:
>>>>>> 
>>>>>> - New framework to track system register traps that are reinjected in
>>>>>>  guest EL2. It is expected to replace the discrete handling we have
>>>>>>  enjoyed so far, which didn't scale at all. This has already fixed a
>>>>>>  number of bugs that were hidden (a bunch of traps were never
>>>>>>  forwarded...). Still a work in progress, but this is going in the
>>>>>>  right direction.
>>>>>> 
>>>>>> - Allow the L1 hypervisor to have a S2 that has an input larger than
>>>>>>  the L0 IPA space. This fixes a number of subtle issues, depending on
>>>>>>  how the initial guest was created.
>>>>>> 
>>>>>> - Consequently, the patch series has gone longer again. Boo. But
>>>>>>  hopefully some of it is easier to review...
>>>>>> 
>>>>>> [1] https://lore.kernel.org/r/20230405154008.3552854-1-maz@kernel.org
>>>>> 
>>>>> I have started testing this and when booting my fedora guest I get
>>>>> 
>>>>> [  151.796544] kvm [7617]: Unsupported guest sys_reg access at:
>>>>> 23f425fd0 [80000209]
>>>>> [  151.796544]  { Op0( 3), Op1( 3), CRn(14), CRm( 3), Op2( 1), func_write },
>>>>> 
>>>>> as soon as the host has kvm-arm.mode=nested
>>>>> 
>>>>> This seems to be triggered very early by EDK2
>>>>> (ArmPkg/Drivers/TimerDxe/TimerDxe.c).
>>>>> 
>>>>> If I am not wrong this CNTV_CTL_EL0. Do you have any idea?
>>>> 
>>>> So here's my current analysis:
>>>> 
>>>> I assume you are running EDK2 as the L1 guest in a nested
>>>> configuration. I also assume that you are not running on an Apple
>>>> CPU. If these assumptions are correct, then EDK2 runs at vEL2, and is
>>>> in nVHE mode.
>>>> 
>>>> Finally, I'm going to assume that your implementation has FEAT_ECV and
>>>> FEAT_NV2, because I can't see how it could fail otherwise.
>>> all the above is correct.
>>>> 
>>>> In these precise conditions, KVM sets the CNTHCTL_EL2.EL1TVT bit so
>>>> that we can trap the EL0 virtual timer and faithfully emulate it (it
>>>> is otherwise written to memory, which isn't very helpful).
>>> 
>>> indeed
>>>> 
>>>> As it turns out, we don't handle these traps. I didn't spot it because
>>>> my test machines are all Apple boxes that don't have a nVHE mode, so
>>>> nothing on the nVHE path is getting *ANY* coverage. Hint: having
>>>> access to such a machine would help (shipping address on request!).
>>>> Otherwise, I'll eventually kill the nVHE support altogether.
>>>> 
>>>> I have written the following patch, which compiles, but that I cannot
>>>> test with my current setup. Could you please give it a go?
>>> 
>>> with the patch below, my guest boots nicely. You did it great on the 1st
>>> shot!!! So this fixes my issue. I will continue testing the v10.
>> 
>> Thanks a lot for reporting the issue and testing my hacks. I'll
>> eventually fold it into the rest of the series.
>> 
>> By the way, what are you using as your VMM? I'd really like to
>> reproduce your setup.
> Sorry I missed your reply. I am using libvirt + qemu (feat Miguel's RFC)
> and fedora L1 guest.
> 

Following this subject, I’ve forward ported Alexandru’s KUT patches
( and I encourage others to do it also =) ) which expose an EL2 test that
does three checks:

- whether VHE is supported and enabled
- disable VHE
- re-enable VHE  

I’m running qemu with virtualization=on as well to run this test and it is passing although
problems seem to happen when running with virtualization=off, which I’m still looking into it.

Thanks
Miguel

> Thanks to your fix, this boots fine. But at the moment it does not
> reboot and hangs in edk2 I think. Unfortunately this time I have no
> trace on host :-( While looking at your series I will add some traces.
> 
> Eric
>> 
>> Cheers,
>> 
>> M.