[PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

[Eric Auger]

Hi Marc,

On 5/17/23 16:12, Marc Zyngier wrote:
> On Wed, 17 May 2023 09:59:45 +0100,
> Eric Auger <eauger at redhat.com> wrote:
>>
>> Hi Marc,
>>Hi Marc,
>> On 5/16/23 22:28, Marc Zyngier wrote:
>>> On Tue, 16 May 2023 17:53:14 +0100,
>>> Eric Auger <eauger at redhat.com> wrote:
>>>>
>>>> Hi Marc,
>>>>
>>>> On 5/15/23 19:30, Marc Zyngier wrote:
>>>>> This is the 4th drop of NV support on arm64 for this year.
>>>>>
>>>>> For the previous episodes, see [1].
>>>>>
>>>>> What's changed:
>>>>>
>>>>> - New framework to track system register traps that are reinjected in
>>>>>   guest EL2. It is expected to replace the discrete handling we have
>>>>>   enjoyed so far, which didn't scale at all. This has already fixed a
>>>>>   number of bugs that were hidden (a bunch of traps were never
>>>>>   forwarded...). Still a work in progress, but this is going in the
>>>>>   right direction.
>>>>>
>>>>> - Allow the L1 hypervisor to have a S2 that has an input larger than
>>>>>   the L0 IPA space. This fixes a number of subtle issues, depending on
>>>>>   how the initial guest was created.
>>>>>
>>>>> - Consequently, the patch series has gone longer again. Boo. But
>>>>>   hopefully some of it is easier to review...
>>>>>
>>>>> [1] https://lore.kernel.org/r/20230405154008.3552854-1-maz@kernel.org
>>>>
>>>> I have started testing this and when booting my fedora guest I get
>>>>
>>>> [  151.796544] kvm [7617]: Unsupported guest sys_reg access at:
>>>> 23f425fd0 [80000209]
>>>> [  151.796544]  { Op0( 3), Op1( 3), CRn(14), CRm( 3), Op2( 1), func_write },
>>>>
>>>> as soon as the host has kvm-arm.mode=nested
>>>>
>>>> This seems to be triggered very early by EDK2
>>>> (ArmPkg/Drivers/TimerDxe/TimerDxe.c).
>>>>
>>>> If I am not wrong this CNTV_CTL_EL0. Do you have any idea?
>>>
>>> So here's my current analysis:
>>>
>>> I assume you are running EDK2 as the L1 guest in a nested
>>> configuration. I also assume that you are not running on an Apple
>>> CPU. If these assumptions are correct, then EDK2 runs at vEL2, and is
>>> in nVHE mode.
>>>
>>> Finally, I'm going to assume that your implementation has FEAT_ECV and
>>> FEAT_NV2, because I can't see how it could fail otherwise.
>> all the above is correct.
>>>
>>> In these precise conditions, KVM sets the CNTHCTL_EL2.EL1TVT bit so
>>> that we can trap the EL0 virtual timer and faithfully emulate it (it
>>> is otherwise written to memory, which isn't very helpful).
>>
>> indeed
>>>
>>> As it turns out, we don't handle these traps. I didn't spot it because
>>> my test machines are all Apple boxes that don't have a nVHE mode, so
>>> nothing on the nVHE path is getting *ANY* coverage. Hint: having
>>> access to such a machine would help (shipping address on request!).
>>> Otherwise, I'll eventually kill the nVHE support altogether.
>>>
>>> I have written the following patch, which compiles, but that I cannot
>>> test with my current setup. Could you please give it a go?
>>
>> with the patch below, my guest boots nicely. You did it great on the 1st
>> shot!!! So this fixes my issue. I will continue testing the v10.
> 
> Thanks a lot for reporting the issue and testing my hacks. I'll
> eventually fold it into the rest of the series.
> 
> By the way, what are you using as your VMM? I'd really like to
> reproduce your setup.
Sorry I missed your reply. I am using libvirt + qemu (feat Miguel's RFC)
and fedora L1 guest.

Thanks to your fix, this boots fine. But at the moment it does not
reboot and hangs in edk2 I think. Unfortunately this time I have no
trace on host :-( While looking at your series I will add some traces.

Eric
> 
> Cheers,
> 
> 	M.
>