Any way to disable KVM VHE extension?

[Robin Murphy]

On 2021-07-15 10:44, Qu Wenruo wrote:
> 
> 
> On 2021/7/15 下午5:28, Robin Murphy wrote:
>> On 2021-07-15 09:55, Qu Wenruo wrote:
>>> Hi,
>>>
>>> Recently I'm playing around the Nvidia Xavier AGX board, which has 
>>> VHE extension support.
>>>
>>> In theory, considering the CPU and memory, it should be pretty 
>>> powerful compared to boards like RPI CM4.
>>>
>>> But to my surprise, KVM runs pretty poor on Xavier.
>>>
>>> Just booting the edk2 firmware could take over 10s, and 20s to fully 
>>> boot the kernel.
>>> Even my VM on RPI CM4 has way faster boot time, even just running on 
>>> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
>>>
>>> This is definitely out of my expectation, I double checked to be sure 
>>> that it's running in KVM mode.
>>>
>>> But further digging shows that, since Xavier AGX CPU supports VHE, 
>>> kvm is running in VHE mode other than HYP mode on CM4.
>>>
>>> Is there anyway to manually disable VHE mode to test the more common 
>>> HYP mode on Xavier?
>>
>> According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
>> low-level equivalent "id_aa64mmfr1.vh=0") on the command line should 
>> do that.
> 
> Thanks for this one, I stupidly only searched modinfo of kvm, and didn't 
> even bother to search arch/arm64/kvm...
> 
>>
>> However I'd imagine the discrepancy is likely to be something more 
>> fundamental to the wildly different microarchitectures. There's 
>> certainly no harm in giving non-VHE a go for comparison, but I 
>> wouldn't be surprised if it turns out even slower...
> 
> You're totally right, with nvhe mode, it's still the same slow speed.
> 
> BTW, what did you mean by the "wildly different microarch"?
> Is ARMv8.2 arch that different from ARMv8 of RPI4?

I don't mean Armv8.x architectural features, I mean the actual 
implementation of NVIDIA's Carmel core is very, very different from 
Cortex-A72 or indeed our newer v8.2 Cortex-A designs.

> And any extra methods I could try to explore the reason of the slowness?

I guess the first check would be whether you're trapping and exiting the 
VM significantly more. I believe there are stats somewhere, but I don't 
know exactly where, sorry - I know very little about actually *using* KVM :)

If it's not that, then it might just be that EDK2 is doing a lot of 
cache maintenance or system register modification or some other 
operation that happens to be slower on Carmel compared to Cortex-A72.

Robin.

> At least RPI CM4 is beyond my expectation and is working pretty fine.
> 
> Thanks,
> Qu
> 
>>
>> Robin.
>>
>>> BTW, this is the dmesg related to KVM on Xavier, running v5.13 
>>> upstream kernel, with 64K page size:
>>> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
>>> [    0.857378] kvm [1]: vgic interrupt IRQ9
>>> [    0.862122] kvm: pmu event creation failed -2
>>> [    0.866734] kvm [1]: VHE mode initialized successfully
>>>
>>> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
>>> dtb), with 4K page size:
>>> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
>>> [    1.278425] kvm [1]: vgic interrupt IRQ9
>>> [    1.278620] kvm [1]: Hyp mode initialized successfully
>>>
>>> Could it be the PAGE size causing problem?
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel at lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
>