Any way to disable KVM VHE extension?

[Mark Rutland]

On Thu, Jul 15, 2021 at 11:00:42AM +0100, Robin Murphy wrote:
> On 2021-07-15 10:44, Qu Wenruo wrote:
> > 
> > 
> > On 2021/7/15 下午5:28, Robin Murphy wrote:
> > > On 2021-07-15 09:55, Qu Wenruo wrote:
> > > > Hi,
> > > > 
> > > > Recently I'm playing around the Nvidia Xavier AGX board, which
> > > > has VHE extension support.
> > > > 
> > > > In theory, considering the CPU and memory, it should be pretty
> > > > powerful compared to boards like RPI CM4.
> > > > 
> > > > But to my surprise, KVM runs pretty poor on Xavier.
> > > > 
> > > > Just booting the edk2 firmware could take over 10s, and 20s to
> > > > fully boot the kernel.
> > > > Even my VM on RPI CM4 has way faster boot time, even just
> > > > running on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> > > > 
> > > > This is definitely out of my expectation, I double checked to be
> > > > sure that it's running in KVM mode.
> > > > 
> > > > But further digging shows that, since Xavier AGX CPU supports
> > > > VHE, kvm is running in VHE mode other than HYP mode on CM4.
> > > > 
> > > > Is there anyway to manually disable VHE mode to test the more
> > > > common HYP mode on Xavier?
> > > 
> > > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its
> > > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should
> > > do that.
> > 
> > Thanks for this one, I stupidly only searched modinfo of kvm, and didn't
> > even bother to search arch/arm64/kvm...
> > 
> > > 
> > > However I'd imagine the discrepancy is likely to be something more
> > > fundamental to the wildly different microarchitectures. There's
> > > certainly no harm in giving non-VHE a go for comparison, but I
> > > wouldn't be surprised if it turns out even slower...
> > 
> > You're totally right, with nvhe mode, it's still the same slow speed.
> > 
> > BTW, what did you mean by the "wildly different microarch"?
> > Is ARMv8.2 arch that different from ARMv8 of RPI4?
> 
> I don't mean Armv8.x architectural features, I mean the actual
> implementation of NVIDIA's Carmel core is very, very different from
> Cortex-A72 or indeed our newer v8.2 Cortex-A designs.
> 
> > And any extra methods I could try to explore the reason of the slowness?
> 
> I guess the first check would be whether you're trapping and exiting the VM
> significantly more. I believe there are stats somewhere, but I don't know
> exactly where, sorry - I know very little about actually *using* KVM :)
> 
> If it's not that, then it might just be that EDK2 is doing a lot of cache
> maintenance or system register modification or some other operation that
> happens to be slower on Carmel compared to Cortex-A72.

It would also be worthchecking tha the CPUs are running at the speed you
expect, in e.g. case the lack of a DVFS driver means they're running
slow, and this just happens to be more noticeable in a VM.

You can estimate that by using `perf stat` on the host on a busy loop,
and looking what the cpu-cycles count implies.

Thanks,
Mark.