[PATCH v2 00/36] Optimize KVM/ARM for VHE systems

[Yury Norov]

On Mon, Dec 11, 2017 at 04:34:58PM +0100, Christoffer Dall wrote:
> Hi Yury,
> 
> On Mon, Dec 11, 2017 at 05:43:23PM +0300, Yury Norov wrote:
> > 
> > On Thu, Dec 07, 2017 at 06:05:54PM +0100, Christoffer Dall wrote:
> > > This series redesigns parts of KVM/ARM to optimize the performance on
> > > VHE systems.  The general approach is to try to do as little work as
> > > possible when transitioning between the VM and the hypervisor.  This has
> > > the benefit of lower latency when waiting for interrupts and delivering
> > > virtual interrupts, and reduces the overhead of emulating behavior and
> > > I/O in the host kernel.
> > > 
> > > Patches 01 through 04 are not VHE specific, but rework parts of KVM/ARM
> > > that can be generally improved.  We then add infrastructure to move more
> > > logic into vcpu_load and vcpu_put, we improve handling of VFP and debug
> > > registers.
> > > 
> > > We then introduce a new world-switch function for VHE systems, which we
> > > can tweak and optimize for VHE systems.  To do that, we rework a lot of
> > > the system register save/restore handling and emulation code that may
> > > need access to system registers, so that we can defer as many system
> > > register save/restore operations to vcpu_load and vcpu_put, and move
> > > this logic out of the VHE world switch function.
> > > 
> > > We then optimize the configuration of traps.  On non-VHE systems, both
> > > the host and VM kernels run in EL1, but because the host kernel should
> > > have full access to the underlying hardware, but the VM kernel should
> > > not, we essentially make the host kernel more privileged than the VM
> > > kernel despite them both running at the same privilege level by enabling
> > > VE traps when entering the VM and disabling those traps when exiting the
> > > VM.  On VHE systems, the host kernel runs in EL2 and has full access to
> > > the hardware (as much as allowed by secure side software), and is
> > > unaffected by the trap configuration.  That means we can configure the
> > > traps for VMs running in EL1 once, and don't have to switch them on and
> > > off for every entry/exit to/from the VM.
> > > 
> > > Finally, we improve our VGIC handling by moving all save/restore logic
> > > out of the VHE world-switch, and we make it possible to truly only
> > > evaluate if the AP list is empty and not do *any* VGIC work if that is
> > > the case, and only do the minimal amount of work required in the course
> > > of the VGIC processing when we have virtual interrupts in flight.
> > > 
> > > The patches are based on v4.15-rc1 plus the fixes sent for v4.15-rc3
> > > [1], the level-triggered mapped interrupts support series [2], and the
> > > first five patches of James' SDEI series [3], a single SVE patch that
> > > moves the CPU ID reg trap setup out of the world-switch path, and v3 of
> > > my vcpu load/put series [4].
> > > 
> > > I've given the patches a fair amount of testing on Thunder-X, Mustang,
> > > Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE
> > > functionality on the Foundation model, running both 64-bit VMs and
> > > 32-bit VMs side-by-side and using both GICv3-on-GICv3 and
> > > GICv2-on-GICv3.
> > > 
> > > The patches are also available in the vhe-optimize-v2 branch on my
> > > kernel.org repository [5].
> > > 
> > > Changes since v1:
> > >  - Rebased on v4.15-rc1 and newer versions of other dependencies,
> > >    including the vcpu load/put approach taken for KVM.
> > >  - Addressed review comments from v1 (detailed changelogs are in the
> > >    individual patches).
> > > 
> > > Thanks,
> > > -Christoffer
> > > 
> > > [1]: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm kvm-arm-fixes-for-v4.15-1
> > > [2]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git level-mapped-v6
> > > [3]: git://linux-arm.org/linux-jm.git sdei/v5/base
> > > [4]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vcpu-load-put-v3
> > > [5]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v2
> > 
> > I just submitted the benchmark I used to test your v1 and v2 series':
> > https://lkml.org/lkml/2017/12/11/364
> > 
> > On ThunderX2, 112 online CPUs test results for v1 are like this:
> > 
> > Host, v4.14:
> > Dry-run:          0         1
> > Self-IPI:         9        18
> > Normal IPI:      81       110
> > Broadcast IPI:    0      2106
> > 
> > Guest, v4.14:
> > Dry-run:          0         1
> > Self-IPI:        10        18
> > Normal IPI:     305       525
> > Broadcast IPI:    0      9729
> > 
> > Guest, v4.14 + VHE:
> > Dry-run:          0         1
> > Self-IPI:         9        18
> > Normal IPI:     176       343
> > Broadcast IPI:    0      9885
> > 
> > And for v2.
> > 
> > Host, v4.15:                   
> > Dry-run:          0         1
> > Self-IPI:         9        18
> > Normal IPI:      79       108
> > Broadcast IPI:    0      2102
> >                         
> > Guest, v4.15-rc:
> > Dry-run:          0         1
> > Self-IPI:         9        18
> > Normal IPI:     291       526
> > Broadcast IPI:    0     10439
> > 
> > Guest, v4.15-rc + VHE:
> > Dry-run:          0         2
> > Self-IPI:        14        28
> > Normal IPI:     370       569
> > Broadcast IPI:    0     11688
> > 
> > All times are normalized to v1 host dry-run time. Smaller - better.
> > 
> 
> Thanks for running this.
> 
> > Results for v1 and v2 may vary because kernel version is changed. 
> > What makes us worry is slowing down the "Normal IPI" test observed in 
> > v2 series.
> 
> I'm wondering if this is not simply variability in your measurements.
> How many times have you run this?  The 100,000 iterations for each run
> is not a lot if you consider the cost of migrating threads.

I ran it more than 100 times, maybe more than 200. Variablity exists,
but ~5% at max, much less than observed changes. I can run 1M iterations
version to handle this concern.

> Is this workload pinned to a single CPU?

No. We are interested in test close to real usecases, so I didn't pin
the test. Inside, sending IPI and waiting for acknowledge is pinned
using {get,put}_cpu(). Tomorrow I'll run test pinned to some CPU. Are
you OK with 'taskset -c 111 insmod ipi_benchmark.ko'?

> Is the system otherwise idle (both host and guest)? 

Yes, this machine is in my exclusive usage, and I don't run something
heavy in background. And this is newly installed Ubuntu. 

> If you run this during boot or during kernel module load, the results
> may be skewed by that.

Hmm... I do it at module load, but there are many tests that measure
performance like this... Anyway, I'll check that.

> Power management can greatly influence results as well.

That's true. I'll check this also. But as you see, all host numbers,
and guest dry-run and self-ipi numbers are stable, except v2 test...

> Just so I'm sure we're reading these reults the same way, your "+ VHE"
> notation means the VHE optimization series, but both the before and
> after picture runs with VHE enabled, right?

Yes.

> Are you using the same guest kernel version and config for both your v1
> and v2 results, and for both the before and after versions?

I rebased v1 on 4.14. For v2 I ran make olddefconfig, the rest is same
as on your branches. I used same kernel image for host and guest, ie 
4.14 host + 4.14 guest for v1, and 4.15-rc host and guest for v2. I
also tested host with and without this series - no difference for both
versions.

> I can't easily come up with a scneario that explains the slowdown on the
> normal IPI test, beyond some unfortunate bug introduced in v2.
> 
> > 
> > Nevertheless, if you find test relevant, for v1 and v2,
> > Tested-by: Yury Norov <ynorov at caviumnetworks.com>
> 
> Thanks,
> -Christoffer