[PATCH v2 0/4] KVM: arm64: Improve PMU support on heterogeneous systems

[Alexandru Elisei]

Hi Reiji,

On Sun, Dec 12, 2021 at 10:36:52PM -0800, Reiji Watanabe wrote:
> On Wed, Dec 8, 2021 at 12:05 AM Marc Zyngier <maz at kernel.org> wrote:
> >
> > Reji,
> >
> > On 2021-12-08 02:36, Reiji Watanabe wrote:
> > > Hi Alex,
> > >
> > > On Mon, Dec 6, 2021 at 9:02 AM Alexandru Elisei
> > > <alexandru.elisei at arm.com> wrote:
> > >>
> > >> (CC'ing Peter Maydell in case this might be of interest to qemu)
> > >>
> > >> The series can be found on a branch at [1], and the kvmtool support at
> > >> [2].
> > >> The kvmtool patches are also on the mailing list [3] and haven't
> > >> changed
> > >> since v1.
> > >>
> > >> Detailed explanation of the issue and symptoms that the patches
> > >> attempt to
> > >> correct can be found in the cover letter for v1 [4].
> > >>
> > >> A brief summary of the problem is that on heterogeneous systems KVM
> > >> will
> > >> always use the same PMU for creating the VCPU events for *all* VCPUs
> > >> regardless of the physical CPU on which the VCPU is running, leading
> > >> to
> > >> events suddenly stopping and resuming in the guest as the VCPU thread
> > >> gets
> > >> migrated across different CPUs.
> > >>
> > >> This series proposes to fix this behaviour by allowing the user to
> > >> specify
> > >> which physical PMU is used when creating the VCPU events needed for
> > >> guest
> > >> PMU emulation. When the PMU is set, KVM will refuse to the VCPU on a
> > >> physical which is not part of the supported CPUs for the specified
> > >> PMU.
> > >
> > > Just to confirm, this series provides an API for userspace to request
> > > KVM to detect a wrong affinity setting due to a userspace bug so that
> > > userspace can get an error at KVM_RUN instead of leading to events
> > > suddenly stopping, correct ?
> >
> > More than that, it allows userspace to select which PMU will be used
> > for their guest. The affinity setting is a byproduct of the PMU's own
> > affinity.
> 
> Thank you for the clarification.
> (I overlooked the change in kvm_pmu_create_perf_event()...)
> 
> 
> > >
> > >> The default behaviour stays the same - without userspace setting the
> > >> PMU,
> > >> events will stop counting if the VCPU is scheduled on the wrong CPU.
> > >
> > > Can't we fix the default behavior (in addition to the current fix) ?
> > > (Do we need to maintain the default behavior ??)
> >
> > Of course we do. This is a behaviour that has been exposed to userspace
> > for years, and *we don't break userspace*.
> 
> I'm wondering if it might be better to have kvm_pmu_create_perf_event()
> set attr.type to pmu_id based on the current (physical) CPU by default
> on such heterogeneous systems (even if userspace don't explicitly
> specify pmu_id with the new API).  Then, by setting the CPU affinity,
> the PMU in that environment can behave predictably even with existing
> userspace (or maybe this won't be helpful at all?).

I think then you would end up with the possible mismatch between
kvm->arch.pmuver and the version of the PMU that is used for creating the
events.

Also, as VCPUs get migrated from one physical CPU to the other, the
semantics of the microarchitectural events change, even if the event ID is
the same.

Thanks,
Alex

> 
> Thanks,
> Reiji