[PATCH v2 0/4] KVM: arm64: Improve PMU support on heterogeneous systems

[Reiji Watanabe]

Hi Alex,

On Mon, Dec 13, 2021 at 3:14 AM Alexandru Elisei
<alexandru.elisei at arm.com> wrote:
>
> Hi Reiji,
>
> On Sun, Dec 12, 2021 at 10:36:52PM -0800, Reiji Watanabe wrote:
> > On Wed, Dec 8, 2021 at 12:05 AM Marc Zyngier <maz at kernel.org> wrote:
> > >
> > > Reji,
> > >
> > > On 2021-12-08 02:36, Reiji Watanabe wrote:
> > > > Hi Alex,
> > > >
> > > > On Mon, Dec 6, 2021 at 9:02 AM Alexandru Elisei
> > > > <alexandru.elisei at arm.com> wrote:
> > > >>
> > > >> (CC'ing Peter Maydell in case this might be of interest to qemu)
> > > >>
> > > >> The series can be found on a branch at [1], and the kvmtool support at
> > > >> [2].
> > > >> The kvmtool patches are also on the mailing list [3] and haven't
> > > >> changed
> > > >> since v1.
> > > >>
> > > >> Detailed explanation of the issue and symptoms that the patches
> > > >> attempt to
> > > >> correct can be found in the cover letter for v1 [4].
> > > >>
> > > >> A brief summary of the problem is that on heterogeneous systems KVM
> > > >> will
> > > >> always use the same PMU for creating the VCPU events for *all* VCPUs
> > > >> regardless of the physical CPU on which the VCPU is running, leading
> > > >> to
> > > >> events suddenly stopping and resuming in the guest as the VCPU thread
> > > >> gets
> > > >> migrated across different CPUs.
> > > >>
> > > >> This series proposes to fix this behaviour by allowing the user to
> > > >> specify
> > > >> which physical PMU is used when creating the VCPU events needed for
> > > >> guest
> > > >> PMU emulation. When the PMU is set, KVM will refuse to the VCPU on a
> > > >> physical which is not part of the supported CPUs for the specified
> > > >> PMU.
> > > >
> > > > Just to confirm, this series provides an API for userspace to request
> > > > KVM to detect a wrong affinity setting due to a userspace bug so that
> > > > userspace can get an error at KVM_RUN instead of leading to events
> > > > suddenly stopping, correct ?
> > >
> > > More than that, it allows userspace to select which PMU will be used
> > > for their guest. The affinity setting is a byproduct of the PMU's own
> > > affinity.
> >
> > Thank you for the clarification.
> > (I overlooked the change in kvm_pmu_create_perf_event()...)
> >
> >
> > > >
> > > >> The default behaviour stays the same - without userspace setting the
> > > >> PMU,
> > > >> events will stop counting if the VCPU is scheduled on the wrong CPU.
> > > >
> > > > Can't we fix the default behavior (in addition to the current fix) ?
> > > > (Do we need to maintain the default behavior ??)
> > >
> > > Of course we do. This is a behaviour that has been exposed to userspace
> > > for years, and *we don't break userspace*.
> >
> > I'm wondering if it might be better to have kvm_pmu_create_perf_event()
> > set attr.type to pmu_id based on the current (physical) CPU by default
> > on such heterogeneous systems (even if userspace don't explicitly
> > specify pmu_id with the new API).  Then, by setting the CPU affinity,
> > the PMU in that environment can behave predictably even with existing
> > userspace (or maybe this won't be helpful at all?).
>
> I think then you would end up with the possible mismatch between
> kvm->arch.pmuver and the version of the PMU that is used for creating the
> events.

Yes, but, I would think we can have kvm_pmu_create_perf_event()
set vcpu->arch.pmu.arm_pmu based on the current physical CPU
when vcpu->arch.pmu.arm_pmu is null (then, the pmuver is handled
as if KVM_ARM_VCPU_PMU_V3_SET_PMU was done implicitly).


> Also, as VCPUs get migrated from one physical CPU to the other, the
> semantics of the microarchitectural events change, even if the event ID is
> the same.

Yes, I understand.  As mentioned, this can work only when the
CPU affinity is set for vCPU threads appropriately (, which could
be done even without changing userspace).

Thanks,
Reiji