[PATCH v2 0/9] perf: Add NVIDIA Tegra410 uncore PMU events and metrics
Besar Wicaksono
bwicaksono at nvidia.com
Mon Nov 2 15:07:55 PST 2026
> -----Original Message-----
> From: Ian Rogers <irogers at google.com>
> Sent: Saturday, October 31, 2026 4:18 AM
> To: Besar Wicaksono <bwicaksono at nvidia.com>
> Cc: Will Deacon <will at kernel.org>; acme at kernel.org;
> namhyung at kernel.org; ctshao at google.com; linux-tegra at vger.kernel.org;
> linux-arm-kernel at lists.infradead.org; linux-perf-users at vger.kernel.org;
> linux-kernel at vger.kernel.org; Vikram Sethi <vsethi at nvidia.com>; Rich Wiley
> <rwiley at nvidia.com>; Sean Kelley <skelley at nvidia.com>; Matt Ochs
> <mochs at nvidia.com>; Raj Joshi <rajjoshi at nvidia.com>; Thierry Reding
> <treding at nvidia.com>; Jon Hunter <jonathanh at nvidia.com>;
> john.garry at linux.dev; james.clark at linaro.org; mike.leach at arm.com;
> leo.yan at linux.dev; mark.rutland at arm.com;
> alexander.shishkin at linux.intel.com; jolsa at kernel.org;
> adrian.hunter at intel.com
> Subject: Re: [PATCH v2 0/9] perf: Add NVIDIA Tegra410 uncore PMU events
> and metrics
>
> External email: Use caution opening links or attachments
>
>
> On Fri, Oct 30, 2026 at 3:19 PM Besar Wicaksono
> <bwicaksono at nvidia.com> wrote:
> >
> > > -----Original Message-----
> > > From: Ian Rogers <irogers at google.com>
> > > Sent: Friday, October 30, 2026 6:07 PM
> > > To: Besar Wicaksono <bwicaksono at nvidia.com>
> > > Cc: acme at kernel.org; namhyung at kernel.org; ctshao at google.com;
> linux-
> > > tegra at vger.kernel.org; linux-arm-kernel at lists.infradead.org; linux-perf-
> > > users at vger.kernel.org; linux-kernel at vger.kernel.org; Vikram Sethi
> > > <vsethi at nvidia.com>; Rich Wiley <rwiley at nvidia.com>; Sean Kelley
> > > <skelley at nvidia.com>; Matt Ochs <mochs at nvidia.com>; Raj Joshi
> > > <rajjoshi at nvidia.com>; Thierry Reding <treding at nvidia.com>; Jon
> Hunter
> > > <jonathanh at nvidia.com>; john.garry at linux.dev; will at kernel.org;
> > > james.clark at linaro.org; mike.leach at arm.com; leo.yan at linux.dev;
> > > mark.rutland at arm.com; alexander.shishkin at linux.intel.com;
> > > jolsa at kernel.org; adrian.hunter at intel.com
> > > Subject: Re: [PATCH v2 0/9] perf: Add NVIDIA Tegra410 uncore PMU
> events
> > > and metrics
> > >
> > > External email: Use caution opening links or attachments
> > >
> > >
> > > On Fri, Oct 30, 2026 at 10:49 AM Besar Wicaksono
> > > <bwicaksono at nvidia.com> wrote:
> > > >
> > > > Add event aliases and derived metrics for the NVIDIA Tegra410 uncore
> > > > PMUs, and fix perf's aggregation of PMU instances that are numbered
> in
> > > > more than one parts in their name.
> > > >
> > > > The series covers the UCF, PCIe, PCIe target, CPU memory latency, and
> > > > chip-to-chip interconnect PMUs. It provides symbolic names for the
> raw
> > > > event encodings and metrics for bandwidth, frequency, and average
> read
> > > > latency.
> > > >
> > > > The PCIe and PCIe target PMUs are named
> > > nvidia_pcie_pmu_<socket>_rc_<rc>
> > > > and nvidia_pcie_tgt_pmu_<socket>_rc_<rc>. Their json Unit uses the
> > > > existing comma-separated form ("nvidia_pcie_pmu_,rc_"), but perf
> only
> > > > merged instances that differ in the trailing number, so these events and
> > > > metrics gave one result per socket and were reported incorrectly (e.g.
> > > > with --metric-only), as discussed in v1. Patches 6-9 fix this:
> > > >
> > > > - Patch 6 adds a check of the cpumasks of all merged PMU instances to
> the
> > > > zero-count skip in perf stat, so a socket that counts nothing shows 0
> > > > instead of having its row hidden.
> > > > - Patch 7 merges PMU instances that match the same comma-
> separated
> > > json
> > > > Unit. This also applies to the HiSilicon hisi_sccl PMUs, which use the
> > > > same json form; not tested on HiSilicon hardware.
> > > > - Patch 8 labels a merged uncore counter after all of its PMUs, e.g.
> > > > nvidia_pcie_pmu_*_rc_*/cycles/ or nvidia_ucf_pmu/cycles/, instead
> of
> > > > after the first instance, such as nvidia_pcie_pmu_0_rc_0/cycles/.
> > > > Output with --no-aggr is unchanged.
> > > > - Patch 9 describes these PMU names in the ABI document.
> > > >
> > > > Changes in v2:
> > > > - Move the uncore JSON out of the sys directory and remove the
> > > > "Compat" fields. Thanks to Ian Rogers for the feedback.
> > > > - Drop "perf jevents: Process model files before sys subdirectories",
> > > > which this series no longer needs.
> > > > - Add UCF SLC and memory bandwidth metrics from all sources to local
> > > > CMEM (read, write and read+write) and from local sources to local
> > > > CMEM and to remote memory, PCIe bandwidth metrics to local
> > > > destinations, and read+write bandwidth metrics for UCF SLC, UCF
> > > > memory and PCIe. Thanks to Chun-Tse Shao for the suggestion.
> > > > - Add patches 6-9 to fix the PCIe instance aggregation. Thanks to Ian
> > > > Rogers and Chun-Tse Shao for the pointers.
> > > >
> > > > v1: https://lore.kernel.org/all/20261019164241.403644-1-
> > > bwicaksono at nvidia.com/
> > >
> > > Why don't we look to fix code in drivers/perf? This series introduces
> > > PMU instances, overloads the meaning of aliases, and a bunch of other
> > > silly complexity due to poor reviews and maintenance of drivers/perf -
> > > specifically by not pointing out the ABI issues. Isn't this a
> > > straightforward, whoops we did a bad suffix and here's the patch to
> > > rename our PMU in the driver? Throwing more complexity at the
> problem
> > > doesn't feel like the right fix.
> >
> > Hi Ian,
> >
> > My apologies for the names not following perf's PMU naming convention.
> > I will be more careful about this in future.
> >
> > These names (nvidia_pcie_pmu_<socket>_rc_<root complex> and
> > nvidia_pcie_tgt_pmu_<socket>_rc_<root complex>) have been upstream
> > since v7.1, and several customers already rely on them. Renaming them in
> > the driver would break their use case.
>
> It only breaks their use case if they name the PMU with an event. You
> have multiple naming conventions for the PMU and only those involving
> multiple sockets have a broken suffix.
>
Naming the PMU in the event is currently the main way to use the PCIE PMUs,
which we can't break.
> > Given that, what would you suggest as the next step? Would handling
> these
> > names in perf, as in patch 7 and 8, be reasonable, or do you have another
> > approach in mind?
>
> So we don't need a new abstraction, I've been reducing aggregation
> abstractions in the perf tool. As I pointed to previously, the suffix
> is computed with pmu_name_len_no_suffix:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
> next.git/tree/tools/perf/util/pmus.c?h=perf-tools-next#n66
> So you can do something like (untested):
> ```
> size_t pmu_name_len_no_suffix(const char *str)
> {
> if (strstarts(str, "nvidia_") && strstr(str, "_rc_")) {
> // nvidia PMU with a non-ABI suffix of nvidia_..._[0-9]+_rc_[0-9]+
> const char *p = str + strlen(str);
> int n = 3;
> while(p > str && n > 0) {
> if (*p-- == '_')
> n--
> }
> return p - str;
> }
> // The rest of the existing pmu_name_len_no_suffix code.
> ...
> }
> ```
Thanks for the suggestion. I can replace the changes in patch 7 and 8
with NVIDIA PCIE RC specific check in pmu_name_len_no_suffix().
While trying it, I noticed two side effects:
1. perf list now combines the nvidia_pcie_pmu_<socket>_rc_<rc> PMUs and
shows their events as nvidia_pcie_pmu/<event>/, but "perf stat" doesn't
accept just "nvidia_pcie_pmu" on the command line, e.g.
"perf stat -e nvidia_pcie_pmu/rd_req/" can't find the PMU.
2. After fixing no.1, "perf stat -A -a -e nvidia_pcie_pmu/rd_req/"
lists the result in sysfs directory order. On my system, it gives the
result for RC 0, 5, 3, 1, 4, 2.
Both look like they can be handled with a similar NVIDIA specific check
in perf_pmu__match_wildcard() and pmu_name_cmp():
/* perf_pmu__match_wildcard() */
if (strstarts(pmu_name, "nvidia") && strstr(pmu_name, "_rc_")) {
/* NVIDIA PMU with a non-ABI suffix of nvidia_..._[0-9]+_rc_[0-9]+. */
if (tok_len == pmu_name_len_no_suffix(pmu_name) && !strncmp(pmu_name, tok, tok_len))
return true;
}
/* pmu_name_cmp()*/
if (lhs_num == rhs_num && strstarts(lhs_pmu_name, "nvidia")) {
/*
* NVIDIA PMU with a non-ABI suffix of nvidia_..._[0-9]+_rc_[0-9]+:
* compare the RC too.
*/
const char *lhs_rc = strstr(lhs_pmu_name, "_rc_");
const char *rhs_rc = strstr(rhs_pmu_name, "_rc_");
if (lhs_rc && rhs_rc) {
lhs_num = strtoull(lhs_rc + 4, NULL, 10);
rhs_num = strtoull(rhs_rc + 4, NULL, 10);
}
}
What do you think about these additional changes?
Regards,
Besar
> I'd much rather we didn't add bespoke behavior to the tool for every
> new PMU. This happened previously when PMUs were merged in
> drivers/perf with hex instead of numeric suffixes. We already handle
> certain PMUs specially due to their lack of hotplug support:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
> next.git/tree/tools/perf/arch/arm/util/pmu.c?h=perf-tools-next#n41
> And due to event name fiascos:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
> next.git/tree/tools/perf/builtin-record.c?h=perf-tools-next#n1395
> At some point, perf's internals become so forked due to mismatched
> conventions that every code path will have a variant. This saddens me
> as something that we've much improved this year is being able to say
> `perf stat record` on one machine type and then `perf stat report` on
> another.
>
> Thanks,
> Ian
>
> > Will, do you have any comment on this?
> >
> > Thanks,
> > Besar
> >
> > >
> > > Thanks,
> > > Ian
> > >
> > > > Besar Wicaksono (9):
> > > > perf jevents: Add support for NVIDIA Tegra410 UCF PMU aliasing
> > > > perf jevents: Add support for NVIDIA Tegra410 PCIE PMU aliasing
> > > > perf jevents: Add support for NVIDIA Tegra410 PCIE-TGT PMU aliasing
> > > > perf jevents: Add support for NVIDIA Tegra410 CPU Memory Latency
> > > PMU
> > > > aliasing
> > > > perf jevents: Add support for NVIDIA Tegra410 C2C PMU aliasing
> > > > perf stat: Check merged PMU instance's cpumasks when skipping zero
> > > > counts
> > > > perf pmu: Merge PMU instances that match the same comma-
> separated
> > > json
> > > > Unit
> > > > perf evsel: Name a merged uncore counter after all of its PMUs
> > > > perf Documentation: Describe PMU names numbered in more than
> one
> > > place
> > > >
> > > > .../testing/sysfs-bus-event_source-devices | 14 +
> > > > .../arm64/nvidia/t410/uncore-cpu_mem.json | 43 +
> > > > .../arm64/nvidia/t410/uncore-nvclink.json | 67 ++
> > > > .../arm64/nvidia/t410/uncore-nvdlink.json | 43 +
> > > > .../arm64/nvidia/t410/uncore-nvlink_c2c.json | 115 +++
> > > > .../arch/arm64/nvidia/t410/uncore-pcie.json | 249 ++++++
> > > > .../arm64/nvidia/t410/uncore-pcie_tgt.json | 83 ++
> > > > .../arch/arm64/nvidia/t410/uncore-ucf.json | 764
> > > ++++++++++++++++++
> > > > tools/perf/tests/pmu.c | 134 +++
> > > > tools/perf/util/evlist.c | 2 +-
> > > > tools/perf/util/evsel.c | 83 +-
> > > > tools/perf/util/evsel.h | 2 +-
> > > > tools/perf/util/parse-events.c | 2 +-
> > > > tools/perf/util/pmu.c | 65 ++
> > > > tools/perf/util/pmu.h | 9 +
> > > > tools/perf/util/pmus.c | 47 ++
> > > > tools/perf/util/pmus.h | 1 +
> > > > tools/perf/util/stat-display.c | 20 +
> > > > 18 files changed, 1737 insertions(+), 6 deletions(-)
> > > > create mode 100644 tools/perf/pmu-
> > > events/arch/arm64/nvidia/t410/uncore-cpu_mem.json
> > > > create mode 100644 tools/perf/pmu-
> > > events/arch/arm64/nvidia/t410/uncore-nvclink.json
> > > > create mode 100644 tools/perf/pmu-
> > > events/arch/arm64/nvidia/t410/uncore-nvdlink.json
> > > > create mode 100644 tools/perf/pmu-
> > > events/arch/arm64/nvidia/t410/uncore-nvlink_c2c.json
> > > > create mode 100644 tools/perf/pmu-
> > > events/arch/arm64/nvidia/t410/uncore-pcie.json
> > > > create mode 100644 tools/perf/pmu-
> > > events/arch/arm64/nvidia/t410/uncore-pcie_tgt.json
> > > > create mode 100644 tools/perf/pmu-
> > > events/arch/arm64/nvidia/t410/uncore-ucf.json
> > > >
> > > > --
> > > > 2.43.0
> > > >
More information about the linux-arm-kernel
mailing list