[v3 00/10] Improve RISC-V Perf support using SBI PMU and sscofpmf extension
Palmer Dabbelt
palmer at dabbelt.com
Mon Oct 4 11:20:14 PDT 2021
On Fri, 10 Sep 2021 12:27:47 PDT (-0700), Atish Patra wrote:
> This series adds improved perf support for RISC-V based system using
> SBI PMU extension[1] and Sscofpmf extension[2]. The SBI PMU extension allows
Last we talked the SBI-0.3 stuff was in an uncertain state and I'm not
sure we ever got to a point of agreement there. I've decided to just
stop worrying about the state of extensions, so if you guys want the
SBI-0.3 stuff merged then just go say it's frozen and that'll be good
enough for me.
> the kernel to program the counters for different events and start/stop counters
> while the sscofpmf extension allows the counter overflow interrupt and privilege
> mode filtering. An hardware platform can leverage SBI PMU extension without
> the sscofpmf extension if it supports mcountinhibit and mcounteren. However,
> the reverse is not true. With both of these extension enabled, a platform can
> take advantage of all both event counting and sampling using perf tool.
>
> This series introduces a platform perf driver instead of a existing arch
> specific implementation. The new perf implementation has adopted a modular
> approach where most of the generic event handling is done in the core library
> while individual PMUs need to only implement necessary features specific to
> the PMU. This is easily extensible and any future RISC-V PMU implementation
> can leverage this. Currently, SBI PMU driver & legacy PMU driver are implemented
> as a part of this series.
>
> The legacy driver tries to reimplement the existing minimal perf under a new
> config to maintain backward compatibility. This implementation only allows
> monitoring of always running cycle/instruction counters. Moreover, they can
> not be started or stopped. In general, this is very limited and not very useful.
> That's why, I am not very keen to carry the support into the new driver.
> However, I don't want to break perf for any existing hardware platforms.
> If nobody really uses perf currently, I will be happy to drop PATCH 4.
>
> This series has been tested in Qemu on both RV64 & RV32. Qemu[5] & OpenSBI [3]
> patches are required to test it. Qemu changes are not backward compatible.
> That means, you can not use perf anymore on older Qemu versions with latest
> OpenSBI and/or Kernel. However, newer kernel will just use legacy pmu driver if
> old OpenSBI is detected or hardware doesn't implement mcountinhibit.
>
> Here is an output of perf stat/report while running hackbench with OpenSBI & Linux
> kernel patches applied [3].
>
> Perf stat:
> =========
>
> [root at fedora-riscv riscv]# perf stat -e r8000000000000005 -e r8000000000000007
> -e r8000000000000006 -e r0000000000020002 -e r0000000000020004 -e branch-misses
> -e cache-misses -e dTLB-load-misses -e dTLB-store-misses -e iTLB-load-misses
> -e cycles -e instructions ./hackbench -pipe 15 process
> Running with 15*40 (== 600) tasks.
> Time: 6.578
>
> Performance counter stats for './hackbench -pipe 15 process':
>
> 6,491 r8000000000000005 (52.59%) --> SBI_PMU_FW_SET_TIMER
> 20,433 r8000000000000007 (60.74%) --> SBI_PMU_FW_IPI_RECVD
> 21,271 r8000000000000006 (68.71%) --> SBI_PMU_FW_IPI_SENT
> 0 r0000000000020002 (76.55%)
> <not counted> r0000000000020004 (0.00%)
> <not counted> branch-misses (0.00%)
> <not counted> cache-misses (0.00%)
> 57,537,853 dTLB-load-misses (9.49%)
> 2,821,147 dTLB-store-misses (18.64%)
> 52,928,130 iTLB-load-misses (27.53%)
> 89,521,791,110 cycles (36.08%)
> 90,678,132,464 instructions # 1.01 insn per cycle (44.44%)
>
> 6.975908032 seconds time elapsed
>
> 3.130950000 seconds user
> 24.353310000 seconds sys
>
> The patches can also be found in the github[4].
>
> Perf record:
> ============
> [root at fedora-riscv riscv]# perf record -e cycles -e instructions -e \
> dTLB-load-misses -e dTLB-store-misses -c 1000 ./hackbench -pipe 15 process 15
> Running with 15*40 (== 600) tasks.
> Time: 1.238
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.106 MB perf.data (1020 samples) ]
>
> [root at fedora-riscv riscv]# perf report
> Available samples
> 372 cycles ◆
> 372 instructions ▒
> 262 dTLB-load-misses ▒
> 14 dTLB-store-misses
>
> The patches can also be found in the github[4].
>
> [1] https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/riscv-sbi.adoc
> [2] https://drive.google.com/file/d/171j4jFjIkKdj5LWcExphq4xG_2sihbfd/edit
> [3] https://github.com/atishp04/opensbi/tree/pmu_sscofpmf
> [4] https://github.com/atishp04/linux/tree/riscv_pmu_v3
> [5] https://github.com/atishp04/qemu/tree/riscv_pmu_v2
>
> Changes from v3->v4:
> 1. Added interrupt overflow support.
> 2. Cleaned up legacy driver initialization.
> 3. Supports perf record now.
> 4. Added the DT binding and maintainers file.
> 5. Changed cpu hotplug notifier to be multi-state.
> 6. OpenSBI doesn't disable cycle/instret counter during boot. Update the
> perf code to disable all the counter during the boot.
>
> Changes from v1->v2
> 1. Implemented the latest SBI PMU extension specification.
> 2. The core platform driver was changed to operate as a library while only
> sbi based PMU is built as a driver. The legacy one is just a fallback if
> SBI PMU extension is not available.
>
> Atish Patra (10):
> RISC-V: Remove the current perf implementation
> RISC-V: Add CSR encodings for all HPMCOUNTERS
> RISC-V: Add a perf core library for pmu drivers
> RISC-V: Add a simple platform driver for RISC-V legacy perf
> RISC-V: Add RISC-V SBI PMU extension definitions
> dt-binding: pmu: Add RISC-V PMU DT bindings
> RISC-V: Add perf platform driver based on SBI PMU extension
> RISC-V: Add interrupt support for perf
> Documentation: riscv: Remove the old documentation
> MAINTAINERS: Add entry for RISC-V PMU drivers
>
> .../devicetree/bindings/perf/riscv,pmu.yaml | 51 ++
> Documentation/riscv/pmu.rst | 255 ------
> MAINTAINERS | 10 +
> arch/riscv/Kconfig | 13 -
> arch/riscv/include/asm/csr.h | 66 +-
> arch/riscv/include/asm/perf_event.h | 72 --
> arch/riscv/include/asm/sbi.h | 97 +++
> arch/riscv/kernel/Makefile | 1 -
> arch/riscv/kernel/perf_event.c | 485 ------------
> drivers/perf/Kconfig | 25 +
> drivers/perf/Makefile | 5 +
> drivers/perf/riscv_pmu.c | 331 ++++++++
> drivers/perf/riscv_pmu_legacy.c | 143 ++++
> drivers/perf/riscv_pmu_sbi.c | 731 ++++++++++++++++++
> include/linux/cpuhotplug.h | 1 +
> include/linux/perf/riscv_pmu.h | 69 ++
> 16 files changed, 1528 insertions(+), 827 deletions(-)
> create mode 100644 Documentation/devicetree/bindings/perf/riscv,pmu.yaml
> delete mode 100644 Documentation/riscv/pmu.rst
> delete mode 100644 arch/riscv/kernel/perf_event.c
> create mode 100644 drivers/perf/riscv_pmu.c
> create mode 100644 drivers/perf/riscv_pmu_legacy.c
> create mode 100644 drivers/perf/riscv_pmu_sbi.c
> create mode 100644 include/linux/perf/riscv_pmu.h
More information about the linux-riscv
mailing list