[External] [PATCH] RISC-V: store percpu offset in CSR_SCRATCH

[yunhui cui]

Hi Radim,

On Thu, Jul 10, 2025 at 2:35 PM Radim Krčmář <rkrcmar at ventanamicro.com> wrote:
>
> 2025-07-10T11:45:06+08:00, yunhui cui <cuiyunhui at bytedance.com>:
> > On Wed, Jul 9, 2025 at 10:20 PM Radim Krčmář <rkrcmar at ventanamicro.com> wrote:
> >> Is the overhead above with this patch?  And when we then use the
> >> CSR_SCRATCH for percpu, does it degrade even further?
> >
> > We can see that the percpu optimization is around 2.5% through the
> > method of fixing registers, and we can consider that the percpu
> > optimization can bring a 2.5% gain. Is there no need to add the percpu
> > optimization logic on the basis of the scratch patch for testing?
> >
> > Reference: https://lists.riscv.org/g/tech-privileged/message/2485
>
> That is when the value is in a GPR, though, and we don't know the
> performance of a CSR_SCRATCH access.
> We can hope that it's not much worse than a GPR, but an implementation
> might choose to be very slow with CSR_SCRATCH.
>
> I have in mind another method where we can use the current CSR_SCRATCH
> without changing CSR_TVAL, but I don't really want to spend time on it
> if reading the CSR doesn't give any benefit.
>
> It would be to store the percpu offset in CSR_SCRATCH permanently, do
> the early exception register shuffling with a percpu area storage, and
> load the thread pointer from there as well.
> That method would also eliminate writing CSR_SCRATCH on every exception
> entry+exit, so maybe it makes sense to try it even if CSRs are slow...
>
> Thanks.


Based on the patch, optimizations for percpu offset have been added,
with the following data:
6.989 7.046 6.976 6.986 7.001 7.017 7.007 7.064 7.008 7.039
Geometric mean: 7.013248303
Compared to reusing the scratch register, the performance has improved
by approximately 0.7%.

If more optimizations can be made to the scratch register, there
should be further performance improvements.

Patch:
---
 arch/riscv/include/asm/percpu.h | 14 ++++++++++++++
 arch/riscv/kernel/asm-offsets.c |  1 +
 arch/riscv/kernel/entry.S       |  7 +++++++
 arch/riscv/kernel/smpboot.c     |  3 +++
 4 files changed, 25 insertions(+)
 create mode 100644 arch/riscv/include/asm/percpu.h

diff --git a/arch/riscv/include/asm/percpu.h b/arch/riscv/include/asm/percpu.h
new file mode 100644
index 000000000000..1fbfcb108f84
--- /dev/null
+++ b/arch/riscv/include/asm/percpu.h
@@ -0,0 +1,14 @@
+#ifndef __ASM_PERCPU_H
+#define __ASM_PERCPU_H
+
+static inline void set_my_cpu_offset(unsigned long off)
+{
+        csr_write(CSR_SCRATCH, off);
+}
+
+#define __my_cpu_offset csr_read(CSR_SCRATCH)
+
+#include <asm-generic/percpu.h>
+
+#endif
+
diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c
index a03129f40c46..0ce96f30bf32 100644
--- a/arch/riscv/kernel/asm-offsets.c
+++ b/arch/riscv/kernel/asm-offsets.c
@@ -35,6 +35,7 @@ void asm_offsets(void)
  OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]);
  OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]);
  OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]);
+ OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu);
  OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags);
  OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count);
  OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp);
diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
index cc2fd4cd54a0..82caeee91c15 100644
--- a/arch/riscv/kernel/entry.S
+++ b/arch/riscv/kernel/entry.S
@@ -75,6 +75,13 @@ SYM_CODE_START_NOALIGN(handle_exception)
  REG_S s4, PT_CAUSE(sp)
  REG_S s5, PT_TP(sp)

+ REG_L s0, TASK_TI_CPU(tp)
+ slli s0, s0, 3
+ la s1, __per_cpu_offset
+ add s1, s1, s0
+ REG_L s1, 0(s1)
+ csrw CSR_SCRATCH, s1
+
  la s1, handle_kernel_exception
  csrw CSR_TVEC, s1

diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
index fb6ab7f8bfbd..6fa12cc84523 100644
--- a/arch/riscv/kernel/smpboot.c
+++ b/arch/riscv/kernel/smpboot.c
@@ -43,6 +43,7 @@ static DECLARE_COMPLETION(cpu_running);

 void __init smp_prepare_boot_cpu(void)
 {
+ set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
 }

 void __init smp_prepare_cpus(unsigned int max_cpus)
@@ -240,6 +241,8 @@ asmlinkage __visible void smp_callin(void)
  mmgrab(mm);
  current->active_mm = mm;

+ set_my_cpu_offset(per_cpu_offset(curr_cpuid));
+
  store_cpu_topology(curr_cpuid);
  notify_cpu_starting(curr_cpuid);

--
2.43.0

Thanks,
Yunhui