[PATCH] riscv: uprobes: Avoid redundant icache flushes for the XOL slot

[Davidlohr Bueso]

arch_uprobe_copy_ixol() unconditionally calls flush_icache_range(),
which has no range form on riscv and is a flush_icache_all(). It
runs once per single-step, as the generic code allocates and frees
an out-of-line slot around each one.

The allocator hands out the lowest free slot and the free path leaves
the slot contents alone, so a hot probe keeps getting the same slot
back holding the instruction it already wrote there. Rewriting it
costs a broadcast for nothing.

Return early when the slot already holds this instruction and its
breakpoint, similar to commit bdf94836c22a ("arm64: uprobes: Optimize
cache flushes for xol slot"). arm64 can compare against the source,
but riscv appends an ebreak after the opcode, so compare the
instruction and that breakpoint word instead.

Measured with the selftests/bpf bench on a SiFive P550, with one
producer pinned to a hart:

                            before       after
  trig-uprobe-nop           0.075 M/s    0.514 M/s    6.9x
  trig-uprobe-push          0.079 M/s    0.425 M/s    5.4x
  trig-uprobe-ret           0.075 M/s    0.514 M/s    6.9x
  trig-uprobe-multi-nop     0.079 M/s    0.548 M/s    6.9x
  trig-uprobe-multi-push    0.078 M/s    0.545 M/s    7.0x
  trig-uprobe-multi-ret     0.081 M/s    0.549 M/s    6.8x

The uretprobe variants gain 4.8x to 6.0x.

Signed-off-by: Davidlohr Bueso <dave at stgolabs.net>
---
 arch/riscv/kernel/probes/uprobes.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/arch/riscv/kernel/probes/uprobes.c b/arch/riscv/kernel/probes/uprobes.c
index eb177d0ce8ab..44e566f6b75b 100644
--- a/arch/riscv/kernel/probes/uprobes.c
+++ b/arch/riscv/kernel/probes/uprobes.c
@@ -168,15 +168,26 @@ void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr,
 	void *kaddr = kmap_local_page(page);
 	void *dst = kaddr + (vaddr & ~PAGE_MASK);
 	unsigned long start = (unsigned long)dst;
+	unsigned long insn_len;
+
+	insn_len = vaddr ? GET_INSN_LENGTH(*(probe_opcode_t *)src) : len;
+
+	/*
+	 * Slots are reused without being cleared, so a hot probe usually gets
+	 * one back already holding this instruction and its breakpoint. Skip
+	 * the rewrite and the flush then.
+	 */
+	if (!memcmp(dst, src, insn_len) &&
+	    (!vaddr || *(uprobe_opcode_t *)(dst + insn_len) == __BUG_INSN_32))
+		goto done;
 
 	memcpy(dst, src, len);
 
 	/* Add ebreak behind opcode to simulate singlestep */
-	if (vaddr) {
-		dst += GET_INSN_LENGTH(*(probe_opcode_t *)src);
-		*(uprobe_opcode_t *)dst = __BUG_INSN_32;
-	}
+	if (vaddr)
+		*(uprobe_opcode_t *)(dst + insn_len) = __BUG_INSN_32;
 
 	flush_icache_range(start, start + len);
+done:
 	kunmap_local(kaddr);
 }
-- 
2.39.5