[PATCH] arm64: mte: switch GCR_EL1 on task switch rather than entry/exit

[Peter Collingbourne]

On Fri, Jul 2, 2021 at 10:44 AM Catalin Marinas <catalin.marinas at arm.com> wrote:
>
> On Thu, Jul 01, 2021 at 08:19:22PM -0700, Peter Collingbourne wrote:
> > Accessing GCR_EL1 and issuing an ISB can be expensive on some
> > microarchitectures. To avoid taking this performance hit on every
> > kernel entry/exit, switch GCR_EL1 on task switch rather than
> > entry/exit.
>
> Is it the ISB that's causing issues or the MRS/MSR as well? I think we
> can avoid the ISB when PtrAuth is enabled by shuffling the entry code a
> bit. We can also simplify the mte_set_gcr macro to avoid an MRS.

This was the first thing that I tried on our hardware:

diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 7312eafec946..8699ab28a924 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -204,7 +204,6 @@ alternative_else_nop_endif
        ldr_l   \tmp, gcr_kernel_excl

        mte_set_gcr \tmp, \tmp2
-       isb
 1:
 #endif
        .endm
@@ -277,13 +276,13 @@ alternative_if ARM64_HAS_ADDRESS_AUTH
        orr     x0, x0, SCTLR_ELx_ENIA
        msr     sctlr_el1, x0
 2:
-       isb
 alternative_else_nop_endif
 #endif

        apply_ssbd 1, x22, x23

        mte_set_kernel_gcr x22, x23
+       isb

        scs_load tsk, x20
        .else

However, on most of the cores this led to only around half of the
performance improvement of the patch that I sent. Which is somewhat
surprising, but it is what it is.

But I would like to get IRG out of the kernel (at least in production
kernels) for other reasons. I would at some point like to add a
deterministic IRG mode (to support record/replay debugging). This will
require setting RRND=0 and a per-task RGSR. If we then allow IRG in
the kernel we would need to manually switch RGSR here as well.

Peter