[PATCH] arm64: mte: switch GCR_EL1 on task switch rather than entry/exit

[Catalin Marinas]

On Fri, Jul 02, 2021 at 11:39:33AM -0700, Peter Collingbourne wrote:
> On Fri, Jul 2, 2021 at 10:44 AM Catalin Marinas <catalin.marinas at arm.com> wrote:
> > On Thu, Jul 01, 2021 at 08:19:22PM -0700, Peter Collingbourne wrote:
> > > Accessing GCR_EL1 and issuing an ISB can be expensive on some
> > > microarchitectures. To avoid taking this performance hit on every
> > > kernel entry/exit, switch GCR_EL1 on task switch rather than
> > > entry/exit.
> >
> > Is it the ISB that's causing issues or the MRS/MSR as well? I think we
> > can avoid the ISB when PtrAuth is enabled by shuffling the entry code a
> > bit. We can also simplify the mte_set_gcr macro to avoid an MRS.
> 
> This was the first thing that I tried on our hardware:
> 
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index 7312eafec946..8699ab28a924 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -204,7 +204,6 @@ alternative_else_nop_endif
>         ldr_l   \tmp, gcr_kernel_excl
> 
>         mte_set_gcr \tmp, \tmp2
> -       isb
>  1:
>  #endif
>         .endm
> @@ -277,13 +276,13 @@ alternative_if ARM64_HAS_ADDRESS_AUTH
>         orr     x0, x0, SCTLR_ELx_ENIA
>         msr     sctlr_el1, x0
>  2:
> -       isb
>  alternative_else_nop_endif
>  #endif
> 
>         apply_ssbd 1, x22, x23
> 
>         mte_set_kernel_gcr x22, x23
> +       isb
> 
>         scs_load tsk, x20
>         .else
> 
> However, on most of the cores this led to only around half of the
> performance improvement of the patch that I sent. Which is somewhat
> surprising, but it is what it is.

BTW, can you also modify mte_set_kernel_gcr to only do a write to the
GCR_EL1 register rather than a read-modify-write?

-- 
Catalin