[PATCH v1 3/3] arm64/sve: Skip flushing Z registers with 128 bit vectors

[Dave P Martin]

On Mon, May 10, 2021 at 01:23:48PM +0100, Mark Brown wrote:
> When the SVE vector length is 128 bits then there are no bits in the Z
> registers which are not shared with the V registers so we can skip them
> when zeroing state not shared with FPSIMD, this results in a minor
> performance improvement.
> 
> Signed-off-by: Mark Brown <broonie at kernel.org>
> ---
>  arch/arm64/include/asm/fpsimd.h  | 2 +-
>  arch/arm64/kernel/entry-fpsimd.S | 9 +++++++--
>  arch/arm64/kernel/fpsimd.c       | 6 ++++--
>  3 files changed, 12 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 2599504674b5..c072161d5c65 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -69,7 +69,7 @@ static inline void *sve_pffr(struct thread_struct *thread)
>  extern void sve_save_state(void *state, u32 *pfpsr);
>  extern void sve_load_state(void const *state, u32 const *pfpsr,
>  			   unsigned long vq_minus_1);
> -extern void sve_flush_live(void);
> +extern void sve_flush_live(unsigned long vq_minus_1);
>  extern void sve_load_from_fpsimd_state(struct user_fpsimd_state const *state,
>  				       unsigned long vq_minus_1);
>  extern unsigned int sve_get_vl(void);
> diff --git a/arch/arm64/kernel/entry-fpsimd.S b/arch/arm64/kernel/entry-fpsimd.S
> index ee8773f4088b..090449e825e7 100644
> --- a/arch/arm64/kernel/entry-fpsimd.S
> +++ b/arch/arm64/kernel/entry-fpsimd.S
> @@ -70,10 +70,15 @@ SYM_FUNC_START(sve_load_from_fpsimd_state)
>  	ret
>  SYM_FUNC_END(sve_load_from_fpsimd_state)
>  
> -/* Zero all SVE registers but the first 128-bits of each vector */
> +/*
> + * Zero all SVE registers but the first 128-bits of each vector
> + *
> + * x0 = VQ - 1
> + */
>  SYM_FUNC_START(sve_flush_live)
> +	cbz		x0, 1f	// A VQ-1 of 0 is 128 bits so no extra Z state

Should we worry about branch mispredicts here?  It may be in the noise,
but I wonder whether it's worth considering use of alternatives here
instead.

I have a suspicion that VL = 128 bits won't be common at runtime, except
in the case of systems where the physical (or max usable) vector length
(i.e., sve_max_vl) is 128 bits.  

Concerns like this could be addressed later instead though, if/when we
have evidence to support changes.

>  	sve_flush_z
> -	sve_flush_p
> +1:	sve_flush_p
>  	sve_flush_ffr
>  	ret
>  SYM_FUNC_END(sve_flush_live)
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index ad3dd34a83cf..e57b23f95284 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -957,8 +957,10 @@ void do_sve_acc(unsigned int esr, struct pt_regs *regs)
>  	 * disabling the trap, otherwise update our in-memory copy.
>  	 */
>  	if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) {
> -		sve_set_vq(sve_vq_from_vl(current->thread.sve_vl) - 1);
> -		sve_flush_live();
> +		unsigned long vq_minus_one =
> +			sve_vq_from_vl(current->thread.sve_vl) - 1;
> +		sve_set_vq(vq_minus_one);
> +		sve_flush_live(vq_minus_one);

Seems reasonable.  sve_flush_live() could alternatively be made a C
function, with asm wrappers for sve_flush_{z,p,ffr} so that the
conditional logic can be inlined -- but I can't see that it would
improve the generated code much.  So I'd be happy with it to stay in
this form.

[...]

Cheers
---Dave