[PATCH RESEND v5 3/3] riscv: word-at-a-time: improve find_zero() for Zbb

[Ze Huang]

On Sat Nov 7, 2026 at 9:46 PM CST, Jisheng Zhang wrote:
> In commit f915a3e5b018 ("arm64: word-at-a-time: improve byte count
> calculations for LE"), Linus improved the find_zero() for arm64 LE.
> Do the same optimization as he did: "do __ffs() on the intermediate value
> that found whether there is a zero byte, before we've actually computed
> the final byte mask.", so that we share the similar improvements:
>
> "The difference between the old and the new implementation is that
> "count_zero()" ends up scheduling better because it is being done on a
> value that is available earlier (before the final mask).
>
> But more importantly, it can be implemented without the insane semantics
> of the standard bit finding helpers that have the off-by-one issue and
> have to special-case the zero mask situation."
>
> Before the patch:
> 0000000000000000 <find_zero>:
>    0:	c909                	beqz	a0,12 <.L1>
>    2:	60051793          	clz	a5,a0
>    6:	03f00513          	li	a0,63
>    a:	8d1d                	sub	a0,a0,a5
>    c:	2505                	addiw	a0,a0,1
>    e:	4035551b          	sraiw	a0,a0,0x3
>
> 0000000000000012 <.L1>:
>   12:	8082                	ret
>
> After the patch:
>
> 0000000000000000 <find_zero>:
>    0:	60151513          	ctz	a0,a0
>    4:	810d                	srli	a0,a0,0x3
>    6:	8082                	ret
>
> 7 instructions vs 3 instructions!
>
> As can be seen, on RV64 w/ Zbb, the new "find_zero()" ends up just
> "ctz" plus the shift right that then ends up being subsumed by the
> "add to final length".
>
> But I have no HW platform which supports Zbb, so I can't get the
> performance improvement numbers by the last patch, only built and
> tested the patch on QEMU.

Hi,

I tested patch 3 on a SpacemiT K3 board (rv64imafdcvbvh, Zbb).

Against patch 1+2, the has_zero -> prep -> create_zero_mask -> find_zero chain
takes 45% less time: 3.74 ns -> 2.05 ns per call (-1.69 ns). Both versions are
built into the same image and measured alternately within one boot (6 cold
boots).

Note that this is a per-call figure, not a workload one: find_zero() runs once
per string operation, which takes less than 1% time of an openat() here.

Method and raw data:
https://gist.github.com/Sutter099/1c51abdb192eeedbf051ae582a1e8145#file-fzbench-md

Tested-by: Ze Huang <huang.ze at linux.dev>

>
> Signed-off-by: Jisheng Zhang <jszhang at kernel.org>
> ---
>  arch/riscv/include/asm/word-at-a-time.h | 18 +++++++++++++++---
>  1 file changed, 15 insertions(+), 3 deletions(-)
>
> diff --git a/arch/riscv/include/asm/word-at-a-time.h b/arch/riscv/include/asm/word-at-a-time.h
> index f2d16c1eea8d..31240133eccd 100644
> --- a/arch/riscv/include/asm/word-at-a-time.h
> +++ b/arch/riscv/include/asm/word-at-a-time.h
> @@ -35,6 +35,10 @@ static inline unsigned long prep_zero_mask(unsigned long val,
>  
>  static inline unsigned long create_zero_mask(unsigned long bits)
>  {
> +	if (IS_ENABLED(CONFIG_RISCV_ISA_ZBB) && IS_ENABLED(CONFIG_TOOLCHAIN_HAS_ZBB) &&
> +	    riscv_has_extension_likely(RISCV_ISA_EXT_ZBB))
> +		return bits;
> +
>  	bits = (bits - 1) & ~bits;
>  	return bits >> 7;
>  }
> @@ -67,13 +71,21 @@ static inline unsigned long find_zero(unsigned long mask)
>  {
>  	if (IS_ENABLED(CONFIG_RISCV_ISA_ZBB) && IS_ENABLED(CONFIG_TOOLCHAIN_HAS_ZBB) &&
>  	    riscv_has_extension_likely(RISCV_ISA_EXT_ZBB))
> -		return !mask ? 0 : ((__fls(mask) + 1) >> 3);
> +		return __ffs(mask) >> 3;
>  
>  	return count_masked_bytes(mask);
>  }
>  
> -/* The mask we created is directly usable as a bytemask */
> -#define zero_bytemask(mask) (mask)
> +static inline unsigned long zero_bytemask(unsigned long bits)
> +{
> +	if (IS_ENABLED(CONFIG_RISCV_ISA_ZBB) && IS_ENABLED(CONFIG_TOOLCHAIN_HAS_ZBB) &&
> +	    riscv_has_extension_likely(RISCV_ISA_EXT_ZBB)) {
> +		bits = (bits - 1) & ~bits;
> +		return bits >> 7;
> +	}
> +
> +	return bits;
> +}
>  
>  #ifdef CONFIG_DCACHE_WORD_ACCESS
>