[PATCH 0/2] arm64: csum: Add fused copy and Internet checksum

[David Laight]

On Mon, 2 Nov 2026 13:44:37 +0200
saeed bishara <saeed.bishara.os at gmail.com> wrote:

> On Sun, Nov 1, 2026 at 3:34 PM Demian Shulhan <demyansh at gmail.com> wrote:
> >
> > I would prefer to keep only the scalar version of the fused copy+csum.
> > It reads the data once instead of twice, and on real Armv9 hardware it
> > cuts end-to-end cycles/byte for bulk TCP over WireGuard by 3-12%
> > depending on the core (+6-8% throughput). It uses only general-purpose
> > registers, so there is no FPSIMD state to save and it is safe in any
> > context.
> >
> > If you are OK with the scalar path, I can send v2 with NEON dropped,
> > the scalar path only, and the KUnit fix for the x86 zero-length case
> > Jakub reported.
> >  
> The scalar path seems like generic C code and not restricted to arm64,
> curious how that performs against existing (e.g. x86) assembly

The x86 asm version manages to do more than one 'add with carry' every
clock. That needs the massive out-of-order execution unit and the ability
to have multiple independent 'flags' registers live at the same time.
The generic C do_csum() (I just looked at lib/checksum.c) is horrid
There are three dependent alu operations per iteration.

It is likely that the pipelineable:
u64 sum64_a(unsigned char *buf, unsigned long count)
{
    u64 val, sum = 0, carry = 0;

    buf += count;
    count = -count;
    do {
        val = *(u64 *)(buf + count);
        sum += val;
        carry += sum < val;
    } while ((count += 8));

    sum += carry;
    sum += sum < carry;
    return sum;
}
or:
u64 sum64_b(unsigned char *buf, unsigned long count)
{
    u64 val, sum = 0, carry = 0;
    unsigned char *end = buf + count;

    do {
        val = *(u64 *)(buf);
        sum += val;
        carry += sum < val;
    } while ((buf += 8) != end);

    sum += carry;
    sum += sum < carry;
    return sum;
}

will generate 'pretty good' code https://godbolt.org/z/K9ob6d8vv.
The first needs 'reg+reg' addressing, the second better for mips-like
cpu that have 'compare and branch' rather than a flags register.

gcc and clang seem to make a reasonable job of compiling those
(provided buf isn't u64 *).
Today it left the 'negative offset from end' loop alone,
but it has a nasty habit of pessimising it.
For x86-64 gcc is generating:
.L4:
        addq    (%rdi,%rdx), %rax
        adcq    $0, %rcx
        addq    $8, %rdx
        jne     .L4

I think a zen5 would execute that at one iteration per clock.
Intel cpu can only execute two clock loops - so it needs unrolling once.
If you unroll it once (with separate sum/carry variables) there
is a moderate chance of it running faster than 'one word per clock'
(two words/clock is probably wishful thinking, zen5 can do 6 add/clock
but some timings I've done imply that memory addressing can use and alu
add).

For slower cpu (especially in-order ones with memory read delays)
you may need interleave the code, eg giving the loop:
    val0 = *(u64 *)buf;
    do {
        val1 = *(u64 *)(buf + 8);
        sum0 += val0;
        carry0 += sum0 < val0;
        val0 = *(u64 *)(buf + 16);
        sum1 += val1;
        carry1 += sum1 < val1;
    } while ((buf += 16) != end);

I didn't try those loops when I was timing the cksum code a few
years back.

They have the big advantage of a small loop stride.

David

> optimized implementation.
> saeed