[PATCH 0/2] arm64: csum: Add fused copy and Internet checksum

[Demian Shulhan]

Hi David,

Sorry for the delay, I was in the middle of a company change.

You are right about SG and TX csum, but only when the traffic goes
straight to a NIC that offloads them. There is another side: software
GSO (skb_segment() with !sg && !csum) hits this path on 100% of TCP
payload over WireGuard (wg_xmit() -> skb_gso_segment(skb, 0)), and on
drivers without SG/TX csum (u_ether/f_ncm, cdc_ncm, mt76, rtw88/89,
ath9k).

So in general the software copy+csum path is still alive.

On the other hand, NEON is useless here. skb_segment() splits GSO
packets into 1-MSS segments before copying, so on these paths calls
never exceed 1448 bytes (1368 on WireGuard). At that size
kernel_neon_begin()/end() costs more than NEON saves, on real hardware
v1 was ~3% slower in cycles/byte than the scalar loop on the slower
cores. So that part can go.

I would prefer to keep only the scalar version of the fused copy+csum.
It reads the data once instead of twice, and on real Armv9 hardware it
cuts end-to-end cycles/byte for bulk TCP over WireGuard by 3-12%
depending on the core (+6-8% throughput). It uses only general-purpose
registers, so there is no FPSIMD state to save and it is safe in any
context.

If you are OK with the scalar path, I can send v2 with NEON dropped,
the scalar path only, and the KUnit fix for the x86 zero-length case
Jakub reported.

Thanks,
Demian

нд, 27 вер. 2026 р. о 19:44 David Laight <david.laight.linux at gmail.com> пише:
>
> On Sun, 27 Sep 2026 15:17:56 +0200
> Demian Shulhan <demyansh at gmail.com> wrote:
>
> > arm64 currently uses the generic csum_partial_copy_nocheck(), which
> > performs memcpy() followed by a second pass for csum_partial(). This
> > double pass exerts unnecessary pressure on the L1 cache.
>
> Which workload actually needs this?
> Most modern ethernet MAC support checksum setting on transmit and
> checking on receive.
> So the software checksum shouldn't be needed very often.
>
> IIRC there is also code to defer UDP checksum validation until the
> copy_to_user().
> I'd bet (a few pints of beer) that the complication this adds isn't
> actually worth while.
> Even Linus can't remember why it was done, my guess is it improved the
> performance of the userspace NFS (over UDP) daemon that would be
> doing 8k UDP send/receive (fragmented by IP).
>
> There is certainly still code to checksum data during copy_from_user()
> in send().
> Last time I looked I couldn't see why send on TCP sockets didn't go through it.
> On x86 (in particular) copies can be done far faster than ones that
> include a checksum.
>
> David
>
> >
> > Replace it with a single-pass implementation. The new implementation
> > provides a general-purpose register path for short buffers and atomic
> > contexts, and a kernel-mode NEON path for lengths >= 1024 bytes.
> >
> > Measured in-kernel on an Ampere Altra (Neoverse-N1):
> > - Scalar path: 1.2x-1.6x faster for lengths < 1024 bytes.
> > - NEON path: 1.2x faster at 1024 bytes, scaling up to 1.6x-1.8x at
> >   4096 bytes.
> > On Apple M-series cores, gains are 1.3-1.7x below 1024 bytes and
> > 1.6-2.4x above. No length or alignment regresses on either
> > microarchitecture.
> >
> > Patch 1 implements the fused routines and the dispatcher.
> > Patch 2 adds KUnit test coverage for the new API and internal paths.
> >
> > Tested: in-kernel benchmark module on Neoverse-N1 with both
> > implementations cross-checked (0 mismatches); KUnit suite under QEMU
> > (with/without KASAN, with PREEMPT_RT), exhaustive and random userspace
> > testing of both routines against a naive reference with PROT_NONE guard
> > pages, gcc 13 and clang 18 W=1 builds, checkpatch --strict.
> >
> > Demian Shulhan (2):
> >   arm64: csum: Add fused copy and Internet checksum
> >   lib/tests: checksum: Add KUnit tests for csum_partial_copy_nocheck()
> >
> >  arch/arm64/include/asm/checksum.h |   3 +
> >  arch/arm64/lib/Makefile           |   7 +-
> >  arch/arm64/lib/csum-copy-neon.c   | 168 +++++++++++
> >  arch/arm64/lib/csum-copy.c        | 108 +++++++
> >  arch/arm64/lib/csum-copy.h        |  84 ++++++
> >  arch/arm64/lib/csum.c             |  54 ++++
> >  lib/Kconfig.debug                 |  10 +
> >  lib/tests/checksum_kunit.c        | 463 ++++++++++++++++++++++++++++++
> >  8 files changed, 896 insertions(+), 1 deletion(-)
> >  create mode 100644 arch/arm64/lib/csum-copy-neon.c
> >  create mode 100644 arch/arm64/lib/csum-copy.c
> >  create mode 100644 arch/arm64/lib/csum-copy.h
> >
> >
> > base-commit: 93f51579e7df248780214094418f205253383cc5
>