[PATCH 0/5] arm64: Use memory copy instructions in kernel routines

[Kristina Martsenko]

Hi,

Here is a small series to make memcpy() and related functions use the
memory copy/set instructions (Armv8.8 FEAT_MOPS).

The kernel uses several library routines for copying or initializing
memory, for example copy_to_user() and memset(). These routines have
been optimized to make their load/store sequence perform well across a
range of CPUs. However the chosen sequence can't be the fastest possible
for every CPU microarchitecture nor for heterogeneous systems, and needs
to be rewritten periodically as hardware changes.

Future arm64 CPUs will have CPY* and SET* instructions that can copy (or
set) a block of memory of arbitrary size and alignment. The kernel
currently supports using these instructions in userspace applications
[1] and KVM guests [2] but does not use them within the kernel.

CPUs are expected to implement the CPY/SET instructions close to
optimally for their microarchitecture (i.e. close to the performance of
the best load/store sequence performing a generic copy/set). Using the
instructions in the kernel's copy/set routines would therefore make the
routines optimal and avoid the need to rewrite them. It could also lead
to a performance improvement for some CPUs and systems.

This series makes the memcpy(), memmove() and memset() routines use the
CPY/SET instructions, as well as copy_page() and clear_page(). I'll send
a follow-up series to update the usercopy routines (copy_to_user() etc)
"soon", as it needs a bit more work.

The patches were tested on an Arm FVP.

Thanks,
Kristina

[1] https://lore.kernel.org/lkml/20230509142235.3284028-1-kristina.martsenko@arm.com/
[2] https://lore.kernel.org/linux-arm-kernel/20230922112508.1774352-1-kristina.martsenko@arm.com/

Kristina Martsenko (5):
  arm64: probes: Disable kprobes/uprobes on MOPS instructions
  arm64: mops: Handle MOPS exceptions from EL1
  arm64: mops: Document booting requirement for HCR_EL2.MCE2
  arm64: lib: Use MOPS for memcpy() routines
  arm64: lib: Use MOPS for copy_page() and clear_page()

 Documentation/arch/arm64/booting.rst    |  3 +++
 arch/arm64/Kconfig                      |  3 +++
 arch/arm64/include/asm/debug-monitors.h |  1 +
 arch/arm64/include/asm/exception.h      |  1 +
 arch/arm64/include/asm/insn.h           |  1 +
 arch/arm64/kernel/debug-monitors.c      |  5 +++++
 arch/arm64/kernel/entry-common.c        | 12 ++++++++++++
 arch/arm64/kernel/probes/decode-insn.c  |  7 +++++--
 arch/arm64/kernel/traps.c               |  7 +++++++
 arch/arm64/lib/clear_page.S             | 13 +++++++++++++
 arch/arm64/lib/copy_page.S              | 13 +++++++++++++
 arch/arm64/lib/memcpy.S                 | 19 ++++++++++++++++++-
 arch/arm64/lib/memset.S                 | 20 +++++++++++++++++++-
 13 files changed, 101 insertions(+), 4 deletions(-)


base-commit: 9852d85ec9d492ebef56dc5f229416c925758edc
-- 
2.34.1