[PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support
Kefeng Wang
wangkefeng.wang at huawei.com
Mon May 18 08:05:47 PDT 2026
On 5/18/2026 4:49 PM, Ruidong Tian wrote:
> This series continues Tong Tiangen's work on arm64 ARCH_HAS_COPY_MC
> support. We encounter the same problem, and from a forward-looking
> perspective, large-memory ARM machines such as Grace and Vera will suffer
> more from this class of issues, which motivates us to push this feature
> upstream.
>
> Problem
> =========
> With the increase of memory capacity and density, the probability of memory
> error also increases. The increasing size and density of server RAM in data
> centers and clouds have shown increased uncorrectable memory errors.
>
> Currently, more and more scenarios that can tolerate memory errors, such as
> COW[1,2], KSM copy[3], coredump copy[4], khugepaged[5,6], uaccess copy[7],
> etc.
We have encountered more scenarios and have made more enhancements, eg,
658be46520ce mm: support poison recovery from copy_present_page()
aa549f923f5e mm: support poison recovery from do_cow_fault()
f00b295b9b61 fs: hugetlbfs: support poisoned recover from
hugetlbfs_migrate_folio()
060913999d7a mm: migrate: support poisoned recover from migrate folio
Hope that the architecture-related sections can receive relevant reviews
and responses.
Thanks.
> Solution
> =========
>
> This patchset introduces a new processing framework on ARM64, which enables
> ARM64 to support error recovery in the above scenarios, and more scenarios
> can be expanded based on this in the future.
>
> In arm64, memory error handling in do_sea(), which is divided into two cases:
> 1. If the user state consumed the memory errors, the solution is to kill
> the user process and isolate the error page.
> 2. If the kernel state consumed the memory errors, the solution is to
> panic.
>
> For case 2, Undifferentiated panic may not be the optimal choice, as it can
> be handled better. In some scenarios, we can avoid panic, such as uaccess,
> if the uaccess fails due to memory error, only the user process will be
> affected, returning an error to the caller and isolating the user page with
> hardware memory errors is a better choice.
>
> [1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits poison, take page offline")
> [2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults")
> [3] commit 6b970599e807 ("mm: hwpoison: support recovery from ksm_might_need_to_copy()")
> [4] commit 245f09226893 ("mm: hwpoison: coredump: support recovery from dump_user_range()")
> [5] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned anonymous memory")
> [6] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-backed memory")
> [7] commit 278b917f8cb9 ("x86/mce: Add _ASM_EXTABLE_CPY for copy user access")
>
> ------------------
> Test result:
>
> Tested on Kunpeng 920.
>
> 1. copy_page(), copy_mc_page() basic function test pass, and the disassembly
> contents remains the same before and after refactor.
>
> 2. copy_to/from_user() access kernel NULL pointer raise translation fault
> and dump error message then die(), test pass.
>
> 3. Test following scenarios: copy_from_user(), get_user(), COW.
>
> Before patched: trigger a hardware memory error then panic.
> After patched: trigger a hardware memory error without panic.
>
> Testing step:
> step1. start an user-process.
> step2. poison(einj) the user-process's page.
> step3: user-process access the poison page in kernel mode, then trigger SEA.
> step4: the kernel will not panic, only the user process is killed, the poison
> page is isolated. (before patched, the kernel will panic in do_sea())
>
> The above tests can also be reproduced using ras-tools, which provides
> einj-based injection and validation for uaccess and COW scenarios.
> Example usage:
>
> einj_mem_uc futex # get_user
> einj_mem_uc copyin # copy_to_user
> einj_mem_uc copy-on-write # COW
>
> Link: https://git.kernel.org/pub/scm/linux/kernel/git/aegl/ras-tools.git
>
> ------------------
>
> Benefits
> =========
> According to Huawei's statistics from their storage products, memory errors
> triggered in kernel-mode by COW and page cache read (uaccess) scenarios
> account for more than 50%. With this patchset deployed, all kernel panics
> caused by COW and page cache memory errors are eliminated.
> Alibaba Cloud has also observed memory errors occurring in uaccess contexts.
>
> Since V13:
> 1. Changed MC-safe functions to return an error rather than kill the user
> process. When a user program invokes a syscall and the kernel encounters
> a memory error during uaccess, killing the process is unexpected; the
> syscall should return an error.
> 2. Added FEAT_MOPS support for the copy_page_mc paths.
> 3. Refactored copy_page() and memcpy() on top of the shared memcpy_template,
> reducing duplicated assembly code.
>
> Since v12:
> Thanks to the suggestions of Jonathan, Mark, and Mauro, the following modifications
> are made:
> 1. Rebase to latest kernel version.
> 2. Patch1, add Jonathan's and Mauro's review-by.
> 3. Patch2, modified do_apei_claim_sea() according to Mark's and Jonathan's suggestions,
> and optimized the commit message according to Mark's suggestions(Added description of
> the impact on regular copy_to_user()).
> 4. Patch3, optimized the commit message according to Mauro's suggestions and add Jonathan's
> review-by.
> 5. Patch4, modified copy_mc_user_highpage() and Optimized the commit message according to
> Jonathan's suggestions(no functional changes).
> 6. Patch5, optimized the commit message according to Mauro's suggestions.
> 7. Patch4/5, FEAT_MOPS is added to the code logic. Currently, the fixup is not performed
> on the MOPS instruction.
> 8. Remove patch6 in v12 according to Jonathan's suggestions.
>
> Since v11:
> 1. Rebase to latest kernel version 6.9-rc1.
> 2. Add patch 5, Since the problem described in "Since V10 Besides 3" has
> been solved in a50026bdb867 ('iov_iter: get rid of 'copy_mc' flag').
> 3. Add the benefit of applying the patch set to our company to the description of patch0.
>
> Since V10:
> Accroding Mark's suggestion:
> 1. Merge V10's patch2 and patch3 to V11's patch2.
> 2. Patch2(V11): use new fixup_type for ld* in copy_to_user(), fix fatal
> issues (NULL kernel pointeraccess) been fixup incorrectly.
> 3. Patch2(V11): refactoring the logic of do_sea().
> 4. Patch4(V11): Remove duplicate assembly logic and remove do_mte().
>
> Besides:
> 1. Patch2(V11): remove st* insn's fixup, st* generally not trigger memory error.
> 2. Split a part of the logic of patch2(V11) to patch5(V11), for detail,
> see patch5(V11)'s commit msg.
> 3. Remove patch6(v10) “arm64: introduce copy_mc_to_kernel() implementation”.
> During modification, some problems that cannot be solved in a short
> period are found. The patch will be released after the problems are
> solved.
> 4. Add test result in this patch.
> 5. Modify patchset title, do not use machine check and remove "-next".
>
> Since V9:
> 1. Rebase to latest kernel version 6.8-rc2.
> 2. Add patch 6/6 to support copy_mc_to_kernel().
>
> Since V8:
> 1. Rebase to latest kernel version and fix topo in some of the patches.
> 2. According to the suggestion of Catalin, I attempted to modify the
> return value of function copy_mc_[user]_highpage() to bytes not copied.
> During the modification process, I found that it would be more
> reasonable to return -EFAULT when copy error occurs (referring to the
> newly added patch 4).
>
> For ARM64, the implementation of copy_mc_[user]_highpage() needs to
> consider MTE. Considering the scenario where data copying is successful
> but the MTE tag copying fails, it is also not reasonable to return
> bytes not copied.
> 3. Considering the recent addition of machine check safe support for
> multiple scenarios, modify commit message for patch 5 (patch 4 for V8).
>
> Since V7:
> Currently, there are patches supporting recover from poison
> consumption for the cow scenario[1]. Therefore, Supporting cow
> scenario under the arm64 architecture only needs to modify the relevant
> code under the arch/.
> [1]https://lore.kernel.org/lkml/20221031201029.102123-1-tony.luck@intel.com/
>
> Since V6:
> Resend patches that are not merged into the mainline in V6.
>
> Since V5:
> 1. Add patch2/3 to add uaccess assembly helpers.
> 2. Optimize the implementation logic of arm64_do_kernel_sea() in patch8.
> 3. Remove kernel access fixup in patch9.
> All suggestion are from Mark.
>
> Since V4:
> 1. According Michael's suggestion, add patch5.
> 2. According Mark's suggestiog, do some restructuring to arm64
> extable, then a new adaptation of machine check safe support is made based
> on this.
> 3. According Mark's suggestion, support machine check safe in do_mte() in
> cow scene.
> 4. In V4, two patches have been merged into -next, so V5 not send these
> two patches.
>
> Since V3:
> 1. According to Robin's suggestion, direct modify user_ldst and
> user_ldp in asm-uaccess.h and modify mte.S.
> 2. Add new macro USER_MC in asm-uaccess.h, used in copy_from_user.S
> and copy_to_user.S.
> 3. According to Robin's suggestion, using micro in copy_page_mc.S to
> simplify code.
> 4. According to KeFeng's suggestion, modify powerpc code in patch1.
> 5. According to KeFeng's suggestion, modify mm/extable.c and some code
> optimization.
>
> Since V2:
> 1. According to Mark's suggestion, all uaccess can be recovered due to
> memory error.
> 2. Scenario pagecache reading is also supported as part of uaccess
> (copy_to_user()) and duplication code problem is also solved.
> Thanks for Robin's suggestion.
> 3. According Mark's suggestion, update commit message of patch 2/5.
> 4. According Borisllav's suggestion, update commit message of patch 1/5.
>
> Since V1:
> 1.Consistent with PPC/x86, Using CONFIG_ARCH_HAS_COPY_MC instead of
> ARM64_UCE_KERNEL_RECOVERY.
> 2.Add two new scenes, cow and pagecache reading.
> 3.Fix two small bug(the first two patch).
>
> V1 in here:
> https://lore.kernel.org/lkml/20220323033705.3966643-1-tongtiangen@huawei.com/
>
> Ruidong Tian (3):
> ACPI: APEI: GHES: use exception context to gate SIGBUS on poison
> consumption
> lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests
> lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test
>
> Tong Tiangen (5):
> uaccess: add generic fallback version of copy_mc_to_user()
> arm64: add support for ARCH_HAS_COPY_MC
> mm/hwpoison: return -EFAULT when copy fail in
> copy_mc_[user]_highpage()
> arm64: support copy_mc_[user]_highpage()
> arm64: introduce copy_mc_to_kernel() implementation
>
> arch/arm64/Kconfig | 1 +
> arch/arm64/include/asm/asm-extable.h | 22 ++-
> arch/arm64/include/asm/asm-uaccess.h | 4 +
> arch/arm64/include/asm/extable.h | 1 +
> arch/arm64/include/asm/mte.h | 9 +
> arch/arm64/include/asm/page.h | 10 ++
> arch/arm64/include/asm/string.h | 5 +
> arch/arm64/include/asm/uaccess.h | 17 ++
> arch/arm64/kernel/acpi.c | 2 +-
> arch/arm64/lib/Makefile | 2 +
> arch/arm64/lib/copy_mc_page.S | 44 +++++
> arch/arm64/lib/copy_page.S | 62 +------
> arch/arm64/lib/copy_page_template.S | 71 ++++++++
> arch/arm64/lib/copy_to_user.S | 10 +-
> arch/arm64/lib/memcpy.S | 253 ++-------------------------
> arch/arm64/lib/memcpy_mc.S | 56 ++++++
> arch/arm64/lib/memcpy_template.S | 249 ++++++++++++++++++++++++++
> arch/arm64/lib/mte.S | 29 +++
> arch/arm64/mm/copypage.c | 75 ++++++++
> arch/arm64/mm/extable.c | 21 +++
> arch/arm64/mm/fault.c | 30 +++-
> arch/powerpc/include/asm/uaccess.h | 1 +
> arch/x86/include/asm/uaccess.h | 1 +
> drivers/acpi/apei/ghes.c | 36 ++--
> include/acpi/ghes.h | 6 +-
> include/linux/highmem.h | 16 +-
> include/linux/uaccess.h | 8 +
> lib/tests/memcpy_kunit.c | 178 ++++++++++++++++++-
> mm/kasan/shadow.c | 12 ++
> mm/khugepaged.c | 4 +-
> 30 files changed, 904 insertions(+), 331 deletions(-)
> create mode 100644 arch/arm64/lib/copy_mc_page.S
> create mode 100644 arch/arm64/lib/copy_page_template.S
> create mode 100644 arch/arm64/lib/memcpy_mc.S
> create mode 100644 arch/arm64/lib/memcpy_template.S
>
More information about the linux-arm-kernel
mailing list