[PATCH] ACPI: APEI: Handle repeated SEA error interrupts storm scenarios
Shuai Xue
xueshuai at linux.alibaba.com
Tue Mar 24 19:12:24 PDT 2026
Hi, junhao
On 3/24/26 6:04 PM, hejunhao wrote:
> Hi shuai xue,
>
>
> On 2026/3/3 22:42, Shuai Xue wrote:
>> Hi, junhao,
>>
>> On 2/27/26 8:12 PM, hejunhao wrote:
>>>
>>>
>>> On 2025/11/4 9:32, Shuai Xue wrote:
>>>>
>>>>
>>>> 在 2025/11/4 00:19, Rafael J. Wysocki 写道:
>>>>> On Thu, Oct 30, 2025 at 8:13 AM Junhao He <hejunhao3 at h-partners.com> wrote:
>>>>>>
>>>>>> The do_sea() function defaults to using firmware-first mode, if supported.
>>>>>> It invoke acpi/apei/ghes ghes_notify_sea() to report and handling the SEA
>>>>>> error, The GHES uses a buffer to cache the most recent 4 kinds of SEA
>>>>>> errors. If the same kind SEA error continues to occur, GHES will skip to
>>>>>> reporting this SEA error and will not add it to the "ghes_estatus_llist"
>>>>>> list until the cache times out after 10 seconds, at which point the SEA
>>>>>> error will be reprocessed.
>>>>>>
>>>>>> The GHES invoke ghes_proc_in_irq() to handle the SEA error, which
>>>>>> ultimately executes memory_failure() to process the page with hardware
>>>>>> memory corruption. If the same SEA error appears multiple times
>>>>>> consecutively, it indicates that the previous handling was incomplete or
>>>>>> unable to resolve the fault. In such cases, it is more appropriate to
>>>>>> return a failure when encountering the same error again, and then proceed
>>>>>> to arm64_do_kernel_sea for further processing.
>>
>> There is no such function in the arm64 tree. If apei_claim_sea() returns
>
> Sorry for the mistake in the commit message. The function arm64_do_kernel_sea() should
> be arm64_notify_die().
>
>> an error, the actual fallback path in do_sea() is arm64_notify_die(),
>> which sends SIGBUS?
>>
>
> If apei_claim_sea() returns an error, arm64_notify_die() will call arm64_force_sig_fault(inf->sig /* SIGBUS */, , , ),
> followed by force_sig_fault(SIGBUS, , ) to force the process to receive the SIGBUS signal.
So the process is expected to killed by SIGBUS?
>
>>>>>>
>>>>>> When hardware memory corruption occurs, a memory error interrupt is
>>>>>> triggered. If the kernel accesses this erroneous data, it will trigger
>>>>>> the SEA error exception handler. All such handlers will call
>>>>>> memory_failure() to handle the faulty page.
>>>>>>
>>>>>> If a memory error interrupt occurs first, followed by an SEA error
>>>>>> interrupt, the faulty page is first marked as poisoned by the memory error
>>>>>> interrupt process, and then the SEA error interrupt handling process will
>>>>>> send a SIGBUS signal to the process accessing the poisoned page.
>>>>>>
>>>>>> However, if the SEA interrupt is reported first, the following exceptional
>>>>>> scenario occurs:
>>>>>>
>>>>>> When a user process directly requests and accesses a page with hardware
>>>>>> memory corruption via mmap (such as with devmem), the page containing this
>>>>>> address may still be in a free buddy state in the kernel. At this point,
>>>>>> the page is marked as "poisoned" during the SEA claim memory_failure().
>>>>>> However, since the process does not request the page through the kernel's
>>>>>> MMU, the kernel cannot send SIGBUS signal to the processes. And the memory
>>>>>> error interrupt handling process not support send SIGBUS signal. As a
>>>>>> result, these processes continues to access the faulty page, causing
>>>>>> repeated entries into the SEA exception handler. At this time, it lead to
>>>>>> an SEA error interrupt storm.
>>
>> In such case, the user process which accessing the poisoned page will be killed
>> by memory_fauilre?
>>
>> // memory_failure():
>>
>> if (TestSetPageHWPoison(p)) {
>> res = -EHWPOISON;
>> if (flags & MF_ACTION_REQUIRED)
>> res = kill_accessing_process(current, pfn, flags);
>> if (flags & MF_COUNT_INCREASED)
>> put_page(p);
>> action_result(pfn, MF_MSG_ALREADY_POISONED, MF_FAILED);
>> goto unlock_mutex;
>> }
>>
>> I think this problem has already been fixed by commit 2e6053fea379 ("mm/memory-failure:
>> fix infinite UCE for VM_PFNMAP pfn").
>>
>> The root cause is that walk_page_range() skips VM_PFNMAP vmas by default when
>> no .test_walk callback is set, so kill_accessing_process() returns 0 for a
>> devmem-style mapping (remap_pfn_range, VM_PFNMAP), making the caller believe
>> the UCE was handled properly while the process was never actually killed.
>>
>> Did you try the lastest kernel version?
>>
>
> I retested this issue on the kernel v7.0.0-rc4 with the following debug patch and was still able to reproduce it.
>
>
> @@ -1365,8 +1365,11 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
> ghes_clear_estatus(ghes, &tmp_header, buf_paddr, fixmap_idx);
>
> /* This error has been reported before, don't process it again. */
> - if (ghes_estatus_cached(estatus))
> + if (ghes_estatus_cached(estatus)) {
> + pr_info("This error has been reported before, don't process it again.\n");
> goto no_work;
> + }
>
> the test log Only some debug logs are retained here.
>
> [2026/3/24 14:51:58.199] [root at localhost ~]# taskset -c 40 busybox devmem 0x1351811824 32 0
> [2026/3/24 14:51:58.369] [root at localhost ~]# taskset -c 40 busybox devmem 0x1351811824 32
> [2026/3/24 14:51:58.458] [ 130.558038][ C40] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 9
> [2026/3/24 14:51:58.459] [ 130.572517][ C40] {1}[Hardware Error]: event severity: recoverable
> [2026/3/24 14:51:58.459] [ 130.578861][ C40] {1}[Hardware Error]: Error 0, type: recoverable
> [2026/3/24 14:51:58.459] [ 130.585203][ C40] {1}[Hardware Error]: section_type: ARM processor error
> [2026/3/24 14:51:58.459] [ 130.592238][ C40] {1}[Hardware Error]: MIDR: 0x0000000000000000
> [2026/3/24 14:51:58.459] [ 130.598492][ C40] {1}[Hardware Error]: Multiprocessor Affinity Register (MPIDR): 0x0000000081010400
> [2026/3/24 14:51:58.459] [ 130.607871][ C40] {1}[Hardware Error]: error affinity level: 0
> [2026/3/24 14:51:58.459] [ 130.614038][ C40] {1}[Hardware Error]: running state: 0x1
> [2026/3/24 14:51:58.459] [ 130.619770][ C40] {1}[Hardware Error]: Power State Coordination Interface state: 0
> [2026/3/24 14:51:58.459] [ 130.627673][ C40] {1}[Hardware Error]: Error info structure 0:
> [2026/3/24 14:51:58.459] [ 130.633839][ C40] {1}[Hardware Error]: num errors: 1
> [2026/3/24 14:51:58.459] [ 130.639137][ C40] {1}[Hardware Error]: error_type: 0, cache error
> [2026/3/24 14:51:58.459] [ 130.645652][ C40] {1}[Hardware Error]: error_info: 0x0000000020400014
> [2026/3/24 14:51:58.459] [ 130.652514][ C40] {1}[Hardware Error]: cache level: 1
> [2026/3/24 14:51:58.551] [ 130.658073][ C40] {1}[Hardware Error]: the error has not been corrected
> [2026/3/24 14:51:58.551] [ 130.665194][ C40] {1}[Hardware Error]: physical fault address: 0x0000001351811800
> [2026/3/24 14:51:58.551] [ 130.673097][ C40] {1}[Hardware Error]: Vendor specific error info has 48 bytes:
> [2026/3/24 14:51:58.551] [ 130.680744][ C40] {1}[Hardware Error]: 00000000: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 14:51:58.551] [ 130.690471][ C40] {1}[Hardware Error]: 00000010: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 14:51:58.552] [ 130.700198][ C40] {1}[Hardware Error]: 00000020: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 14:51:58.552] [ 130.710083][ T9767] Memory failure: 0x1351811: recovery action for free buddy page: Recovered
> [2026/3/24 14:51:58.638] [ 130.790952][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:51:58.903] [ 131.046994][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:51:58.991] [ 131.132360][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:51:59.969] [ 132.071431][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:00.860] [ 133.010255][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:01.927] [ 134.034746][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:02.906] [ 135.058973][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:03.971] [ 136.083213][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:04.860] [ 137.021956][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:06.018] [ 138.131460][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:06.905] [ 139.070280][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:07.886] [ 140.009147][ C40] This error has been reported before, don't process it again.
> [2026/3/24 14:52:08.596] [ 140.777368][ C40] {2}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 9
> [2026/3/24 14:52:08.683] [ 140.791921][ C40] {2}[Hardware Error]: event severity: recoverable
> [2026/3/24 14:52:08.683] [ 140.798263][ C40] {2}[Hardware Error]: Error 0, type: recoverable
> [2026/3/24 14:52:08.683] [ 140.804606][ C40] {2}[Hardware Error]: section_type: ARM processor error
> [2026/3/24 14:52:08.683] [ 140.811641][ C40] {2}[Hardware Error]: MIDR: 0x0000000000000000
> [2026/3/24 14:52:08.684] [ 140.817895][ C40] {2}[Hardware Error]: Multiprocessor Affinity Register (MPIDR): 0x0000000081010400
> [2026/3/24 14:52:08.684] [ 140.827274][ C40] {2}[Hardware Error]: error affinity level: 0
> [2026/3/24 14:52:08.684] [ 140.833440][ C40] {2}[Hardware Error]: running state: 0x1
> [2026/3/24 14:52:08.684] [ 140.839173][ C40] {2}[Hardware Error]: Power State Coordination Interface state: 0
> [2026/3/24 14:52:08.684] [ 140.847076][ C40] {2}[Hardware Error]: Error info structure 0:
> [2026/3/24 14:52:08.684] [ 140.853241][ C40] {2}[Hardware Error]: num errors: 1
> [2026/3/24 14:52:08.684] [ 140.858540][ C40] {2}[Hardware Error]: error_type: 0, cache error
> [2026/3/24 14:52:08.684] [ 140.865055][ C40] {2}[Hardware Error]: error_info: 0x0000000020400014
> [2026/3/24 14:52:08.684] [ 140.871917][ C40] {2}[Hardware Error]: cache level: 1
> [2026/3/24 14:52:08.684] [ 140.877475][ C40] {2}[Hardware Error]: the error has not been corrected
> [2026/3/24 14:52:08.764] [ 140.884596][ C40] {2}[Hardware Error]: physical fault address: 0x0000001351811800
> [2026/3/24 14:52:08.764] [ 140.892499][ C40] {2}[Hardware Error]: Vendor specific error info has 48 bytes:
> [2026/3/24 14:52:08.766] [ 140.900145][ C40] {2}[Hardware Error]: 00000000: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 14:52:08.767] [ 140.909872][ C40] {2}[Hardware Error]: 00000010: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 14:52:08.767] [ 140.919598][ C40] {2}[Hardware Error]: 00000020: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 14:52:08.768] [ 140.929346][ T9767] Memory failure: 0x1351811: already hardware poisoned
> [2026/3/24 14:52:08.768] [ 140.936072][ T9767] Memory failure: 0x1351811: Sending SIGBUS to busybox:9767 due to hardware memory corruption
Did you cut off some logs here?
The error log also indicates that the SIGBUS is delivered as expected.
>
>
> Apply the patch:
>
> @@ -1365,8 +1365,11 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
> ghes_clear_estatus(ghes, &tmp_header, buf_paddr, fixmap_idx);
>
> /* This error has been reported before, don't process it again. */
> - if (ghes_estatus_cached(estatus))
> + if (ghes_estatus_cached(estatus)) {
> + pr_info("This error has been reported before, don't process it again.\n");
> + rc = -ECANCELED;
> goto no_work;
> + }
>
> [2026/3/24 16:45:40.084] [root at localhost ~]# taskset -c 40 busybox devmem 0x1351811824 32 0
> [2026/3/24 16:45:40.272] [root at localhost ~]# taskset -c 40 busybox devmem 0x1351811824 32
> [2026/3/24 16:45:40.362] [ 112.279324][ C40] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 9
> [2026/3/24 16:45:40.362] [ 112.293797][ C40] {1}[Hardware Error]: event severity: recoverable
> [2026/3/24 16:45:40.362] [ 112.300139][ C40] {1}[Hardware Error]: Error 0, type: recoverable
> [2026/3/24 16:45:40.363] [ 112.306481][ C40] {1}[Hardware Error]: section_type: ARM processor error
> [2026/3/24 16:45:40.363] [ 112.313516][ C40] {1}[Hardware Error]: MIDR: 0x0000000000000000
> [2026/3/24 16:45:40.363] [ 112.319771][ C40] {1}[Hardware Error]: Multiprocessor Affinity Register (MPIDR): 0x0000000081010400
> [2026/3/24 16:45:40.363] [ 112.329151][ C40] {1}[Hardware Error]: error affinity level: 0
> [2026/3/24 16:45:40.363] [ 112.335317][ C40] {1}[Hardware Error]: running state: 0x1
> [2026/3/24 16:45:40.363] [ 112.341049][ C40] {1}[Hardware Error]: Power State Coordination Interface state: 0
> [2026/3/24 16:45:40.363] [ 112.348953][ C40] {1}[Hardware Error]: Error info structure 0:
> [2026/3/24 16:45:40.363] [ 112.355119][ C40] {1}[Hardware Error]: num errors: 1
> [2026/3/24 16:45:40.363] [ 112.360418][ C40] {1}[Hardware Error]: error_type: 0, cache error
> [2026/3/24 16:45:40.363] [ 112.366932][ C40] {1}[Hardware Error]: error_info: 0x0000000020400014
> [2026/3/24 16:45:40.363] [ 112.373795][ C40] {1}[Hardware Error]: cache level: 1
> [2026/3/24 16:45:40.453] [ 112.379354][ C40] {1}[Hardware Error]: the error has not been corrected
> [2026/3/24 16:45:40.453] [ 112.386475][ C40] {1}[Hardware Error]: physical fault address: 0x0000001351811800
> [2026/3/24 16:45:40.453] [ 112.394378][ C40] {1}[Hardware Error]: Vendor specific error info has 48 bytes:
> [2026/3/24 16:45:40.453] [ 112.402027][ C40] {1}[Hardware Error]: 00000000: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 16:45:40.453] [ 112.411754][ C40] {1}[Hardware Error]: 00000010: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 16:45:40.453] [ 112.421480][ C40] {1}[Hardware Error]: 00000020: 00000000 00000000 00000000 00000000 ................
> [2026/3/24 16:45:40.453] [ 112.431639][ T9769] Memory failure: 0x1351811: recovery action for free buddy page: Recovered
> [2026/3/24 16:45:40.531] [ 112.512520][ C40] This error has been reported before, don't process it again.
> [2026/3/24 16:45:40.757] Bus error (core dumped)
>
Thanks.
Shuai
More information about the linux-arm-kernel
mailing list