alignment handler fault when irq enabled with CONFIG_PREEMPT
minghua d
minghuadev at gmail.com
Sat Jun 15 16:12:10 EDT 2013
Hi, We are seeing a kernel oops caused by alignment handler.
It's a NXP LPC3250 based system with ARM 926 EJS processor.
Are there any known issues or fixed issues in either kernel or silicon
that could cause this? Any hint how to further debug this?
Below is a detailed account of the problem and tests.
Thanks.
-Minghua
To have the problem happen it needs three conditions:
[1] An udp app that opens 16 ports, keeps receiving and dropping packets.
[2] A shell that grep dmesg for "Oops" once every second in a loop.
[3] Large volume of packets sent to 16 udp ports through ethernet.
Once the setup is running, the shell sub-processes in the loop will
see random segment faults at a rate of 2 to 3 faults per minute.
Sometimes the rate is much higher or much lower. The fault happens
mostly in "__copy_from_user" and "file_read_actor", but also at:
__copy_to_user_std
__get_user_4
__copy_from_user
__strnlen_use
When it runs for about a few minutes upto an hour, eventually it crashes
due to an oops in alignment handler. I have been tracking this crash.
It appears to be consistent and reproducible with kernel 2.6.39 and 2.6.27.
Have not tried kernel 3.x due to build issues.
The specific chunk of code that triggers the oops is in "do_alignment()":
fs = get_fs();
set_fs(KERNEL_DS);
fault = __get_user(instr, (u32 *)instrptr);
set_fs(fs);
The oops will not happen if one of the three conditions is taken out;
or if disable irq around the above "__get_user()" and around subsequent
data featching code in alighment handler; or if !CONFIG_PREEMPT.
There is an interesting work-around by doing "__get_user()" again if there
is a fault at the first attempt. I interpret this as the paging tables are
correct but momentarily loaded incorrectly. The code goes like this:
fs = get_fs();
set_fs(KERNEL_DS);
fault = __get_user(instr, (u32 *)instrptr);
+ if (fault) fault = __get_user(instr, (u32 *)instrptr);
set_fs(fs);
The sequence of events in the test is like:
A packet is received and processed by softirq.
In ip/udp stack it does unaligned access causing alighment exception.
Alignment handler fetches the instruction.
The next packet is transferred by DMA from Eth to memory, using
bus bandwidth.
An interrupt kicks in thus the instruction fetching faults.
I'm guessing a few reasons:
__get_user() cannot be interrupted;
__get_user() is corrupted if DMA occupies too much bus bandwidth;
address space switching/flushing is corrupted by interrupts or DMA;
other MMU issues or caching issues;
not a logic error in software or hardware, but just noises.
Below is the dump of two consecutive faults in alignment handler. The kernel
is modified in "__do_kernel_fault()" like the following code to help debug:
__show_regs(regs);
WARN_ON(1);
printk("Pre-fixup pc %pS va %pS\n", (void*)(regs->ARM_pc), (void*)addr);
printk("Pre-fixup pc %lx va %lx\n",regs->ARM_pc, addr);
//decode pgd/pud/pmd, follwing the logic in show_pte(), using values from
// "pgd = cpu_get_pgd()", and from "pgd = init_mm.pgd".
----first fault: fetching instruction:
CPU: 0 Tainted: G W (2.6.39.2 #479)
PC is at do_alignment+0x78/0x3e4
LR is at do_alignment+0x28/0x3e4
pc : [<c003163c>] lr : [<c00315ec>] psr: 40000013
sp : c3ac7c60 ip : bf000000 fp : c02b11a8
r10: 00000001 r9 : c02c52d0 r8 : c3b73042
r7 : c01e590c r6 : c3ac7d40 r5 : c3ac7d40 r4 : 00000001
r3 : 00000017 r2 : 00000000 r1 : 00000000 r0 : 00000000
Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 0005317f Table: 83a5c000 DAC: 00000015
------------[ cut here ]------------
WARNING: at arch/arm/mm/fault.c:580 __do_kernel_fault+0x60/0x238()
Modules linked in: DaliPeer DaliGpio DaliMem DaliI2c DaliSpi
[<c002e4a0>] (unwind_backtrace+0x0/0xe4) from [<c003d7fc>]
(warn_slowpath_common+0x4c/0x64)
[<c003d7fc>] (warn_slowpath_common+0x4c/0x64) from [<c003d830>]
(warn_slowpath_null+0x1c/0x24)
[<c003d830>] (warn_slowpath_null+0x1c/0x24) from [<c002f840>]
(__do_kernel_fault+0x60/0x238)
[<c002f840>] (__do_kernel_fault+0x60/0x238) from [<c002fc88>]
(do_bad_area+0x70/0x7c)
[<c002fc88>] (do_bad_area+0x70/0x7c) from [<c002fca0>] (do_sect_fault+0xc/0x18)
[<c002fca0>] (do_sect_fault+0xc/0x18) from [<c001e348>]
(do_DataAbort+0xb8/0x138)
[<c001e348>] (do_DataAbort+0xb8/0x138) from [<c0028fdc>] (__dabt_svc+0x5c/0x80)
Exception stack(0xc3ac7c18 to 0xc3ac7c60)
7c00: 00000000 00000000
7c20: 00000000 00000017 00000001 c3ac7d40 c3ac7d40 c01e590c c3b73042 c02c52d0
7c40: 00000001 c02b11a8 bf000000 c3ac7c60 c00315ec c003163c 40000013 ffffffff
[<c0028fdc>] (__dabt_svc+0x5c/0x80) from [<c003163c>] (do_alignment+0x78/0x3e4)
[<c003163c>] (do_alignment+0x78/0x3e4) from [<c001e348>]
(do_DataAbort+0xb8/0x138)
[<c001e348>] (do_DataAbort+0xb8/0x138) from [<c0028fdc>] (__dabt_svc+0x5c/0x80)
Exception stack(0xc3ac7d40 to 0xc3ac7d88)
7d40: c28903e0 00000007 c3ac6000 c3b73036 c02d1414 c28903e0 c3ac6008 c3ac3260
7d60: c3ac7f5c c3ac7edc 0047f7a2 0000251c c3ac7d78 c3ac7d88 c01e2564 c01e590c
7d80: 40000013 ffffffff
[<c0028fdc>] (__dabt_svc+0x5c/0x80) from [<c01e590c>] (udp_recvmsg+0x35c/0x52c)
[<c01e590c>] (udp_recvmsg+0x35c/0x52c) from [<c01ea7d8>]
(inet_recvmsg+0x44/0x58)
[<c01ea7d8>] (inet_recvmsg+0x44/0x58) from [<c01940f4>] (sock_recvmsg+0x9c/0xc4)
[<c01940f4>] (sock_recvmsg+0x9c/0xc4) from [<c0194964>] (sys_recvfrom+0x84/0xe0)
[<c0194964>] (sys_recvfrom+0x84/0xe0) from [<c0029440>]
(ret_fast_syscall+0x0/0x2c)
---[ end trace 65f8ea860415c06c ]---
Pre-fixup pc do_alignment+0x78/0x3e4 va udp_recvmsg+0x35c/0x52c
Pre-fixup pc c003163c va c01e590c
cpu table
pgd table: c3a5c000 addr 0xc01e590c index 0x00000600
pgd entry: c3a5f000 8000041e 8010041e
pud entry: c3a5f000
pmd entry: c3a5f000 8010041e index 1 (section index, 0 or 1)
pmd sect entry = 8010041e
base addr bit 31..20 80100000
sbz bit 19..12 00000000
ap bit 11..10 00000400
domain bit 8..5 00000000
1cb bit 4..2 0000001c
sect bit 1..0 00000002
addr offset bit 19..12 000e5000
init mm
pgd table: c0004000 addr 0xc01e590c index 0x00000600
pgd entry: c0007000 8000041e 8010041e
pud entry: c0007000
pmd entry: c0007000 8010041e index 1 (0 or 1)
pmd sect entry = 8010041e
base addr bit 31..20 80100000
sbz bit 19..12 00000000
ap bit 11..10 00000400
domain bit 8..5 00000000
1cb bit 4..2 0000001c
sect bit 1..0 00000002
addr offset bit 19..12 000e5000
----second fault: due to failure in fetching instruction
CPU: 0 Tainted: G W (2.6.39.2 #479)
PC is at udp_recvmsg+0x35c/0x52c
LR is at dma_ptr_unlock+0x1c/0x48
pc : [<c01e590c>] lr : [<c01e2564>] psr: 40000013
sp : c3ac7d88 ip : c3ac7d78 fp : 0000251c
r10: 0047f7a2 r9 : c3ac7edc r8 : c3ac7f5c
r7 : c3ac3260 r6 : c3ac6008 r5 : c28903e0 r4 : c02d1414
Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 83a5c000 DAC: 00000015
------------[ cut here ]------------
WARNING: at arch/arm/mm/alignment.c:822 do_alignment+0xc4/0x3e4()
[<c002e4a0>] (unwind_backtrace+0x0/0xe4) from [<c003d7fc>]
(warn_slowpath_common+0x4c/0x64)
[<c003d7fc>] (warn_slowpath_common+0x4c/0x64) from [<c003d830>]
(warn_slowpath_null+0x1c/0x24)
[<c003d830>] (warn_slowpath_null+0x1c/0x24) from [<c0031688>]
(do_alignment+0xc4/0x3e4)
[<c0031688>] (do_alignment+0xc4/0x3e4) from [<c001e348>]
(do_DataAbort+0xb8/0x138)
[<c001e348>] (do_DataAbort+0xb8/0x138) from [<c0028fdc>] (__dabt_svc+0x5c/0x80)
Exception stack(0xc3ac7d40 to 0xc3ac7d88)
7d40: c28903e0 00000007 c3ac6000 c3b73036 c02d1414 c28903e0 c3ac6008 c3ac3260
7d60: c3ac7f5c c3ac7edc 0047f7a2 0000251c c3ac7d78 c3ac7d88 c01e2564 c01e590c
7d80: 40000013 ffffffff
[<c0028fdc>] (__dabt_svc+0x5c/0x80) from [<c01e590c>] (udp_recvmsg+0x35c/0x52c)
[<c01e590c>] (udp_recvmsg+0x35c/0x52c) from [<c01ea7d8>]
(inet_recvmsg+0x44/0x58)
[<c01ea7d8>] (inet_recvmsg+0x44/0x58) from [<c01940f4>] (sock_recvmsg+0x9c/0xc4)
[<c01940f4>] (sock_recvmsg+0x9c/0xc4) from [<c0194964>] (sys_recvfrom+0x84/0xe0)
[<c0194964>] (sys_recvfrom+0x84/0xe0) from [<c0029440>]
(ret_fast_syscall+0x0/0x2c)
---[ end trace 65f8ea860415c06d ]---
cpu table
pgd table: c3a5c000 addr 0xc01e590c index 0x00000600
pgd entry: c3a5f000 8000041e 8010041e
pud entry: c3a5f000
pmd entry: c3a5f000 8010041e index 1 (section index, 0 or 1)
pmd sect entry = 8010041e
base addr bit 31..20 80100000
sbz bit 19..12 00000000
ap bit 11..10 00000400
domain bit 8..5 00000000
1cb bit 4..2 0000001c
sect bit 1..0 00000002
addr offset bit 19..12 000e5000
init mm
pgd table: c0004000 addr 0xc01e590c index 0x00000600
pgd entry: c0007000 8000041e 8010041e
pud entry: c0007000
pmd entry: c0007000 8010041e index 1 (0 or 1)
pmd sect entry = 8010041e
base addr bit 31..20 80100000
sbz bit 19..12 00000000
ap bit 11..10 00000400
domain bit 8..5 00000000
1cb bit 4..2 0000001c
sect bit 1..0 00000002
addr offset bit 19..12 000e5000
More information about the linux-arm-kernel
mailing list