[LEDE-DEV] ath9k stability in LEDE master branch - kernel panic every ~5 minutes

Nick Lowe nick.lowe at gmail.com
Sun Feb 5 08:50:29 PST 2017 

Hi all,

I have an apu2c4 (quad core AMD GX-412TC) with two AR9390 cards, one
offering service at 2.4 GHz and one offering service at 5 GHz with the
same SSID.

I decided to test out the latest x86-64 LEDE snapshot, as of this
Saturday morning:

lede-x86-64-combined-ext4.img.gz 5582.0 KB Sat Feb 4 21:37:05 2017

LEDE Reboot SNAPSHOT r3293-f791fb4 / LuCI Master (git-17.034.72963-f1820de)

With this, service comes up as expected but I observe regular kernel
panics and reboots, roughly every 5 minutes, usually within the ath9k
driver.

The hardware tests out fine so this looks to be a software issue in
the ath9k driver.

First example:

root at LEDE:/# [  345.531745] general protection fault: 0000 [#1] SMP
[  345.536800] Modules linked in: ath9k ath9k_common pppoe ppp_async
ath9k_hw ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat
ipt_REJECT ipt_MASQUERADE cfg80211 xt_ti
me xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit
xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT slhc r8169
nf_reject_ipv4 nf_nat_redirect nf_nat_masqu
erade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4
nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack
iptable_mangle iptable_filter ip_tables e1000e crc_ccit
t compat i2c_dev ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common
ip6table_mangle ip6table_filter ip6_tables x_tables igb i2c_algo_bit
i2c_core e1000 button_hotplug ptp pps_co
re mii
[  345.601401] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.46 #0
[  345.607344] Hardware name: PC Engines apu2/apu2, BIOS 6f1e98a-dirty
05/15/2016
[  345.614605] task: ffff88011a0e8000 ti: ffff88011a0f0000 task.ti:
ffff88011a0f0000
[  345.622124] RIP: 0010:[<ffffffff81092804>]  [<ffffffff81092804>]
native_queued_spin_lock_slowpath+0xf4/0x160
[  345.632039] RSP: 0018:ffff88011ed83d38  EFLAGS: 00010202
[  345.637374] RAX: 00000000000036e6 RBX: ffff8800da9173f8 RCX: ffff88011ed945c0
[  345.644523] RDX: 0000000000100000 RSI: 0022ad3800376879 RDI: ffff8800da88b4b0
[  345.651682] RBP: ffff88011ed83e38 R08: 0000000000000101 R09: 0000000000000000
[  345.658841] R10: 000000561eb7d9aa R11: 0000000000000246 R12: ffff8800da88b4b0
[  345.665992] R13: 0000000000000030 R14: ffff88011a3afe00 R15: ffff8800da889c00
[  345.673161] FS:  00007fc45bd40b48(0000) GS:ffff88011ed80000(0000)
knlGS:0000000000000000
[  345.681302] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  345.687079] CR2: 000000000060c430 CR3: 00000000db98b000 CR4: 00000000000406e0
[  345.694236] Stack:
[  345.696270]  ffffffff8149f403 ffffffffa02ba330 0000000000000000
0000000000000000
[  345.703801]  0000000000000000 0000000000000000 0000000000000000
0000000000000400
[  345.711329]  0000000000000001 0000000100000000 ffff8800dafa4820
ffffffff00000000
[  345.718855] Call Trace:
[  345.721327]  <IRQ>
[  345.723284]  [<ffffffff8149f403>] ? _raw_spin_lock_bh+0x23/0x30
[  345.729460]  [<ffffffffa02ba330>] ? ath_tx_aggr_wakeup+0x610/0xe90 [ath9k]
[  345.736381]  [<ffffffffa02bb5b8>] ? ath_tx_edma_tasklet+0x188/0x340 [ath9k]
[  345.743371]  [<ffffffffa02b42cf>] ? ath9k_tasklet+0xef/0x230 [ath9k]
[  345.749757]  [<ffffffff81068282>] ? tasklet_action+0xa2/0xb0
[  345.755438]  [<ffffffff81068606>] ? __do_softirq+0xc6/0x1d0
[  345.761035]  [<ffffffff8106882f>] ? irq_exit+0x6f/0x80
[  345.766205]  [<ffffffff8100419a>] ? do_IRQ+0x4a/0xc0
[  345.771205]  [<ffffffff814a027f>] ? common_interrupt+0x7f/0x7f
[  345.777056]  <EOI>
[  345.779005]  [<ffffffff81374cdd>] ? cpuidle_enter_state+0x12d/0x1e0
[  345.785529]  [<ffffffff81374cd6>] ? cpuidle_enter_state+0x126/0x1e0
[  345.791822]  [<ffffffff81091284>] ? cpu_startup_entry+0x1f4/0x250
[  345.797945]  [<ffffffff8102bb8d>] ? start_secondary+0x12d/0x140
[  345.803891] Code: 87 47 02 c1 e0 10 85 c0 74 33 48 89 c6 c1 e8 12
48 c1 ee 0c 83 e8 01 83 e6 30 48 98 48 81 c6 c0 45 01 00 48 03 34 c5
20 dc 6c 81 <48> 89 0e eb 02 f3 90 8b 41
08 85 c0 74 f7 eb 02 f3 90 8b 07 66
[  345.824448] RIP  [<ffffffff81092804>]
native_queued_spin_lock_slowpath+0xf4/0x160
[  345.832016]  RSP <ffff88011ed83d38>
[  345.835563] ---[ end trace b85322db3e31494d ]---
[  345.840213] Kernel panic - not syncing: Fatal exception in interrupt
[  345.846616] Kernel Offset: disabled
[  345.850127] Rebooting in 3 seconds..

Second example:

[  243.130690] general protection fault: 0000 [#1] SMP
[  243.135744] Modules linked in: ath9k ath9k_common pppoe ppp_async
ath9k_hw ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat
ipt_REJECT ipt_MASQUERADE cfg80211 xt_ti
me xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit
xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT slhc r8169
nf_reject_ipv4 nf_nat_redirect nf_nat_masqu
erade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4
nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack
iptable_mangle iptable_filter ip_tables e1000e crc_ccit
t compat i2c_dev ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common
ip6table_mangle ip6table_filter ip6_tables x_tables igb i2c_algo_bit
i2c_core e1000 button_hotplug ptp pps_co
re mii
[  243.200369] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.46 #0
[  243.206321] Hardware name: PC Engines apu2/apu2, BIOS 6f1e98a-dirty
05/15/2016
[  243.213598] task: ffff88011a0e8000 ti: ffff88011a0f0000 task.ti:
ffff88011a0f0000
[  243.221109] RIP: 0010:[<ffffffff81092804>]  [<ffffffff81092804>]
native_queued_spin_lock_slowpath+0xf4/0x160
[  243.231017] RSP: 0018:ffff88011ed83d48  EFLAGS: 00010202
[  243.236350] RAX: 00000000000036a6 RBX: ffff8800da8f1e18 RCX: ffff88011ed945c0
[  243.243508] RDX: 0000000000100000 RSI: 3e29fc21a7aee49d RDI: ffff8800da89b4b0
[  243.250676] RBP: ffff88011ed83e50 R08: 0000000000000101 R09: 0000000000000100
[  243.257835] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800da89b4b0
[  243.265003] R13: 0000000000000037 R14: ffff8800dbbcd100 R15: ffff8800da899c00
[  243.272164] FS:  00007f0c7e15eb48(0000) GS:ffff88011ed80000(0000)
knlGS:0000000000000000
[  243.280292] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  243.286059] CR2: 000000000060c430 CR3: 00000000da775000 CR4: 00000000000406e0
[  243.293212] Stack:
[  243.295244]  ffffffff8149f403 ffffffffa02ba330 0000000000000000
0000000000000000

  345.328010] BUG: unable to handle kernel paging request at 0000000000451885
[  345.335293] IP: [<ffffffff81092804>]
native_queued_spin_lock_slowpath+0xf4/0x160
[  345.342965] PGD dac4a067 PUD daaf3067 PMD da8df067 PTE 0
[  345.348564] Oops: 0002 [#1] SMP
[  345.351976] Modules linked in: ath9k ath9k_common pppoe ppp_async
ath9k_hw ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat
ipt_REJECT ipt_MASQUERADE cfg80211 xt_ti
me xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit
xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT slhc r8169
nf_reject_ipv4 nf_nat_redirect nf_nat_masqu
erade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4
nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack
iptable_mangle iptable_filter ip_tables e1000e crc_ccit
t compat i2c_dev ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common
ip6table_mangle ip6table_filter ip6_tables x_tables igb i2c_algo_bit
i2c_core e1000 button_hotplug ptp pps_co
re mii
[  345.418299] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.4.46 #0
[  345.424454] Hardware name: PC Engines apu2/apu2, BIOS 6f1e98a-dirty
05/15/2016
[  345.431879] task: ffff88011a0a6300 ti: ffff88011a0e0000 task.ti:
ffff88011a0e0000
[  345.439589] RIP: 0010:[<ffffffff81092804>]  [<ffffffff81092804>]
native_queued_spin_lock_slowpath+0xf4/0x160
[  345.449790] RSP: 0018:ffff88011ec83d38  EFLAGS: 00010202
[  345.455327] RAX: 00000000000036a6 RBX: ffff8800da8ca7b8 RCX: ffff88011ec945c0
[  345.462761] RDX: 0000000000080000 RSI: 0000000000451885 RDI: ffff8800da8ab4b0
[  345.470115] RBP: ffff88011ec83e38 R08: 0000000000000101 R09: ffff88011a001900
[  345.477552] R10: 0000000000000010 R11: 0000000000003c15 R12: ffff8800da8ab4b0
[  345.484861] R13: 0000000000000033 R14: ffff8800dac4c400 R15: ffff8800da8a9c00
[  345.492162] FS:  00007f454b902b48(0000) GS:ffff88011ec80000(0000)
knlGS:0000000000000000
[  345.500547] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  345.506401] CR2: 0000000000451885 CR3: 00000000da39d000 CR4: 00000000000406e0
[  345.513796] Stack:
[  345.515900]  ffffffff8149f403 ffffffffa02ba330 ffff8800db2a2c00
ffff8800da744d70
[  345.523611]  ffff8800da8a9638 ffff8800da9e2700 ffff8800da744d7a
ffffffffa01d579a
[  345.531345]  0000000000000101 ffff8800da744d70 ffff8800db024820
ffffffffda958e00

root at LEDE:/# [  365.266745] general protection fault: 0000 [#1] SMP
[  365.271799] Modules linked in: ath9k ath9k_common pppoe ppp_async
ath9k_hw ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat
ipt_REJECT ipt_MASQUERADE cfg80211 xt_ti
me xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit
xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT slhc r8169
nf_reject_ipv4 nf_nat_redirect nf_nat_masqu
erade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4
nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack
iptable_mangle iptable_filter ip_tables e1000e crc_ccit
t compat i2c_dev ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common
ip6table_mangle ip6table_filter ip6_tables x_tables igb i2c_algo_bit
i2c_core e1000 button_hotplug ptp pps_co
re mii
[  365.336398] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.46 #0
[  365.342339] Hardware name: PC Engines apu2/apu2, BIOS 6f1e98a-dirty
05/15/2016
[  365.349596] task: ffff88011a0e8000 ti: ffff88011a0f0000 task.ti:
ffff88011a0f0000
[  365.357107] RIP: 0010:[<ffffffff81092804>]  [<ffffffff81092804>]
native_queued_spin_lock_slowpath+0xf4/0x160
[  365.367005] RSP: 0018:ffff88011ed83d38  EFLAGS: 00010206
[  365.372336] RAX: 000000000000368e RBX: ffff8800da8a6d98 RCX: ffff88011ed945c0
[  365.379492] RDX: 0000000000100000 RSI: 026f019c019a4788 RDI: ffff8800da8834b0
[  365.386641] RBP: ffff88011ed83e38 R08: 0000000000000101 R09: 0000000000000006
[  365.393797] R10: 0000000000000003 R11: 0000000000000246 R12: ffff8800da8834b0
[  365.400962] R13: 0000000000000070 R14: ffff88011a3a7900 R15: ffff8800da881c00
[  365.408120] FS:  00007fc203a45b48(0000) GS:ffff88011ed80000(0000)
knlGS:0000000000000000
[  365.416249] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  365.422022] CR2: 0000000000eceff0 CR3: 00000000db2e7000 CR4: 00000000000406e0
[  365.429171] Stack:
[  365.431204]  ffffffff8149f403 ffffffffa02ba330 0000000000000000
0000000000000000

Thanks,

Nick

More information about the Lede-dev mailing list