[FS#1085] Streaming ieee80211 cmd timeouts followed by crash

Thu Oct 19 15:25:23 PDT 2017

A new Flyspray task has been opened.  Details are below. 

User who did this - ratsputin (ratsputin) 

Attached to Project - LEDE Project
Summary - Streaming ieee80211 cmd timeouts followed by crash
Task Type - Bug Report
Category - Kernel
Status - Unconfirmed
Assigned To - 
Operating System - All
Severity - Critical
Priority - Very Low
Reported Version - Trunk
Due in Version - Undecided
Due Date - Undecided
Details - Supply the following if possible:
 - Device problem occurs on: **Linksys WRT3200ACM
**
 - Software versions of LEDE release, packages, etc.: **LEDE Reboot SNAPSHOT r5096-a29848c / LuCI Master (git-17.260.43176-678c15c)**

Router had been up for less than 24 hours from last crash.  Started streaming the following errors in syslog:

Thu Oct 19 16:50:41 2017 kern.err kernel: [82666.284359] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
Thu Oct 19 16:50:41 2017 kern.err kernel: [82666.290490] ieee80211 phy0: return code: 0x1122
Thu Oct 19 16:50:41 2017 kern.err kernel: [82666.295069] ieee80211 phy0: timeout: 0x1122
Thu Oct 19 16:50:41 2017 kern.err kernel: [82666.299280] wlan0: failed to set key (0, e0:b5:2d:xx:xx:xx) to hardware (-5)
Thu Oct 19 16:50:41 2017 daemon.notice hostapd: wlan0: AP-STA-CONNECTED e0:b5:2d:xx:xx:xx
Thu Oct 19 16:50:45 2017 kern.err kernel: [82670.356504] ieee80211 phy0: cmd 0x801d=MEMAddrAccess timed out
Thu Oct 19 16:50:45 2017 kern.err kernel: [82670.362367] ieee80211 phy0: return code: 0x051d
Thu Oct 19 16:50:45 2017 kern.err kernel: [82670.366924] ieee80211 phy0: timeout: 0x001d
Thu Oct 19 16:50:49 2017 kern.err kernel: [82674.372313] ieee80211 phy0: cmd 0x801d=MEMAddrAccess timed out
Thu Oct 19 16:50:49 2017 kern.err kernel: [82674.378197] ieee80211 phy0: return code: 0x001d
Thu Oct 19 16:50:49 2017 kern.err kernel: [82674.382749] ieee80211 phy0: timeout: 0x001d

**Interspersed with these messages were a few:
**Thu Oct 19 16:52:43 2017 daemon.notice hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5

**These then started showing up:
**Thu Oct 19 16:54:56 2017 daemon.notice netifd: wan6 (1414): Command failed: Request timed out

**Then a handful of these:
**
Thu Oct 19 16:56:00 2017 kern.err kernel: [82985.426468] ieee80211 phy0: cmd 0x9143=GetSeqno timed out
Thu Oct 19 16:56:00 2017 kern.err kernel: [82985.431902] ieee80211 phy0: return code: 0x1143
Thu Oct 19 16:56:00 2017 kern.err kernel: [82985.436450] ieee80211 phy0: timeout: 0x1143

**Followed by this, including a crash:
**
Thu Oct 19 16:56:36 2017 kern.err kernel: [83021.558164] wlan0: failed to remove key (0, 78:7e:61:xx:xx:xx) from hardware (-5)
Thu Oct 19 16:56:36 2017 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED e0:b5:2d:2a:89:30
Thu Oct 19 16:56:36 2017 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED dc:41:5f:b9:0d:ba
Thu Oct 19 16:56:40 2017 kern.err kernel: [83025.588715] ieee80211 phy0: cmd 0x9122=UpdateEncryption timed out
Thu Oct 19 16:56:40 2017 kern.err kernel: [83025.594898] ieee80211 phy0: return code: 0x1122
Thu Oct 19 16:56:40 2017 kern.err kernel: [83025.599463] ieee80211 phy0: timeout: 0x1122
Thu Oct 19 16:56:40 2017 kern.err kernel: [83025.603666] wlan0: failed to remove key (0, dc:41:5f:xx:xx:xx) from hardware (-5)
Thu Oct 19 16:56:44 2017 kern.err kernel: [83029.646987] ieee80211 phy0: cmd 0x9125=BAStream timed out
Thu Oct 19 16:56:44 2017 kern.err kernel: [83029.652419] ieee80211 phy0: return code: 0x1125
Thu Oct 19 16:56:44 2017 kern.err kernel: [83029.656973] ieee80211 phy0: timeout: 0x1125
Thu Oct 19 16:56:48 2017 kern.err kernel: [83033.654789] ieee80211 phy0: cmd 0x9125=BAStream timed out
Thu Oct 19 16:56:48 2017 kern.err kernel: [83033.660221] ieee80211 phy0: return code: 0x1125
Thu Oct 19 16:56:48 2017 kern.err kernel: [83033.664769] ieee80211 phy0: timeout: 0x1125
Thu Oct 19 16:56:52 2017 kern.err kernel: [83037.666085] ieee80211 phy0: cmd 0x9125=BAStream timed out
Thu Oct 19 16:56:52 2017 kern.err kernel: [83037.671519] ieee80211 phy0: return code: 0x1125
Thu Oct 19 16:56:52 2017 kern.err kernel: [83037.676066] ieee80211 phy0: timeout: 0x1125

Thu Oct 19 16:56:52 2017 kern.alert kernel: [83037.680345] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
Thu Oct 19 16:56:52 2017 kern.alert kernel: [83037.688494] pgd = d303c000
Thu Oct 19 16:56:53 2017 kern.alert kernel: [83037.691221] [0000000c] *pgd=1369c831, *pte=00000000, *ppte=00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.697545] Internal error: Oops: 17 [#1] SMP ARM
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.702266] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc rfcomm nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipvThu Oct 19 16:56:53 2017 kern.warn kernel: [83037.774820]  ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables tun regmap_mmio ecb cmac uhci_hcd ohci_platform ohci_hcd gpio_button_hotplug
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.812372] CPU: 1 PID: 3044 Comm: hostapd Not tainted 4.9.54 #0
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.818403] Hardware name: Marvell Armada 380/385 (Device Tree)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.824346] task: df564e40 task.stack: d3718000
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.828909] PC is at mwl_fwcmd_destroy_ba+0x98/0xf4 [mwlwifi]
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.834683] LR is at mwl_fwcmd_destroy_ba+0x34/0xf4 [mwlwifi]
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.840451] pc : []    lr : []    psr: 60000013
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.840451] sp : d3719c90  ip : 00000000  fp : 00000000
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.851975] r10: dde78000  r9 : dde78048  r8 : dde78054
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.857220] r7 : ddfc1620  r6 : ddfc2924  r5 : 00000000  r4 : ddcb0000
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.863771] r3 : 00000000  r2 : 00000002  r1 : 00000000  r0 : 00000000
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.870324] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83037.877487] Control: 10c5387d  Table: 1303c04a  DAC: 00000051
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.883255] Process hostapd (pid: 3044, stack limit = 0xd3718210)
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.889372] Stack: (0xd3719c90 to 0xd371a000)
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.893745] 9c80:                                     dde78054 00000007 d8008de8 ddfc45e0
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.901957] 9ca0: ddfc0be0 bf18aac8 d8008de8 ddfc1620 ddfc0be0 d3c51e14 d36b9ce0 de7e0000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.910168] 9cc0: 00000014 bf185b94 d8008800 00000002 00000001 d3c51e14 d81f5e40 de7e0000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.918379] 9ce0: 00000014 bf118c98 ddfc0be0 d36b94c0 d8008800 bf11b2dc 00000002 00000002
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.926589] 9d00: d8008800 ddfc0be0 d36b94c0 bf11c270 d8008800 00000000 00000000 d8008800
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.934800] 9d20: d3c51e20 bf11c3bc d36b94c0 00000454 d3c51e20 bf11c434 ddfc0000 d36b9000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.943012] 9d40: d3719d68 bf0e38e4 d3c51e20 0002000c 00000000 bf0ff534 bf108c84 c03c0e1c
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.951223] 9d60: bf0fee9a 80150014 59e7e961 0000068e d3c51e00 d3c51e10 d3c51e14 de7e0000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.959434] 9d80: ddfc0000 d36b9000 d3c51e00 d81f5e40 c03c0b94 d3475c00 d3719dcc 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.967645] 9da0: 00000008 c03c01d0 c0629cb0 d81f5e40 d81f5e40 c03c0b80 df4ba400 00000028
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.975857] 9dc0: d81f5e40 c03bfbd4 d3475c00 7fffffff 00000000 d81f5e40 00000000 d3719f68
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.984068] 9de0: 00000028 d3475c00 00000000 c03c0040 00000000 c021cb0c 00000028 d0851040
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83037.992278] 9e00: 00000000 00000be4 00000000 00000000 00000001 d3719f68 00000000 d3719e8c
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.000489] 9e20: df30f200 00000000 c0604fc8 00000000 00000000 c0380060 d3719f68 c03805f0
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.008701] 9e40: 00000000 00000000 00000000 00376010 00000028 beadcbec 00000000 d3719eb4
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.016912] 9e60: df30f200 c03807e8 c05ffec0 00000000 beadcc54 0038a030 00004000 c004f000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.025124] 9e80: dfbd2ef0 ffffffff e813b2d8 c0628400 d3719ecc 00000051 00008933 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.033334] 9ea0: 00000000 beadcc8c 00000020 00000010 00000000 00000000 00000000 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.041545] 9ec0: dfbd2ef0 c0605998 c05ffec0 6e616c77 d32552c0 60000013 df569500 d3255080
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.049755] 9ee0: 00000000 d3255080 d0029b20 c00f188c d0029b84 00000007 00000000 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.057966] 9f00: df564e40 c005aae0 d3719f08 d3719f08 df401180 d0029b00 df29dc38 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.066177] 9f20: df401580 df29dc38 df29dc38 00040936 d0029b20 df30f200 beadcc70 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.074388] 9f40: 00000128 c000ef04 d3718000 00000000 b6e9c740 c038138c 00000000 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.082599] 9f60: 00000000 fffffff7 d3719eac 0000000c 00000001 00000000 00000000 d3719e54
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.090809] 9f80: 00000000 00000006 00000000 00000000 00000000 00000000 00000000 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.099021] 9fa0: b6f1610c c000ed40 00000000 00000000 00000005 beadcc70 00000000 00000000
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.107231] 9fc0: 00000000 00000000 b6f1610c 00000128 b6e9cef0 00000000 0002516c b6e9c740
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.115443] 9fe0: beadcc18 beadcc04 b6eececc b6eec398 60000010 00000005 00000000 00000000
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.123678] [] (mwl_fwcmd_destroy_ba [mwlwifi]) from [] (mwl_fwcmd_del_sta_streams+0x1d4/0x330 [mwlwifi])
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.135037] [] (mwl_fwcmd_del_sta_streams [mwlwifi]) from [] (mwl_mac80211_sta_remove+0x38/0x1c4 [mwlwifi])
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.146598] [] (mwl_mac80211_sta_remove [mwlwifi]) from [] (drv_sta_state+0x16c/0x184 [mac80211])
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.157290] [] (drv_sta_state [mac80211]) from [] (sta_info_move_state+0xbc/0x1d0 [mac80211])
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.167629] [] (sta_info_move_state [mac80211]) from [] (sta_set_sinfo+0x954/0xa80 [mac80211])
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.178054] [] (sta_set_sinfo [mac80211]) from [] (__sta_info_destroy+0x20/0x28 [mac80211])
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.188218] [] (__sta_info_destroy [mac80211]) from [] (sta_info_destroy_addr_bss+0x2c/0x44 [mac80211])
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.199429] [] (sta_info_destroy_addr_bss [mac80211]) from [] (nl80211_del_station+0xe8/0xf0 [cfg80211])
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.210711] [] (nl80211_del_station [cfg80211]) from [] (genl_rcv_msg+0x288/0x310)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.220059] [] (genl_rcv_msg) from [] (netlink_rcv_skb+0x58/0xb4)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.227922] [] (netlink_rcv_skb) from [] (genl_rcv+0x20/0x34)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.235436] [] (genl_rcv) from [] (netlink_unicast+0x138/0x1fc)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.243124] [] (netlink_unicast) from [] (netlink_sendmsg+0x2f0/0x310)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.251425] [] (netlink_sendmsg) from [] (sock_sendmsg+0x14/0x24)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.259287] [] (sock_sendmsg) from [] (___sys_sendmsg+0x184/0x228)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.267238] [] (___sys_sendmsg) from [] (__sys_sendmsg+0x40/0x64)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.275104] [] (__sys_sendmsg) from [] (ret_fast_syscall+0x0/0x3c)
Thu Oct 19 16:56:53 2017 kern.emerg kernel: [83038.283055] Code: e5840010 e5d81004 e5c41014 e598c000 (e59c300c)
Thu Oct 19 16:56:53 2017 kern.warn kernel: [83038.289200] ---[ end trace 6c4677423c79f036 ]---

More information can be found at the following URL:
https://bugs.lede-project.org/index.php?do=details&task_id=1085