[BUG] Kernel crashes with ath10k radio and Nexus 5X clients

Fri Nov 18 01:30:41 PST 2016

Hi again,

in the meantime, I have some more information to add to the issue mentioned in
my email quoted further down below.

Ben Greear approached me off-list and suggested to try the Candela Tech ath10k
driver and firmware and see if the issue occurs with that as well. So, for the
last 3 weeks I've been testing the CT driver and firmware and I can happily
report that the issue with the driver crashing after a while when a Nexus 5X
device is connected is not occuring with the current BETA 18
firmware-2-ct-full-community.bin. So, this really seems like a regression in
the official API level 5 ath10k firmware blobs.

The CT firmware is not perfect either, since it seems to suffer from a different
bug that has been resolved in the official firmwares, and that is that after a
reboot of my TP-Link Archer C7 v2, the ath10k driver won't load. Only after a 
hard reset or "cold boot" it will come up. That's an issue I had with older
official firmwares as well, but it has been resolved with the recent API level
5 firmwares.

Nevertheless, for the time being, I will stick to the CT firmware because I can
work around the reboot issue and having the 5GHz wifi working for my Nexus 5X
clients is more important.

Over the next weeks, I will test different combinations of ath10k(-ct) driver
and firmware to see if there's a combination that resolves all issues. This
morning I flashed a LEDE build with the official ath10k driver and the CT
firmware binary.

But of course, if someone has more suggestions on what I could try or what
information I could collect to help resolve the issue related to the Nexus 5X
clients in the official firmware binaries, I think that would be beneficial
for a larger audience.

Regards,

Timo

P.S.: Please include my email address in any reply, since I'm not subscribed
to the mailing list. Thank you.

Timo Sigurdsson schrieb am 29.10.2016 22:19:

> Hi everybody,
> 
> I have a TP-Link Archer C7 v2 running a fairly recent build of LEDE (r1952, 
> Linux 4.4.26, compat-wireless-2016-10-08). It all works well except for the
> fact that when I connect a Nexus 5X device to the 5GHz radio, the kernel or
> ath10k driver will crash after a while. 5Ghz wifi will be dead after that
> until I reboot the system.
> 
> This issue has been reported before [1] and it also has been declared as
> solved with newer firmwares [2] (but reopened by other users). However, even
> with the latest firmware 10.2.4.70.58 from Kalle Valo's Github repository the
> issue is far from resolved. I have tried many different firmware revisions
> over the time (more recently 10.2.4.70.56 and 10.2.4.70.54), and I can could
> only find that the issue sometimes takes longer to trigger with some firmwares
> (which might just be random), but it would always occur at some point with API
> level 5 firmwares. With API level 2 firmwares (which I testesd when I was
> still using OpenWrt 15.05) I never saw these crashes, but the Nexus 5X had
> other connectivity issues with these older firmwares that made this
> combination no fun to use either. But this shows that the firmware itself
> makes the difference here.
> 
> I actually have two Nexus 5X on my network (my wife's and my own). I can
> trigger the crash with either one of them. And if both Nexus 5X are connected
> to the 5Ghz radio, then the issue triggers much faster (can be as low as 15
> minutes). My workaround is to let the Nexus 5X devices only connect to the
> 2.4GHz radio. This way, the device can runs for weeks without any issue or
> crash, but of course I would prefer the actual bug being fixed rather than to
> circumvent it.
> 
> I'm appending a syslog from my access point with a crash happening while one
> Nexus 5X was connected to the 5GHz radio starting from the time the system
> booted. I randomized the MAC addresses. but left the first two characters
> unique so different clients can be distinguished.
> 
> If there is more info I could collect to help identify the cause of this
> issue, please let me know (and possibly how to do that as well).
> 
> Thank you and regards,
> 
> Timo
> 
> [1] http://lists.infradead.org/pipermail/ath10k/2015-November/006413.html
> [2] https://dev.openwrt.org/ticket/20854
> 
> And here's the log:
> Fri Oct 28 02:01:35 2016 kern.notice kernel: [    0.000000] Linux version
> 4.4.26 (user at buildsystem) (gcc version 5.4.0 (LEDE GCC 5.4.0 r1952) ) #0 Fri
> Oct 21 15:52:28 2016
> [...]
> Fri Oct 28 02:01:35 2016 kern.info kernel: [    9.468751] Loading modules
> backported from Linux version wt-2016-10-03-1-g6fcb1a6
> Fri Oct 28 02:01:35 2016 kern.info kernel: [    9.476481] Backport generated by
> backports.git backports-20160324-9-g0e38f5c
> Fri Oct 28 02:01:35 2016 kern.warn kernel: [    9.576570] PCI: Enabling device
> 0000:01:00.0 (0000 -> 0002)
> Fri Oct 28 02:01:35 2016 kern.info kernel: [    9.582475] ath10k_pci
> 0000:01:00.0: pci irq legacy oper_irq_mode 1 irq_mode 0 reset_mode 0
> Fri Oct 28 02:01:35 2016 kern.warn kernel: [   10.087609] ath10k_pci
> 0000:01:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:01:00.0.bin
> failed with error -2
> Fri Oct 28 02:01:35 2016 kern.warn kernel: [   10.098492] ath10k_pci
> 0000:01:00.0: Falling back to user helper
> Fri Oct 28 02:01:35 2016 kern.err kernel: [   10.176500] firmware
> ath10k!pre-cal-pci-0000:01:00.0.bin: firmware_loading_store: map pages failed
> Fri Oct 28 02:01:35 2016 kern.info kernel: [   10.677026] ath10k_pci
> 0000:01:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000
> Fri Oct 28 02:01:35 2016 kern.info kernel: [   10.686424] ath10k_pci
> 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
> Fri Oct 28 02:01:35 2016 kern.info kernel: [   10.699498] ath10k_pci
> 0000:01:00.0: firmware ver 10.2.4.70.58 api 5 features no-p2p,raw-mode,mfp
> crc32 e1af076f
> Fri Oct 28 02:01:35 2016 kern.warn kernel: [   10.709932] ath10k_pci
> 0000:01:00.0: Direct firmware load for ath10k/QCA988X/hw2.0/board-2.bin failed
> with error -2
> Fri Oct 28 02:01:35 2016 kern.warn kernel: [   10.720531] ath10k_pci
> 0000:01:00.0: Falling back to user helper
> Fri Oct 28 02:01:35 2016 kern.err kernel: [   10.798719] firmware
> ath10k!QCA988X!hw2.0!board-2.bin: firmware_loading_store: map pages failed
> Fri Oct 28 02:01:35 2016 kern.info kernel: [   10.823845] ath10k_pci
> 0000:01:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
> Fri Oct 28 02:01:35 2016 kern.info kernel: [   11.954723] ath10k_pci
> 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 cal file max-sta 128 raw 0 hwcrypto
> 1
> [...]
> // Evertyhing runs fine for a bit more than a day, but then this happens ... //
> // Note: The ath10k radio in question is wlan0 //
> Sat Oct 29 10:38:21 2016 daemon.info hostapd: wlan1: STA 9c:12:34:56:78:9a IEEE
> 802.11: disassociated
> Sat Oct 29 10:38:22 2016 daemon.info hostapd: wlan1: STA 9c:12:34:56:78:9a IEEE
> 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
> Sat Oct 29 10:42:57 2016 daemon.info hostapd: wlan1: STA 00:12:34:56:78:9a WPA:
> group key handshake completed (RSN)
> Sat Oct 29 10:42:57 2016 daemon.info hostapd: wlan0: STA a0:12:34:56:78:9a WPA:
> group key handshake completed (RSN)
> Sat Oct 29 10:42:57 2016 daemon.info hostapd: wlan0: STA 70:12:34:56:78:9a WPA:
> group key handshake completed (RSN)
> Sat Oct 29 10:43:05 2016 kern.warn kernel: [117635.147861] ath10k_pci
> 0000:01:00.0: failed to delete peer 7f:12:34:56:78:9a for vdev 0: -145
> Sat Oct 29 10:43:05 2016 kern.warn kernel: [117635.156679] ath10k_pci
> 0000:01:00.0: found sta peer 7f:12:34:56:78:9a (ptr 86ebf200 id 62) entry on
> vdev 0 after it was supposedly removed
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.169489] ------------[ cut
> here ]------------
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.174314] WARNING: CPU: 0 PID:
> 1885 at compat-wireless-2016-10-08/net/mac80211/sta_info.c:964 0x875870a4
> [mac80211 at 87580000+0x5f500]()
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.186854] Modules linked in:
> pppoe ppp_async iptable_nat ath9k pppox ppp_generic nf_nat_ipv4
> nf_conntrack_ipv6 nf_conntrack_ipv4 ipt_REJECT ipt_MASQUERADE ax88179_178a
> ath9k_common xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat
> xt_multiport xt_mark xt_mac xt_limit xt_length xt_id xt_hl xt_helper xt_ecn
> xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS
> xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY usbnet Sat Oct 29 10:43:06
> 2016 kern.warn kernel: [117635.294282] CPU: 0 PID: 1885 Comm: hostapd Not
> tainted 4.4.26 #0
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.300471] Stack : 8036cff4
> 00000000 00000001 803c0000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.300471] 	  00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.300471] 	  00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.300471] 	  00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.300471] 	  00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.300471] 	  ...
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.336918] Call
> Trace:[<800716f0>] 0x800716f0
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.341551] [<800716f0>]
> 0x800716f0
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.345176] [<8008144c>]
> 0x8008144c
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.348832] [<875870a4>]
> 0x875870a4 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.354943] [<80081504>]
> 0x80081504
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.358611] [<875870a4>]
> 0x875870a4 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.364742] [<87587188>]
> 0x87587188 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.370861] [<875849b4>]
> 0x875849b4 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.376951] [<80115770>]
> 0x80115770
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.380613] [<87587238>]
> 0x87587238 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.386727] [<874cc114>]
> 0x874cc114 [cfg80211 at 874c0000+0x35b60]
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.392854] [<80293074>]
> 0x80293074
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.396485] [<80291900>]
> 0x80291900
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.400176] [<80292df8>]
> 0x80292df8
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.403801] [<80292238>]
> 0x80292238
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.407439] [<80292de0>]
> 0x80292de0
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.411079] [<80257794>]
> 0x80257794
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.414711] [<80291aa0>]
> 0x80291aa0
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.418345] [<80256c90>]
> 0x80256c90
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.421991] [<80292004>]
> 0x80292004
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.425616] [<801b5304>]
> 0x801b5304
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.429260] [<8025b478>]
> 0x8025b478
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.432902] [<8024d0e4>]
> 0x8024d0e4
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.436540] [<8024e4f8>]
> 0x8024e4f8
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.440195] [<801b5304>]
> 0x801b5304
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.443835] [<8027e338>]
> 0x8027e338
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.447464] [<802e8244>]
> 0x802e8244
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.451136] [<8011aa58>]
> 0x8011aa58
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.454762] [<8011ae3c>]
> 0x8011ae3c
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.458402] [<8011acd4>]
> 0x8011acd4
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.462057] [<80117a10>]
> 0x80117a10
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.465700] [<8024f424>]
> 0x8024f424
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.469344] [<800b1dd0>]
> 0x800b1dd0
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.472986] [<8006295c>]
> 0x8006295c
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.476615] 
> Sat Oct 29 10:43:06 2016 kern.warn kernel: [117635.478223] ---[ end trace
> 2c07f06df2595aa5 ]---
> Sat Oct 29 10:43:06 2016 daemon.info hostapd: wlan0: STA 7f:12:34:56:78:9a IEEE
> 802.11: deauthenticated due to local deauth request
> Sat Oct 29 10:43:15 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a IEEE
> 802.11: authenticated
> Sat Oct 29 10:43:15 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a IEEE
> 802.11: associated (aid 2)
> Sat Oct 29 10:43:15 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a
> RADIUS: starting accounting session D1227001A194DA09
> Sat Oct 29 10:43:15 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a WPA:
> pairwise key handshake completed (RSN)
> Sat Oct 29 10:43:31 2016 daemon.info hostapd: wlan1: STA a0:12:34:56:78:9a IEEE
> 802.11: authenticated
> Sat Oct 29 10:43:31 2016 daemon.info hostapd: wlan1: STA a0:12:34:56:78:9a IEEE
> 802.11: associated (aid 3)
> Sat Oct 29 10:43:31 2016 daemon.info hostapd: wlan1: STA a0:12:34:56:78:9a
> RADIUS: starting accounting session 92084E55AFE512B3
> Sat Oct 29 10:43:31 2016 daemon.info hostapd: wlan1: STA a0:12:34:56:78:9a WPA:
> pairwise key handshake completed (RSN)
> Sat Oct 29 10:48:02 2016 daemon.info hostapd: wlan0: STA a0:12:34:56:78:9a IEEE
> 802.11: disassociated due to inactivity
> Sat Oct 29 10:48:05 2016 kern.warn kernel: [117935.165013] ath10k_pci
> 0000:01:00.0: failed to install key for vdev 0 peer a0:12:34:56:78:9a: -145
> Sat Oct 29 10:48:05 2016 kern.err kernel: [117935.174193] wlan0: failed to
> remove key (0, a0:12:34:56:78:9a) from hardware (-145)
> Sat Oct 29 10:48:05 2016 daemon.info hostapd: wlan0: STA 70:12:34:56:78:9a IEEE
> 802.11: disassociated due to inactivity
> Sat Oct 29 10:48:08 2016 kern.warn kernel: [117938.174996] ath10k_pci
> 0000:01:00.0: failed to install key for vdev 0 peer 70:12:34:56:78:9a: -11
> Sat Oct 29 10:48:08 2016 kern.err kernel: [117938.184092] wlan0: failed to
> remove key (0, 70:12:34:56:78:9a) from hardware (-11)
> Sat Oct 29 10:48:08 2016 daemon.info hostapd: wlan0: STA a0:12:34:56:78:9a IEEE
> 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.193251] ath10k_pci
> 0000:01:00.0: cipher 0 is not supported
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.199320] ath10k_pci
> 0000:01:00.0: failed to remove peer wep key 0: -122
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.206409] ath10k_pci
> 0000:01:00.0: failed to clear all peer wep keys for vdev 0: -122
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.214621] ath10k_pci
> 0000:01:00.0: failed to disassociate station: a0:12:34:56:78:9a vdev 0: -122
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.223892] ------------[ cut
> here ]------------
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.228706] WARNING: CPU: 0 PID:
> 1885 at compat-wireless-2016-10-08/net/mac80211/sta_info.c:956 0x87587050
> [mac80211 at 87580000+0x5f500]()
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.241226] Modules linked in:
> pppoe ppp_async iptable_nat ath9k pppox ppp_generic nf_nat_ipv4
> nf_conntrack_ipv6 nf_conntrack_ipv4 ipt_REJECT ipt_MASQUERADE ax88179_178a
> ath9k_common xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat
> xt_multiport xt_mark xt_mac xt_limit xt_length xt_id xt_hl xt_helper xt_ecn
> xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS
> xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY usbnet Sat Oct 29 10:48:09
> 2016 kern.warn kernel: [117938.348645] CPU: 0 PID: 1885 Comm: hostapd Tainted:
> G        W       4.4.26 #0
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.356067] Stack : 8036cff4
> 00000000 00000001 803c0000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.356067] 	  00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.356067] 	  00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.356067] 	  00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.356067] 	  00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.356067] 	  ...
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.392511] Call
> Trace:[<800716f0>] 0x800716f0
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.397147] [<800716f0>]
> 0x800716f0
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.400770] [<8008144c>]
> 0x8008144c
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.404414] [<87587050>]
> 0x87587050 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.410531] [<80081504>]
> 0x80081504
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.414184] [<87587050>]
> 0x87587050 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.420297] [<8759cb40>]
> 0x8759cb40 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.426431] [<87587188>]
> 0x87587188 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.432531] [<875849b4>]
> 0x875849b4 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.438629] [<801018e4>]
> 0x801018e4
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.442276] [<87587238>]
> 0x87587238 [mac80211 at 87580000+0x5f500]
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.448406] [<874cc114>]
> 0x874cc114 [cfg80211 at 874c0000+0x35b60]
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.454516] [<80293074>]
> 0x80293074
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.458147] [<80291900>]
> 0x80291900
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.461822] [<80292df8>]
> 0x80292df8
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.465461] [<80292238>]
> 0x80292238
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.469092] [<800a15d0>]
> 0x800a15d0
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.472726] [<80292de0>]
> 0x80292de0
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.476371] [<80257794>]
> 0x80257794
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.480002] [<80291aa0>]
> 0x80291aa0
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.483630] [<80256c90>]
> 0x80256c90
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.487280] [<80292004>]
> 0x80292004
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.490908] [<801b5304>]
> 0x801b5304
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.494541] [<8025b478>]
> 0x8025b478
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.498198] [<8024d0e4>]
> 0x8024d0e4
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.501840] [<8024e4f8>]
> 0x8024e4f8
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.505492] [<8027e338>]
> 0x8027e338
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.509121] [<802e8244>]
> 0x802e8244
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.512780] [<8011aa58>]
> 0x8011aa58
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.516413] [<8011ae3c>]
> 0x8011ae3c
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.520043] [<8011acd4>]
> 0x8011acd4
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.523697] [<80117a10>]
> 0x80117a10
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.527352] [<8024f424>]
> 0x8024f424
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.530981] [<801059e8>]
> 0x801059e8
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.534625] [<8006295c>]
> 0x8006295c
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.538265] 
> Sat Oct 29 10:48:09 2016 kern.warn kernel: [117938.539870] ---[ end trace
> 2c07f06df2595aa6 ]---
> Sat Oct 29 10:48:12 2016 kern.warn kernel: [117941.534956] ath10k_pci
> 0000:01:00.0: failed to delete peer a0:12:34:56:78:9a for vdev 0: -11
> Sat Oct 29 10:48:12 2016 kern.warn kernel: [117941.543716] ath10k_pci
> 0000:01:00.0: found sta peer a0:12:34:56:78:9a (ptr 86ebf400 id 196) entry on
> vdev 0 after it was supposedly removed
> Sat Oct 29 10:48:12 2016 daemon.info hostapd: wlan0: STA 70:12:34:56:78:9a IEEE
> 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
> Sat Oct 29 10:48:12 2016 kern.warn kernel: [117941.558962] ath10k_pci
> 0000:01:00.0: cipher 0 is not supported
> Sat Oct 29 10:48:12 2016 kern.warn kernel: [117941.565049] ath10k_pci
> 0000:01:00.0: failed to remove peer wep key 0: -122
> Sat Oct 29 10:48:15 2016 kern.warn kernel: [117944.564917] ath10k_warn: 2
> callbacks suppressed
> Sat Oct 29 10:48:15 2016 kern.warn kernel: [117944.569617] ath10k_pci
> 0000:01:00.0: failed to delete peer 70:12:34:56:78:9a for vdev 0: -11
> Sat Oct 29 10:48:15 2016 kern.warn kernel: [117944.578309] ath10k_pci
> 0000:01:00.0: found sta peer 70:12:34:56:78:9a (ptr 86320c00 id 251) entry on
> vdev 0 after it was supposedly removed
> Sat Oct 29 10:48:18 2016 kern.warn kernel: [117947.584908] ath10k_pci
> 0000:01:00.0: failed to set beacon mode for vdev 0: -11
> Sat Oct 29 10:48:21 2016 kern.warn kernel: [117950.584876] ath10k_pci
> 0000:01:00.0: failed to set dtim period for vdev 0: -11
> Sat Oct 29 10:48:24 2016 kern.warn kernel: [117953.594843] ath10k_pci
> 0000:01:00.0: failed to recalculate rts/cts prot for vdev 0: -11
> Sat Oct 29 10:48:27 2016 kern.warn kernel: [117956.594833] ath10k_pci
> 0000:01:00.0: failed to set protection mode 0 on vdev 0: -11
> Sat Oct 29 10:48:30 2016 kern.warn kernel: [117959.594805] ath10k_pci
> 0000:01:00.0: failed to set preamble for vdev 0: -11
> Sat Oct 29 10:52:57 2016 daemon.info hostapd: wlan1: STA 00:12:34:56:78:9a WPA:
> group key handshake completed (RSN)
> Sat Oct 29 10:52:57 2016 daemon.info hostapd: wlan1: STA a0:12:34:56:78:9a WPA:
> group key handshake completed (RSN)
> Sat Oct 29 10:53:00 2016 kern.warn kernel: [118229.632250] ath10k_pci
> 0000:01:00.0: failed to install key for vdev 0 peer 30:12:34:56:78:9a: -11
> Sat Oct 29 10:53:00 2016 kern.err kernel: [118229.641421] wlan0: failed to
> remove key (2, ff:ff:ff:ff:ff:ff) from hardware (-11)
> Sat Oct 29 10:53:03 2016 kern.warn kernel: [118232.642204] ath10k_pci
> 0000:01:00.0: failed to install key for vdev 0 peer 30:12:34:56:78:9a: -11
> Sat Oct 29 10:53:03 2016 kern.err kernel: [118232.651309] wlan0: failed to set
> key (2, ff:ff:ff:ff:ff:ff) to hardware (-11)
> Sat Oct 29 10:53:08 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a IEEE
> 802.11: deauthenticated due to local deauth request
> Sat Oct 29 10:53:11 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a IEEE
> 802.11: authenticated
> Sat Oct 29 10:53:11 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a IEEE
> 802.11: associated (aid 2)
> Sat Oct 29 10:53:11 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a
> RADIUS: starting accounting session 56977F3FD253EA9C
> Sat Oct 29 10:53:11 2016 daemon.info hostapd: wlan1: STA 70:12:34:56:78:9a WPA:
> pairwise key handshake completed (RSN)
> 