[LEDE-DEV] ath10k firmware crashes in mesh mode on QCA9880

[Benjamin Morgan]

When we try to transmit traffic (ping) between two meshed ath10k devices 
running latest lede we keep experiencing ath10k firmware crashes. This 
seems to only happen when running in 802.11n/ac mode but not in 
802.11a/g mode. Also, from the station dumps it appears that management 
traffic is flowing between the devices, however when we try to send 
unicast data traffic the firmware crashes immediately.

Platform: Archer C7 AC1750 v2
Software Image: LEDE (HEAD, r2299) Commit: 
https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58

Crypto: wpa_supplicant
wpa_supplicant-wlan0.conf
network={
ssid="bmorgan_lede_mesh"
key_mgmt=SAE
mode=5
frequency=5180
psk="meshpassword"
}

Backports Verstion:
[    9.818007] Loading modules backported from Linux version 
wt-2016-10-03-1-g6fcb1a6
[    9.825736] Backport generated by backports.git 
backports-20160324-9-g0e38f5c

​​Ath10k Initialization on Station A (dmesg)
[    9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
[    9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode 1 
irq_mode 0 reset_mode 0
[   10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for 
ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
[   10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
[   10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin: 
firmware_loading_store: map pages failed
[   10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043202ff sub 0000:0000
[   10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 
tracing 0 dfs 1 testmode 1
[   10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5 
features no-p2p,raw-mode,mfp crc32 9d340dd9
[   10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for 
ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
[   10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
[   10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin: 
firmware_loading_store: map pages failed
[   11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A 
crc32 bebc7c08
[   12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 
cal file max-sta 128 raw 0 hwcrypto 1

Station A (wlan0):
18:A6:F7:23:6E:66
10.230.5.41

Station B (wlan0):
18:a6:f7:26:0f:21
10.230.5.42

Station Dump on Station A before ping:
Station 18:a6:f7:26:0f:21 (on wlan0)
inactive time:340 ms
rx bytes:2472
rx packets:28
tx bytes:1204
tx packets:9
tx retries:0
tx failed:0
rx drop misc:1
signal: -14 dBm
signal avg:-14 dBm
Toffset:18142530 us
tx bitrate:6.0 MBit/s
rx bitrate:6.0 MBit/s
rx duration:1524 us
mesh llid:0
mesh plid:0
mesh plink:ESTAB
mesh local PS mode:ACTIVE
mesh peer PS mode:UNKNOWN
mesh non-peer PS mode:ACTIVE
authorized:yes
authenticated:yes
associated:yes
preamble:long
WMM/WME:yes
MFP:yes
TDLS peer:no
DTIM period:2
beacon interval:1000
connected time:10 seconds

​Crash Log on Station B (10.230.5.42)
[245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid 
2bab5ee9-08ff-4a17-95b1-636d212acebc)
[245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043202ff sub 0000:0000
[245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 
0 dfs 1 testmode 1
[245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5 
features no-p2p,raw-mode,mfp crc32 9d340dd9
[245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A crc32 
bebc7c08
[245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 cal 
file max-sta 128 raw 0 hwcrypto 1
[245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
[245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3 
0x009A4577 0x00955B31
[245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130 
0x00000002 0x00439E98
[245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074 
0x00407120 0x004436CC
[245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000 
0x009A3518 0x009A3526
[245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D 
0x00000000 0x00000000
[245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4 
0x0040AC60 0x0040AC09
[245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24 
0x00400000 0xC09A4577
[245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84 
0x0044110C 0x00442074
[245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4 
0x0044110C 0x00407120
[245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4 
0x0040AC14 0x00001580
[245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04 
0x009C643C 0x004436CC
[245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64 
0x004208FC 0x00439E4C
[245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84 
0x004208FC 0x00425404
[245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4 
0x00424FE8 0x00000002
[245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14 
0x00000004 0x004039D0
[245.767023] ieee80211 phy0: Hardware restart was requested
[245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor vdev 
1 stop: -143
[245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
​
With wpa_supplicant turned off:

Station Dump on Station A before ping:
Station 18:a6:f7:26:0f:21 (on wlan0)
inactive time:60 ms
rx bytes:3874
rx packets:67
tx bytes:707
tx packets:7
tx retries:0
tx failed:1
rx drop misc:3
signal: -14 dBm
signal avg:-14 dBm
Toffset:18446744073684250999 us
tx bitrate:6.0 MBit/s
rx bitrate:6.0 MBit/s
rx duration:1312 us
mesh llid:24953
mesh plid:59412
mesh plink:ESTAB
mesh local PS mode:ACTIVE
mesh peer PS mode:UNKNOWN
mesh non-peer PS mode:ACTIVE
authorized:yes
authenticated:yes
associated:yes
preamble:long
WMM/WME:yes
MFP:no
TDLS peer:no
DTIM period:2
beacon interval:1000
short slot time:yes
connected time:33 seconds

Firmware crash on Station A (10.230.5.41)
[ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid 
42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
[ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043202ff sub 0000:0000
[ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 
tracing 0 dfs 1 testmode 1
[ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5 
features no-p2p,raw-mode,mfp crc32 9d340dd9
[ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A 
crc32 bebc7c08
[ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 
cal file max-sta 128 raw 0 hwcrypto 1
[ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
[ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3 
0x009A4577 0x00955B31
[ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130 
0x00000001 0x00435354
[ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074 
0x00407120 0x004436CC
[ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000 
0x009A43A0 0x009A43DE
[ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D 
0x00000000 0x00000000
[ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4 
0x0040AC60 0x0040AC09
[ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24 
0x00955A00 0xC09A4577
[ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84 
0x0044110C 0x00442074
[ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4 
0x0044110C 0x00407120
[ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4 
0x0040AC10 0x00001580
[ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04 
0x009C643C 0x004436CC
[ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64 
0x004208FC 0x00439E4C
[ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84 
0x004208FC 0x00425758
[ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4 
0x00424FE8 0x00000002
[ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14 
0x00000004 0x004039D0
[ 1041.267025] ieee80211 phy0: Hardware restart was requested
[ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize monitor 
vdev 1 stop: -143
[ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143

​
Thanks in advance for your time and help,

~Benjamin