wifi: ath12k: start-up crash with WCN7850 hw2.0 on TI AM69-SK board
Parth Panchoil
parth105105 at gmail.com
Wed Jan 29 08:20:23 PST 2025
Hi,
I would like to share more details regarding this issue.
A similar bug was previously reported:
https://lore.kernel.org/all/CAFED-jma7ZB17Rcgx9H+eVZ=N6Pty4__gz3B+O2V34XtcEzOGg@mail.gmail.com/
Continuing my debugging on the mainline ath12k driver, I have been
investigating a startup hang/crash on my system.
So far, I have identified potential issues with memory mappings, where
the driver attempts to read an offset like GCC_GCC_PCIE_HOT_RST
(0x1e38338), which is significantly farther than PCIE_PCIE_PARF_LTSSM
(0x1e081b0).
I have tried several approaches, including:
- Adjusting the MMIO window dynamically based on the offset instead of
using a fixed window.
- Increasing the MMIO window size beyond 512 KiB.
However, these changes did not resolve the issue.
I am sharing my debug logs below (The patch enabling these exact logs
is pasted), which may be useful for someone working on this driver to
identify the root cause and help resolve this bug.
Any insights or suggestions would be greatly appreciated.
root at am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #4 SMP PREEMPT Wed
Jan 29 08:55:48 CET 2025 aarch64 GNU/Linux
root at am69-sk:~# insmod ./ath12k.ko debug_mask=0xffffffff
[ 50.148099] ath12k_pci 0002:01:00.0: BAR 0 [mem 0x4410200000-
0x44103fffff 64bit]: assigned
[ 50.156434] ath12k_pci 0002:01:00.0: enabling device (0000 -> 0002)
[ 50.162775] ath12k_pci 0002:01:00.0: boot pci_mem 0x000000005f7b1667
[ 50.169126] ath12k_pci 0002:01:00.0: pci probe 17cb:1107 17cb:1107
[ 50.175294] ### DBG: ath12k_pci_read32 1199 offset =1b00000
[ 50.180856] ### DBG: ath12k_pci_read32 1217 offset = 1b00000
[ 50.186504] ### DBG: ath12k_pci_select_window 177 window = 36
ab_pci->register_window = 0 WINDOW_REG_ADDRESS = 310c
[ 50.196920] ### DBG: ath12k_pci_select_window 178 (WINDOW_ENABLE_BIT
| window) = 40000036
[ 50.205084] ### DBG: ath12k_pci_select_window 179 ab->mem +
WINDOW_REG_ADDRESS = 8aa0310c
[ 50.213248] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window
= 36
[ 50.220023] ### DBG: ath12k_pci_read32 1221 window_start = 80000
offset = 1b00000 WINDOW_RANGE_MASK = 7ffff
[ 50.229831] ### DBG: ath12k_pci_read32 1229 window_start = 80000
offset = 1b00000 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000
[ 50.241200] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start +
(offset & WINDOW_RANGE_MASK) = 8aa80000
[ 50.251010] ### DBG: ath12k_pci_read32 1233
[ 50.255185] ### DBG: ath12k_pci_read32 1243 val = 40170200
[ 50.260658] ath12k_pci 0002:01:00.0: pci tcsr_soc_hw_version major 2
minor 0
[ 50.268230] ath12k_pci 0002:01:00.0: MSI vectors: 16
[ 50.273212] ath12k_pci 0002:01:00.0: msi base data is 0
[ 50.278430] ath12k_pci 0002:01:00.0: Hardware name: wcn7850 hw2.0
[ 50.290532] ath12k_pci 0002:01:00.0: failed to load firmware-2.bin:
-2
[ 50.297063] ath12k_pci 0002:01:00.0: using fw api 1
[ 50.301959] ath12k_pci 0002:01:00.0: Assign MSI to user: MHI,
num_vectors: 3, user_base_data: 0, base_vector: 0
[ 50.312038] ath12k_pci 0002:01:00.0: Number of assigned MSI for MHI
is 3, base vector is 0
[ 50.322527] ath12k_pci 0002:01:00.0: Assign MSI to user: CE,
num_vectors: 5, user_base_data: 3, base_vector: 3
[ 50.332673] ath12k_pci 0002:01:00.0: Assign MSI to user: DP,
num_vectors: 8, user_base_data: 8, base_vector: 8
[ 50.342697] ath12k_pci 0002:01:00.0: irq:604 group:0
[ 50.347696] ath12k_pci 0002:01:00.0: irq:605 group:1
[ 50.352694] ath12k_pci 0002:01:00.0: irq:606 group:2
[ 50.357698] ath12k_pci 0002:01:00.0: irq:607 group:3
[ 50.362696] ath12k_pci 0002:01:00.0: irq:608 group:4
[ 50.367693] ath12k_pci 0002:01:00.0: irq:609 group:5
[ 50.372689] ath12k_pci 0002:01:00.0: irq:610 group:6
[ 50.377716] ath12k_pci 0002:01:00.0: pci after request_irq
msi_ep_base_data 0
[ 50.384846] ath12k_pci 0002:01:00.0: unable to get wsi info from dt,
grouping single device
[ 50.393196] ath12k_pci 0002:01:00.0: wsi group-id 255 num-devices 1
index 0
[ 50.400148] ath12k_pci 0002:01:00.0: num devices 1 num probed 1
[ 50.423637] NET: Registered PF_QIPCRTR protocol family
[ 50.429598] ### DBG: ath12k_pci_enable_ltssm 290
[ 50.434225] ### DBG: ath12k_pci_read32 1199 offset =1e081b0
[ 50.439790] ### DBG: ath12k_pci_read32 1217 offset = 1e081b0
[ 50.445437] ### DBG: ath12k_pci_select_window 177 window = 3c
ab_pci->register_window = 0 WINDOW_REG_ADDRESS = 310c
[ 50.455852] ### DBG: ath12k_pci_select_window 178 (WINDOW_ENABLE_BIT
| window) = 4000003c
[ 50.464014] ### DBG: ath12k_pci_select_window 179 ab->mem +
WINDOW_REG_ADDRESS = 8aa0310c
[ 50.472178] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window
= 3c
[ 50.478952] ### DBG: ath12k_pci_read32 1221 window_start = 80000
offset = 1e081b0 WINDOW_RANGE_MASK = 7ffff
[ 50.488759] ### DBG: ath12k_pci_read32 1229 window_start = 80000
offset = 1e081b0 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000
[ 50.500126] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start +
(offset & WINDOW_RANGE_MASK) = 8aa881b0
[ 50.509936] ### DBG: ath12k_pci_read32 1233
[ 50.514109] ### DBG: ath12k_pci_read32 1243 val = 111
[ 50.519150] ### DBG: ath12k_pci_enable_ltssm 293 val=111
[ 50.524452] ath12k_pci 0002:01:00.0: pci ltssm 0x111
[ 50.529407] ### DBG: ath12k_pci_enable_ltssm 305
[ 50.534014] ### DBG: ath12k_pci_read32 1199 offset =1e38338
[ 50.539577] ### DBG: ath12k_pci_read32 1217 offset = 1e38338
[ 50.545227] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window
= 3c
[ 50.552001] ### DBG: ath12k_pci_read32 1221 window_start = 80000
offset = 1e38338 WINDOW_RANGE_MASK = 7ffff
[ 50.561808] ### DBG: ath12k_pci_read32 1229 window_start = 80000
offset = 1e38338 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000
[ 50.573175] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start +
(offset & WINDOW_RANGE_MASK) = 8aab8338
Thank you.
Regards,
Parth P
=======================================================================
diff --git a/drivers/net/wireless/ath/ath12k/pci.c
b/drivers/net/wireless/ath/ath12k/pci.c
index 06cff3849ab8..2984064783f9 100644
--- a/drivers/net/wireless/ath/ath12k/pci.c
+++ b/drivers/net/wireless/ath/ath12k/pci.c
@@ -174,6 +174,9 @@ static void ath12k_pci_select_window(struct
ath12k_pci *ab_pci, u32 offset)
window |= static_window;
if (window != ab_pci->register_window) {
+ printk("### DBG: %s %d window = %x ab_pci-
>register_window = %x WINDOW_REG_ADDRESS =
%x\n",__func__,__LINE__,window, ab_pci->register_window,
WINDOW_REG_ADDRESS);
+ printk("### DBG: %s %d (WINDOW_ENABLE_BIT | window) =
%x\n",__func__,__LINE__,(WINDOW_ENABLE_BIT | window) );
+ printk("### DBG: %s %d ab->mem + WINDOW_REG_ADDRESS =
%x\n",__func__,__LINE__,(ab->mem + WINDOW_REG_ADDRESS) );
iowrite32(WINDOW_ENABLE_BIT | window,
ab->mem + WINDOW_REG_ADDRESS);
ioread32(ab->mem + WINDOW_REG_ADDRESS);
@@ -279,8 +282,15 @@ static void ath12k_pci_enable_ltssm(struct
ath12k_base *ab)
u32 val;
int i;
+#if 0
+ /* Prevent startup crash on BPI-Rx */
+ return;
+#endif
+
+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM);
+ printk("### DBG: %s %d val=%x\n",__func__,__LINE__,val);
/* PCIE link seems very unstable after the Hot Reset*/
for (i = 0; val != PARM_LTSSM_VALUE && i < 5; i++) {
if (val == 0xffffffff)
@@ -292,11 +302,15 @@ static void ath12k_pci_enable_ltssm(struct
ath12k_base *ab)
ath12k_dbg(ab, ATH12K_DBG_PCI, "pci ltssm 0x%x\n", val);
+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST);
+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
val |= GCC_GCC_PCIE_HOT_RST_VAL;
+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
ath12k_pci_write32(ab, GCC_GCC_PCIE_HOT_RST, val);
val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST);
+ printk("### DBG: %s %d\n",__func__,__LINE__);
ath12k_dbg(ab, ATH12K_DBG_PCI, "pci pcie_hot_rst 0x%x\n",
val);
mdelay(5);
@@ -1182,6 +1196,7 @@ u32 ath12k_pci_read32(struct ath12k_base *ab, u32
offset)
u32 val, window_start;
int ret = 0;
+ printk("### DBG: %s %d offset
=%x\n",__func__,__LINE__,offset);
/* for offset beyond BAR + 4K - 32, may
* need to wakeup MHI to access.
*/
@@ -1196,26 +1211,36 @@ u32 ath12k_pci_read32(struct ath12k_base *ab,
u32 offset)
window_start = ath12k_pci_get_window_start(ab,
offset);
else
window_start = WINDOW_START;
+ //window_start = (offset &
~WINDOW_RANGE_MASK);
if (window_start == WINDOW_START) {
+ printk("### DBG: %s %d offset =
%x\n",__func__,__LINE__,offset);
spin_lock_bh(&ab_pci->window_lock);
ath12k_pci_select_window(ab_pci, offset);
+ printk("### DBG: %s %d ab_pci->register_window
= %x \n",__func__,__LINE__, ab_pci->register_window);
+ printk("### DBG: %s %d window_start = %x
offset = %x WINDOW_RANGE_MASK = %x \n",__func__,__LINE__,
window_start,offset, WINDOW_RANGE_MASK);
if
(ath12k_pci_is_offset_within_mhi_region(offset)) {
+ printk("### DBG: %s
%d\n",__func__,__LINE__);
offset = offset - PCI_MHIREGLEN_REG;
val = ioread32(ab->mem +
(offset &
WINDOW_RANGE_MASK));
} else {
+ printk("### DBG: %s %d window_start =
%x offset = %x WINDOW_RANGE_MASK = %x ab->mem =
%x\n",__func__,__LINE__,window_start,offset, WINDOW_RANGE_MASK,ab-
>mem);
+ printk("### DBG: %s %d ab->mem +
window_start + (offset & WINDOW_RANGE_MASK) =
%x\n",__func__,__LINE__,ab->mem + window_start + (offset &
WINDOW_RANGE_MASK));
val = ioread32(ab->mem + window_start
+
(offset &
WINDOW_RANGE_MASK));
+ printk("### DBG: %s
%d\n",__func__,__LINE__);
}
spin_unlock_bh(&ab_pci->window_lock);
} else {
+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ioread32(ab->mem + window_start +
(offset & WINDOW_RANGE_MASK));
}
}
+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
if (test_bit(ATH12K_PCI_FLAG_INIT_DONE, &ab_pci->flags) &&
offset >= ACCESS_ALWAYS_OFF && ab_pci->pci_ops->release &&
!ret)
=======================================================================
On Mon, 2025-01-27 at 15:01 +0100, Parth Panchoil wrote:
> Hi,
>
> I am currently debugging the ath12k_pci_enable_ltssm start up
> crash/bug
> with the mainline kernel on my system and would like to share my
> observations so far:
>
> The ath12k mainline driver gets stuck at this specific line:
> https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
> in the ath12k_pci_enable_ltssm which attempts to read
> GCC_GCC_PCIE_HOT_RST, particularly
> https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209
>
> Interestingly, within the same function, the line val =
> ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM) successfully reads the
> expected value 0x111 for PCIE_PCIE_PARF_LTSSM.
>
> I am continuing to debug from my end, although my understanding of
> the
> ath12k driver is limited. Any leads, suggestions, or hints to help
> resolve this issue would be greatly appreciated.
>
> Thank you.
>
> Regards,
> Parth P
>
>
> On Fri, 2025-01-24 at 10:02 +0000, Parth Pancholi wrote:
> > I appreciate your response, Baochen.
> >
> > I have been working on enabling mainline kernel support on my TI
> > AM69-
> > SK board to test the mainline ath12k driver on my system.
> >
> > Using the mainline kernel repository for the ath drivers [1], I
> > made
> > the following observation:
> > While the exact crash observed earlier is no longer present, the
> > system
> > hangs upon loading the ath12k mainline driver, displaying the
> > messages
> > below.
> >
> > root at am69-sk:~# modprobe ath12k debug_mask=0xffffffff
> > [ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000-
> > 0x44103fffff 64bit]: assigned
> > [ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 ->
> > 0002)
> > [ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16
> > [ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850
> > hw2.0
> > [ 1122.040183] NET: Registered PF_QIPCRTR protocol family
> >
> > root at am69-sk:~# uname -a
> > Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT
> > Wed
> > Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
> >
> > root at am69-sk:~# lspci
> > 0000:00:00.0 PCI bridge: Texas Instruments Device b012
> > 0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x
> > Wi-
> > Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
> > 0001:00:00.0 PCI bridge: Texas Instruments Device b012
> > 0002:00:00.0 PCI bridge: Texas Instruments Device b012
> >
> > Do you have any insights into what might still be missing or
> > incorrect
> > in my setup?
> >
> > Regards,
> > Parth P
> >
> > On Wed, 2025-01-22 at 15:20 +0800, Baochen Qiang wrote:
> > >
> > >
> > > On 1/21/2025 10:19 PM, Parth Panchoil wrote:
> > > > Hi All,
> > > >
> > > > I am performing tests on the SX-PCEBE Wi-Fi module, which
> > > > utilizes
> > > > the
> > > > ATH12k driver, on the Texas Instruments AM69-SK board.
> > > > The board is running the TI Linux Kernel from the ti-linux-
> > > > 6.6.y
> > >
> > > 6.6 is too old, and besides we don;t support customer kernel.
> > >
> > > Could you try latest ath tree [1] or the mainline tree [2]?
> > >
> > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
> > > [2]
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
> > >
> > > If the issue is still seen, please enable verbose ath12k log
> > > using
> > > below command and help
> > > collect dmesg logs:
> > >
> > > sudo modprobe ath12k debug_mask=0xffffffff
> > >
> > > One more thing, the open-WRT patch is overkill, can you narrow
> > > down
> > > to find which line of
> > > code in ath12k_pci_enable_ltssm() is causing this issue?
> > >
> > >
> > > > branch. During testing, I observed a kernel crash from the
> > > > ATH12k
> > > > driver as soon as the probe is called. The crash log is as
> > > > follows:
> > > >
> > > > [ 9.492631] Kernel panic - not syncing: Asynchronous SError
> > > > Interrupt
> > > > [ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted
> > > > 6.6.58-
> > > > 01497-ga7758da17c28-dirty #1
> > > > [ 9.492638] Hardware name: Texas Instruments AM69 SK (DT)
> > > > [ 9.492640] Call trace:
> > > > [ 9.492642] dump_backtrace+0x94/0xec
> > > > [ 9.492658] show_stack+0x18/0x24
> > > > [ 9.492662] dump_stack_lvl+0x48/0x60
> > > > [ 9.492669] dump_stack+0x18/0x24
> > > > [ 9.492672] panic+0x320/0x378
> > > > [ 9.492677] nmi_panic+0x8c/0x90
> > > > [ 9.492681] arm64_serror_panic+0x6c/0x78
> > > > [ 9.492686] do_serror+0x3c/0x78
> > > > [ 9.492692] el1h_64_error_handler+0x34/0x4c
> > > > [ 9.492697] el1h_64_error+0x64/0x68
> > > > [ 9.492700] ath12k_pci_read32+0x1bc/0x1e8 [ath12k]
> > > > [ 9.492725] ath12k_pci_power_up+0xdc/0x340 [ath12k]
> > > > [ 9.492747] ath12k_core_init+0x2c/0xa8 [ath12k]
> > > > [ 9.492769] ath12k_pci_probe+0x698/0x908 [ath12k]
> > > > [ 9.492791] pci_device_probe+0xa8/0x16c
> > > > [ 9.492800] really_probe+0x110/0x27c
> > > > [ 9.492805] __driver_probe_device+0x78/0x12c
> > > > [ 9.492808] driver_probe_device+0x3c/0x118
> > > > [ 9.492810] __driver_attach+0x74/0x124
> > > > [ 9.492813] bus_for_each_dev+0x78/0xd8
> > > > [ 9.492819] driver_attach+0x24/0x30
> > > > [ 9.492824] bus_add_driver+0xe4/0x208
> > > > [ 9.492828] driver_register+0x60/0x128
> > > > [ 9.492831] __pci_register_driver+0x44/0x50
> > > > [ 9.492835] ath12k_pci_init+0x2c/0x6c [ath12k]
> > > > [ 9.492858] do_one_initcall+0x70/0x1b4
> > > > [ 9.492861] do_init_module+0x58/0x1e4
> > > > [ 9.492867] load_module+0x19bc/0x1a8c
> > > > [ 9.492869] init_module_from_file+0x88/0xc4
> > > > [ 9.492873] __arm64_sys_finit_module+0x1c0/0x2ac
> > > > [ 9.492877] invoke_syscall+0x44/0x108
> > > > [ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0
> > > > [ 9.492885] do_el0_svc+0x1c/0x28
> > > > [ 9.492889] el0_svc+0x2c/0x84
> > > > [ 9.492892] el0t_64_sync_handler+0xc0/0xc4
> > > > [ 9.492895] el0t_64_sync+0x190/0x194
> > > > [ 9.492899] SMP: stopping secondary CPUs
> > > > [ 9.492908] Kernel Offset: disabled
> > > > [ 9.492909] CPU features: 0x0,80000200,28020000,1000420b
> > > > [ 9.492913] Memory Limit: none
> > > >
> > > > Upon searching online, I found the OpenWRT patch that appears
> > > > to
> > > > address a similar issue: OpenWRT Patch: Prevent LTSSM Startup
> > > > Crash.
> > > > https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
> > > > With the above patch applied, I do not see the crash anymore.
> > > >
> > > > Could anyone confirm if this issue has been reported
> > > > before/known
> > > > bug
> > > > or provide any insights?
> > > > Any additional information or suggestions would be greatly
> > > > appreciated.
> > > >
> > > > Details about the test setup,
> > > > TI-AM69-SK board:
> > > > https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
> > > > Silex WiFi card SX-PCEBE:
> > > > https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
> > > > TI Linux Repo:
> > > > https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y
> > > >
> > > > Thank you.
> > > >
> > > > Regards,
> > > > Parth P
> > > >
> > >
> > >
> >
> >
>
More information about the ath12k
mailing list