ath11k: crashes with 1 MSI vector, workaround disable MHI M2 state

[Kalle Valo]

Kalle Valo <kvalo at codeaurora.org> writes:

> To keep the discussion organised I'll start a new thread about weird
> kernel crashes we are seeing on ath11k, and include MHI folks as well in
> case they have any ideas. This is a long story, but I try to summarise
> this as short as I can :)
>
> Recently Dell released laptops with QCA6390. Unfortunately there's a
> BIOS bug[1] and ath11k only receives 1 MSI vector, opposed to 32 vectors
> it needs. Carl implemented a proof of concept patch[2] which worked fine
> on some platforms, for example I didn't see any issues on my Intel NUC
> with QCA6390.
>
> But once we people with Dell XPS 13 9310 started testing Carl's patches
> started reporting weird kernel crashes. This is what wink reported[3]:
>
> ----------------------------------------------------------------------
> So up until this point, everything is working without issues.
> Everything seems to spiral out of control a couple of seconds later
> when my system attempts to actually bring up the adapter.  In most of
> the crash states I will see this:
>
> [   31.286725] wlp85s0: send auth to ec:08:6b:27:01:ea (try 1/3)
> [   31.390187] wlp85s0: send auth to ec:08:6b:27:01:ea (try 2/3)
> [   31.391928] wlp85s0: authenticated
> [   31.394196] wlp85s0: associate with ec:08:6b:27:01:ea (try 1/3)
> [   31.396513] wlp85s0: RX AssocResp from ec:08:6b:27:01:ea
> (capab=0x411 status=0 aid=6)
> [   31.407730] wlp85s0: associated
> [   31.434354] IPv6: ADDRCONF(NETDEV_CHANGE): wlp85s0: link becomes ready
>
> And then either somewhere in that pile of messages, or a second or two
> after this my machine will start to stutter as I mentioned before, and
> then it either hangs, or I see this message (I'm truncating the
> timestamp):
>
> [   35.xxxx ] sched: RT throttling activated
>
> After that moment, the machine is unresponsive.  Sorry I can't seem to
> extract this data other than screenshots from my phone at the moment,
> you can see the dmesg output from 6 different hangs here:
>
> https://github.com/w1nk/ath11k-debug
> ----------------------------------------------------------------------
>
> Wink even made videos available[3].
>
> After extensive debugging from wink he found out that disabling M2 state
> makes the all problems go away:
>
> --- a/drivers/bus/mhi/core/pm.c
> +++ b/drivers/bus/mhi/core/pm.c
> @@ -55,12 +55,12 @@ static struct mhi_pm_transitions const dev_state_transitions[] = {
>         },
>         {
>                 MHI_PM_M0,
> -               MHI_PM_M0 | MHI_PM_M2 | MHI_PM_M3_ENTER |
> +               MHI_PM_M0 | MHI_PM_M3_ENTER |
>                 MHI_PM_SYS_ERR_DETECT | MHI_PM_SHUTDOWN_PROCESS |
>                 MHI_PM_LD_ERR_FATAL_DETECT | MHI_PM_FW_DL_ERR
>         },
>         {
> -               MHI_PM_M2,
> +               MHI_PM_M0,
>                 MHI_PM_M0 | MHI_PM_SYS_ERR_DETECT | MHI_PM_SHUTDOWN_PROCESS |
>                 MHI_PM_LD_ERR_FATAL_DETECT
>         },
>
> And indeed now we have numerous people reporting that with this
> workaround ath11k is stable on their Dell XPS 13 9310 laptops. What on
> earth could cause these kernel crashes/interrupt storms? And why is it
> visible only on Dell laptops? Why does disabling M2 state fix it?
>
> Also something to investigate is does AC power vs battery power have
> something to do with this? Can that affect M2 states somehow?
>
> Any other ideas how to debug this? This is a very weird problem.

I was told that some registers are not allowed to be accessed during M2
state, so it looks like wink was spot on with his workaround. And ASPM
is also related, which might explain why not everyone see these
problems.

This is all still very sketchy and I'm trying to get more information.

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches