ath11k-qca6390-bringup-202011191920: new suspend implementation

Sun Nov 22 10:07:13 EST 2020

On Sun, Nov 22, 2020 at 2:15 PM Mitchell Nordine
<mitchell.nordine at gmail.com> wrote:
>
> > Unfortunately there's no solution still for the weird
> crashes some people are seeing.
>
> Can confirm, the spurious system freezing still continues. This time
> while typing my password into the gdm UI for login.
>
> On Sun, Nov 22, 2020 at 12:44 AM Mitchell Nordine
> <mitchell.nordine at gmail.com> wrote:
> >
> > Thanks for the update!
> >
> > I no longer notice any errors related to ath11k during boot of NixOS
> > on my XPS 13 9310 with these patches:
> >
> > [mindtree at mindtree:~]$ dmesg | grep -e ath11
> > [    4.084314] ath11k_pci 0000:56:00.0: WARNING: ath11k PCI support is
> > experimental!
> > [    4.084358] ath11k_pci 0000:56:00.0: BAR 0: assigned [mem
> > 0x8c300000-0x8c3fffff 64bit]
> > [    4.084377] ath11k_pci 0000:56:00.0: enabling device (0000 -> 0002)
> > [    4.084442] ath11k_pci 0000:56:00.0: MSI vectors: 1
> > [    4.320847] ath11k_pci 0000:56:00.0: qmi req mem_seg[0] 0x59c00000 3522560 1
> > [    4.320849] ath11k_pci 0000:56:00.0: qmi req mem_seg[1] 0x5a200000 884736 4
> > [    4.330816] ath11k_pci 0000:56:00.0: chip_id 0x0 chip_family 0xb
> > board_id 0xff soc_id 0xffffffff
> > [    4.330818] ath11k_pci 0000:56:00.0: fw_version 0x101c06cc
> > fw_build_timestamp 2020-06-24 19:50 fw_build_id
> > [    4.521522] ath11k_pci 0000:56:00.0 wlp86s0: renamed from wlan0
> >
> > Everything appears to run smoothly for the first 5-10 minutes, then
> > the firmware appears to crash and the internet drops out:
> >
> > [  293.677300] ath11k_pci 0000:56:00.0: firmware crashed:
> > MHI_CB_SYS_ERROR
> > [  385.774509] mhi 0000:56:00.0: Device failed to exit MHI Reset state
> >
> > I haven't yet been able to identify an action that consistently causes
> > the crash.
> >
> > Following the crash, the gnome shell appears to still believe that the
> > connection is up, however upon clicking on the wifi in the top-right
> > drop-down menu and clicking the "Turn Off" option, the shell freezes
> > for a few seconds and a few more errors show up in dmesg:
> >
> > [  634.018718] wlp86s0: deauthenticating from 7a:8a:20:d5:98:d7 by
> > local choice (Reason: 3=DEAUTH_LEAVING)
> > [  639.151611] ath11k_pci 0000:56:00.0: failed to flush transmit queue
> > 0
> > [  642.159384] ath11k_pci 0000:56:00.0: wmi command 24595 timeout
> > [  642.159388] ath11k_pci 0000:56:00.0: failed to send
> > WMI_PEER_REORDER_QUEUE_SETUP
> > [  642.159394] ath11k_pci 0000:56:00.0: failed to send wmi to delete rx tid -11
> > [  642.159400] wlp86s0: HW problem - can not stop rx aggregation for
> > 7a:8a:20:d5:98:d7 tid 0
> > [  645.168070] ath11k_pci 0000:56:00.0: wmi command 24595 timeout
> > [  645.168072] ath11k_pci 0000:56:00.0: failed to send
> > WMI_PEER_REORDER_QUEUE_SETUP
> > [  645.168074] ath11k_pci 0000:56:00.0: failed to send wmi to delete rx tid -11
> > [  645.168077] wlp86s0: HW problem - can not stop rx aggregation for
> > 7a:8a:20:d5:98:d7 tid 1
> > [  648.174960] ath11k_pci 0000:56:00.0: wmi command 24595 timeout
> > [  648.174965] ath11k_pci 0000:56:00.0: failed to send
> > WMI_PEER_REORDER_QUEUE_SETUP
> > [  648.174971] ath11k_pci 0000:56:00.0: failed to send wmi to delete rx tid -11
> > [  648.174976] wlp86s0: HW problem - can not stop rx aggregation for
> > 7a:8a:20:d5:98:d7 tid 6
> > [  651.183596] ath11k_pci 0000:56:00.0: wmi command 20489 timeout
> > [  651.183601] ath11k_pci 0000:56:00.0: failed to send WMI_VDEV_INSTALL_KEY cmd
> > [  651.183606] ath11k_pci 0000:56:00.0: ath11k_install_key failed (-11)
> > [  651.183610] wlp86s0: failed to remove key (0, 7a:8a:20:d5:98:d7)
> > from hardware (-11)
> > [  654.190511] ath11k_pci 0000:56:00.0: wmi command 24578 timeout
> > [  654.190516] ath11k_pci 0000:56:00.0: failed to send WMI_PEER_DELETE cmd
> > [  654.190523] ath11k_pci 0000:56:00.0: failed to delete peer vdev_id
> > 0 addr 7a:8a:20:d5:98:d7 ret -11
> > [  654.190526] ath11k_pci 0000:56:00.0: Failed to delete peer:
> > 7a:8a:20:d5:98:d7 for VDEV: 0
> > [  654.190528] ath11k_pci 0000:56:00.0: Found peer entry
> > 9c:b6:d0:3e:43:4a n vdev 0 after it was supposedly removed
> > [  654.190574] ------------[ cut here ]------------
> > [  654.190594] WARNING: CPU: 5 PID: 1208 at
> > net/mac80211/sta_info.c:1098 __sta_info_destroy_part2+0x11c/0x140
> > [mac80211]
> > [  654.190595] Modules linked in: ath9k_htc ath9k_common ath9k_hw ath
> > fuse ctr ccm michael_mic af_packet cdc_ether usbnet r8152 mii
> > typec_displayport uvcvideo
> > videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common
> > videodev mc hid_sensor_als hid_sensor_trigger
> > industrialio_triggered_buffer kfifo_buf hid_se
> > nsor_iio_common industrialio hid_sensor_hub intel_ishtp_loader joydev
> > mousedev intel_ishtp_hid wacom usbhid hid_multitouch hid_generic
> > qrtr_mhi iTCO_wdt intel_
> > pmc_bxt 8250_dw watchdog mei_hdcp i2c_designware_platform
> > i2c_designware_core intel_rapl_msr snd_sof_pci snd_sof_intel_byt
> > snd_sof_intel_ipc qrtr dell_wmi wmi_
> > bmof ns snd_sof_intel_hda_common dell_laptop ath11k_pci
> > snd_soc_hdac_hda dell_smbios snd_sof_xtensa_dsp snd_hda_codec_hdmi mhi
> > snd_sof_intel_hda dell_wmi_descr
> > iptor dcdbas snd_sof ath11k x86_pkg_temp_thermal intel_powerclamp
> > dell_smm_hwmon qmi_helpers snd_hda_ext_core coretemp crc32_pclmul
> > ghash_clmulni_intel snd_soc
> > _acpi_intel_match aesni_intel snd_soc_acpi
> > [  654.190666]  snd_hda_codec_realtek libaes mac80211 crypto_simd
> > cryptd glue_helper snd_hda_codec_generic ledtrig_audio intel_cstate
> > snd_soc_core intel_uncore
> >  snd_compress sha256_ssse3 ac97_bus snd_pcm_dmaengine sha256_generic
> > input_leds led_class deflate snd_hda_intel intel_spi_pci efi_pstore
> > snd_intel_dspcfg cfg80
> > 211 intel_spi serio_raw pstore spi_nor snd_hda_codec mtd nls_iso8859_1
> > nls_cp437 snd_hda_core vfat i2c_i801 snd_hwdep i2c_smbus rfkill
> > tpm_crb fat libarc4 sch_
> > fq_codel intel_ish_ipc mei_me intel_lpss_pci tpm_tis intel_ishtp
> > intel_lpss tpm_tis_core mei ucsi_acpi idma64 processor_thermal_device
> > virt_dma tpm typec_ucsi
> > intel_rapl_common 8250_pci intel_soc_dts_iosf typec snd_pcm_oss
> > rng_core snd_mixer_oss tiny_power_button snd_pcm wmi battery button
> > snd_timer snd i2c_hid sound
> > core hid msr int3403_thermal evdev int340x_thermal_zone mac_hid
> > int3400_thermal acpi_thermal_rel intel_hid sparse_keymap
> > pinctrl_tigerlake intel_pmc_core acpi_
> > tad ac acpi_pad loop cpufreq_powersave tun tap
> > [  654.190754]  macvlan bridge stp llc kvm_intel kvm irqbypass
> > efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache
> > jbd2 xhci_pci xhci_pci_ren
> > esas rtsx_pci_sdmmc xhci_hcd mmc_core atkbd libps2 usbcore thunderbolt
> > nvme nvme_core rtsx_pci crc32c_intel t10_pi crc_t10dif
> > crct10dif_generic crct10dif_pclmu
> > l usb_common crct10dif_common i8042 rtc_cmos serio dm_mod i915 video
> > intel_gtt i2c_algo_bit cec drm_kms_helper syscopyarea sysfillrect
> > sysimgblt fb_sys_fops dr
> > m i2c_core backlight agpgart
> > [  654.190811] CPU: 5 PID: 1208 Comm: NetworkManager Tainted: G
> > W I       5.10.0-rc4 #1-NixOS
> > [  654.190813] Hardware name: Dell Inc. XPS 13 9310/0F7M4C, BIOS 1.1.1
> > 10/05/2020
> > [  654.190825] RIP: 0010:__sta_info_destroy_part2+0x11c/0x140 [mac80211]
> > [  654.190829] Code: ff 0f 0b 80 bd 14 01 00 00 00 74 82 45 31 c0 b9
> > 01 00 00 00 48 89 ea 48 89 de 4c 89 e7 e8 ac ad ff ff 85 c0 0f 84 64
> > ff ff ff <0f> 0b e9 5
> > d ff ff ff be 03 00 00 00 48 89 ef e8 10 ea ff ff 85 c0
> > [  654.190831] RSP: 0018:ffffac81c0897b80 EFLAGS: 00010286
> > [  654.190834] RAX: 00000000fffffff5 RBX: ffff9d54d5800900 RCX:
> > 0000000000000000
> > [  654.190836] RDX: ffff9d54c3d0bf00 RSI: 000000000020001a RDI:
> > ffff9d54d629b5d8
> > [  654.190837] RBP: ffff9d54c778f000 R08: 0000000000000000 R09:
> > ffffffffc1245800
> > [  654.190838] R10: ffff9d54cde07800 R11: 0000000000000001 R12:
> > ffff9d54d6298800
> > [  654.190840] R13: ffff9d54d5800900 R14: 0000000000000001 R15:
> > ffff9d54d6298de0
> > [  654.190842] FS:  00007f1bf8509040(0000) GS:ffff9d5c2f740000(0000)
> > knlGS:0000000000000000
> > [  654.190844] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  654.190845] CR2: 00007f6d3cb34000 CR3: 0000000118d9a006 CR4:
> > 0000000000770ee0
> > [  654.190847] PKRU: 55555554
> > [  654.190848] Call Trace:
> > [  654.190866]  __sta_info_flush+0x123/0x180 [mac80211]
> > [  654.190885]  ieee80211_set_disassoc+0xba/0x5d0 [mac80211]
> > [  654.190902]  ieee80211_mgd_deauth.cold+0x49/0x1bf [mac80211]
> > [  654.190923]  cfg80211_mlme_deauth+0xb1/0x1b0 [cfg80211]
> > [  654.190939]  cfg80211_mlme_down+0x66/0x90 [cfg80211]
> > [  654.190955]  cfg80211_disconnect+0x128/0x1b0 [cfg80211]
> > [  654.190967]  cfg80211_leave+0x27/0x40 [cfg80211]
> > [  654.190977]  cfg80211_netdev_notifier_call+0xec/0x440 [cfg80211]
> > [  654.190984]  raw_notifier_call_chain+0x44/0x60
> > [  654.190991]  __dev_close_many+0x5f/0x110
> > [  654.190995]  dev_close_many+0x81/0x130
> > [  654.190999]  dev_close.part.0+0x3e/0x70
> > [  654.191008]  cfg80211_shutdown_all_interfaces+0x71/0xd0 [cfg80211]
> > [  654.191017]  cfg80211_rfkill_set_block+0x22/0x30 [cfg80211]
> > [  654.191022]  rfkill_set_block+0x92/0x140 [rfkill]
> > [  654.191026]  rfkill_fop_write+0x11f/0x1c0 [rfkill]
> > [  654.191032]  vfs_write+0xc7/0x280
> > [  654.191035]  ksys_write+0xa7/0xe0
> > [  654.191041]  do_syscall_64+0x33/0x40
> > [  654.191045]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [  654.191048] RIP: 0033:0x7f1bf93906f7
> > [  654.191052] Code: 1f 40 00 41 54 49 89 d4 55 48 89 f5 53 89 fb 48
> > 83 ec 10 e8 fb fc ff ff 4c 89 e2 48 89 ee 89 df 41 89 c0 b8 01 00 00
> > 00 0f 05 <48> 3d 00 f
> > 0 ff ff 77 35 44 89 c7 48 89 44 24 08 e8 54 fd ff ff 48
> > [  654.191053] RSP: 002b:00007ffc79f67e10 EFLAGS: 00000293 ORIG_RAX:
> > 0000000000000001
> > [  654.191056] RAX: ffffffffffffffda RBX: 000000000000001d RCX: 00007f1bf93906f7
> > [  654.191057] RDX: 0000000000000008 RSI: 00007ffc79f67e48 RDI: 000000000000001d
> > [  654.191059] RBP: 00007ffc79f67e48 R08: 0000000000000000 R09: 0000000000000001
> > [  654.191060] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000008
> > [  654.191061] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000001b10c20
> > [  654.191075] ---[ end trace 4fd47da3698c4a9f ]---
> > [  657.198288] ath11k_pci 0000:56:00.0: wmi command 20488 timeout
> > [  657.198293] ath11k_pci 0000:56:00.0: failed to send WMI_VDEV_SET_PARAM_CMDID
> > [  657.198299] ath11k_pci 0000:56:00.0: Failed to set CTS prot for VDEV: 0
> > [  660.205991] ath11k_pci 0000:56:00.0: wmi command 20488 timeout
> > [  660.205995] ath11k_pci 0000:56:00.0: failed to send WMI_VDEV_SET_PARAM_CMDID
> > [  660.206000] ath11k_pci 0000:56:00.0: Failed to set erp slot for VDEV: 0
> > [  663.213835] ath11k_pci 0000:56:00.0: wmi command 20488 timeout
> > [  663.213840] ath11k_pci 0000:56:00.0: failed to send WMI_VDEV_SET_PARAM_CMDID
> > [  663.213846] ath11k_pci 0000:56:00.0: Failed to set preamble for VDEV: 0
> > [  666.221628] ath11k_pci 0000:56:00.0: wmi command 20487 timeout
> > [  666.221633] ath11k_pci 0000:56:00.0: failed to submit WMI_VDEV_DOWN cmd
> > [  666.221639] ath11k_pci 0000:56:00.0: failed to down vdev 0: -11
> > [  669.229407] ath11k_pci 0000:56:00.0: wmi command 20493 timeout
> > [  669.229412] ath11k_pci 0000:56:00.0: failed to send
> > WMI_VDEV_SET_WMM_PARAMS_CMDID
> > [  669.229417] ath11k_pci 0000:56:00.0: failed to set wmm params: -11
> > [  672.237193] ath11k_pci 0000:56:00.0: wmi command 20493 timeout
> > [  672.237198] ath11k_pci 0000:56:00.0: failed to send
> > WMI_VDEV_SET_WMM_PARAMS_CMDID
> > [  672.237203] ath11k_pci 0000:56:00.0: failed to set wmm params: -11
> > [  675.244963] ath11k_pci 0000:56:00.0: wmi command 20493 timeout
> > [  675.244968] ath11k_pci 0000:56:00.0: failed to send
> > WMI_VDEV_SET_WMM_PARAMS_CMDID
> > [  675.244971] ath11k_pci 0000:56:00.0: failed to set wmm params: -11
> > [  678.252682] ath11k_pci 0000:56:00.0: wmi command 20493 timeout
> > [  678.252689] ath11k_pci 0000:56:00.0: failed to send
> > WMI_VDEV_SET_WMM_PARAMS_CMDID
> > [  678.252695] ath11k_pci 0000:56:00.0: failed to set wmm params: -11
> > [  681.260582] ath11k_pci 0000:56:00.0: wmi command 20486 timeout
> > [  681.260587] ath11k_pci 0000:56:00.0: failed to submit WMI_VDEV_STOP
> > cmd
> > [  681.260594] ath11k_pci 0000:56:00.0: failed to stop WMI vdev 0: -11
> > [  681.260596] ath11k_pci 0000:56:00.0: failed to stop vdev 0: -11
> > [  686.764099] ath11k_pci 0000:56:00.0: failed to flush transmit queue
> > 0
> > [  689.771891] ath11k_pci 0000:56:00.0: wmi command 20482 timeout
> > [  689.771897] ath11k_pci 0000:56:00.0: failed to submit
> > WMI_VDEV_DELETE_CMDID
> > [  689.771904] ath11k_pci 0000:56:00.0: failed to delete WMI vdev 0:
> > -11
> > [  719.529733] ath11k_pci 0000:56:00.0: wmi command 16387 timeout
> > [  719.529740] ath11k_pci 0000:56:00.0: failed to send
> > WMI_PDEV_SET_PARAM cmd
> > [  719.529748] ath11k_pci 0000:56:00.0: failed to enable PMF QOS: (-11
> > [  722.793499] ath11k_pci 0000:56:00.0: wmi command 16387 timeout
> > [  722.793517] ath11k_pci 0000:56:00.0: failed to send
> > WMI_PDEV_SET_PARAM cmd
> > [  722.793524] ath11k_pci 0000:56:00.0: failed to enable PMF QOS: (-11
> >
> > Apologies for the long output, hopefully something here is useful.
> >
> > I haven't had my whole system freeze yet like I did prior to these
> > patches, however I've only been running these patches for a few hours
> > so far, currently on my third boot.
> >
> > You can find the nix configuration I'm working on for the xps 9310
> > that includes the new patches here:
> >
> > https://github.com/NixOS/nixos-hardware/pull/207
> >
> > On Thu, Nov 19, 2020 at 8:52 PM Kalle Valo <kvalo at codeaurora.org> wrote:
> > >
> > > Kalle Valo <kvalo at codeaurora.org> writes:
> > >
> > > > (Bcc: people reporting qca6390 problems)
> > > >
> > > > Hi,
> > > >
> > > > I collected all important QCA6390 fixes to ath11k-qca6390 branch so that
> > > > there's a good baseline for all testing:
> > > >
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/log/?h=ath11k-qca6390-bringup
> > > >
> > > > At the moment it's based on v5.10-rc4 and I will try to update it to a
> > > > recent -rc release every few weeks or so. Everytime I update the branch
> > > > I create a new tag and the latest tag is now:
> > > >
> > > > ath11k-qca6390-bringup-202011191920
> > > >
> > > > In this tag there's now a brand new implementation for suspend, which
> > > > relies that the platform provides power to QCA6390 during suspend. Not
> > > > all platforms do, but most of them should do that. ath11k also prints a
> > > > warning whenever it notices that the firmware has crashed, but I'm not
> > > > sure yet if it (the MHI subsystem to be exact) can detect every case.
> > > >
> > > > The MSI patch is mostly the same, it had just some refactoring since the
> > > > last version. Unfortunately there's no solution still for the weird
> > > > crashes some people are seeing.
> > >
> > > Forgot to mention when debugging ath11k PCI issues it's a good idea to
> > > enable MHI debug messages. To do that enable CONFIG_MHI_BUS_DEBUG and
> > > CONFIG_DYNAMIC_DEBUG and run:
> > >
> > > sudo sh -c "echo -n 'module mhi +p' > /sys/kernel/debug/dynamic_debug/control"
> > >
> > > --
> > > https://patchwork.kernel.org/project/linux-wireless/list/
> > >
> > > https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
>
> --
> ath11k mailing list
> ath11k at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath11k

So after your message I was wracking my brain to sort out any
differences between our configs.  I did disable vt / vt-d and that
seems to have increased the stability of things some, but I still see
occasional hangs on initialization / association.