rk3588(CM3588) USB issues

Thu Aug 8 01:38:01 PDT 2024

(sent again because android-gmail doesn't support non-html mails and
the original was rejected by some recipients)

Hi Sebastian,

Do you have the same USB issues? I'm asking you because you added the
board to Mainline. If not, it'd be great if you can describe your
software setup. Maybe I'm doing something wrong on the bootloader
side?

Thanks
Michael

On Thu, Aug 1, 2024 at 9:05 PM Michael Zimmermann
<sigmaepsilon92 at gmail.com> wrote:
>
> Hi,
>
> I have a friendlyelec CM3588 and USB doesn't quite seem to work. I
> started with 6.8 but am now mostly testing with 6.11-rc1 since it
> fixed the typec ports and pcie bifurcation and both of them seem to
> have issues. I'm using fedora coreOS 40 with a self compiled
> mainline-mcuboot which simply adds a config and dtb for that board. I
> tried many different situations and I'll list a few but here are the
> symptoms that can happen when a device is already connected before
> powering on the SoC:
>
> 1) The system dies after these log lines:
> [    6.446287] hub 1-0:1.0: USB hub found
> [    6.446713] hub 1-0:1.0: 1 port detected
> No watchdog will trigger, the power button doesn't work.
>
> 2) The watchdog complains after the same lines as in 1, but the system
> is still stuck:
> [   36.025461] Modules linked in: dw_mmc(+) phy_rockchip_usbdp(+)
> nvme_core rockchip_saradc phy_rockchip_naneng_combphy
> phy_rockchip_snps_pcie3 industrialio_triggered_buffer
> nvmem_rockchip_otp rtc_hym8563(+) rockchip_dfi typec pl330 kfifo_buf
> dw_wdt nvme_auth dwc3 phy_rockchip_inno_usb2 ohci_platform udc_core
> adc_keys pwm_rockchip industrialio ohci_hcd ehci_platform ulpi
> cpufreq_dt be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio
> libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp
> libiscsi scsi_transport_iscsi scsi_dh_rdac scsi_dh_emc scsi_dh_alua
> ip6_tables ip_tables fuse dm_multipath
>
> 3) The system boots fine but USB doesn't work and the system is
> spammed with these logs:
> [   29.439659] usb usb6-port1: Cannot enable. Maybe the USB cable is bad?
> [   30.583618] usb usb6-port1: Cannot enable. Maybe the USB cable is bad?
> [   31.727508] usb usb6-port1: Cannot enable. Maybe the USB cable is bad?
>
> Things I have tried: using an unpowered USB3-hub, using a powered
> USB3-hub. Connecting USB3 devices to the USB2 port and vice-versa.
> Below only list test results which were done through a powered hub to
> exclude skewed results due to power issues.
>
> Keep in mind that the CM3588 has 4 USB ports:
> - typeC(which I am not testing)
> - USB2
> - USB3A
> - USB3A connected to the SoCs second typeC port. In the results below
> I'll call this USB3AC
>
> I also don't have any low-power usb3 dongles or usb3 drives whose
> controllers are expensive enough that I trust them for these tests, So
> I'm mostly testing with my Pixel8.
>
> Some test results:
>
> 1) Boot with Pixel8 connected to USB3AC. The behavior is random.
> Sometimes I have this:
> ...
> [    6.237348] ohci-platform fc840000.usb: Generic Platform OHCI controller
> [    6.238187] ohci-platform fc840000.usb: new USB bus registered,
> assigned bus number 1
> [    6.239009] ohci-platform fc840000.usb: irq 42, io mem 0xfc840000
> [    6.242667] ehci-platform fc800000.usb: EHCI Host Controller
> [    6.244480] ehci-platform fc800000.usb: new USB bus registered,
> assigned bus number 2
> [    6.245294] ehci-platform fc800000.usb: irq 44, io mem 0xfc800000
> [    6.251160] ehci-platform fc880000.usb: EHCI Host Controller
> [    6.251694] ohci-platform fc8c0000.usb: Generic Platform OHCI controller
> [    6.252503] ohci-platform fc8c0000.usb: new USB bus registered,
> assigned bus number 3
> [    6.253271] ohci-platform fc8c0000.usb: irq 46, io mem 0xfc8c0000
> [    6.253643] ehci-platform fc800000.usb: USB 2.0 started, EHCI 1.00
> [    6.253850] ehci-platform fc880000.usb: new USB bus registered,
> assigned bus number 4
> [    6.254502] usb usb2: New USB device found, idVendor=1d6b,
> idProduct=0002, bcdDevice= 6.11
> [    6.255121] ehci-platform fc880000.usb: irq 47, io mem 0xfc880000
> [    6.255777] usb usb2: New USB device strings: Mfr=3, Product=2,
> SerialNumber=1
> [    6.256967] usb usb2: Product: EHCI Host Controller
> [    6.257407] usb usb2: Manufacturer: Linux
> 6.11.0-0.rc1.20240729gitdc1c8034e31b.16.fc41.aarch64 ehci_hcd
> [    6.258231] usb usb2: SerialNumber: fc800000.usb
> [    6.259129] hub 2-0:1.0: USB hub found
> [    6.259521] hub 2-0:1.0: 1 port detected
> [    6.263349] ehci-platform fc880000.usb: USB 2.0 started, EHCI 1.00
> [    6.264046] usb usb4: New USB device found, idVendor=1d6b,
> idProduct=0002, bcdDevice= 6.11
> [    6.264794] usb usb4: New USB device strings: Mfr=3, Product=2,
> SerialNumber=1
> [    6.265436] usb usb4: Product: EHCI Host Controller
> [    6.265867] usb usb4: Manufacturer: Linux
> 6.11.0-0.rc1.20240729gitdc1c8034e31b.16.fc41.aarch64 ehci_hcd
> [    6.266695] usb usb4: SerialNumber: fc880000.usb
> [    6.268464] hub 4-0:1.0: USB hub found
> [    6.268876] hub 4-0:1.0: 1 port detected
> [    6.297227] usb usb1: New USB device found, idVendor=1d6b,
> idProduct=0001, bcdDevice= 6.11
> [    6.297982] usb usb1: New USB device strings: Mfr=3, Product=2,
> SerialNumber=1
> [    6.298627] usb usb1: Product: Generic Platform OHCI controller
> [    6.299148] usb usb1: Manufacturer: Linux
> 6.11.0-0.rc1.20240729gitdc1c8034e31b.16.fc41.aarch64 ohci_hcd
> [    6.299976] usb usb1: SerialNumber: fc840000.usb
> [    6.301706] hub 1-0:1.0: USB hub found
> [    6.302069] hub 1-0:1.0: 1 port detected
> [    6.311456] usb usb3: New USB device found, idVendor=1d6b,
> idProduct=0001, bcdDevice= 6.11
> [    6.312198] usb usb3: New USB device strings: Mfr=3, Product=2,
> SerialNumber=1
> [    6.312845] usb usb3: Product: Generic Platform OHCI controller
> [    6.313371] usb usb3: Manufacturer: Linux
> 6.11.0-0.rc1.20240729gitdc1c8034e31b.16.fc41.aarch64 ohci_hcd
> [    6.314206] usb usb3: SerialNumber: fc8c0000.usb
> [    6.315073] hub 3-0:1.0: USB hub found
> [    6.315457] hub 3-0:1.0: 1 port detected
> [    6.316795] sdhci: Secure Digital Host Controller Interface driver
> [    6.317390] sdhci: Copyright(c) Pierre Ossman
> [    6.325998] Synopsys Designware Multimedia Card Interface Driver
> [    6.335776] dma-pl330 fea10000.dma-controller: Loaded driver for
> PL330 DMAC-241330
> [    6.336514] dma-pl330 fea10000.dma-controller:
> DBUFF-128x8bytes Num_Chans-8 Num_Peri-32 Num_Events-16
> [    6.366285] sdhci-pltfm: SDHCI platform and OF driver helper
> [   36.026322] watchdog: Watchdog detected hard LOCKUP on cpu 5
> [   36.026837] Modules linked in: phy_rockchip_usbdp(+) typec
> phy_rockchip_naneng_combphy(+) sdhci_pltfm phy_rockchip_snps_pcie3
> dw_mmc_rockchip nvme_core(+) rockchip_saradc rockchip_dfi dw_mmc_pltfm
> industrialio_triggered_buffer cqhci pl330(+) dw_mmc kfifo_buf sdhci
> dwc3 nvme_auth ehci_platform phy_rockchip_inno_usb2 pwm_rockchip
> adc_keys cpufreq_dt udc_core industrialio ulpi ohci_platform ohci_hcd
> be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi
> libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6_tables
> ip_tables fuse dm_multipath
> [   36.029321] watchdog: Watchdog detected hard LOCKUP on cpu 0
> [   36.031733] Sending NMI from CPU 4 to CPUs 5:
> [   36.032228] Modules linked in: phy_rockchip_usbdp(+) typec
> phy_rockchip_naneng_combphy(+) sdhci_pltfm phy_rockchip_snps_pcie3
> dw_mmc_rockchip nvme_core(+) rockchip_saradc rockchip_dfi dw_mmc_pltfm
> industrialio_triggered_buffer cqhci pl330(+) dw_mmc kfifo_buf sdhci
> dwc3 nvme_auth ehci_platform phy_rockchip_inno_usb2 pwm_rockchip
> adc_keys cpufreq_dt udc_core industrialio ulpi ohci_platform ohci_hcd
> be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi
> libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6_tables
> ip_tables fuse dm_multipath
> [   66.371321] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> [   66.371862] rcu:     0-...!: (0 ticks this GP) idle=47c0/0/0x0
> softirq=1091/1091 fqs=0 (false positive?)
> [   66.372678] rcu:     6-...!: (1 GPs behind)
> idle=7b84/1/0x4000000000000000 softirq=1557/1558 fqs=0
> [   66.373447] rcu:     (detected by 4, t=60004 jiffies, g=1689, q=29 ncpus=8)
> [   66.374039] Sending NMI from CPU 4 to CPUs 0:
> [   76.375676] Sending NMI from CPU 4 to CPUs 6:
> [   86.377312] rcu: rcu_preempt kthread timer wakeup didn't happen for
> 80004 jiffies! g1689 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
> [   86.378313] rcu:     Possible timer handling issue on cpu=5 timer-softirq=315
> [   86.378917] rcu: rcu_preempt kthread starved for 80009 jiffies!
> g1689 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=5
> [   86.379834] rcu:     Unless rcu_preempt kthread gets sufficient CPU
> time, OOM is now expected behavior.
> [   86.380633] rcu: RCU grace-period kthread stack dump:
> [   86.381077] task:rcu_preempt     state:I stack:0     pid:18
> tgid:18    ppid:2      flags:0x00000008
> [   86.381899] Call trace:
> [   86.382119]  __switch_to+0xc4/0xe8
> [   86.382432]  __schedule+0x2a0/0x798
> [   86.382747]  schedule+0x3c/0xe0
> [   86.383031]  schedule_timeout+0x9c/0x1a8
> [   86.383382]  rcu_gp_fqs_loop+0x124/0x580
> [   86.383733]  rcu_gp_kthread+0x180/0x200
> [   86.384075]  kthread+0xf4/0x110
> [   86.384359]  ret_from_fork+0x10/0x20
> [   86.384681] rcu: Stack dump where RCU GP kthread last ran:
> [   86.385163] Sending NMI from CPU 4 to CPUs 5:
> ...
> and then it hangs. And sometimes it boots, doesn't detect a devices
> and keeps spamming the log with "usb usb6-port1: Cannot enable. Maybe
> the USB cable is bad?".
>
> 2) Boot without anything connected, then connect Pixel8 to USB3A: No
> device detected, logs spammed with "usb usb6-port1: Cannot enable.
> Maybe the USB cable is bad?".
>
> 3) Boot with Pixel8 connected to USB3AC: works just fine. Also works
> when I connect it at runtime.
>
> 4) Boot with Pixel 8 connected to USB2 port:
> ...
> [    6.331359] hub 4-0:1.0: USB hub found
> [    6.331439] input: gpio-keys as /devices/platform/gpio-keys/input/input0
> [    6.331756] hub 4-0:1.0: 1 port detected
> [    6.334789] usb usb3: New USB device found, idVendor=1d6b,
> idProduct=0001, bcdDevice= 6.11
> [    6.335530] usb usb3: New USB device strings: Mfr=3, Product=2,
> SerialNumber=1
> [    6.336176] usb usb3: Product: Generic Platform OHCI controller
> [    6.336698] usb usb3: Manufacturer: Linux
> 6.11.0-0.rc1.20240729gitdc1c8034e31b.16.fc41.aarch64 ohci_hcd
> [    6.337526] usb usb3: SerialNumber: fc8c0000.usb
> [   36.022707] watchdog: Watchdog detected hard LOCKUP on cpu 2
> [   36.023240] Modules linked in: rtc_hym8563(+) tcpm dw_mmc gpio_keys
> pwm_fan phy_rockchip_usbdp nvme_core phy_rockchip_snps_pcie3
> phy_rockchip_naneng_combphy typec cqhci dw_wdt nvmem_rockchip_otp
> pl330(+) spi_rockchip sdhci kfifo_buf rockchip_dfi nvme_auth dwc3
> phy_rockchip_inno_usb2 pwm_rockchip ohci_platform adc_keys udc_core
> ohci_hcd industrialio ehci_platform ulpi cpufreq_dt be2iscsi bnx2i
> cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx
> iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6_tables ip_tables fuse
> dm_multipath
> [   36.024708] watchdog: Watchdog detected hard LOCKUP on cpu 4
> [   36.028171] Sending NMI from CPU 1 to CPUs 2:
> [   36.028669] Modules linked in: rtc_hym8563(+) tcpm dw_mmc gpio_keys
> pwm_fan phy_rockchip_usbdp nvme_core phy_rockchip_snps_pcie3
> phy_rockchip_naneng_combphy typec cqhci dw_wdt nvmem_rockchip_otp
> pl330(+) spi_rockchip sdhci kfifo_buf rockchip_dfi nvme_auth dwc3
> phy_rockchip_inno_usb2 pwm_rockchip ohci_platform adc_keys udc_core
> ohci_hcd industrialio ehci_platform ulpi cpufreq_dt be2iscsi bnx2i
> cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx
> iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6_tables ip_tables fuse
> dm_multipath
> [   66.336708] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> [   66.337272] rcu:     4-...0: (1 GPs behind)
> idle=afbc/1/0x4000000000000000 softirq=1897/1898 fqs=6846
> [   66.338089] rcu:     (detected by 1, t=60005 jiffies, g=1757, q=91 ncpus=8)
> [   66.338699] Sending NMI from CPU 1 to CPUs 4:
> [   76.340352] rcu: rcu_preempt kthread timer wakeup didn't happen for
> 17792 jiffies! g1757 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
> [   76.341374] rcu:     Possible timer handling issue on cpu=3
> timer-softirq=6205
> [   76.342000] rcu: rcu_preempt kthread starved for 17795 jiffies!
> g1757 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=3
> [   76.342937] rcu:     Unless rcu_preempt kthread gets sufficient CPU
> time, OOM is now expected behavior.
> [   76.343754] rcu: RCU grace-period kthread stack dump:
> [   76.344211] task:rcu_preempt     state:I stack:0     pid:18
> tgid:18    ppid:2      flags:0x00000008
> [   76.345056] Call trace:
> [   76.345287]  __switch_to+0xc4/0xe8
> [   76.345617]  __schedule+0x2a0/0x798
> [   76.345948]  schedule+0x3c/0xe0
> [   76.346247]  schedule_timeout+0x9c/0x1a8
> [   76.346613]  rcu_gp_fqs_loop+0x124/0x580
> [   76.346979]  rcu_gp_kthread+0x180/0x200
> [   76.347337]  kthread+0xf4/0x110
> [   76.347637]  ret_from_fork+0x10/0x20
> [   76.347975] rcu: Stack dump where RCU GP kthread last ran:
> [   76.348470] Sending NMI from CPU 1 to CPUs 3:
> ...
> And then it hangs. It works if I connect it after booting.
>
> 5) USB2 stick works on all ports when connected at runtime.
>
> 6) Booting with USB2 stick connected to USB3A port: works
> 7) Booting with USB2 stick connected to USB3AC port: hangs after:
> [    6.423500] hub 4-0:1.0: USB hub found
> [    6.423870] hub 4-0:1.0: 1 port detected
> [    6.602224] rtc-hym8563 6-0051: registered as rtc0
> [    6.610981] rtc-hym8563 6-0051: setting system clock to
> 2024-08-01T18:56:52 UTC (1722538612)
>
> 8) Booting with USB2 stick connected to USB2 port: hangs after:
> [    6.337067] hub 3-0:1.0: USB hub found
> [    6.337494] hub 3-0:1.0: 1 port detected
> [    6.338460] rtc-hym8563 6-0051: registered as rtc0
> [    6.339744] rtc-hym8563 6-0051: setting system clock to
> 2024-08-01T18:59:56 UTC (1722538796)
> [    6.372577] usb usb2: New USB device found, idVendor=1d6b,
> idProduct=0001, bcdDevice= 6.11
> [    6.373361] usb usb2: New USB device strings: Mfr=3, Product=2,
> SerialNumber=1
> [    6.374018] usb usb2: Product: Generic Platform OHCI controller
> [    6.374573] usb usb2: Manufacturer: Linux
> 6.11.0-0.rc1.20240729gitdc1c8034e31b.16.fc41.aarch64 ohci_hcd
> [    6.375431] usb usb2: SerialNumber: fc8c0000.usb
>
> 9) Boot with ch341-uart connected to USB2: works.
>
> I hope that this is enough information for you to start investigating
> the issue. I also hope this is reproducible on your side.
>
> Thanks
> Michael