[BUG][RT][ti-j72xx] spi: cadence-quadspi: boot failed with rtmutex deadlock detected

[He, Guocai (CN)]

Hi all,

I am seeing an reproducible issue in
drivers/spi/spi-cadence-quadspi.c on TI J72xx platform.

Kernel:
  v6.1/standard/preempt-rt/ti-sdk-6.1/ti-j7xxx
  (based on Linux v6.1.159 PREEMPT_RT)

Platform:
  TI J72xx
  QSPI controller: Cadence QuadSPI

Problem description:
when boot fail with the following call trace:
[    2.028205] ------------[ cut here ]------------
[    2.028213] rtmutex deadlock detected
[    2.028227] WARNING: CPU: 1 PID: 50 at kernel/locking/rtmutex.c:1642 __rt_mutex_slowlock_locked.constprop.0+0x184/0x2dc
[    2.028246] Modules linked in:
[    2.028256] CPU: 1 PID: 50 Comm: kworker/u8:1 Not tainted 6.1.159-rt32-yocto-preempt-rt #1
[    2.028263] Hardware name: Texas Instruments AM62A7 SK (DT)
[    2.028269] Workqueue: events_unbound deferred_probe_work_func
[    2.028282] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    2.028288] pc : __rt_mutex_slowlock_locked.constprop.0+0x184/0x2dc
[    2.028294] lr : __rt_mutex_slowlock_locked.constprop.0+0x184/0x2dc
[    2.028300] sp : fffffe000abef090
[    2.028302] x29: fffffe000abef110 x28: fffffe00090dc620 x27: 0000000000000001
[    2.028311] x26: fffffc0801af98f8 x25: 0000000000000000 x24: fffffc0804ac4088
[    2.028319] x23: 0000000000000000 x22: fffffe000abef098 x21: 00000000ffffffdd
[    2.028327] x20: fffffc0801a01d80 x19: fffffc08034b53e0 x18: ffffffffffffffff
[    2.028335] x17: 0000000000000000 x16: 0000000000000001 x15: fffffe000abef5b0
[    2.028343] x14: fffffc0804d1ad8e x13: 0a64657463657465 x12: 64206b636f6c6461
[    2.028351] x11: fffffffffffe0000 x10: 2d2d2d2d2d2d2d2d x9 : fffffe00080fc87c
[    2.028359] x8 : fffffe000abef090 x7 : 6b636f6c64616564 x6 : 000000000000000c
[    2.028367] x5 : 00000000fffff141 x4 : fffffe000abeee40 x3 : 0000000000000000
[    2.028375] x2 : 0000000000000000 x1 : 0000000000000000 x0 : fffffc0801a01d80
[    2.028384] Call trace:
[    2.028386]  __rt_mutex_slowlock_locked.constprop.0+0x184/0x2dc
[    2.028393]  mutex_lock+0x98/0xbc
[    2.028401]  spi_controller_resume+0x28/0xbc
[    2.028410]  cqspi_resume+0x58/0xa4
[    2.028418]  pm_generic_runtime_resume+0x34/0x50
[    2.028426]  __genpd_runtime_resume+0x38/0x90
[    2.028434]  genpd_runtime_resume+0x148/0x27c
[    2.028443]  __rpm_callback+0x50/0x1b4
[    2.028452]  rpm_callback+0x74/0x80
[    2.028459]  rpm_resume+0x5e4/0x85c
[    2.028467]  __pm_runtime_resume+0x5c/0xc0
[    2.028474]  cqspi_exec_mem_op+0x64/0xf50
[    2.028481]  spi_mem_exec_op+0x3bc/0x410
[    2.028487]  spinand_reset_op.constprop.0+0x64/0xa0
[    2.028495]  spinand_probe+0xdc/0x650
[    2.028502]  spi_mem_probe+0x74/0xc4
[    2.028508]  spi_probe+0x8c/0xec
[    2.028515]  really_probe+0xc4/0x2e0
[    2.028523]  __driver_probe_device+0x80/0x120
[    2.028531]  driver_probe_device+0xdc/0x170
[    2.028539]  __device_attach_driver+0xc0/0x14c
[    2.028548]  bus_for_each_drv+0x88/0xec
[    2.028555]  __device_attach+0xa8/0x1ac
[    2.028563]  device_initial_probe+0x1c/0x30
[    2.028572]  bus_probe_device+0xa4/0xb0
[    2.028579]  device_add+0x380/0x784
[    2.028586]  __spi_add_device+0x74/0x114
[    2.028592]  spi_add_device+0x68/0xa0
[    2.028597]  spi_register_controller+0x730/0xb00
[    2.028602]  cqspi_probe+0x690/0x98c
[    2.028608]  platform_probe+0x70/0xcc
[    2.028615]  really_probe+0xc4/0x2e0
[    2.028623]  __driver_probe_device+0x80/0x120
[    2.028631]  driver_probe_device+0xdc/0x170
[    2.028639]  __device_attach_driver+0xc0/0x14c
[    2.028648]  bus_for_each_drv+0x88/0xec
[    2.028655]  __device_attach+0xa8/0x1ac
[    2.028663]  device_initial_probe+0x1c/0x30
[    2.028671]  bus_probe_device+0xa4/0xb0
[    2.028679]  deferred_probe_work_func+0x90/0xd0
[    2.028687]  process_one_work+0x1f8/0x4b0
[    2.028696]  worker_thread+0x238/0x440
[    2.028703]  kthread+0x138/0x140
[    2.028710]  ret_from_fork+0x10/0x20
[    2.028718] ---[ end trace 0000000000000000 ]---

Analysis:
* 18db8c7f20a6 (HEAD) Revert "spi: cadence-qspi: fix pointer reference in runtime PM hooks"
*   7083c6620486 Merge branch 'v6.1/standard/base' into v6.1/standard/preempt-rt/ti-sdk-6.1/ti-j7xxx
|\
| *   2c1b513331dc Merge tag 'v6.1.140' into v6.1/standard/base
| |\
| | * da3c5173c55f (tag: v6.1.140) Linux 6.1.140
| | * 05e85d376720 x86/modules: Set VM_FLUSH_RESET_PERMS in module_alloc()

It is ok at da3c5173c55f，but it is bad at 7083c6620486.
I revert the commit(2c914aac952) which merged from stable release,  it also ok at 18db8c7f20a6.
Maybe it is a compatibility issue between v6.1/standard/preempt-rt/ti-sdk-6.1/ti-j7xxx and the stable release.

if this is an known issue ? 

Thanks,
Guocai He