Hard lock-up on RK3328 with no kernel console output

[Craig McQueen]

I have a Rockchip RK3328 SOM based embedded Linux system. I'm using Yocto scarthgap with kernel v6.6.123. 
 
I'm seeing complete system lock-ups occur roughly every 1 to 72 hours, with no diagnostic information shown in the kernel serial console output. It is abrupt and looks like it's a total shut-down. But it does get rebooted by the RK3328 HW watchdog. By reading the CRU_GLB_CNT_TH register after reboot, I confirm that the watchdog triggered the reset. If I don't enable the RK3328 HW watchdog, it stays locked-up indefinitely until an external reset/power-cycle.

I am struggling to diagnose/debug this issue because I have been unable to get any kernel log output on the serial console. Normally I would expect to see a kernel message about a hard lock-up, or kernel oops, etc. I have enabled (among other things):

CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR_BUDDY=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y

I have also tried using the kernel PSTORE, PSTORE_RAM, PSTORE_CONSOLE feature, in conjunction with U-Boot `pstore` command. After a lock-up and reboot, I was able to read the kernel dmesg log with U-Boot `pstore display` command. But it also didn't show any lock-up diagnostics.

In this abrupt lock-up scenario, I would appreciate any advice on:

* How to get any diagnostics out of the system (eg any other kernel config changes to try).
* What sort of known fault mechanisms could lead to such a hard lock-up.

I may try using a SWD/JTAG debugger, once I can get a J-Link debugger.

-- 
Craig McQueen