Deadlock under load with Linux 5.9 and other recent kernels

[Christian Hewitt]

> On 26 Sep 2020, at 4:28 pm, Christian Hewitt <christianshewitt at gmail.com> wrote:
> 
>> 
>> On 26 Sep 2020, at 4:13 pm, Jens Axboe <axboe at kernel.dk> wrote:
>> 
>> On 9/26/20 5:55 AM, Christian Hewitt wrote:
>>>> 
>>>> On 26 Sep 2020, at 2:51 pm, Jens Axboe <axboe at kernel.dk> wrote:
>>>> 
>>>> On 9/26/20 1:55 AM, Christian Hewitt wrote:
>>>>> I am using an ARM SBC device with Amlogic S922X chip (Beelink
>>>>> GS-King-X, an Android STB) to boot the Kodi mediacentre distro
>>>>> LibreELEC (which I work on) although the issue is also reproducible
>>>>> with Manjaro and Armbian on the same hardware, and with the GT-King
>>>>> and GT-King Pro devices from the same vendor - all three devices are
>>>>> using a common dtsi:
>>>>> 
>>>>> https://github.com/chewitt/linux/blob/amlogic-5.9-integ/arch/arm64/boot/dts/amlogic/meson-g12b-gsking-x.dts
>>>>> https://github.com/chewitt/linux/blob/amlogic-5.9-integ/arch/arm64/boot/dts/amlogic/meson-g12b-gtking-pro.dts
>>>>> https://github.com/chewitt/linux/blob/amlogic-5.9-integ/arch/arm64/boot/dts/amlogic/meson-g12b-gtking.dts
>>>>> https://github.com/chewitt/linux/blob/amlogic-5.9-integ/arch/arm64/boot/dts/amlogic/meson-g12b-w400.dtsi
>>>>> 
>>>>> I have schematics for the devices, but can only share those privately
>>>>> on request.
>>>>> 
>>>>> For testing I am booting LibreELEC from SD card. The box has a 4TB
>>>>> SATA drive internally connected with a USB > SATA bridge, see dmesg:
>>>>> http://ix.io/2yLh and I connect a USB stick with a 4GB ISO file that I
>>>>> copy to the internal SATA drive. Within 10-20 seconds of starting the
>>>>> copy the box deadlocks needing a hard power cycle to recover. The
>>>>> timing of the deadlock is variable but the device _always_ deadlocks.
>>>>> Although I am using a simple copy use-case, there are similar reports
>>>>> in Armbian forums performing tasks like installs/updates that involve
>>>>> I/O loads.
>>>>> 
>>>>> Following advice in the #linux-amlogic IRC channel I added
>>>>> CONFIG_SOFTLOCKUP_DETECTOR and CONFIG_DETECT_HUNG_TASK and was able to
>>>>> get output on the HDMI screen (it is not possible to connect to UART
>>>>> pins without destroying the box case). If you advance the following
>>>>> video frame by frame in VLC you can see the output:
>>>>> 
>>>>> https://www.dropbox.com/s/klvcizim8cs5lze/lockup_clip.mov?dl=0
>>>> 
>>>> Try with this patch:
>>>> 
>>>> https://lore.kernel.org/linux-block/20200925191902.543953-1-shakeelb@google.com/
>>> 
>>> It still locks up approx. 25 seconds into the copy operation. Here’s the output in video again (a little blurry):
>>> 
>>> https://www.dropbox.com/s/3j2czaq509arg6g/lockup_clip2.mov?dl=0
>> 
>> Can you try and set CONFIG_SLUB in your .config instead of CONFIG_SLAB?
> 
> CONFIG_SLUB is already set, here’s the full defconfig http://paste.ubuntu.com/p/5BNdZv6J3c/
> 
> # dmesg | grep -i slub
> [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
> 
>> Also, just take a picture, should be easier to get readable than a video.
>> And the static trace is all that is needed.
> 
> This is from a GT-King Pro which someone reminded me has a large RS232 port on the rear:
> 
> https://pastebin.com/raw/sGtzgreN

from 5.9—rc7 https://pastebin.com/raw/nbHJmrqe

Christian