AER: Malformed TLP recovery deadlock with NVMe drives

[Alex G.]

On 05/07/2018 03:30 PM, okaya at codeaurora.org wrote:
> On 2018-05-07 21:16, Alex G. wrote:
>> On 05/07/2018 01:46 PM, okaya at codeaurora.org wrote:
>>> On 2018-05-07 19:36, Alex G. wrote:
>>>> Hi! Me again!
>>>>
>>>> I'm seeing what appears to be a deadlock in the AER recovery path. It
>>>> seems that the device_lock() call in report_slot_reset() never returns.
>>>> How we get there is interesting:
>>>
>>> Can you give this patch a try?
>>>
>> Oh! Patches so soon? Yay!
>>
>>> https://patchwork.kernel.org/patch/10351515/
>>
>> Thank you! I tried a few runs. there was one run where we didn't lock
>> up, but the other runs all went like before.
>>
>> For comparison, the run that didn't deadlock looked like [2].
>>
> 
> 
> Sounds like there are multiple problems.

If it were easy, somebody would have patched it by now ;)

> With this patch, you shouldn't
> see link down and up interrupts during reset but i do see them in the log.

You will see the messages from the link up/down events regardless if any
action is actually taken.

> Can you also share a fail case log with this patch and a diff of your
> hacks so that we know where prints are coming from.

Of course. Example of failing case [3], and is identical to the fail log
without any patches. Although prints have the function name, the diff is
in [4].

Alex

[3] http://gtech.myftp.org/~mrnuke/nvme_logs/log-20180507-1509.log
[4] http://gtech.myftp.org/~mrnuke/nvme_logs/print_hacks.patch


>> [2] http://gtech.myftp.org/~mrnuke/nvme_logs/log-20180507-1429.log
>>>> [1] http://gtech.myftp.org/~mrnuke/nvme_logs/log-20180507-1308.log