AER: Malformed TLP recovery deadlock with NVMe drives

[okaya at codeaurora.org]

On 2018-05-07 21:58, Alex G. wrote:
> On 05/07/2018 03:30 PM, okaya at codeaurora.org wrote:
>> On 2018-05-07 21:16, Alex G. wrote:
>>> On 05/07/2018 01:46 PM, okaya at codeaurora.org wrote:
>>>> On 2018-05-07 19:36, Alex G. wrote:
>>>>> Hi! Me again!
>>>>> 
>>>>> I'm seeing what appears to be a deadlock in the AER recovery path. 
>>>>> It
>>>>> seems that the device_lock() call in report_slot_reset() never 
>>>>> returns.
>>>>> How we get there is interesting:
>>>> 
>>>> Can you give this patch a try?
>>>> 
>>> Oh! Patches so soon? Yay!
>>> 
>>>> https://patchwork.kernel.org/patch/10351515/
>>> 
>>> Thank you! I tried a few runs. there was one run where we didn't lock
>>> up, but the other runs all went like before.
>>> 
>>> For comparison, the run that didn't deadlock looked like [2].
>>> 
>> 
>> 
>> Sounds like there are multiple problems.
> 
> If it were easy, somebody would have patched it by now ;)

Can you file a bugzilla CC me, keith and bjorn and attach all of your 
logs?

Let's debug this there.


> 
>> With this patch, you shouldn't
>> see link down and up interrupts during reset but i do see them in the 
>> log.
> 
> You will see the messages from the link up/down events regardless if 
> any
> action is actually taken.
> 
>> Can you also share a fail case log with this patch and a diff of your
>> hacks so that we know where prints are coming from.
> 
> Of course. Example of failing case [3], and is identical to the fail 
> log
> without any patches. Although prints have the function name, the diff 
> is
> in [4].
> 
> Alex
> 
> [3] http://gtech.myftp.org/~mrnuke/nvme_logs/log-20180507-1509.log
> [4] http://gtech.myftp.org/~mrnuke/nvme_logs/print_hacks.patch
> 
> 
>>> [2] http://gtech.myftp.org/~mrnuke/nvme_logs/log-20180507-1429.log
>>>>> [1] http://gtech.myftp.org/~mrnuke/nvme_logs/log-20180507-1308.log