[External] : Re: way to unbind a bad nvme device/controller without powering off system

[James Puthukattukaran]

On 11/8/22 18:15, Keith Busch wrote:
> On Tue, Nov 08, 2022 at 02:13:25PM -0500, James Puthukattukaran wrote:
>> On 10/25/22 12:56, Keith Busch wrote:
>>> On Mon, Oct 24, 2022 at 08:26:54PM -0600, Keith Busch wrote:
>>>> On Mon, Oct 24, 2022 at 08:02:33PM -0400, James Puthukattukaran wrote:
>>>>> On 10/24/22 18:36, Keith Busch wrote:
>>>>>
>>>>>>
>>>>>> Generally, the default timeout is really long. If you have a broken
>>>>>> controller, it could take several minutes before the driver unblocks
>>>>>> forward progress to unbind.
>>>>> One concern is that the reset controller flow attempts to reinitialze the controller and this will cause problems if the controller is bad. Would it make sense to have a sysfs "remove_controller" interface that simply goes through and does a nvme_dev_disable() with the assumption that the controller is dead? Will the nvme_kill_queues() in nvme_dev_disadble() unwedge any potential nvme reset thread that is blocked and thus allow the nvme_remove() flow to complete?
>>>>> thanks
>>>>
>>>> In your log snippet, there's this line:
>>>>
>>>>   kernel:warning: [10416608.580157] nvme nvme3: I/O 209 QID 1 timeout, disable controller
>>>>
>>>> The next action the driver takes after logging that is to drain any
>>>> outstanding IO through a forced reset, and all subsequent tasks *should*
>>>> be unblocked after that completes to allow the unbinding, so I don't
>>>> think adding any new sysfs knobs is going to help if it's not already
>>>> succeeding.
>>>>
>>>> The only other thing that looks odd is that one of your stuck tasks is a
>>>> user passthrough command, but that should have also been cleared out by
>>>> the reset. Do you know what command that process is sending? I'll need
>>>> to double check your kernel version to see if there's anything missing
>>>> in that driver to ensure the unbinding succeeds. 
>>>
>>> I think there could be a mismatched queue quiesce state happening, but
>>> there's some fixes for this in later kernels. Could you possibly try
>>> something newer, like 6.0-stable, as an experiment?
>>
>> Is this the patch for the mismatch?
>>
>> commit d4060d2be1132596154f31f4d57976bd103e969d
>> Author: Tao Chiu <taochiu at synology.com>
>> Date:   Mon Apr 26 10:53:55 2021 +0800
>>
>>     nvme-pci: fix controller reset hang when racing with nvme_timeout
> 
> That doesn't look like what's happening here. I was thinking of this
> one:
> 
>   commit 9e6a6b1212100148c109675e003369e3e219dbd9
>   Author: Ming Lei <ming.lei at redhat.com>
>   Date:   Thu Oct 14 16:17:08 2021 +0800
>   
>       nvme: paring quiesce/unquiesce
> 
> It's not a clean cherry-pick, though.

Thanks. We are desperately trying to reproduce the issue so we know the patch fixes it. The plan is to run nvme IO along with attempting to do a reset controller (via the sysfs) while another thread repeatedly attempts nvme user IO commands. fingers crossed.