[bug report] NVMe/IB: reset_controller need more than 1min

[Yi Zhang]

Hello

Gentle ping here, this issue still exists on latest 5.13-rc7

# time nvme reset /dev/nvme0

real 0m12.636s
user 0m0.002s
sys 0m0.005s
# time nvme reset /dev/nvme0

real 0m12.641s
user 0m0.000s
sys 0m0.007s
# time nvme reset /dev/nvme0

real 1m16.133s
user 0m0.000s
sys 0m0.007s

On Sat, May 22, 2021 at 12:27 PM Yi Zhang <yi.zhang at redhat.com> wrote:
>
> On Sat, May 22, 2021 at 2:00 AM Sagi Grimberg <sagi at grimberg.me> wrote:
> >
> >
> > > Hi
> > > I found this issue on 5.13-rc2 with NVMe/IB environment, could anyone
> > > help check it?
> > > Thanks.
> > >
> > > $ time echo 1 >/sys/block/nvme0n1/device/reset_controller
> > > real 0m10.678s
> > > user 0m0.000s
> > > sys 0m0.000s
> > > $ time echo 1 >/sys/block/nvme0n1/device/reset_controller
> > > real 1m11.530s
> > > user 0m0.000s
> > > sys 0m0.000s
> > >
> > > target:
> > > $ dmesg | grep nvme
> > > [  276.891454] nvmet: creating controller 1 for subsystem testnqn for
> > > NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> > > [  287.374412] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
> > > [  287.399317] nvmet: ctrl 1 fatal error occurred!
> > > [  348.412672] nvmet: creating controller 1 for subsystem testnqn for
> > > NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> > >
> > > client:
> > > $ dmesg | grep nvme
> > > [  281.704475] nvme nvme0: creating 40 I/O queues.
> > > [  285.557759] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> > > [  353.187809] nvme nvme0: I/O 8 QID 0 timeout
> > > [  353.193100] nvme nvme0: Property Set error: 881, offset 0x14
> > > [  353.226082] nvme nvme0: creating 40 I/O queues.
> > > [  357.088266] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> >
> > It appears that there is an admin timeout that is either triggered
> > by the reset or unrelated.
> >
> > Can you run nvme reset /dev/nvme0 instead so we can see the "resetting
> > controller" print?
> >
> Yes, here is the log:
> ------------------------0
> + nvme reset /dev/nvme0
> real 0m10.737s
> user 0m0.004s
> sys 0m0.004s
> ------------------------1
> + nvme reset /dev/nvme0
> real 1m11.335s
> user 0m0.002s
> sys 0m0.005s
>
> target:
> [  934.306016] nvmet: creating controller 1 for subsystem testnqn for
> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> [  944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
> [  944.900051] nvmet: ctrl 1 fatal error occurred!
> [ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for
> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
>
> client:
> [  857.264029] nvme nvme0: resetting controller
> [  864.115369] nvme nvme0: creating 40 I/O queues.
> [  867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> [  868.001673] nvme nvme0: resetting controller
> [  935.396789] nvme nvme0: I/O 9 QID 0 timeout
> [  935.402036] nvme nvme0: Property Set error: 881, offset 0x14
> [  935.438080] nvme nvme0: creating 40 I/O queues.
> [  939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues.
>
>
> --
> Best Regards,
>   Yi Zhang