[PATCH v2 0/5] avoid race for time out

Wed Oct 28 07:36:06 EDT 2020

Hello

This series fixed the WARNING issue I reported [1], but now the nvme/012 
[2] will be hang there and never finished, here is the log[3]
[1]
https://lore.kernel.org/linux-nvme/1934331639.3314730.1602152202454.JavaMail.zimbra@redhat.com/

[2]
[root at hpe-xw9400-02 blktests]# nvme_trtype=tcp ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block 
device-backed ns)
     runtime  1199.651s  ...

[3]
[  120.550409] run blktests nvme/012 at 2020-10-28 06:50:11
[  121.138234] loop: module loaded
[  121.170869] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[  121.215930] nvmet_tcp: enabling port 0 (127.0.0.1:4420)
[  121.288229] nvmet: creating controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:ffe2b140e76a45649005853f3b871859.
[  121.302597] nvme nvme0: creating 12 I/O queues.
[  121.308361] nvme nvme0: mapped 12/0/0 default/read/poll queues.
[  121.320030] nvme nvme0: new ctrl: NQN "blktests-subsystem-1", addr 
127.0.0.1:4420
[  123.278903] XFS (nvme0n1): Mounting V5 Filesystem
[  123.291608] XFS (nvme0n1): Ending clean mount
[  123.297321] xfs filesystem being mounted at /mnt/blktests supports 
timestamps until 2038 (0x7fffffff)
[  183.872118] nvme nvme0: queue 1: timeout request 0x6c type 4
[  183.877792] nvme nvme0: starting error recovery
[  183.882376] nvme nvme0: queue 8: timeout request 0x11 type 4
[  183.888149] nvme nvme0: queue 8: timeout request 0x12 type 4
[  183.893805] nvme nvme0: queue 8: timeout request 0x13 type 4
[  183.899469] nvme nvme0: queue 8: timeout request 0x14 type 4
[  183.905130] nvme nvme0: queue 8: timeout request 0x15 type 4
[  183.910792] nvme nvme0: queue 8: timeout request 0x16 type 4
[  183.916453] nvme nvme0: queue 8: timeout request 0x17 type 4
[  183.922114] nvme nvme0: queue 8: timeout request 0x18 type 4
[  183.927777] nvme nvme0: queue 8: timeout request 0x19 type 4
[  183.933450] nvme nvme0: queue 8: timeout request 0x1a type 4
[  183.939110] nvme nvme0: queue 8: timeout request 0x1b type 4
[  183.944771] nvme nvme0: queue 8: timeout request 0x1c type 4
[  183.950431] nvme nvme0: queue 8: timeout request 0x1d type 4
[  183.956095] nvme nvme0: queue 8: timeout request 0x1e type 4
[  183.961755] nvme nvme0: queue 8: timeout request 0x1f type 4
[  183.967414] nvme nvme0: queue 8: timeout request 0x20 type 4
[  183.973218] block nvme0n1: no usable path - requeuing I/O
[  183.978623] block nvme0n1: no usable path - requeuing I/O
[  183.982492] nvme nvme0: Reconnecting in 10 seconds...
[  183.984022] block nvme0n1: no usable path - requeuing I/O
[  183.994476] block nvme0n1: no usable path - requeuing I/O
[  183.999870] block nvme0n1: no usable path - requeuing I/O
[  184.005264] block nvme0n1: no usable path - requeuing I/O
[  184.010669] block nvme0n1: no usable path - requeuing I/O
[  184.016080] block nvme0n1: no usable path - requeuing I/O
[  184.021463] block nvme0n1: no usable path - requeuing I/O
[  184.026858] block nvme0n1: no usable path - requeuing I/O
[  209.472647] nvmet: ctrl 2 keep-alive timer (15 seconds) expired!
[  209.478662] nvmet: ctrl 2 fatal error occurred!
[  213.568765] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  213.574782] nvmet: ctrl 1 fatal error occurred!
[  238.064572] nvmet: creating controller 2 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:ffe2b140e76a45649005853f3b871859.
[  256.577658] nvme nvme0: queue 0: timeout request 0x0 type 4
[  256.583333] nvme nvme0: Connect command failed, error wo/DNR bit: 881
[  256.589806] nvme nvme0: failed to connect queue: 0 ret=881
[  256.595326] nvme nvme0: Failed reconnect attempt 1
[  256.600119] nvme nvme0: Reconnecting in 10 seconds...
[  266.818455] nvmet: creating controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:ffe2b140e76a45649005853f3b871859.
[  266.832356] nvme_ns_head_submit_bio: 30 callbacks suppressed
[  266.832362] block nvme0n1: no usable path - requeuing I/O
[  266.843443] block nvme0n1: no usable path - requeuing I/O
[  266.848848] block nvme0n1: no usable path - requeuing I/O
[  266.854244] block nvme0n1: no usable path - requeuing I/O
[  266.859663] block nvme0n1: no usable path - requeuing I/O
[  266.865059] block nvme0n1: no usable path - requeuing I/O
[  266.870454] block nvme0n1: no usable path - requeuing I/O
[  266.875845] block nvme0n1: no usable path - requeuing I/O
[  266.881234] block nvme0n1: no usable path - requeuing I/O
[  266.886632] block nvme0n1: no usable path - requeuing I/O
[  266.892237] nvme nvme0: creating 12 I/O queues.
[  266.903744] nvme nvme0: mapped 12/0/0 default/read/poll queues.
[  266.911929] nvme nvme0: Successfully reconnected (2 attempt)
[  327.747177] nvme nvme0: queue 2: timeout request 0x1e type 4
[  327.752883] nvme nvme0: starting error recovery
[  327.757450] nvme nvme0: queue 4: timeout request 0x63 type 4
[  327.763511] nvme_ns_head_submit_bio: 14 callbacks suppressed
[  327.763520] block nvme0n1: no usable path - requeuing I/O
[  327.774614] block nvme0n1: no usable path - requeuing I/O
[  327.780053] block nvme0n1: no usable path - requeuing I/O
[  327.785450] block nvme0n1: no usable path - requeuing I/O
[  327.790876] block nvme0n1: no usable path - requeuing I/O
[  327.796316] block nvme0n1: no usable path - requeuing I/O
[  327.801727] block nvme0n1: no usable path - requeuing I/O
[  327.807231] block nvme0n1: no usable path - requeuing I/O
[  327.812627] block nvme0n1: no usable path - requeuing I/O
[  327.818025] block nvme0n1: no usable path - requeuing I/O
[  353.859745] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
[  353.865761] nvmet: ctrl 1 fatal error occurred!

On 10/22/20 10:14 AM, Chao Leng wrote:
> First avoid race between time out and tear down for rdma and tcp.
> Second avoid repeated request completion in time out for rdma and tcp.
>
> V2:
> 	- add avoiding repeated request completion in time out
>
> Chao Leng (3):
>    nvme-core: introduce sync io queues
>    nvme-rdma: avoid race between time out and tear down
>    nvme-tcp: avoid race between time out and tear down
>
> Sagi Grimberg (2):
>    nvme-rdma: avoid repeated request completion
>    nvme-tcp: avoid repeated request completion
>
>   drivers/nvme/host/core.c |  8 ++++++--
>   drivers/nvme/host/nvme.h |  1 +
>   drivers/nvme/host/rdma.c | 14 +++-----------
>   drivers/nvme/host/tcp.c  | 16 ++++------------
>   4 files changed, 14 insertions(+), 25 deletions(-)
>