Request timeout seen with NVMEoF TCP

[Sagi Grimberg]

> Hi Sagi,
> With above patch I still see the issue but less frequently. Without patch I was
> able to consistently reproduce the timouts with 4 target devices. With patch I
> see IO running fine for 4 targets. Tried the same test with 8 target devices
> and I see the below timeout. I've observed only one instance of timeout. So,
> I'll let it run for somemore time or rerun and update.

Hey Potnuri,

Have you observed this further?

I'd think that if the io_work reschedule itself when it races
with the direct send path this should not happen, but we may be
seeing a different race going on here, adding Samuel who saw
a similar phenomenon.

> 
> Target dmesg:
> ---
> [ 1704.132366] nvmet: creating controller 1 for subsystem nvme-ram0 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.185987] nvmet: creating controller 2 for subsystem nvme-ram1 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.230065] nvmet: creating controller 3 for subsystem nvme-ram2 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.277712] nvmet: creating controller 4 for subsystem nvme-ram3 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.314457] nvmet: creating controller 5 for subsystem nvme-ram4 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.370124] nvmet: creating controller 6 for subsystem nvme-ram5 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.435581] nvmet: creating controller 7 for subsystem nvme-ram6 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.501813] nvmet: creating controller 8 for subsystem nvme-ram7 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 2103.965017] nvmet: creating controller 6 for subsystem nvme-ram5 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> ^^^^^^^^^^^^^^^
> ---
> 
> Initiator dmesg:
> ---
> [ 1735.038634] EXT4-fs (nvme7n1): mounted filesystem with ordered data mode. Opts: (null)
> [ 2111.990419] nvme nvme5: queue 7: timeout request 0x57 type 4
> [ 2111.991835] nvme nvme5: starting error recovery
> [ 2111.998796] block nvme5n1: no usable path - requeuing I/O
> [ 2111.998816] nvme nvme5: Reconnecting in 10 seconds...
> [ 2122.253431] block nvme5n1: no usable path - requeuing I/O
> [ 2122.254732] nvme nvme5: creating 16 I/O queues.
> [ 2122.301169] nvme nvme5: mapped 16/0/0 default/read/poll queues.
> [ 2122.314229] nvme nvme5: Successfully reconnected (1 attempt)
> ---
> 
> 
> Thanks.
>