[PATCH v7] nvme-fabrics: reject I/O to offline device

[James Smart]

On 7/8/2020 8:07 AM, Victor Gladkov wrote:
> Commands get stuck while Host NVMe controller (TCP or RDMA) is in
> reconnect state. NVMe controller enters into reconnect state when it
> loses connection with the target. It tries to reconnect every 10
> seconds (default) until successful reconnection or until reconnect
> time-out is reached. The default reconnect time out is 10 minutes.
>
> To fix this long delay due to the default timeout we introduce new
> session parameter "fast_io_fail_tmo". The timeout is measured in
> seconds from the controller reconnect, any command beyond that
> timeout is rejected. The new parameter value may be passed during
> 'connect'.
> The default value of 0 means no timeout (similar to current behavior).
>
> We add a new controller NVME_CTRL_FAILFAST_EXPIRED and respective
> delayed work that updates the NVME_CTRL_FAILFAST_EXPIRED flag.
>
> When the controller is entering the CONNECTING state, we schedule
> the delayed_work based on failfast timeout value. If the transition
> is out of CONNECTING, terminate delayed work item and ensure
> failfast_expired is false. If delayed work item expires then set
> "NVME_CTRL_FAILFAST_EXPIRED" flag to true.
>
> We also update nvmf_fail_nonready_command() and
> nvme_available_path() functions with check the
> "NVME_CTRL_FAILFAST_EXPIRED" controller flag.
>
> Signed-off-by: Victor Gladkov <victor.gladkov at kioxia.com>
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> Reviewed-by: Hannes Reinecke <hare at suse.de>

Victor,

when we talked about what fast fail does, we discussed how it was 
different from SCSI's notion of fast fail, which was in support of 
mpio.  nvme didn't need scsi's fast_fail behavior as the fabric check 
ready's are spotting the mpath attributes immediately. So this patch set 
is for "normal" io that normally enters retries.  You replied with a 
very important distinction about this patch that isn't described by the 
above.

Please add this paragraph to the patch description:

 From email from Victor on 12/3/2019 at 2:04 am US PST:
Applications are expecting commands to complete with success or error 
within a certain timeout (30 seconds by default).  The NVMe host is 
enforcing that timeout while it is connected, never the less, during 
reconnection, the timeout is not enforced and commands may get stuck for 
a long period or even forever.

-- james