[PATCH 5/6] nvme-rdma: fix timeout handler

[Chao Leng]

On 2020/8/5 15:19, Sagi Grimberg wrote:
> 
>>>>> The request being timed out cannot be completed after the queue is
>>>>> stopped, that is the point of nvme_rdma_stop_queue. if it is only
>>>>> ALLOCATED, we did not yet connect hence there is zero chance for
>>>>> any command to complete.
>>>> The request may already complete before stop queue, it is in the cq, but
>>>> is not treated by software.
>>>
>>> Not possible, ib_drain_cq completion guarantees that all cqes were
>>> reaped and handled by SW.
>>>
>>>> If nvme_rdma_stop_queue concurrent
>>>
>>> Before we complete we make sure the queue is stopped (and drained and
>>> reaped).
>>>
>>> , for
>>>> example:
>>>> The error recovery run first, it will clear the flag:NVME_RDMA_Q_LIVE,
>>>> and then wait drain cq. At the same time nvme_rdma_timeout
>>>> call nvme_rdma_stop_queue will return immediately, and then may call
>>>> blk_mq_complete_request. but error recovery may drain cq at the same
>>>> time, and may also treat the same request.
>>>
>>> We flush the err_work before running nvme_rdma_stop_queue exactly
>>> because of that. your example cannot happen.
>> Flush work is not safe. See my previous email.
> 
> How is it not safe? when flush_work returns, the work is guaranteed
> to have finished execution, and we only do that for states
> RESETTING/CONNECTING which means that it either has already started
> or already finished.

Though the state is NVME_CTRL_RESETTING, but it does not mean the work
is already queued(started) or finished. There is a hole between Change state
and queue work.

Like this:
static void nvme_rdma_error_recovery(struct nvme_rdma_ctrl *ctrl)
{
     if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING))
         return;
--------------------------------
may interrupt by hard interrupt, and then timeout progress flush work
at this time. Thus error recovery and nvme_rdma_complete_timed_out may
concurrent to stop queue. will cause: error recovery may cancel request
or nvme_rdma_complete_timed_out may complete request, but the queue may
not be stoped. Thus will cause abnormal.
--------------------------------
     queue_work(nvme_reset_wq, &ctrl->err_work);
}

Another, although the probability of occurrence is very low, reset work
and nvme_rdma_complete_timed_out may also concurrent to stop queue, may
also cause abnormal.