[PATCH] nvmet-rdma: Release connections synchronously

[Bart Van Assche]

On 5/14/23 03:38, Shinichiro Kawasaki wrote:
> Bart, thanks for the v2 patch. I applied to kernel v6.3 and ran blktests nvme
> test group with siw transport. Unfortunately, I observed test process hang at
> nvme/003 [1]. The kernel reported lockdep WARN of id_priv->handler_mutex. The
> patch adds the "from_cm_handler" argument to __nvmet_rdma_queue_disconnect() to
> skip rdma_destroy_id() if the call is from CM handler. However,
> rdma_destroy_id() is still called in the call path below:
> 
> __nvmet_rdma_queue_disconnect()
>    nvmet_rdma_free_queue()
>      nvmet_rdma_destroy_queue_ib()
>        rdma_destroy_id()
> 
> So I think, the from_cm_handler should be added to nvmet_rdma_free_queue() and
> nvmet_rdma_destroy_queue_ib() also, so that from_cm_handler is propagated and
> rdma_destroy_id() is not called in nvmet_rdma_destroy_queue_ib(). I created a
> quick patch with this approach [2], and observed it avoids the process hang.
> 
> However, when I did further test using the v2 patch and the fix [1], I observed
> another process hang at nvme/009 [3]. This time, deadlock of subsys->lock
> happened in nvmet_ctrl_free(). I created a dirty patch to avoid it [4], but it
> caused lockdep WARN for possible deadlock between subsys->lock and
> id_priv->handler_mutex. I also observed hang at nvme/044 related
> nvmet_fatal_error_handler.

Hi Shinichiro,

We must be running different tests since the issues discovered by your 
tests did not show up in my tests. Anyway, I think this is a good time 
to ask for feedback from the NVMe maintainers. The only approach I see 
to suppress the lockdep complaints triggered by test nvme/003 about the 
NVMe RDMA transport driver is to something like patch [4]: track the 
owner of subsys->lock, find all code paths that could trigger recursive 
locking and make sure that recursive locking is not triggered. Does 
everyone agree with this approach?

Thanks,

Bart.