[bug report] nvme/rdma: nvme connect failed after offline one cpu on host side

[Ming Lei]

On Tue, Jul 05, 2022 at 02:04:53AM +0300, Sagi Grimberg wrote:
> 
> > update the subject to better describe the issue:
> > 
> > So I tried this issue on one nvme/rdma environment, and it was also
> > reproducible, here are the steps:
> > 
> > # echo 0 >/sys/devices/system/cpu/cpu0/online
> > # dmesg | tail -10
> > [  781.577235] smpboot: CPU 0 is now offline
> > # nvme connect -t rdma -a 172.31.45.202 -s 4420 -n testnqn
> > Failed to write to /dev/nvme-fabrics: Invalid cross-device link
> > no controller found: failed to write to nvme-fabrics device
> > 
> > # dmesg
> > [  781.577235] smpboot: CPU 0 is now offline
> > [  799.471627] nvme nvme0: creating 39 I/O queues.
> > [  801.053782] nvme nvme0: mapped 39/0/0 default/read/poll queues.
> > [  801.064149] nvme nvme0: Connect command failed, error wo/DNR bit: -16402
> > [  801.073059] nvme nvme0: failed to connect queue: 1 ret=-18
> 
> This is because of blk_mq_alloc_request_hctx() and was raised before.
> 
> IIRC there was reluctance to make it allocate a request for an hctx even
> if its associated mapped cpu is offline.
> 
> The latest attempt was from Ming:
> [PATCH V7 0/3] blk-mq: fix blk_mq_alloc_request_hctx
> 
> Don't know where that went tho...

The attempt relies on that the queue for connecting io queue uses
non-admined irq, unfortunately that can't be true for all drivers,
so that way can't go.

So far, I'd suggest to fix nvme_*_connect_io_queues() to ignore failed
io queue, then the nvme host still can be setup with less io queues.

Otherwise nvme_*_connect_io_queues() could fail easily, especially for
1:1 mapping.


Thanks,
Ming