[PATCH] nvme-tcp: Don't update queue count when failing to set io queues

[liruozhu]

On 2021/8/14 4:02, Keith Busch wrote:

> On Sat, Aug 07, 2021 at 11:50:23AM +0800, Ruozhu Li wrote:
>> We update ctrl->queue_count and schedule another reconnect when io queue
>> count is zero.But we will never try to create any io queue in next reco-
>> nnection, because ctrl->queue_count already set to zero.We will end up
>> having an admin-only session in Live state, which is exactly what we try
>> to avoid in the original patch.
>> Update ctrl->queue_count after queue_count zero checking to fix it.
> I think this patch fixes more than just an admin-only session. We've observed
> an issue where a subsequent reconnect has the wrong queue_count and tries to
> use uninitialized sockets that leads to the below panic. We have not seen the
> panic with this patch included since it reconfigures the IO queues, so I feel
> this patch is good for either 5.14 or stable.
Agree. Eventually this problem will cause nr_hw_queue and the real hardware
queue number to be inconsistent, and lead to the use of uninitialized 
resources.
I also found a crash using rdma, because driver detected that nr_hw_queue is
greater than ctrl.queue_count. This patch(rdma version) also solves that 
problem.
--
BUG_ON(hctx_idx >= ctrl->ctrl.queue_count); // nvme_rdma_init_hctx
--
Call trace:
nvme_rdma_init_hctx+0x58/0x60 [nvme_rdma]
blk_mq_realloc_hw_ctxs+0x140/0x4c0
blk_mq_init_allocated_queue+0x130/0x410
blk_mq_init_queue+0x40/0x88
nvme_validate_ns+0xb8/0x740
nvme_scan_work+0x29c/0x460
process_one_work+0x1f8/0x490
worker_thread+0x50/0x4b8
kthread+0x134/0x138
ret_from_fork+0x10/0x18
--

Thanks,
Ruozhu
>   [00:30:04 2021] nvme nvme3: queue 0: timeout request 0x16 type 4
>   [00:30:04 2021] nvme nvme3: Could not set queue count (881)
>   [00:30:04 2021] nvme nvme3: unable to set any I/O queues
>   [00:30:04 2021] nvme nvme3: Failed reconnect attempt 4
>   [00:30:04 2021] nvme nvme3: Reconnecting in 10 seconds...
>   [00:30:15 2021] nvme nvme3: queue_size 128 > ctrl sqsize 16, clamping down
>   [00:30:15 2021] BUG: kernel NULL pointer dereference, address: 00000000000000a0
>   [00:30:15 2021] #PF: supervisor read access in kernel mode
>   [00:30:15 2021] #PF: error_code(0x0000) - not-present page
>   [00:30:15 2021] PGD 0 P4D 0
>   [00:30:15 2021] Oops: 0000 [#1] SMP NOPTI
>   [00:30:15 2021] CPU: 0 PID: 3776 Comm: kworker/0:2H Not tainted 5.12.9 #1
>   [00:30:15 2021] Hardware name: Dell Inc. PowerEdge R7525/0YHMCJ, BIOS 2.0.3 01/15/2021
>   [00:30:15 2021] Workqueue: nvme_tcp_wq nvme_tcp_io_work [nvme_tcp]
>   [00:30:15 2021] RIP: 0010:kernel_sendpage+0x16/0xc0
>   [00:30:15 2021] Code: 5d c3 48 63 03 45 31 e4 48 89 1c c5 20 b1 55 a7 eb cc 66 90 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc 48 83 ec 18 48 8b 47 20 <4c> 8b 88 a0 00 00 00 4d 85 c9 74 3d 48 8b 7e 08 48 8d 47 ff 83 e7
>   [00:30:15 2021] RSP: 0018:ffffa8e90aef3d60 EFLAGS: 00010286
>   [00:30:15 2021] RAX: 0000000000000000 RBX: ffff98bd601ca130 RCX: 0000000000000048
>   [00:30:15 2021] RDX: 0000000000000bc8 RSI: ffffd241a3c577c0 RDI: ffff98c4c64a3400
>   [00:30:15 2021] RBP: ffffa8e90aef3d80 R08: 0000000000028040 R09: ffff98bd9892b998
>   [00:30:15 2021] R10: ffff98bd9892b498 R11: 000000000000000c R12: ffff98c4c64a3400
>   [00:30:15 2021] R13: 0000000000000048 R14: 0000000000000000 R15: ffff98bd9892b998
>   [00:30:15 2021] FS:  0000000000000000(0000) GS:ffff98c49e800000(0000) knlGS:0000000000000000
>   [00:30:15 2021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>   [00:30:15 2021] CR2: 00000000000000a0 CR3: 00000008901fc000 CR4: 0000000000350ef0
>   [00:30:15 2021] Call Trace:
>   [00:30:15 2021]  ? tcp_read_sock+0x199/0x270
>   [00:30:15 2021]  nvme_tcp_try_send+0x160/0x7f0 [nvme_tcp]
>   [00:30:15 2021]  ? _raw_spin_unlock_bh+0x1e/0x20
>   [00:30:15 2021]  ? release_sock+0x8f/0xa0
>   [00:30:15 2021]  ? nvme_tcp_try_recv+0x74/0xa0 [nvme_tcp]
>   [00:30:15 2021]  nvme_tcp_io_work+0x81/0xd0 [nvme_tcp]
>   [00:30:15 2021]  process_one_work+0x220/0x3c0
>   [00:30:15 2021]  worker_thread+0x4d/0x3f0
>   [00:30:15 2021]  kthread+0x114/0x150
>   [00:30:15 2021]  ? process_one_work+0x3c0/0x3c0
>   [00:30:15 2021]  ? kthread_park+0x90/0x90
>   [00:30:15 2021]  ret_from_fork+0x22/0x30
>
>> ---
>>   drivers/nvme/host/tcp.c | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
>> index 8cb15ee5b249..18bd68b82d78 100644
>> --- a/drivers/nvme/host/tcp.c
>> +++ b/drivers/nvme/host/tcp.c
>> @@ -1769,13 +1769,13 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
>>   	if (ret)
>>   		return ret;
>>   
>> -	ctrl->queue_count = nr_io_queues + 1;
>> -	if (ctrl->queue_count < 2) {
>> +	if (nr_io_queues == 0) {
>>   		dev_err(ctrl->device,
>>   			"unable to set any I/O queues\n");
>>   		return -ENOMEM;
>>   	}
>>   
>> +	ctrl->queue_count = nr_io_queues + 1;
>>   	dev_info(ctrl->device,
>>   		"creating %d I/O queues.\n", nr_io_queues);
>>   
>> -- 
> .