[PATCH 0/3] nvme-cli: NIC topology aware I/O queue scaling

[Sagi Grimberg]

On 24/08/2026 11:48, Nilay Shroff wrote:
> On 8/23/26 3:13 AM, Sagi Grimberg wrote:
>>
>>
>> On 21/08/2026 17:43, Nilay Shroff wrote:
>>> Hi,
>>>
>>> This series is a rework of the earlier patchset[1]. The main
>>> difference is that --nr-io-queues is now calculated in nvme-cli
>>> instead of in the kernel when establishing an NVMe/TCP connection.
>>>
>>> This rework is based on the feedback received[2] from the netdev
>>> maintainers.
>>>
>>> The original patchset determined the number of NVMe/TCP I/O queues
>>> based on the number of online CPUs and the number of hardware queues
>>> available on the NIC in kernel driver. This series moves that logic
>>> to nvme-cli.
>>>
>>> When --nr-io-queues is not explicitly specified, nvme-cli determines
>>> the egress netdev for the NVMe/TCP connection, retrieves its current
>>> hardware queue count, and calculates the default as:
>>>
>>>     min(nr_hw_queues, num_online_cpus)
>>
>> This looks reasonable Nilay.
>>
> Thank you...
>
>> I am wandering tho if we want to place some lower limit here.
>> For example, my laptop has a virtio device with 4 cpu cores and
>> a single combined ring:
>> -- 
>> $ lscpu | grep NUMA
>> NUMA node(s):                            1
>> NUMA node0 CPU(s):                       0-3
>> $ ethtool -l enp7s0
>> Channel parameters for enp7s0:
>> Pre-set maximums:
>> RX:        n/a
>> TX:        n/a
>> Other:        n/a
>> Combined:    1
>> Current hardware settings:
>> RX:        n/a
>> TX:        n/a
>> Other:        n/a
>> Combined:    1
>> -- 
>>
>> It would be kinda annoying for me to now explicitly pass the 
>> nr-io-queues...
>> I am wandering if some sort of threshold make sense as what you are 
>> aiming for
>> is reducing the amount of queues for large cpu counts...
>
> I think you're running a QEMU guest using user-mode (SLIRP) 
> networking, so having
> a combined queue count of 1 is expected.
>
> I also tested this setup before posting the change. With QEMU 
> user-mode networking,
> increasing --nr-io-queues beyond 1 (I tried 4 and 8 with vCPU set to 
> match those
> numbers) did not improve performance. In fact, limiting --nr-io-queues 
> to 1, which
> matches the netdev's single combined queue, gave slightly better 
> performance.
>
> My understanding is that in this topology there is only a single 
> underlying
> virtqueue/network queue, so creating multiple NVMe/TCP I/O queues does 
> not provide
> additional network parallelism. Instead, those NVMe/TCP queues end up 
> contending
> on the same virtqueue/network queue, which can add overhead without 
> providing additional
> throughput.

I don't care about performance. I care that if I am testing stuff, I 
want more than a single
queue. And it is annoying to explicitly change the queue count...

Also, I don't know if your performance statements are correct for TLS.