[PATCH RFC] nvme-tcp: allow multiple queues per hctx

[Sagi Grimberg]

On 03/09/2026 18:26, Keith Busch wrote:
> From: Keith Busch<kbusch at kernel.org>
>
> We've observed many workloads may burst large amounts of IO from a one
> thread, which has the nvme tcp driver queue all the work onto a single
> socket and serialized by a single CPU. We can hit a CPU bottleneck
> before saturating the link when this happens.
>
> Provide an option to ask for more queues for each blk-mq hctx. The
> option is provided in the nvme-fabrics connection string by appending
> "queues_per_hctx=N", where N is the number of queues you want to share
> each hctx. The nvme tcp driver will group those queues under that hctx
> and perform a simple round-robin to dispatch the requests and subsequent
> work. The workqueue provided for this is always unbounded, and we
> provide the CPU hint from the original dispatching CPU in order to
> maintain locality for the completion side.
>
> The default is the pre-existing 1 queue per hctx, so this patch should
> be a no-op if you don't explicitly opt into this feature.
>
> Testing with a very capable NIC, a high bandwith workload of QD16 1MB
> reads improved from 7GB/s on the default settings to 24GB/s with 4
> queues per hctx.
>
> For a more IOPs intensive workload, a QD64 4k random read test improved
> from 184k to 252k.
>
> This also gets a tighter grouping on application observed latency,
> significantly bringing in the P99 outliers.
>
> On the down side, there is a minor regression on low depth workloads.

Hmm... While this is a very meaningful boost, we are adding a new IO 
scheduling
logic really deep down at the driver...

Most (though not all) the nvme-tcp implementations that I am aware of, 
usually
avoid this problem by creating a round-robin/queue-depth multipath to 
multiple
controllers (which are actually separate controllers)...

I'm wandering if you guys can do the same --duplicate-connect and setting
mpath with round-robin/queue-depth would get you the same result? Perhaps
we can add this flag to nvme discover/connect-all for simplicity?

BTW, how many cpus/queues do you have in your test keith?

Moreover, I suspect that this is not a problem that is specific for 
nvme-tcp...
I'm not sure creating more 4x queues is the right tool for solving this
single-threaded workload issue. I would expect that the overall number
of connections remain, and nr_hw_queues reducing by a factor of 
queue_per_hctx.

I am wandering if there is any desire to have the block layer adding 
something like hctx
groups which would share a tag space but have multiple hctxs?