[PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Sagi Grimberg]

On 02/09/2026 9:24, Saravanan D wrote:
> On Mon, 31 Aug 2026 01:30:08 +0300 Sagi Grimberg <sagi at grimberg.me> wrote:
>> I just think that the adoption approach has downsides in many other
>> workloads. I wouldn't categorize your use-case as the common use-case.
>> Not to say that it shouldn't be solved.
> Adoption in v2 is opt in per controller through the io_cpu_adopt
> connect option, so every workload that does not enable it
> keeps today's behavior unchanged. The downsides would only be borne by
> deployments that choose it. I concur with the broader point. Adoption
> makes the driver itself decide where socket work belongs, through a
> fixed rule that cannot fit every deployment, while a writable io_cpu
> is very flexible and leaves the decision to the control plane.
>
>> I agree with you that queue io_cpu selection at connection time is too
>> limiting. Perhaps we can have each nvme-tcp queue expose a sysfs entry
>> to set the io_cpu?
>>
>> Something like:
>> echo $CPU > /sys/class/nvme/nvme0/tcp_queues/$QUEUE/io_cpu
>>
>> I am just trying to think of a way that will allow some control plane
>> to set it exactly as it wants, without the guessing part...
> The per queue sysfs io_cpu works for us, and in one respect it works
> better than a connect option because a sysfs attribute can be set after
> the fact on any existing connection. The control plane already knows each
> VM's cpuset and can read the queue to cpu grouping from mq/*/cpu_list,
> so it can nail the socket work placement.
>
> Two questions before I spin the next version around this.
>
> 1. On reconnect the queues come back with the connect time io_cpu, so
> the control plane has to notice and reapply its placement. Is relying
> on the existing nvme udev events for controller state the intended
> signal, or would you want a dedicated notification ?

I think that io_cpu should not change in reconnect. We can use a
NVME_TCP_Q_ flag for it.

>
> 2. What should the write path enforce ? I would validate the cpu is
> online, rebalance the nvme_tcp_cpu_queues accounting, and let the new
> value take effect from the next queued io_work. I intend to restrict the
> value to the queue's map group.

I'd just do basic validation that override the queue io_cpu.. perhaps
xchg(queue->io_cpu, cpu).

>
> If this direction sounds right I will drop the adoption heuristic and
> send the next version v3 exposing each queue's io_cpu as a writeable
> sysfs attribute.
>
> Thanks,
> Saravanan D.