[PATCH v5 0/2] nvme-tcp: allow setting per-queue io_cpu through sysfs
Saravanan D
saravanand at crusoe.ai
Sat Oct 17 18:00:57 PDT 2026
On hosts that partition their CPUs after connect time, the connect
time io_cpu selection can land a queue's socket work on CPUs owned by
a different workload. This series lets a control plane set each
queue's io_cpu directly through sysfs.
The first patch adds core helpers for a transport neutral
/sys/class/nvme/nvmeX/queues/<qid>/
directory per queue under the controller device. The second patch has
nvme-tcp expose io_cpu and managed attributes there.
Changes since v4 [1]:
- Rebased onto nvme-7.4, which sets up the I/O queues in parallel
(61b0ed3be81b) and serializes the io_cpu accounting with
nvme_tcp_cpu_queues_lock. The per controller io_cpu_lock is gone,
and the io_cpu writes take nvme_tcp_cpu_queues_lock instead.
- A queue's directory now follows the queue's allocation, added in
nvme_tcp_alloc_queue() and removed in nvme_tcp_free_queue(), so the
directories match the queue count after a reconnect (Nilay). The
core allocates each directory and its release frees only that
allocation, which drops the transport's reference counting and
completion (Nilay) and fixes the use after free, the controller
leak and the teardown hang found by the Sashiko review of v4 [2].
- The transport creates the queues directory once before the parallel
queue setup, and the kerneldoc states which calls may run
concurrently (Nilay).
- The user assignment flag moved from the core into the nvme-tcp
queue flags (Nilay).
- io_cpu is read through nvme_tcp_queue_io_cpu() and written through
nvme_tcp_queue_change_io_cpu(), which carries __must_hold() for the
context analysis (Nilay).
- The sysfs store collapsed to a single path and no longer checks
NVME_TCP_Q_ALLOCATED, since the directory only exists while the
queue is allocated (Nilay).
- The queue stop now clears NVME_TCP_Q_IO_CPU_SET under
nvme_tcp_cpu_queues_lock, since a store may move io_cpu between the
clear and the decrement.
- nvme_tcp_free_queue() drops NVME_TCP_Q_IO_CPU_SET, which also fixes
a ledger count leaked by every failed I/O queue connect.
- io_cpu parses a decimal cpu number, and the ABI entry describes the
directory lifetime, persistence across CPU offline and the behavior
with wq_unbound.
[1] https://lore.kernel.org/linux-nvme/20260927070925.47209-1-saravanand@crusoe.ai/
[2] https://sashiko.dev/#/patchset/20260927070925.47209-1-saravanand%40crusoe.ai
Saravanan D (2):
nvme: add per-queue sysfs directories
nvme-tcp: allow setting per-queue io_cpu through sysfs
Documentation/ABI/stable/sysfs-nvme | 29 +++++
drivers/nvme/host/core.c | 6 +
drivers/nvme/host/nvme.h | 23 ++++
drivers/nvme/host/sysfs.c | 88 ++++++++++++++
drivers/nvme/host/tcp.c | 173 ++++++++++++++++++++++++----
5 files changed, 299 insertions(+), 20 deletions(-)
base-commit: 2e8e30b3cf9f165e1a1d079331543cb7f2ad0e0c
--
2.55.0
More information about the Linux-nvme
mailing list