[Question] Configuring CPU Cores for NVMe Target & Host in Linux Kernel

[Keith Busch]

On Tue, Jun 04, 2024 at 11:10:30PM +0200, Jigao Luo wrote:
> I am benchmarking NVMe over Fabrics Target and Host using both the Linux
> kernel driver and the SPDK driver. In SPDK, there is an option to specify a
> core mask that limits the CPU cores on which SPDK can execute work items.
> The SPDK team provides a benchmark report that demonstrates the core
> scalability of NVMe over Fabrics Target and Host: https://ci.spdk.io/download/performance-reports/SPDK_rdma_mlx_perf_report_2401.pdf
> 
> I am curious if there is a similar parameter available in the Linux kernel
> target/host driver. I have not been able to find such a parameter. Can you
> confirm if it is possible to configure the number of cores used by the Linux
> kernel NVMe driver? If this configuration is not available, could you
> provide me guidance on understanding core scalability in the Linux
> implementation?

You're using RDMA, so you can assign CPU completion interrupts however
you like through the /proc/irq/*/smp_affinity interface.

On the submission side, you can steer user space programs to some subset
of CPUs with the "taskset -c" command.

For the kernel side, you can mostly set which CPUs can dispatch IO with
the "isolcpus=" kernel parameter.