[Question] Configuring CPU Cores for NVMe Target & Host in Linux Kernel

[Jigao Luo]

Hi Keith,

Thank you for your response.

My question is more focused on the Linux NVMe-oF RDMA target side. 
Specifically, is there a parameter for e.g. `modprobe nvmet-rdma` (or 
similar approach) that allows configuring the number of cores the 
NVMe-oF RDMA target can run on? Additionally, how can we benchmark and 
collect data on the number of kernel cycles used solely for the NVMe-oF 
RDMA target? I understand isolcpus, but want to find a way only for 
NVMe-oF RDMA target.

In comparison, SPDK, due to its user-space properties, can be easily 
configured with the number of cores and benchmarked to collect CPU cycles.

Thank you for your assistance.

Best regards,
Jigao

On 6/4/24 11:37 PM, Keith Busch wrote:
> On Tue, Jun 04, 2024 at 11:10:30PM +0200, Jigao Luo wrote:
>> I am benchmarking NVMe over Fabrics Target and Host using both the Linux
>> kernel driver and the SPDK driver. In SPDK, there is an option to specify a
>> core mask that limits the CPU cores on which SPDK can execute work items.
>> The SPDK team provides a benchmark report that demonstrates the core
>> scalability of NVMe over Fabrics Target and Host: https://ci.spdk.io/download/performance-reports/SPDK_rdma_mlx_perf_report_2401.pdf
>>
>> I am curious if there is a similar parameter available in the Linux kernel
>> target/host driver. I have not been able to find such a parameter. Can you
>> confirm if it is possible to configure the number of cores used by the Linux
>> kernel NVMe driver? If this configuration is not available, could you
>> provide me guidance on understanding core scalability in the Linux
>> implementation?
> You're using RDMA, so you can assign CPU completion interrupts however
> you like through the /proc/irq/*/smp_affinity interface.
>
> On the submission side, you can steer user space programs to some subset
> of CPUs with the "taskset -c" command.
>
> For the kernel side, you can mostly set which CPUs can dispatch IO with
> the "isolcpus=" kernel parameter.