Seeking for help with NVMe arbitration questions

[Keith Busch]

On Thu, Apr 27, 2023 at 02:21:21PM -0700, Wang Yicheng wrote:
> Thanks a lot for the prompt and detailed reply!
> 
> Before asking for further clarification of your answers (thanks for
> being patient with me :) ), I'd like to sort out one fundamental
> question first. When we're talking about configuring the queues
> (default/read/poll), is it really the software queue or the hardware
> queue in the blk-mq model (figure 5 in this paper:
> https://kernel.dk/blk-mq.pdf)?

We're talking about hardware queues.
 
> I've always been understanding the configuration procedure as
> distributing that number of queues among the CPUs. But as you pointed
> out, if I only distribute 1 queue as default, then the cpu_list will
> contain all CPUs. It gives me an impression that the queue type is
> really about the hardware queues. We don't need to set up the software
> queues, as they're automatically affinitized to the underlying
> hardware queues, in order to make sure all submitted IOs, whichever
> the queue type is, don't need to transfer across CPUs. In other words,
> if I allocate all 3 types of queues, there will be at least 3 software
> queues in each CPU, corresponding to each queue type.

Each CPU has one software context, and each of those have 3 possible
hardware queues. Any given software context may be sharing one or
more of its hardware queues with another software context.

The set of hardware queues of any given type is fully affinitized
to every CPU you can run on.