[PATCH] nvme-tcp: align I/O cpu with blk-mq mapping

[Hannes Reinecke]

On 6/19/24 07:30, Christoph Hellwig wrote:
> On Tue, Jun 18, 2024 at 02:03:45PM +0200, Hannes Reinecke wrote:
>> Add a new module parameter 'wq_affinity' to spread the I/O
>> over all cpus within the blk-mq hctx mapping for the queue.
>> This avoids bouncing I/O between cpus when we have less
>> hardware queues than cpus.
> 
> What is the benefit when setting it?  What is the downside?  Why do you
> think it needs to be conditional?
> 
We could make it the default option, as the current way of just cycling
through the cpu_online_mask() works only for trivial cpu configurations 
(ie not on AMD).
Downsides are none; at worst we're ending up with the same behaviour
which we have now (ie scheduling I/O on a cpu which is not part of the
blk-mq hwctx map).

>> +	}
>>   	if (wq_unbound)
>>   		queue->io_cpu = WORK_CPU_UNBOUND;
>> +	else if (wq_affinity) {
> 
> Missing curly braces.  But this also means the wq_unbound options
> is incompatible with your new one.
> 
Correct. The options are mutually incompatible.
I was looking into merging them, but then 'wq_unbound' is already 
existing and I wasn't sure if one can replace existing module options.

>> +	} else
>>   		queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false);
> 
> Overly long line here.
> 
Yeah, sorry.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Ivo Totev, Andrew McDonald,
Werner Knoblich