[PATCH 0/7] nvme-tcp scalability improvements

[Hannes Reinecke]

On 6/26/24 15:37, Sagi Grimberg wrote:
> 
> 
> On 26/06/2024 15:13, Hannes Reinecke wrote:
>> From: Hannes Reinecke <hare at suse.de>
>>
>> Hi all,
>>
>> we have had reports from partners that nvme-tcp suffers from scalability
>> problems with the number of controllers; they even managed to run into
>> a request timeout by just connecting enough controllers to the host.
>>
>> Looking into it I have found several issues with the nvme-tcp 
>> implementation:
>> - the 'io_cpu' assignment is static, leading to the same calculation
>>    for each controller. Thus each queue with the same number is
>>    assigned the same CPU, leading to CPU starvation.
>> - The blk-mq cpu mapping is not taken into account when calculating
>>    the 'io_cpu' number, leading to excessive thread bouncing during I/O
>> - The socket state is not evaluating, so we're piling more and more
>>    requests onto the socket even when it's already full.
>>
>> This patchset addresses these issues, leading to a better I/O
>> distribution for several controllers.
> 
> Hannes, Please quantify what every change gives us please. Each
> change on its own should present merit, otherwise we should consider
> weather it is actually needed.
> 
Oh, I do agree with you. It's just hard to quantify, as with the current
implementation we vastly favour writes over reads. If we try to balance
things out (as these patches do) read performance goes up significantly,
and write performance goes down. So one can't really say its a 
performance regression in general, we're just shifting things around.

I'll try to get some more meaningful data, but this is a hard task as
I first need to setup a scale-out target such that I can demonstrate the
issues more clearly.

In the end I've send out these patches primarily to show the direction 
I'm investigating in, in case other people want to play around with 
them, too.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Ivo Totev, Andrew McDonald,
Werner Knoblich