[PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets

[Hannes Reinecke]

On 5/13/25 21:24, Kamaljit Singh wrote:
> Hi Sagi, Hannes,
> 
> On 09/11/2025 02:11, Sagi Grimberg wrote:
>>> IO timeouts are still occurring with Writes. The only Read that timed
>>> out was most likely due to the path error. It takes ~4.5 hours to fail.
>>>
>>> However, this test does not fail if either ECN is off or if digests
>>> are not enabled. These passing combinations were run for 16+ hours
>>> without any issues. Both ECN and Header+Data Digests need to be turned
>>> on for it to fail.
>>>
>>> Do you have a failing test as well? If so, is it quicker to cause the
>>> failure? Would you mind sharing any details?
>>>
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode 0x2 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout
>>>      [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O
>>>      [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...
>>>
>>> In the current build I had these patches on top of the "nvme-6.16" branch:
>>>      41b2c90a51bd nvme-tcp: sanitize request list handling
>>>      9260acd6c230 nvme-tcp: fix I/O stalls on congested sockets
>>
>> Kamaljit, with the prior version of the patchset (the proposal with the
>> wake_sender flag) did this not reproduce regardless of ECN?
>   
> With the last patchset, when ECN=off, we did not see any IO timeouts
> even with a weekend long test. This was true for both cases, i.e. with
> Inband Auth and with SecureConcat.
> 
> With ECN=on & HD+DD=on IO timeout still persists for both Inband Auth &
> SC. I’m currently debugging a possible target side issue with ECN. I’ll
> let you know once I have some resolution.
> 
> I don't have any clear indications of the original kernel issue to be able to
> differentiate against the current target-side issue. So, if you want to go
> ahead and merge those two patchsets that may be fine for now.
> 
Thanks a lot for your confirmation.
We continue to have issues with high load or oversubscribed fabrics.
But this patchset is addressing the problem of I/O timeouts during
_connect_, which I would argue is a different story.


Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare at suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich