Unable to reconnect namespace via NVMe/TCP

[Yi Zhang]

Hi Hannes

I tried with the upstream kernel v6.17-rc2, and it can still be reproduced.

# dmesg | tail -30
[  219.560691] nvme nvme0: Failed reconnect attempt 3/-1
[  219.565784] nvme nvme0: Reconnecting in 10 seconds...
[  229.795215] nvme nvme0: failed to connect socket: -111
[  229.800369] nvme nvme0: Failed reconnect attempt 4/-1
[  229.805450] nvme nvme0: Reconnecting in 10 seconds...
[  240.034918] nvme nvme0: failed to connect socket: -111
[  240.040093] nvme nvme0: Failed reconnect attempt 5/-1
[  240.045165] nvme nvme0: Reconnecting in 10 seconds...
[  250.274619] nvme nvme0: failed to connect socket: -111
[  250.279776] nvme nvme0: Failed reconnect attempt 6/-1
[  250.284855] nvme nvme0: Reconnecting in 10 seconds...
[  260.514102] nvme nvme0: failed to connect socket: -111
[  260.519261] nvme nvme0: Failed reconnect attempt 7/-1
[  260.524340] nvme nvme0: Reconnecting in 10 seconds...
[  270.754031] nvme nvme0: failed to connect socket: -111
[  270.759184] nvme nvme0: Failed reconnect attempt 8/-1
[  270.764263] nvme nvme0: Reconnecting in 10 seconds...
[  280.993410] nvme nvme0: failed to connect socket: -111
[  280.998591] nvme nvme0: Failed reconnect attempt 9/-1
[  281.003653] nvme nvme0: Reconnecting in 10 seconds...
[  291.249090] nvme nvme0: creating 4 I/O queues.
[  291.264959] nvme nvme0: mapped 4/0/0 default/read/poll queues.
[  291.271975] nvme nvme0: Successfully reconnected (attempt 10/-1)
[  291.273897] nvme nvme0: identifiers changed for nsid 2
[  291.283631] block nvme0n1: no available path - failing I/O
[  291.289139] block nvme0n1: no available path - failing I/O
[  291.294649] block nvme0n1: no available path - failing I/O
[  291.300159] block nvme0n1: no available path - failing I/O
[  291.305665] block nvme0n1: no available path - failing I/O
[  291.311197] block nvme0n1: no available path - failing I/O

On Tue, Aug 19, 2025 at 2:11 PM Hannes Reinecke <hare at suse.de> wrote:
>
> On 8/12/25 17:48, Anton Gavriliuk wrote:
> > Hi
> >
> > There are NVMe/TCP target and initiator servers, both running on
> > RHEL10 (6.12.0-55.25.1.el10_0.x86_64)
> >
> > NVMe/TCP target exports single NVMe SSD
> >
> >        "namespaces": [
> >          {
> >            "device": {
> >              "nguid": "01000000-0000-0000-8ce3-8ee3064aa4f2",
> >              "path": "/dev/nvme0n1"
> >            },
> >            "enable": 1,
> >            "nsid": 1
> >          }
> >        ],
> >
> > If NVMe/TCP target is not available, initiator tries to reconnect
> > every 10 seconds
> >
> > [ 2586.071048] nvme nvme9: failed to connect socket: -111
> > [ 2586.071403] nvme nvme9: Failed reconnect attempt 16/-1
> > [ 2586.071565] nvme nvme9: Reconnecting in 10 seconds...
> > [ 2596.310921] nvme nvme9: failed to connect socket: -111
> > [ 2596.311186] nvme nvme9: Failed reconnect attempt 17/-1
> > [ 2596.311349] nvme nvme9: Reconnecting in 10 seconds...
> > [ 2606.550772] nvme nvme9: failed to connect socket: -111
> > [ 2606.551252] nvme nvme9: Failed reconnect attempt 18/-1
> > [ 2606.551592] nvme nvme9: Reconnecting in 10 seconds...
> >
> > when NVMe/TCP target become available, initiator failed reconnect the namespace
> >
> > [ 2606.551592] nvme nvme9: Reconnecting in 10 seconds...
> > [ 2616.793080] nvme nvme9: creating 16 I/O queues.
> > [ 2616.829881] nvme nvme9: mapped 16/0/0 default/read/poll queues.
> > [ 2616.833685] nvme nvme9: Successfully reconnected (attempt 19/-1)
> > [ 2616.834446] nvme nvme9: identifiers changed for nsid 1
> > [ 2616.835618] block nvme9n1: no usable path - requeuing I/O
> > [ 2616.856602] block nvme9n1: no available path - failing I/O
> > [ 2616.856811] block nvme9n1: no available path - failing I/O
> >
> > and there is no nvme9n1 namespace in the "nvme list" output.
> >
> This looks like the missed re-scan issue I found recently.
> Should be fixed with
> 9546ad1a9bda ("nvme: requeue namespace scan on missed AENs")
>
> (And you are running RHEL. Please open a bugzilla with RH.)
> (And why am I even answering that?)
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                  Kernel Storage Architect
> hare at suse.de                                +49 911 74053 688
> SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
> HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
>


--
Best Regards,
  Yi Zhang