Unable to reconnect namespace via NVMe/TCP

[Anton Gavriliuk]

Hi Yi Zhang

> Please try to add the uuid in the device field, which should fix your issue.

On the NVMe/TCP target for the given device (/dev/nvme0n1) nguid and
uuid are exactly the same

[root at memverge4 ~]# cat /sys/class/block/nvme0n1/uuid
01000000-0000-0000-8ce3-8ee3064aa4f2
[root at memverge4 ~]# cat /sys/class/block/nvme0n1/nguid
01000000-0000-0000-8ce3-8ee3064aa4f2

So I added uuid

      "namespaces": [
        {
          "device": {
            "nguid": "01000000-0000-0000-8ce3-8ee3064aa4f2",
            "uuid": "01000000-0000-0000-8ce3-8ee3064aa4f2",
            "path": "/dev/nvme0n1"
          },
          "enable": 1,
           "nsid": 1
        }
      ],

Yes, this fixed my issue - automatically reconnect namespace after
NVMe/TCP target reboot.

Anton

вт, 19 авг. 2025 г. в 16:43, Yi Zhang <yi.zhang at redhat.com>:
>
> Hi Anton
>
> Please try to add the uuid in the device field, which should fix your issue.
>
> On Tue, Aug 19, 2025 at 6:55 PM Yi Zhang <yi.zhang at redhat.com> wrote:
> >
> > Hi Hannes
> >
> > I tried with the upstream kernel v6.17-rc2, and it can still be reproduced.
> >
> > # dmesg | tail -30
> > [  219.560691] nvme nvme0: Failed reconnect attempt 3/-1
> > [  219.565784] nvme nvme0: Reconnecting in 10 seconds...
> > [  229.795215] nvme nvme0: failed to connect socket: -111
> > [  229.800369] nvme nvme0: Failed reconnect attempt 4/-1
> > [  229.805450] nvme nvme0: Reconnecting in 10 seconds...
> > [  240.034918] nvme nvme0: failed to connect socket: -111
> > [  240.040093] nvme nvme0: Failed reconnect attempt 5/-1
> > [  240.045165] nvme nvme0: Reconnecting in 10 seconds...
> > [  250.274619] nvme nvme0: failed to connect socket: -111
> > [  250.279776] nvme nvme0: Failed reconnect attempt 6/-1
> > [  250.284855] nvme nvme0: Reconnecting in 10 seconds...
> > [  260.514102] nvme nvme0: failed to connect socket: -111
> > [  260.519261] nvme nvme0: Failed reconnect attempt 7/-1
> > [  260.524340] nvme nvme0: Reconnecting in 10 seconds...
> > [  270.754031] nvme nvme0: failed to connect socket: -111
> > [  270.759184] nvme nvme0: Failed reconnect attempt 8/-1
> > [  270.764263] nvme nvme0: Reconnecting in 10 seconds...
> > [  280.993410] nvme nvme0: failed to connect socket: -111
> > [  280.998591] nvme nvme0: Failed reconnect attempt 9/-1
> > [  281.003653] nvme nvme0: Reconnecting in 10 seconds...
> > [  291.249090] nvme nvme0: creating 4 I/O queues.
> > [  291.264959] nvme nvme0: mapped 4/0/0 default/read/poll queues.
> > [  291.271975] nvme nvme0: Successfully reconnected (attempt 10/-1)
> > [  291.273897] nvme nvme0: identifiers changed for nsid 2
> > [  291.283631] block nvme0n1: no available path - failing I/O
> > [  291.289139] block nvme0n1: no available path - failing I/O
> > [  291.294649] block nvme0n1: no available path - failing I/O
> > [  291.300159] block nvme0n1: no available path - failing I/O
> > [  291.305665] block nvme0n1: no available path - failing I/O
> > [  291.311197] block nvme0n1: no available path - failing I/O
> >
> > On Tue, Aug 19, 2025 at 2:11 PM Hannes Reinecke <hare at suse.de> wrote:
> > >
> > > On 8/12/25 17:48, Anton Gavriliuk wrote:
> > > > Hi
> > > >
> > > > There are NVMe/TCP target and initiator servers, both running on
> > > > RHEL10 (6.12.0-55.25.1.el10_0.x86_64)
> > > >
> > > > NVMe/TCP target exports single NVMe SSD
> > > >
> > > >        "namespaces": [
> > > >          {
> > > >            "device": {
> > > >              "nguid": "01000000-0000-0000-8ce3-8ee3064aa4f2",
> > > >              "path": "/dev/nvme0n1"
> > > >            },
> > > >            "enable": 1,
> > > >            "nsid": 1
> > > >          }
> > > >        ],
> > > >
> > > > If NVMe/TCP target is not available, initiator tries to reconnect
> > > > every 10 seconds
> > > >
> > > > [ 2586.071048] nvme nvme9: failed to connect socket: -111
> > > > [ 2586.071403] nvme nvme9: Failed reconnect attempt 16/-1
> > > > [ 2586.071565] nvme nvme9: Reconnecting in 10 seconds...
> > > > [ 2596.310921] nvme nvme9: failed to connect socket: -111
> > > > [ 2596.311186] nvme nvme9: Failed reconnect attempt 17/-1
> > > > [ 2596.311349] nvme nvme9: Reconnecting in 10 seconds...
> > > > [ 2606.550772] nvme nvme9: failed to connect socket: -111
> > > > [ 2606.551252] nvme nvme9: Failed reconnect attempt 18/-1
> > > > [ 2606.551592] nvme nvme9: Reconnecting in 10 seconds...
> > > >
> > > > when NVMe/TCP target become available, initiator failed reconnect the namespace
> > > >
> > > > [ 2606.551592] nvme nvme9: Reconnecting in 10 seconds...
> > > > [ 2616.793080] nvme nvme9: creating 16 I/O queues.
> > > > [ 2616.829881] nvme nvme9: mapped 16/0/0 default/read/poll queues.
> > > > [ 2616.833685] nvme nvme9: Successfully reconnected (attempt 19/-1)
> > > > [ 2616.834446] nvme nvme9: identifiers changed for nsid 1
> > > > [ 2616.835618] block nvme9n1: no usable path - requeuing I/O
> > > > [ 2616.856602] block nvme9n1: no available path - failing I/O
> > > > [ 2616.856811] block nvme9n1: no available path - failing I/O
> > > >
> > > > and there is no nvme9n1 namespace in the "nvme list" output.
> > > >
> > > This looks like the missed re-scan issue I found recently.
> > > Should be fixed with
> > > 9546ad1a9bda ("nvme: requeue namespace scan on missed AENs")
> > >
> > > (And you are running RHEL. Please open a bugzilla with RH.)
> > > (And why am I even answering that?)
> > >
> > > Cheers,
> > >
> > > Hannes
> > > --
> > > Dr. Hannes Reinecke                  Kernel Storage Architect
> > > hare at suse.de                                +49 911 74053 688
> > > SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
> > > HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
> > >
> >
> >
> > --
> > Best Regards,
> >   Yi Zhang
>
>
>
> --
> Best Regards,
>   Yi Zhang
>