nvme-tcp: io queue shutdown ordering

[Sagi Grimberg]

> When disconnecting an nvme-tcp device using nvme cli (i.e., nvme
> disconnect), nvme_tcp_teardown_ctrl() destroys I/O queues before
> issuing the shutdown notification on the admin queue.

Yes, same for rdma, and also pci.

> According to the fabrics spec version 1.1a section 1.5.2, an
> association is terminated if an NVMe Transport connection is lost
> between the host and controller for any I/O Queue when the host or
> controller does not support individual I/O Queue deletion.

The connection lost implies that the controller (or host) is not
reachable I think. disconnecting a controller is an explicit
operation triggered, and an orderly shutdown done (or at least
attempted). But none the less, the association is being terminated.

The message in the paragraph that you are referring to was designed
to articulate that there is no mode where the controller will continue
to function and accept I/O with a subset of the queues available which 
requires logic for I/O failover within queues of a controller ala iscsi
erl=2. Instead the spec says something like "if a connectivity issue was
detected on any of the queues, just terminate the entire association".

An orderly teardown/disconnection does not qualify as a connection loss.

> The initiator in mainline does not advertise support for I/O queue
> deletion in the fabrics connect attributes (i.e., bit 3). Therefore,
> termination of the association in response to connection loss of an
> I/O queue should terminate the admin queue connection. This could
> result in an inability to process the shutdown notification, causing a
> disconnect request to hang until a timeout occurs.
> 
> Is this possibly a bug or my misunderstanding of the spec?

The spec does not say anything about making sure that no I/O is sent
when the association is terminated, it actually says that its up to
the host implementation (see the shutdown section 4.6, which is probably
more relevant to the orderly shutdown you are describing).