[PATCH v2] nvmet: force reconnect when number of queue changes

[Daniel Wagner]

On Wed, Sep 28, 2022 at 12:39:44PM +0000, Knight, Frederick wrote:
> Would this be a case for a new AEN - controller configuration changed?
> I'm also wondering exactly what changed in the controller?  It can't
> be the "Number of Queues" feature (that can't change - The controller
> shall not change the value allocated between resets.).  Is it the MQES
> field in the CAP property that changes (queue size)?
>
> We already have change reporting for: Namespace attribute, Predictable
> Latency, LBA status, EG aggregate, Zone descriptor, Discovery Log,
> Reservations. We've questioned whether we need a Controller Attribute
> Changed.
>
> Would this be a case for an exception?  Does the DNR bit apply only to
> commands sent on queues that already exist (i.e., NOT the connect
> command since that command is actually creating the queue)?  FWIW - I
> don't like exceptions.
>
> Can you elaborate on exactly what is changing?

The background story is, that we have a setup with two targets (*) and
the host is connected two both of them (HA setup). Both server run the
same software version. The host is told via Number of Queues (Feature
Identifier 07h) how many queues are supported (N queues).

Now, a software upgrade is started which takes first one server offline,
updates it and brings it online again. Then the same process with the
second server.

In the meantime the host tries to reconnect. Eventually, the reconnect
is successful but the Number of Queues (Feature Identifier 07h) has
changed to a smaller value, e.g N-2 queues.

My test case here is trying to replicated this scenario but just with
one target. Hence the problem how to notify the host that it should
reconnect. As you mentioned this is not to supposed to change as long a
connection is established.

My understanding is that the current nvme target implementation in Linux
doesn't really support this HA setup scenario hence my attempt to get it
flying with one target. The DNR bit comes into play because I was toying
with removing the subsystem from the port, changing the number of queues
and re-adding the subsystem to the port.

This resulted in any request posted from the host seeing the DNR
bit. The second attempt here was to delete the controller to force a
reconnect. I agree it's also not really the right thing to do.

As far I can tell, what's is missing from a testing point of view is the
ability to fail requests without the DNR bit set or the ability to tell
the host to reconnect. Obviously, an AEN would be nice for this but I
don't know if this is reason enough to extend the spec.

I can't really say if this is a real world scenario or just a result of
trying to cut corners. Anyway, I am glad to do the work if we can agree
on how this test case could be implemented.

Daniel

(*) not sure how to call it properly, is this one target or two targets?