[PATCH v2] nvmet: force reconnect when number of queue changes

[Knight, Frederick]

I sure didn't intend to said that.  Section 3.6.2 talks about shutting down the controller (which leaves the connection intact).

Section 3.7.1 talks about NVM Subsystem resets which causes controller level resets (where again, I believe the intent is that the connection remains intact - so the host can use Get/Set Properties commands to restore operation (if the host so chooses)).

Those are cases where resets happen but the connection remains intact.  The spec suggests a 2 minute window (see section 3.5.2) - last sentence:
The association may be removed if step 5 (i.e., setting CC.EN to ‘1’) is not completed within 2 minutes after establishing the association.
Section 3.6.2 on shutdown says:
After CC.EN transitions to ‘0’ (i.e., due to Controller Level Reset), the association between the host and controller shall be preserved for at least 2 minutes. After this time, the association may be removed if the controller has not been re-enabled.
You've got 2 minutes to reenable the controller before disconnection.

There are also statements in those sections that talk about fatal errors - which do cause the connection to go down (and which therefore, reset the controller).

Section 3.7.2 talks about Controller Level Resets (which uses CC.EN). Although, this section does say - that for message-based transports (fabrics), all internal controller state is reset - no exceptions.  If the host is trying to reenable a controller after a reset on an existing connection, then it must reestablish everything - because everything (all state) has been reset.  The host can't assume anything has been retained.

If the controller doesn't get reenabled, then a disconnect will happen.

But, I’m not sure what that has to do with the Number of Queues feature setting.
The host can do a SET FEATURES to request a Number of Queues - ONCE, and the controller then tells the host how many it really gets.
Then, the host can't do that again (Command Sequence Error if you try); and
The controller can't change it's mind about how many it gave to the host.
That value is STUCK - until a RESET happens.
Once a RESET happens, then there are no queues, so everything must be initialized again.
The host goes through the whole controller enabling process again - including a new SET FEATURES to request a number of queues and the controller tells the host how many it gets this time.
And again, that number is stuck until a RESET happens.

If the controller wants to change the number of IO Queues - then it must force a RESET - and cause the host to perform a full reinitialization (and reestablish whatever is the appropriate new number of IO Queues).

Yes, when that RESET happens, the connection could be dropped, but that is not required.

	Fred


> -----Original Message-----
> From: John Meneghini <jmeneghi at redhat.com>
> Sent: Wednesday, September 28, 2022 10:14 PM
> To: Knight, Frederick <Frederick.Knight at netapp.com>; Daniel Wagner
> <dwagner at suse.de>
> Cc: Sagi Grimberg <sagi at grimberg.me>; linux-nvme at lists.infradead.org;
> Shinichiro Kawasaki <shinichiro.kawasaki at wdc.com>; hare at suse.de
> Subject: Re: [PATCH v2] nvmet: force reconnect when number of queue
> changes
> 
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
> 
> 
> 
> 
> On 9/28/22 14:02, Knight, Frederick wrote:
> > Reset and Disconnect are different.
> >
> > Reset uses registers (and therefore Get Property and Set Property
> commands) over a valid connection.
> >
> > Disconnect causes a reset (you have to reconnect to the reset
> > controller) Reset does not cause an immediate disconnect.
> 
> OK, so you are saying that the rule "The controller shall not change the value
> allocated between resets" in NVMe 5.27.1.5 doesn't apply to fabric
> controllers?
> 
> /John
> 
> >       Fred
> >