[PATCH 1/9] nvme-apple: fix controller shutdown in apple_nvme_disable

[Janne Grunau]

On 2022-11-30 00:41:45 +0900, Hector Martin wrote:
> On 29/11/2022 22.22, Christoph Hellwig wrote:
> > nvme_shutdown_ctrl already shuts the controller down, there is no
> > need to also call nvme_disable_ctrl for the shutdown case.
> > 
> > Signed-off-by: Christoph Hellwig <hch at lst.de>
> > ---
> >  drivers/nvme/host/apple.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c
> > index 94ef797e8b4a5f..56d9e9be945b76 100644
> > --- a/drivers/nvme/host/apple.c
> > +++ b/drivers/nvme/host/apple.c
> > @@ -831,7 +831,8 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown)
> >  
> >  		if (shutdown)
> >  			nvme_shutdown_ctrl(&anv->ctrl);
> > -		nvme_disable_ctrl(&anv->ctrl);
> > +		else
> > +			nvme_disable_ctrl(&anv->ctrl);
> >  	}
> >  
> >  	WRITE_ONCE(anv->ioq.enabled, false);
> 
> Reviewed-by: Hector Martin <marcan at marcan.st>
> 
> I looked at some of our other implementations and we always seem to do
> both, but this makes sense. If it breaks something we'll notice and
> shout loudly when it makes it into an -rc at the latest :)

This breaks resume for the apple nvme controller. The immediate problem 
is that apple_nvme_reset_work() uses NVME_CC_ENABLE to decide whether to 
call apple_nvme_disable(). Only nvme_disable_ctrl(ctrl, false) clears 
NVME_CC_ENABLE.

Even after extending the check to consider NVME_CC_SHN_NORMAL resume is 
still broken. The host specific reset to re-enable the controller works 
but IO is blocked. The controller logs following messages into its 
syslog:

| RTKit: syslog message: cmd.c:728: Oldest tag 12 is 19 seconds VERY OLD with cmdQe:2
| RTKit: syslog message: cmd.c:732: Oldest tag is about to crash!!
| RTKit: syslog message: cmd.c:720: Oldest high water mark moved up to 21,tag 12, cmdQ CQ_PRI0W

This looks looks like the controller reset after/during shutdown is required
on this controller.

Below patch fixes resume for me. I can send this with comment to
prevent repeating this regression or if preferred handle this in
nvme_disable_ctrl() either via a quirk or an additional parameter.

Janne

--- 
diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c
index e13a992b6096..82396f9486e1 100644
--- a/drivers/nvme/host/apple.c
+++ b/drivers/nvme/host/apple.c
@@ -867,7 +867,9 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown)
			apple_nvme_remove_cq(anv);
		}
 
-               nvme_disable_ctrl(&anv->ctrl, shutdown);
+               if (shutdown)
+                       nvme_disable_ctrl(&anv->ctrl, true);
+               nvme_disable_ctrl(&anv->ctrl, false);
	}
 
	WRITE_ONCE(anv->ioq.enabled, false);