[PATCH v5 10/16] nvme-tcp: Use CCR to recover controller that hits an error

[Mohamed Khalfella]

On Fri 2026-09-04 15:52:39 -0700, Mohamed Khalfella wrote:
> On Sun 2026-08-23 04:03:18 +0300, Sagi Grimberg wrote:
> > 
> > 
> > On 12/07/2026 5:23, Mohamed Khalfella wrote:
> > > An alive nvme controller that hits an error now will move to FENCING
> > > state instead of RESETTING state. ctrl->fencing_work attempts CCR to
> > > terminate inflight IOs. Regardless of the success or failure of CCR
> > > operation the controller is transitioned to RESETTING state to continue
> > > error recovery process.
> > >
> > > Signed-off-by: Mohamed Khalfella <mkhalfella at purestorage.com>
> > > ---
> > >   drivers/nvme/host/tcp.c | 30 +++++++++++++++++++++++++++++-
> > >   1 file changed, 29 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> > > index ba5c7b3e2a7c..a1711dd1d3c2 100644
> > > --- a/drivers/nvme/host/tcp.c
> > > +++ b/drivers/nvme/host/tcp.c
> > > @@ -161,6 +161,7 @@ struct nvme_tcp_ctrl {
> > >   	struct sockaddr_storage src_addr;
> > >   	struct nvme_ctrl	ctrl;
> > >   
> > > +	struct work_struct	fencing_work;
> > >   	struct work_struct	err_work;
> > >   	struct delayed_work	connect_work;
> > >   	struct nvme_tcp_request async_req;
> > > @@ -605,6 +606,12 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue)
> > >   
> > >   static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
> > >   {
> > > +	if (nvme_change_ctrl_state(ctrl, NVME_CTRL_FENCING)) {
> > > +		dev_warn(ctrl->device, "starting controller fencing\n");
> > > +		queue_work(nvme_wq, &to_tcp_ctrl(ctrl)->fencing_work);
> > > +		return;
> > > +	}
> > > +
> > >   	if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
> > >   		return;
> > >   
> > > @@ -2494,12 +2501,29 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work)
> > >   	nvme_tcp_reconnect_or_remove(ctrl, ret);
> > >   }
> > >   
> > > +static void nvme_tcp_fencing_work(struct work_struct *work)
> > > +{
> > > +	struct nvme_tcp_ctrl *tcp_ctrl = container_of(work,
> > > +			struct nvme_tcp_ctrl, fencing_work);
> > > +	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
> > > +	unsigned long rem;
> > > +
> > > +	rem = nvme_fence_ctrl(ctrl);
> > > +	if (rem)
> > > +		dev_info(ctrl->device, "CCR failed, starting error recovery\n");
> > > +
> > > +	nvme_change_ctrl_state(ctrl, NVME_CTRL_FENCED);
> > > +	if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
> > > +		queue_work(nvme_reset_wq, &tcp_ctrl->err_work);
> > > +}
> > > +
> > >   static void nvme_tcp_error_recovery_work(struct work_struct *work)
> > >   {
> > >   	struct nvme_tcp_ctrl *tcp_ctrl = container_of(work,
> > >   				struct nvme_tcp_ctrl, err_work);
> > >   	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
> > >   
> > > +	flush_work(&to_tcp_ctrl(ctrl)->fencing_work);
> > 
> > Agree we shouldn't be here with fencing work running.
> 
> Right, nvme_tcp_fencing_work() above queus tcp_ctrl->err_work. This
> flush makes aure that fencing is 100% done before we proceed with
> resetting.
> 
> > 
> > >   	if (nvme_tcp_key_revoke_needed(ctrl))
> > >   		nvme_auth_revoke_tls_key(ctrl);
> > >   	nvme_stop_keep_alive(ctrl);
> > > @@ -2542,6 +2566,7 @@ static void nvme_reset_ctrl_work(struct work_struct *work)
> > >   		container_of(work, struct nvme_ctrl, reset_work);
> > >   	int ret;
> > >   
> > > +	flush_work(&to_tcp_ctrl(ctrl)->fencing_work);
> > 
> > Isn't it being called in nvme_stop_ctrl? - perhaps it should be called 
> > in ->stop_ctrl() callback.
> > 
> > Other than that, this looks reasonable to me.
> 
> This flush_work() is needed in case nvme_tcp_fencing_work() loses the
> race of transitioning the controller from FENCED to RESETTING. The
> moment we move to FENCED anything can reset the controller. For example,
> userspace can do that. If we lose the race then tcp_ctrl->err_work will
> not be queued. That means reset work needs to flush fencing_work.

Now I am thinking about it again, what you suggested makes more sense
for both fencing and fenced work. Both should be flushed ->stop_ctrl().
I will do that.