nvme-tcp crashes the system when overloading the backend device.

[Sagi Grimberg]

> Hi all,
> 
> I can consistently crash a system when I sufficiently overload the nvme-tcp target.
> The easiest way to reproduce the problem is by creating a raid5.
> 
> While this R5 is resyncing export it with the nvmet-tcp target driver and start a high queue-depth 4K random fio workload from the initiator.
> At some point the target system will start logging these messages:
> [ 2865.725069] nvmet: ctrl 238 keep-alive timer (15 seconds) expired!
> [ 2865.725072] nvmet: ctrl 236 keep-alive timer (15 seconds) expired!
> [ 2865.725075] nvmet: ctrl 238 fatal error occurred!
> [ 2865.725076] nvmet: ctrl 236 fatal error occurred!
> [ 2865.725080] nvmet: ctrl 237 keep-alive timer (15 seconds) expired!
> [ 2865.725083] nvmet: ctrl 237 fatal error occurred!
> [ 2865.725087] nvmet: ctrl 235 keep-alive timer (15 seconds) expired!
> [ 2865.725094] nvmet: ctrl 235 fatal error occurred!

This is the target not getting a keep-alive commands from the initiator, 
hence it triggers a controller removal. btw what kernel version is this?

While the target is tearing down the controller queues, and waits for
all the inflight commands to complete and drop the final reference on
the queue. I'm wandering if something is preventing these inflight
commands from getting completed by the backend R5 device.

> 
> Even when you stop all IO from the initiator some of the nvmet_tcp_wq workers will keep running forever.

That is because the target is waiting for the commands to complete, but 
they don't.

> Eventually the system runs out of resources.
> At some point the system will reach a workload of 2000+ and crash.
> 
> So far, I have been unable to determine why the number of nvmet_tcp_wq keeps increasing.
> It must be because the current failed worker gets replaced by a new worker without the old being terminated.

Is it possible to check if the R5 device has inflight commands? if not 
there is some race condition or misaccounting that prevents an orderly 
shutdown of the queues.