[PATCH] nvme: allow to re-attach namespaces after all paths are down

[Sagi Grimberg]

>> On Mon, May 10, 2021 at 03:25:04PM -0700, Sagi Grimberg wrote:
>>> Hannes, you sent a patch like this before, my comment was that a nshead
>>> should be removed before the final reference (which who knows when it
>>> will ever arrive) as if a nsid were to be reused by the controller for
>>> a different namespace then we'd reject it so I'm not sure how this helps?
>>
>> Hm, you're considering something like if a user does namespace
>> management commands to delete one that is in-use (mounted, raid, etc),
>> then creates and attaches a new namespace that happens to be assigned
>> the same NSID.
>>
>> The new one may get different EUI/NGUID values, so this approach here
>> would prevent the namespace from being usable without additional user
>> intervention. I think we'd want the driver to allocate an entirely new
>> head in that case if we could detect that's what happened. There is also
>> a possibility that the controller recycles the same NGUID for the new
>> namespace, and that could be confusing for the application holding the
>> open reference on the old namespace.
>>
> ... and, of course, none of this would happen if we would _not_ keep the
> old nshead around. I'm still not sure _why_ this is considered
> important. After all, things work perfectly when the CMIC bit is not
> set; even device hot-removal and re-insertion (one of our IHVs tested
> the crap out of that one, and I can confidently state the nvme stack did
> not cause any issue there. Unlike other subsystems ...).
> 
> So I'm really curious which use-case we're trying to support with that;
> EIO is EIO, and the application couldn't care less if it's generated due
> to the nshead running out of paths or the nshead not existing...

I don't disagree with you Hannes, hence I think this should be the
default behavior. Christoph wants to see a queue_if_no_path behavior
that would be the existing functionality (although I don't think we
actually provide this type of functionality because we _do_ fail
the I/O as we don't have any paths).

So I'm on the camp of restoring the behavior that matches non-mpath
and then add queue_if_no_path properly as an opt-in.