Should NVME_SC_INVALID_NS be translated to BLK_STS_IOERR instead of BLK_STS_NOTSUPP so that multipath(both native and dm) can failover on the failure?

[Keith Busch]

On Mon, Dec 04, 2023 at 03:58:14PM +0800, Jirong Feng wrote:
> Hi all,
> 
> I have two storage servers, each of which has an NVMe SSD. Recently I'm
> trying nvmet-tcp with DRBD, steps are:
> 1. Configure DRBD for the two SSDs in two-primary mode, so that each server
> can accept IO on DRBD device.
> 2. On each server, add the corresponding DRBD device to nvmet subsystem with
> same device uuid, so that multipath on the host side can group them into one
> device(My fabric type is tcp).
> 3. On client host, nvme discover & connect the both servers, making sure DM
> multipath device is generated, and both paths are online.
> 4. Execute fio randread on DM device continuously.
> 5. On the server whose multipath status is active, under nvmet namespace
> configfs directory, execute "echo 0 > enable" to disable the namespace.
> what I expect is that IO can be automatically retried and switched to the
> other storage server by multipath, fio goes on. But actually I see an
> "Operation not supported" error, and fio fails and stops. I've also tried
> iSCSI target, after I delete mapped lun from acl, fio continues running
> without any error.
> 
> My kernel version is 4.18.0-147.5.1(rhel 8.1). After checked out the kernel
> code, I found that:
> 1. On target side, nvmet returns NVME_SC_INVALID_NS to host due to namespace
> not found.

So the controller through that path used to be able to access the
Namespace, then suddenly lost ability to do so, but some other path can
still access it if we retry on a failover/alternate path? I think your
target is returning the wrong error code. It should be SCT/SC 303h,
Asymmetric Access Persistent Loss (NVME_SC_ANA_TRANSITION), for what
you're describing.