"Connect rejected: status 28" from "nvme discover"

[Keith Busch]

On Tue, Aug 25, 2020 at 11:59:28AM -1000, Chris Worley wrote:
> Running CentOS 8.2, using native drivers but following instructions at
> https://community.mellanox.com/s/article/howto-configure-nvme-over-fabrics
> (but not overriding the native drivers with MOFED).
> 
> Hardware is connectX-3 HCAs.
> 
> Target setup goes okay:
> 
> [root at localhost 1]# dmesg | grep nvmet_rdma
> [ 6366.612458] nvmet_rdma: enabling port 1 (10.4.0.6:4420)
> [ 6376.078826] nvmet_rdma: enabling port 2 (10.4.0.6:4421)
> [ 6385.662419] nvmet_rdma: enabling port 3 (10.4.0.6:4423)
> 
> But discover goes awry on client:
> 
> [root at localhost network-scripts]# nvme discover -t rdma -a 10.4.0.6 -s 4420
> Failed to write to /dev/nvme-fabrics: Connection reset by peer
> [root at localhost network-scripts]# dmesg | tail
> [ 1688.058291] nvme nvme4: Connect rejected: status 28 (consumer defined) nvme status 0 (unrecognized reason).
> [ 1688.058410] nvme nvme4: rdma connection establishment failed (-104)
> 
> Systems ping each other.
> 
> Hnts?

Does the target's dmesg mention anything after the failed connection?