Fail to configure NVMe-fabric over soft-RoCE

[Moni Shoua]

Max, please  take it and let me know if you see any connectivity issues 

-----Original Message-----
From: Youngjae Lee [mailto:leeyo at linux.vnet.ibm.com] 
Sent: Tuesday, March 07, 2017 5:39 PM
To: Max Gurtovoy <maxg at mellanox.com>; linux-nvme at lists.infradead.org; Moni Shoua <monis at mellanox.com>
Subject: Re: Fail to configure NVMe-fabric over soft-RoCE

Hi, Max,


Sure, I passed some basic rdma tests like udaddy and ibv_rc_pingtest....

Thanks.


On 3/7/17 3:35 AM, Max Gurtovoy wrote:
> adding Moni.
>
> Youngjae Lee,
>
> did you run some basic rdma tests before NVMEoF ?
> This is a precondition.
>
> Moni,
> please advise.
>
> Max.
>
> On 3/7/2017 1:19 AM, Youngjae Lee wrote:
>> Hi, all
>>
>> Has anyone succeeded to configure NVMe over Fabrics with soft-RoCE
>> (rxe) ?
>> I'm trying it with the latest rc kernel (4.11.0-rc1), but the 
>> discover operation (of nvme-cli) on the client side fails. (please 
>> see the attached nvme-cli/dmesg logs below..)
>>
>> I'm following the instructions from this page to configure it.
>> https://community.mellanox.com/docs/DOC-2504
>> A NVMe target seems to be perfectly set up on the target server side.
>>
>> Dmesg log on the target server,
>> [ 5574.892787] nvmet: adding nsid 10 to subsystem test [ 5574.897461] 
>> nvmet_rdma: enabling port 1 (10.1.1.17:1023) [ 5612.369855] nvmet: 
>> creating controller 1 for subsystem 
>> nqn.2014-08.org.nvmexpress.discovery for NQN 
>> nqn.2014-08.org.nvmexpress:NVMf:uuid:15b61008-8a88-4d7b-b9be-66600269a9e7.
>>
>> [ 5673.040744] nvmet_rdma: freeing queue 0
>>
>> nvme-cli output and dmesg log on the client, root at rxe2:~/nvme-cli# 
>> ./nvme discover -t rdma -a 10.1.1.17 -s 1023 Failed to write to 
>> /dev/nvme-fabrics: Input/output error
>>
>> [  386.091648] rdma_rxe: qp#17 moved to error state [  446.756855] 
>> nvme nvme0: Identify Controller failed (16391)
>>
>> I enabled debug msgs of rdma_rxe to see what happened in rdma_rxe and 
>> it looks like there were some errors in rdma communications during 
>> the nvme discover operation.
>> ....
>> [ 8908.806021] rdma_rxe: qp#17 state = GET_REQ [ 8908.806022] 
>> rdma_rxe: qp#17 state = CHK_PSN [ 8908.806023] rdma_rxe: qp#17 state 
>> = CHK_OP_SEQ [ 8908.806025] rdma_rxe: qp#17 state = CHK_OP_VALID [ 
>> 8908.806026] rdma_rxe: qp#17 state = CHK_RESOURCE [ 8908.806028] 
>> rdma_rxe: qp#17 state = CHK_LENGTH [ 8908.806030] rdma_rxe: qp#17 
>> state = CHK_RKEY [ 8908.806033] rdma_rxe: qp#17 state = ERR_LENGTH [ 
>> 8908.806035] rdma_rxe: qp#17 state = COMPLETE [ 8908.806036] 
>> rdma_rxe: qp#17 state = CLEANUP [ 8908.806037] rdma_rxe: qp#17 state 
>> = DONE [ 8908.806039] rdma_rxe: qp#17 state = ERROR [ 8908.806040] 
>> rdma_rxe: qp#17 moved to error state .....
>>
>> Any advice to resolve this issue ???
>>
>> Thanks.
>>
>> - Youngjae Lee
>>
>>
>> _______________________________________________
>> Linux-nvme mailing list
>> Linux-nvme at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
>

--
Best Regards.

- Youngjae Lee