Failure with 8K Write operations

Narayan Ayalasomayajula narayan.ayalasomayajula at kazan-networks.com
Thu Sep 15 06:36:21 PDT 2016


Hi Jay,

Thanks for pointing out that I was not running the latest version of the kernel. I updated to 4.8rc6 and my FIO test that had previously failed with the Linux NVMeF target (using null_blk device as the target) is now completing successfully. I am still seeing the same NAK (Remote Access Error) failure when I use our target instead. I will debug this further but updating to 4.8rc6 did improve things.

(Sagi, thanks for the print statements to display what the driver is passing down to the RNIC. I will use that change in 4.8rc6 to debug further). 

Thanks,
Narayan

-----Original Message-----
From: J Freyensee [mailto:james_p_freyensee at linux.intel.com] 
Sent: Wednesday, September 14, 2016 9:52 AM
To: Narayan Ayalasomayajula <narayan.ayalasomayajula at kazan-networks.com>; Sagi Grimberg <sagi at grimberg.me>; linux-nvme at lists.infradead.org
Subject: Re: Failure with 8K Write operations

On Wed, 2016-09-14 at 00:03 +0000, Narayan Ayalasomayajula wrote:
> Hi Jay,
> 
> Thanks for taking the effort to emulate the behavior.
> 
> I did not mention this in my last email but had indicated it in the 
> earlier posting. I am using null_blk as the target (so the IOs are not 
> being serviced by a real nvme target). I am not sure if that could 
> somehow be the catalyst for the failure. Is it possible for you to 
> re-run your test with null_blk as the target?

As we talked off-line, try the latest mainline kernel from kernel.org and see if you see anything different.

> 
> Thanks,
> Narayan
> 
> -----Original Message-----
> From: J Freyensee [mailto:james_p_freyensee at linux.intel.com]
> Sent: Tuesday, September 13, 2016 4:51 PM
> To: Narayan Ayalasomayajula <narayan.ayalasomayajula at kazan-networks.c
> om>; Sagi Grimberg <sagi at grimberg.me>; linux-nvme at lists.infradead.org
> Subject: Re: Failure with 8K Write operations
> 
> On Tue, 2016-09-13 at 20:04 +0000, Narayan Ayalasomayajula wrote:
> > 
> > Hi Sagi,
> > 
> > Thanks for the print statement to verify that the sgls in the 
> > command capsule match what the Host programmed. I added this print 
> > statement and compared the Virtual Address and R_Key information in 
> > the /var/log to the NVMe Commands in the trace file and found the 
> > two to match.
> > I
> > have the trace and Host log files from this failure (trace is ~6M)
> > -
> > will it be useful for someone who may be looking into this issue?
> > 
> > Regarding the host side log information you mentioned, I had 
> > attached that in my prior email (attached again). Is this what you 
> > are requesting? That was collected prior to adding the print 
> > statement that you suggested.
> > 
> > Just to summarize, the failure is seen in the following
> > configuration:
> > 
> > 1. Host is an 8-core Ubuntu server running the 4.8.0 kernel. It has 
> > a
> > ConnectX-4 RNIC (1x100G) and is connected to a Mellanox Switch.
> > 2. Target is an 8-core Ubuntu server running the 4.8.0 kernel. It 
> > has a ConnectX-3 RNIC (1x10G) and is connected to a Mellanox Switch.
> > 3. Switch has normal Pause and Jumbo frame support enabled on all 
> > ports.
> > 4. Test fails with Host sending a NAK (Remote Access Error) for the 
> > following FIO workload:
> > 
> > 	[global]
> > 	ioengine=libaio
> > 	direct=1
> > 	runtime=10m
> > 	size=800g
> > 	time_based
> > 	norandommap
> > 	group_reporting
> > 	bs=8k
> > 	numjobs=8
> > 	iodepth=16
> > 
> > 	[rand_write]
> > 	filename=/dev/nvme0n1
> > 	rw=randwrite
> > 
> 
> Hi Narayan:
> 
> I have a 2 host, 2 target 1RU server data network using a 32x Arista 
> switch and using your FIO setup above, I am not seeing any errors.  I 
> tried running your script on each Host at the same time targeting the 
> same NVMe Target (but different SSDs targeted by each Host) as well as 
> only running the script on 1 Host only and didn't see any errors.
> Also
> tried 'numjobs=1' and didn't reproduce what you see.
> 
> Both Host and Targets for me are using the 4.8-rc4 kernel.  Both the 
> Host and Target are using dual port Mellanox ConnectX-3 Pro EN 40Gb 
> (so I'm using a RoCE setup). My Hosts are 32 processor machines and 
> Targets are 28 Processor machine.  All filled w/various Intel SSDs.
> 
> Something unique about your setup.
> 
> Jay
> 
> 
> > 
> > I have found that the failure happens with numjobs set to 1 as well.
> > 
> > Thanks again for your response,
> > Narayan
> > 
> > -----Original Message-----
> > From: Sagi Grimberg [mailto:sagi at grimberg.me]
> > Sent: Tuesday, September 13, 2016 2:16 AM
> > To: Narayan Ayalasomayajula <narayan.ayalasomayajula at kazan-networks
> > .c
> > om>; linux-nvme at lists.infradead.org
> > Subject: Re: Failure with 8K Write operations
> > 
> > 
> > > 
> > > 
> > > Hello All,
> > 
> > Hi Narayan,
> > 
> > > 
> > > 
> > > I am running into a failure with the 4.8.0 branch and wanted to 
> > > see this is a known issue or whether there is something I am not 
> > > doing right in my setup/configuration. The issue that I am running 
> > > into is that the Host is indicating a NAK (Remote Access Error) 
> > > condition when executing an FIO script that is performing 100% 8K 
> > > Write operations. Trace analysis shows that the target has the 
> > > expected Virtual Address and R_KEY values in the READ REQUEST but 
> > > for some reason, the Host flags the request as an access 
> > > violation.
> > > I ran a similar test with iWARP Host and Target systems and the 
> > > did see a Terminate followed by FIN from the Host. The cause for 
> > > both failures appears to be the same.
> > > 
> > 
> > I cannot reproduce what you are seeing on my setup (Steve, can
> > you?)
> > I'm running 2 VMs connected over SRIOV on the same PC though...
> > 
> > Can you share the log on the host side?
> > 
> > Can you also add this print to verify that the host driver 
> > programmed the same sgl as it sent the target:
> > --
> > diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c 
> > index c2c2c28e6eb5..248fa2e5cabf 100644
> > --- a/drivers/nvme/host/rdma.c
> > +++ b/drivers/nvme/host/rdma.c
> > @@ -955,6 +955,9 @@ static int nvme_rdma_map_sg_fr(struct 
> > nvme_rdma_queue *queue,
> >          sg->type = (NVME_KEY_SGL_FMT_DATA_DESC << 4) |
> >                          NVME_SGL_FMT_INVALIDATE;
> > 
> > +       pr_err("%s: rkey=%#x iova=%#llx length=%#x\n",
> > +               __func__, req->mr->rkey, req->mr->iova,
> > + req->mr->length);
> > +
> >          return 0;
> >   }
> > --
> > _______________________________________________
> > Linux-nvme mailing list
> > Linux-nvme at lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-nvme


More information about the Linux-nvme mailing list