nvme-fabrics: crash at nvme connect-all
swise at opengridcomputing.com
Thu Jun 9 07:29:33 PDT 2016
> > >
> > > >>> Steve, did you see this before? I'm wandering if we need some sort
> > > >>> of logic handling with resource limitation in iWARP (global mrs
> > > >>
> > > >> Haven't seen this. Does 'cat /sys/kernel/debug/iw_cxgb4/blah/stats'
> > > >> anything interesting? Where/why is it crashing?
> > > >>
> > > >
> > > > So this is the failure:
> > > >
> > > > [ 703.239462] rdma_rw_init_mrs: failed to allocated 128 MRs
> > > > [ 703.239498] failed to init MR pool ret= -12
> > > > [ 703.239541] nvmet_rdma: failed to create_qp ret= -12
> > > > [ 703.239582] nvmet_rdma: nvmet_rdma_alloc_queue: creating RDMA queue
> > > failed
> > > > (-12).
> > > >
> > > > Not sure why it would fail. I would think my setup would be allocating
> > > > given I have 16 cores on the host and target. The debugfs "stats" file
> > > > mentioned above should show us something if we're running out of adapter
> > > > resources for MR or PBL records.
> > >
> > > Note that Marta ran both the host and the target on the same machine.
> > > So, 8 (cores) x 128 (queue entries) x 2 (host and target) gives 2048
> > > MRs...
> > >
> > > What is the T5 limitation?
> > It varies based on a config file that gets loaded when cxgb4 loads. Note
> > error has nothing to do with the low fastreg sg depth limit of T5. If we
> > hitting that then we would be seeing EINVAL and not ENOMEM. Looking at
> > c4iw_alloc_mr(), the ENOMEM paths are either failures from kzalloc() or
> > dma_alloc_coherent(), or failures to allocate adapter resources for MR and
> > records. Each MR takes a 32B record in adapter mem, and the PBL takes
> > based on the max sg depth (roughly sg_depth * 8 + some rounding up). The
> > debugfs "stats" file will show us what is being exhausted and how much
> > mem is available for these resources.
> > Also, the amount of available adapter mem depends on the type of T5 adapter.
> > The T5 adapter info should be in the dmesg log when cxgb4 is loaded.
> > Steve
> Here is an example of the iw_cxgb4 debugfs "stats" output. This is for a
> T580-CR with the "default" configuration, which means there is no config file
> named t5-config.txt in /lib/firmware/cxgb4/.
> [root at stevo1 linux-2.6]# cat /sys/kernel/debug/iw_cxgb4/0000\:82\:00.4/stats
> Object: Total Current Max Fail
> PDID: 65536 0 0 0
> QID: 24576 0 0 0
> TPTMEM: 36604800 0 0 0
> PBLMEM: 91512064 0 0 0
> RQTMEM: 128116864 0 0 0
> OCQPMEM: 0 0 0 0
> DB FULL: 0
> DB EMPTY: 0
> DB DROP: 0
> DB State: NORMAL Transitions 0 FC Interruptions 0
> TCAM_FULL: 0
> ACT_OFLD_CONN_FAILS: 0
> PAS_OFLD_CONN_FAILS: 0
> NEG_ADV_RCVD: 0
> AVAILABLE IRD: 589824
> Note it shows the total, currently allocated, max ever allocated, and failures
> for each rdma resource, most of which are tied to HW resources. So if we see
> failures, then we know the adapter resources were exhausted.
> TPTMEM is the available adapter memory for MR records. Each record is 32B.
> a total of 1143900 MRs (TPTMEM / 32) can be created. The PBLMEM resource is
> holding the dma addresses for all pages in a MR, so each MR uses some number
> depending on the sg depth passed in when allocating a FRMR. So if we allocate
> 128 deep page lists, we should be able to allocate 89367 PBLs (PBLMEM / 8 /
> Seems like we shouldn't be exhausting the adapter resources with 2048 MRs...
I don't see this on my 16 core/64GB memory note, I successfully did a
discover/connect-all with the target/host on the same node with 7 target devices
w/o any errors. Note I'm using the nvmf-all.2 branch Christoph setup up
Marta, I need to learn more about your T5 setup and the "stats" file output.
More information about the Linux-nvme