target crash / host hang with nvme-all.3 branch of nvme-fabrics

[Steve Wise]

> 
> >> On Thu, Jun 16, 2016 at 09:53:45AM -0500, Steve Wise wrote:
> >>> [11436.603807] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
> >>> [11436.609866] BUG: unable to handle kernel NULL pointer dereference at
> >>> 0000000000000050
> >>> [11436.617764] IP: [<ffffffffa09c6dff>] nvmet_rdma_delete_ctrl+0x6f/0x100
> >>
> >> Can you check using gdb where in the code this is?
> >
> >
> > nvmet_rdma_delete_ctrl():
> > /root/nvmef/nvme-fabrics/drivers/nvme/target/rdma.c:1302
> >                          &nvmet_rdma_queue_list, queue_list) {
> >                  if (queue->nvme_sq.ctrl->cntlid == ctrl->cntlid)
> >       df6:       48 8b 40 38             mov    0x38(%rax),%rax
> >       dfa:       41 0f b7 4d 50          movzwl 0x50(%r13),%ecx
> >       dff:       66 39 48 50             cmp    %cx,0x50(%rax)
<===========
> > here
> >       e03:       75 cd                   jne    dd2
<nvmet_rdma_delete_ctrl+0x42>
> 
> Umm, I think this might be happening because we get to delete_ctrl when
> one of our queues has a NULL ctrl. This means that either:
> 1. we never got a chance to initialize it, or
> 2. we already freed it.
> 
> (1) doesn't seem possible as we have a very short window (that we're
> better off eliminating) between when we start the keep-alive timer (in
> alloc_ctrl) and the time we assign the sq->ctrl (install_queue).
> 
> (2) doesn't seem likely either to me at least as from what I followed,
> delete_ctrl should be mutual exclusive with other deletions, moreover,
> I didn't see an indication in the logs that any other deletions are
> happening.
> 
> Steve, is this something that started happening recently? does the
> 4.6-rc3 tag suffer from the same phenomenon?

I'll try and reproduce this on the older code, but the keep-alive timer fired
for some other reason, so I'm not sure the target side keep-alive has been
tested until now.  But it is easy to test over iWARP, just do this while a heavy
fio is running:

ifconfig ethX down; sleep 15; ifconfig ethX <ipaddr>/<mask> up