Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 29, 2017, at 1:35 AM, Sagi Grimberg <sagi at grimberg.me> wrote:
> 
> 
>>> How do you know when that happens?
>> The RPC Call send path sets up the Send SGE array. If it includes
>> page cache pages, it can set IB_SEND_SIGNALED.
>> The SGE array and the ib_cqe for the send are in the same data
>> structure, so the Send completion handler can find the SGE array
>> and figure out what needs to be unmapped.
>> The only problem is if a POSIX signal fires. In that case the
>> data structure can be released before the Send completion fires,
>> and we get touch-after-free in the completion handler.
>> I'm thinking that it just isn't going to be practical to handle
>> unmapping this way, and I should just revert back to using RDMA
>> Read instead of adding page cache pages to the Send SGE.
> 
> Or wait for the send completion before completing the I/O?

In the normal case, that works.

If a POSIX signal occurs (^C, RPC timeout), the RPC exits immediately
and recovers all resources. The Send can still be running at that
point, and it can't be stopped (without transitioning the QP to
error state, I guess).

The alternative is reference-counting the data structure that has
the ib_cqe and the SGE array. That adds one or more atomic_t
operations per I/O that I'd like to avoid.


--
Chuck Lever