Unexpected issues with 2 NVME initiators using the same target

[Chuck Lever]

> On Jun 28, 2017, at 3:08 AM, Sagi Grimberg <sagi at grimberg.me> wrote:
> 
> 
>> Thinking out loud:
>> IIRC the message head and tail reside in the persistently registered
>> and DMA mapped buffers with few exceptions.
>> However, when page cache pages are involved, xprtrdma will do a DMA
>> unmap as you say.
>> So:
>> - we don't have a problem transmitting a garbled request thanks to
>>   exactly-once receive semantics
>> - we don't have a problem with the timing of registration and
>>   invalidation on the initiator because the PD's DMA lkey is used
>> - we do have a problem with DMA unmap
>> Using only persistently registered and DMA mapped Send buffers
>> should avoid the need to signal all Sends. However, when page
>> cache pages are involved, then the Send needs to be signaled,
>> and the pages unmapped only after Send completion, to be completely
>> safe.
> 
> How do you know when that happens?

The RPC Call send path sets up the Send SGE array. If it includes
page cache pages, it can set IB_SEND_SIGNALED.

The SGE array and the ib_cqe for the send are in the same data
structure, so the Send completion handler can find the SGE array
and figure out what needs to be unmapped.

The only problem is if a POSIX signal fires. In that case the
data structure can be released before the Send completion fires,
and we get touch-after-free in the completion handler.

I'm thinking that it just isn't going to be practical to handle
unmapping this way, and I should just revert back to using RDMA
Read instead of adding page cache pages to the Send SGE.


--
Chuck Lever