[PATCH 0/3] Fix request completion holes

[Jason Gunthorpe]

On Thu, Nov 02, 2017 at 11:18:42AM -0500, Steve Wise wrote:

> iWARP spec too.   This is in regard to completion ordering though.  The local
> invalidate send WR must have the IB_SEND_FENCE flag set if you want it to only
> be executed after all prior send WRs are executed.

Oh right, yes, you need to have local invalidate fence to guarentee
that, but usually a local operation will not refer to a RKEY, so
execution ordering wont matter.

But this is a good general point, doesn't a ULP need to set FENCE on
SEND, eg:

 RDMA READ (rkey)
 RDMA READ (rkey)
 SEND w/ FENCE (tell remote to invalidate rkey)

Otherwise IBA Table 79 says RDMA READ can pass SEND and we have a
situation where the rkey has become invalidated when the remote is
still trying to use it.

IBA actually explicitly calls this out (pg 538):

 RDMA Read operations may be executed at the target after subse-
 quent Send and Invalidate operation already performed the invalida-
 tion at the target. That may cause the RDMA Read operation to fail.
 Setting the Fence Indicator on the subsequent operations guarantees
 that the RDMA Read will fully complete before the invalidation takes
 place.

None of our ULPs use IB_SEND_FENCE - so this is a bug!

> Either way, the completions are always inserted into the cq
> in-sq-submission-order and a signaled completion implies completions
> of all prior unsignaled WRs.

Yes, excactly what is relevant for this nvme case..

Jason