Unexpected issues with 2 NVME initiators using the same target

[Sagi Grimberg]

> Hello Sagi,

Hi Bart,

> Can you clarify why you think that the SRP initiator needs to be changed?
> The SRP initiator submits the local invalidate work request after the RDMA
> write request. According to table 79 "Work Request Operation Ordering" the
> order of these work requests must be maintained by the HCA. I think if a HCA
> would start with invalidating the MR before the remote HCA has acknowledged
> the written data that that's a firmware bug.

That flow is fine, we were discussing immediate data sends.

SRP only needs fixing by waiting for the all local invalidates to
complete before unmapping the user buffers and completing the I/O.

BTW, did the efforts on standardizing remote invalidate in SRP ever
evolved to something? Would it be acceptable to add a non-standard
ib_srp and ib_srpt? We can default to off and allow the user to opt
it in if it knows the two sides comply...

We need to fix that in nvmf and iser too btw (luckily we have remote
invalidate so its not a big issue).

> The upstream SRP initiator does not use inline data.

Yet :)

IIRC you have a branch with immediate-data support against scst so
it might be relevant there...