[PATCH RFC] nvme-rdma: support devices with queue size < 32

[Doug Ledford]

On 3/29/17 11:44 AM, Doug Ledford wrote:
> On 3/29/17 11:39 AM, Sagi Grimberg wrote:
>>
>>>> For each I/O we post up to 2 work requests, 1 for memory registration
>>>> and 1 for sending an I/O request (and 1 for local invalidation if the
>>>> target doesn't do it for us, but that is not the case here). So if our
>>>> queue depth is X, we size our completion queue to be X*3, and we need
>>>> to make sure we signal every (X*3)/2.
>>>
>>> ??? If your SQ is X and your CQ is X*3 you need to signal at X/2.
>>
>> Sorry, I confused SQ with CQ (which made it even more confusing..)
>>
>> Our application queue-depth is X, we size our SQ to be X*3
>> (send+reg+inv), we size our RQ to be X (resp) and our CQ to be
>> X*4 (SQ+RQ).
>>
>> So we should signal every (X*3)/2
>
> You say above "we post *up to* 2 work requests", unless you wish to
> change that to "we always post at least 2 work requests per queue
> entry", Jason is right, your frequency of signaling needs to be X/2
> regardless of your CQ size, you need the signaling to control the queue
> depth tracking.

If you would like to spread things out farther between signaling, then 
you can modify your send routine to only increment the send counter for 
actual send requests, ignoring registration WQEs and invalidate WQES, 
and then signal every X/2 sends.