[PATCH v15 06/20] nvme-tcp: Add DDP data-path

[Sagi Grimberg]

>>> @@ -1235,6 +1330,9 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
>>>        else
>>>                msg.msg_flags |= MSG_EOR;
>>>
>>> +     if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags))
>>> +             nvme_tcp_setup_ddp(queue, blk_mq_rq_from_pdu(req));
>>
>> Didn't we agree that setup_ddp should move to setup time and
>> not send time?
> 
> We believe we haven't reached a conclusion last time [1].
> 
> Moving the setup_ddp() call earlier at setup time is less efficient (up
> to ~15% less IOPS) when it does the work on a different CPU.
> 
> 1: https://lore.kernel.org/all/253h6oxvlwd.fsf@nvidia.com/

Can you please explain why? sk_incoming_cpu is updated from the network
recv path while you are arguing that the timing matters before you even
send the pdu. I don't understand why should that matter.