nvme tcp receive errors

[Sagi Grimberg]

>>> This one took a bit to go through. The traces all only have a single r2t
>>> pdu with 0 offset for the full length of the requested transfer. I had
>>> to add trace events to see what the heck the driver is thinking, and
>>> the result is even more confusing.
>>>
>>> The kernel message error:
>>>
>>>       nvme5: req 5 op 1 r2t len 16384 exceeded data len 16384 (4096 sent)
>>>
>>> And all the new trace events for this request are:
>>>
>>>       fio-25086   [011] ....  9630.542669: nvme_tcp_queue_rq: nvme5: qid=4 tag=5 op=1 data_len=16384
>>>       fio-25093   [007] ....  9630.542854: nvme_tcp_cmd_pdu: nvme5: qid=4 tag=5 op=1 page_offset=3664 send_len=72
>>>       <...>-22670 [003] ....  9630.544377: nvme_tcp_r2t: nvme5: qid=4 tag=5 op=1 r2t_len=16384 r2t_offset=0 data_sent=4096 data_len=16384
>>>
>>> The fact "data_sent" is non-zero on the very first r2t makes no sense to me.
>>
>> Yep, not supposed to happen... Like the traces though! very useful and
>> absolutely worth having.
>>
>>> I so far can not find any sequence where that could happen.
>>
>> The only way that data_sent can increment is either by getting an r2t
>> solicitation or by sending incapsule data.
>>
>> What is the ioccsz the controller is exposing?
> 
> In capsule data is not supported by this target, so ioccsz is 4.
> 
>> in nvme_tcp_sned_cmd_pdu we have:
>> --
>>                  if (inline_data) {
>>                          req->state = NVME_TCP_SEND_DATA;
>>                          if (queue->data_digest)
>>                                  crypto_ahash_init(queue->snd_hash);
>>                  } else {
>>                          nvme_tcp_done_send_req(queue);
>>                  }
>> --
>>
>> Where inline_data flag is:
>> 	bool inline_data = nvme_tcp_has_inline_data(req);
>> which essentially boils down to:
>> 	req->data_len <= queue->cmnd_capsule_len - sizeof(struct nvme_command);
>>
>> I wonder how does the nvme command sgl look like? is it an offset
>> sgl?
> 
> Just a single data block descriptor SGL. The target supports only 1 and
> reports that through ID_CTRL.MSDBD.
> 
> What do you mean by "offset" SGL?

In tcp.c:
--
static void nvme_tcp_set_sg_inline(struct nvme_tcp_queue *queue,
                 struct nvme_command *c, u32 data_len)
{
         struct nvme_sgl_desc *sg = &c->common.dptr.sgl;

         sg->addr = cpu_to_le64(queue->ctrl->ctrl.icdoff);
         sg->length = cpu_to_le32(data_len);
         sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET;
}

static void nvme_tcp_set_sg_host_data(struct nvme_command *c,
                 u32 data_len)
{
         struct nvme_sgl_desc *sg = &c->common.dptr.sgl;

         sg->addr = 0;
         sg->length = cpu_to_le32(data_len);
         sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
                         NVME_SGL_FMT_TRANSPORT_A;
}
--

What is the sgl type you see in the traces? transport specific sgl
(host-data i.e. non-incapsule) or inline?