consultation about NVME_SGL_FMT_INVALIDATE not set when using nvme-rdma t10-pi

[liruozhu]

Hi Max,

On 2021/10/27 18:49, Max Gurtovoy wrote:
> Hi Ruozhu,
>
> The performance tests we did for all the IO sizes showed better 
> performance with local invalidation in most of the cases (mainly for 
> large IOs, the remote invalidation task is harder for the HW).
>
> Maybe there is 1 or 2 cases, such as small IO latency, that improves 
> with remote-invalidation but for now we decided to disable it.

I test it with single thread 4K write IO, maybe this is why I get better 
latency data.

I will test various IO models when I have the opportunity, and then give 
you feedback.

In addition, I would like to know if the remote invalidation task is 
harder for the HW, Why do we need to enable it in the general(no t10-pi) 
scene?

Thanks,
Ruozhu

>
> I guess we can add a capability for the device that will indicate it 
> in the future.
>
> What is the difference you see ? and what is the scenario ?
>
> Thanks,
>
> -Max.
>
> On 10/27/2021 11:50 AM, liruozhu wrote:
>> Hi Max,
>>
>> I was testing T10-PI feature of nvme-rdma recently, and found that 
>> when using T10-PI, nvme-rdma driver did not set 
>> NVME_SGL_FMT_INVALIDATE flag in the sgl type field, so the host 
>> software needs to do invalid rkey by itself. I tried to add the flag 
>> to test it. Turns out T10-PI feature is still working, and I get 
>> better IO latency with it enable.
>> I read the original patches in the mailing list, and found that patch 
>> v1 set this flag. But it was silent dropped on patch v5 without any 
>> comments. Is it for any special considerations to delete it?
>>
>> Thanks,
>> Ruozhu
>>
> .