consultation about NVME_SGL_FMT_INVALIDATE not set when using nvme-rdma t10-pi

[liruozhu]

Hi Max,

On 2021/10/27 18:49, Max Gurtovoy wrote:
> Hi Ruozhu,
>
> The performance tests we did for all the IO sizes showed better 
> performance with local invalidation in most of the cases (mainly for 
> large IOs, the remote invalidation task is harder for the HW).
>
> Maybe there is 1 or 2 cases, such as small IO latency, that improves 
> with remote-invalidation but for now we decided to disable it.
>
> I guess we can add a capability for the device that will indicate it 
> in the future.
>
> What is the difference you see ? and what is the scenario ?

Sorry for the late reply, it is not  very convenient for me to use the 
test environment recently. I tested various IO models last week.

In the low-stress model, such as 8k single-threaded writing, using 
remote-invalidation can increase latency and IOPS by about 20%.

In high pressure scenarios, such as 8k 32 threads and 256k 2 threads, 
the performance of whether to use remote-invalidation is almost the same.

In various scenarios, I have not measured that remote invalidation is worse.

The network card I use is BlueField(TM) SmartNIC 25GbE dual-port SFP28, 
PN: MBF1M332A-ASNAT. And I use vdbench for testing.

I would like to know what IO model you have measured that the 
remote-invalidation is worse?

Thanks,

Ruozhu

>
> Thanks,
>
> -Max.
>
> On 10/27/2021 11:50 AM, liruozhu wrote:
>> Hi Max,
>>
>> I was testing T10-PI feature of nvme-rdma recently, and found that 
>> when using T10-PI, nvme-rdma driver did not set 
>> NVME_SGL_FMT_INVALIDATE flag in the sgl type field, so the host 
>> software needs to do invalid rkey by itself. I tried to add the flag 
>> to test it. Turns out T10-PI feature is still working, and I get 
>> better IO latency with it enable.
>> I read the original patches in the mailing list, and found that patch 
>> v1 set this flag. But it was silent dropped on patch v5 without any 
>> comments. Is it for any special considerations to delete it?
>>
>> Thanks,
>> Ruozhu
>>
> .