consultation about NVME_SGL_FMT_INVALIDATE not set when using nvme-rdma t10-pi

[Max Gurtovoy]

Hi Ruozhu,

The performance tests we did for all the IO sizes showed better 
performance with local invalidation in most of the cases (mainly for 
large IOs, the remote invalidation task is harder for the HW).

Maybe there is 1 or 2 cases, such as small IO latency, that improves 
with remote-invalidation but for now we decided to disable it.

I guess we can add a capability for the device that will indicate it in 
the future.

What is the difference you see ? and what is the scenario ?

Thanks,

-Max.

On 10/27/2021 11:50 AM, liruozhu wrote:
> Hi Max,
>
> I was testing T10-PI feature of nvme-rdma recently, and found that 
> when using T10-PI, nvme-rdma driver did not set 
> NVME_SGL_FMT_INVALIDATE flag in the sgl type field, so the host 
> software needs to do invalid rkey by itself. I tried to add the flag 
> to test it. Turns out T10-PI feature is still working, and I get 
> better IO latency with it enable.
> I read the original patches in the mailing list, and found that patch 
> v1 set this flag. But it was silent dropped on patch v5 without any 
> comments. Is it for any special considerations to delete it?
>
> Thanks,
> Ruozhu
>