[bug report] NVMe/IB: reset_controller need more than 1min

[Yi Zhang]

On Wed, Mar 16, 2022 at 11:16 PM Sagi Grimberg <sagi at grimberg.me> wrote:
>
>
> >> Hi Yi Zhang,
> >>
> >> thanks for testing the patches.
> >>
> >> Can you provide more info on the time it took with both kernels ?
> >
> > Hi Max
> > Sorry for the late response, here are the test results/dmesg on
> > debug/non-debug kernel with your patch:
> > debug kernel: timeout
> > # time nvme connect -t rdma -a 172.31.0.202 -s 4420 -n testnqn
> > real    0m16.956s
> > user    0m0.000s
> > sys     0m0.237s
> > # time nvme reset /dev/nvme0
> > real    1m33.623s
> > user    0m0.000s
> > sys     0m0.024s
> > # time nvme disconnect-all
> > real    1m26.640s
> > user    0m0.000s
> > sys     0m9.969s
> >
> > host dmesg:
> > https://pastebin.com/8T3Lqtkn
> > target dmesg:
> > https://pastebin.com/KpFP7xG2
> >
> > non-debug kernel: no timeout issue, but still 12s for reset, and 8s
> > for disconnect
> > host:
> > # time nvme connect -t rdma -a 172.31.0.202 -s 4420 -n testnqn
> >
> > real    0m4.579s
> > user    0m0.000s
> > sys     0m0.004s
> > # time nvme reset /dev/nvme0
> >
> > real    0m12.778s
> > user    0m0.000s
> > sys     0m0.006s
> > # time nvme reset /dev/nvme0
> >
> > real    0m12.793s
> > user    0m0.000s
> > sys     0m0.006s
> > # time nvme reset /dev/nvme0
> >
> > real    0m12.808s
> > user    0m0.000s
> > sys     0m0.006s
> > # time nvme disconnect-all
> >
> > real    0m8.348s
> > user    0m0.000s
> > sys     0m0.189s
>
> These are very long times for a non-debug kernel...
> Max, do you see the root cause for this?
>
> Yi, does this happen with rxe/siw as well?

Hi Sagi

rxe/siw will take less than 1s
with rdma_rxe
# time nvme reset /dev/nvme0
real 0m0.094s
user 0m0.000s
sys 0m0.006s

with siw
# time nvme reset /dev/nvme0
real 0m0.097s
user 0m0.000s
sys 0m0.006s

This is only reproducible with mlx IB card, as I mentioned before, the
reset operation time changed from 3s to 12s after the below commit,
could you check this commit?

commit 5ec5d3bddc6b912b7de9e3eb6c1f2397faeca2bc
Author: Max Gurtovoy <maxg at mellanox.com>
Date:   Tue May 19 17:05:56 2020 +0300

    nvme-rdma: add metadata/T10-PI support


>


-- 
Best Regards,
  Yi Zhang