Error when running fio against nvme-of rdma target (mlx5 driver)

[Mark Ruijter]

Hi Robin,

I ran into the exact same problem while testing with 4 connect-x6 cards, kernel 5.18-rc6.

[ 4878.273016] nvme nvme0: Successfully reconnected (3 attempts)
[ 4879.122015] nvme nvme0: starting error recovery
[ 4879.122028] infiniband mlx5_4: mlx5_handle_error_cqe:332:(pid 0): WC error: 4, Message: local protection error
[ 4879.122035] infiniband mlx5_4: dump_cqe:272:(pid 0): dump error cqe
[ 4879.122037] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 4879.122039] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 4879.122040] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 4879.122040] 00000030: 00 00 00 00 a9 00 56 04 00 00 00 ed 0d da ff e2
[ 4881.085547] nvme nvme3: Reconnecting in 10 seconds...

I assume this means that the problem has still not been resolved?
If so, I'll try to diagnose the problem.

Thanks,

--Mark

On 11/02/2022, 12:35, "Linux-nvme on behalf of Robin Murphy" <linux-nvme-bounces at lists.infradead.org on behalf of robin.murphy at arm.com> wrote:

    On 2022-02-10 23:58, Martin Oliveira wrote:
    > On 2/9/22 1:41 AM, Chaitanya Kulkarni wrote:
    >> On 2/8/22 6:50 PM, Martin Oliveira wrote:
    >>> Hello,
    >>>
    >>> We have been hitting an error when running IO over our nvme-of setup, using the mlx5 driver and we are wondering if anyone has seen anything similar/has any suggestions.
    >>>
    >>> Both initiator and target are AMD EPYC 7502 machines connected over RDMA using a Mellanox MT28908. Target has 12 NVMe SSDs which are exposed as a single NVMe fabrics device, one physical SSD per namespace.
    >>>
    >>
    >> Thanks for reporting this, if you can bisect the problem on your setup
    >> it will help others to help you better.
    >>
    >> -ck
    > 
    > Hi Chaitanya,
    > 
    > I went back to a kernel as old as 4.15 and the problem was still there, so I don't know of a good commit to start from.
    > 
    > I also learned that I can reproduce this with as little as 3 cards and I updated the firmware on the Mellanox cards to the latest version.
    > 
    > I'd be happy to try any tests if someone has any suggestions.

    The IOMMU is probably your friend here - one thing that might be worth 
    trying is capturing the iommu:map and iommu:unmap tracepoints to see if 
    the address reported in subsequent IOMMU faults was previously mapped as 
    a valid DMA address (be warned that there will likely be a *lot* of 
    trace generated). With 5.13 or newer, booting with "iommu.forcedac=1" 
    should also make it easier to tell real DMA IOVAs from rogue physical 
    addresses or other nonsense, as real DMA addresses should then look more 
    like 0xffff24d08000.

    That could at least help narrow down whether it's some kind of 
    use-after-free race or a completely bogus address creeping in somehow.

    Robin.