[External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[John Garry]

On 10/9/26 10:11, Adrian Huang12 wrote:
>> -----Original Message-----
>> From: Christoph Hellwig <hch at lst.de>
>> Sent: Thursday, October 8, 2026 8:37 PM
>> To: John Garry <john.garry at linux.dev>
>> Cc: Christoph Hellwig <hch at lst.de>; kbusch at kernel.org; Adrian Huang12
>> <ahuang12 at lenovo.com>; sagi at grimberg.me; linux-nvme at lists.infradead.org;
>> Jiwei JW10 Sun <sunjw10 at lenovo.com>; iommu at lists.linux.dev
>> Subject: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors
>> based on DMA optimized limitation"
>>
>> On Thu, Oct 08, 2026 at 08:36:54AM +0100, John Garry wrote:
>>> On 10/6/26 13:02, Christoph Hellwig wrote:
>>>> This turns down to cause up to 15% performance regressions on
>>>> standard large transfer workloads.  Revert it and hope the IOVA
>>>> allocator can eventually be fixed to remove the need for the original
>> workaround.
>>>>
>>>> This reverts commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.
>>>>
>>>
>>> What was the block size and conditions for the original issue? The
>>> commit just mentions that mdts=10 and we get a 4096KB max HW sectors.
>>
>> Good question.  The original submitter and reporter are on Cc.
> 
> 1. The block size depends on the configuration of the fio parameter 'bs'. See below for details.
> 2. Conditions for the original issue: The issue is triggered if the IO size > 128KB.
> 
> [Detail]
> 1. A NVME with mdts=10 (maximum data transfer size = 4096 KB)
>    1.1 fio with bs=128k or below => Passed
>    1.2 fio with bs > 128k => Failed


That block size limit is an important detail missing from 
3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd commit message.

And now we seem to have a conflict, as Christoph wants block size > IOVA 
rcache limit but you find it may cause this soft lockup.

> 2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
>    2.1 The issue does not reproduce regardless of the 'bs' configuration
> 3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
> 4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
> 5. The issue only appears with high-core-count servers
>    5.1 Reproduced servers: 448-core server and 384-core server
>    5.2 Non-reproduced server: 128-core server

How long does it take for the lockup to trigger? Pretty quickly, I guess.

Does changing the number of CPUs affect throughput (before the lockup)? 
Are you using just a single disk?

> 
> BTW, with this reverted patch, I'm setting up the test environment to check whether the issue does not appear with a newer kernel that includes the updated IOMMU subsystem. I'll share the results by the end of next week.
> 

thanks