[External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[Adrian Huang12]

> -----Original Message-----
> From: John Garry <john.garry at linux.dev>
> Sent: Thursday, October 15, 2026 6:31 PM
> To: Christoph Hellwig <hch at lst.de>; Adrian Huang12
> <ahuang12 at lenovo.com>
> Cc: kbusch at kernel.org; sagi at grimberg.me; linux-nvme at lists.infradead.org;
> Jiwei JW10 Sun <sunjw10 at lenovo.com>; iommu at lists.linux.dev
> Subject: Re: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors
> based on DMA optimized limitation"
> 
> On 10/12/26 15:39, John Garry wrote:
> > On 10/12/26 13:17, Christoph Hellwig wrote:
> >> On Mon, Oct 12, 2026 at 09:42:04AM +0100, John Garry wrote:
> >>> And now we seem to have a conflict, as Christoph wants block size >
> >>> IOVA rcache limit but you find it may cause this soft lockup.
> >>>
> >>>> 2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
> >>>>     2.1 The issue does not reproduce regardless of the 'bs'
> >>>> configuration
> >>>> 3. The condition '128 KB' is identified because the IOMMU scalable
> >>>> IOVA mechanism supports a maximum transfer size of 128 KB.
> >>>> 4. IOMMU pass-through mode does not trigger the issue with the
> >>>> 'mdts=10' nvme disk.
> >>>> 5. The issue only appears with high-core-count servers
> >>>>     5.1 Reproduced servers: 448-core server and 384-core server
> >>>>     5.2 Non-reproduced server: 128-core server
> >>>
> >>> How long does it take for the lockup to trigger? Pretty quickly, I
> >>> guess.
> >>>
> >>> Does changing the number of CPUs affect throughput (before the
> >>> lockup)? Are you using just a single disk?
> >>
> >> If the iova cache locks up under a perfectly normal workloads it is
> >> broken.
> >
> >
> >> We really need to fix the code there.
> >
> > Sure, but it would help to first fully understand the conditions to fix it.
> 
> BTW, @Adrian, it would be good to run your test with the IOMMU in strict
> mode as well, just to give an idea if there is a problem there as well.

Sorry for the delayed update. The original test system used three years ago
is no longer available, so I needed additional time to assemble two new
test systems.

== Test Environment ==
  1. 448-core AMD server (2-socket) with an NVMe device (mdts=10)
  2. 960-core Intel server (8-socket) with an NVMe device (mdts=10)
  3. The issue can be reproduced with a single NVMe device (mdts=10)
  4. IOMMU lazy mode is enabled. We did not test strict mode three years
    ago (and now). @John, let me know if you need me to run the
    strict mode test.
  5. stress-ng and fio used for reproduction
  6. A 100-run test was performed for each tested kernel

== Summary ==
  1. Since the original patch was merged in v6.4-rc3, I used v6.3 as the
    baseline to verify whether the issue could be reproduced on these
    two systems. The issue was successfully reproduced on both platforms.
  2. On v6.7 and later kernels, after reverting commit 3710e2b056cb, the
    issue no longer occurs.
  3. Detailed test information and logs can be found in [1].

== Detailed Results (Both Systems) ==
  * v7.3.0-rc7 + revert 3710e2b056cb: Passed
  * v6.12 + revert 3710e2b056cb: Passed
  * v6.8 + revert 3710e2b056cb: Passed
  * v6.7 + revert 3710e2b056cb: Passed
  * v6.6 + revert 3710e2b056cb: Failed
  * v6.5 + revert 3710e2b056cb: Failed
  * Vanilla 6.3: Failed

Based on these results, I believe it is safe to apply the revert. Please feel
free to add:

Tested-by: Adrian Huang <ahuang12 at lenovo.com>

Also, my apologies for introducing the performance regression. The results
show that changes in the IOMMU subsystem introduced around v6.7
significantly improved scalability, although I have not yet investigated 
which specific commit resolved the underlying issue.

[1] Detailed test info: https://hackmd.io/@lQBEPrYxRoOVV2MatUYusg/B14XRbZ1nGx

-- Adrian