[External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[Adrian Huang12]

> -----Original Message-----
> From: Christoph Hellwig <hch at lst.de>
> Sent: Thursday, October 8, 2026 8:37 PM
> To: John Garry <john.garry at linux.dev>
> Cc: Christoph Hellwig <hch at lst.de>; kbusch at kernel.org; Adrian Huang12
> <ahuang12 at lenovo.com>; sagi at grimberg.me; linux-nvme at lists.infradead.org;
> Jiwei JW10 Sun <sunjw10 at lenovo.com>; iommu at lists.linux.dev
> Subject: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors
> based on DMA optimized limitation"
> 
> On Thu, Oct 08, 2026 at 08:36:54AM +0100, John Garry wrote:
> > On 10/6/26 13:02, Christoph Hellwig wrote:
> >> This turns down to cause up to 15% performance regressions on
> >> standard large transfer workloads.  Revert it and hope the IOVA
> >> allocator can eventually be fixed to remove the need for the original
> workaround.
> >>
> >> This reverts commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.
> >>
> >
> > What was the block size and conditions for the original issue? The
> > commit just mentions that mdts=10 and we get a 4096KB max HW sectors.
> 
> Good question.  The original submitter and reporter are on Cc.

1. The block size depends on the configuration of the fio parameter 'bs'. See below for details.
2. Conditions for the original issue: The issue is triggered if the IO size > 128KB.

[Detail]
1. A NVME with mdts=10 (maximum data transfer size = 4096 KB)
  1.1 fio with bs=128k or below => Passed
  1.2 fio with bs > 128k => Failed
2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
  2.1 The issue does not reproduce regardless of the 'bs' configuration
3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
5. The issue only appears with high-core-count servers
  5.1 Reproduced servers: 448-core server and 384-core server
  5.2 Non-reproduced server: 128-core server

BTW, with this reverted patch, I'm setting up the test environment to check whether the issue does not appear with a newer kernel that includes the updated IOMMU subsystem. I'll share the results by the end of next week.

-- Adrian