next-20250627: IOMMU DMA warning during NVMe I/O completion after 06cae0e3f61c

[Keith Busch]

On Mon, Jun 30, 2025 at 03:33:43PM +0200, Christoph Hellwig wrote:
> Hi Ben,
> 
> > [   32.857521] iommu_dma_unmap_page+0xc4/0xe8 (P)
> 
> Can you resolve this to a source location for me. i.e.
> 
> gdb vmlinux
> 
> l *(iommu_dma_unmap_page+0xc4)
> 
> Also what IOMMU driver is this device using?  It looks like it
> might not support a 4k IOMMU page size.

I think the PRP handling is broken. At the very least, handling the last
element is wrong if it appears at the end of the list, so I think we
need something like this:

---
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -701,7 +701,7 @@ static void nvme_free_prps(struct request *req)
        prp_list = iod->descriptors[desc];
        do {
                dma_unmap_page(dma_dev, dma_addr, dma_len, dir);
-               if (i == NVME_CTRL_PAGE_SIZE >> 3) {
+               if (i == NVME_CTRL_PAGE_SIZE >> 3 && length > NVME_CTRL_PAGE_SIZE) {
                        prp_list = iod->descriptors[++desc];
                        i = 0;
                }
--

But even that, the PRP setup doesn't match the teardown. We're calling
dma_map_page() on each PRP even if consecutive PRP's came from the same
dma mapping segment. So even if it had been coalesced, but if the device
doesn't support SGLs, then it would use the prp unmap path.

Ben, can you check if your device supports sgl?

  # nvme id-ctrl /dev/nvme0 | grep sgl