DMA pool lock contention

[Keith Busch]

On Thu, Apr 17, 2025 at 12:27:43PM -0700, Caleb Sander Mateos wrote:
> On Thu, Apr 17, 2025 at 11:18 AM Keith Busch <kbusch at kernel.org> wrote:
> >
> > On Thu, Apr 17, 2025 at 10:06:11AM -0700, Caleb Sander Mateos wrote:
> > > Hi Linux NVMe folks,
> > > On a 32 KB NVMe passthru read workload, we see 2.4% of CPU time is
> > > spent in _raw_spin_lock_irqsave called from dma_pool_alloc and
> > > dma_pool_free.
> >
> > Just curious, do you have multiple cpu NUMA nodes here? Or do these CPUs
> > share a last-level-cache?
> 
> CPUs from 2 NUMA nodes. Half of them are hypertwins of the others, so
> the contention isn't quite as bad as it otherwise would be. Just
> making the DMA pools per-NUMA-node would help considerably.

Cool, I was hoping that was the case. I measure some contention, but
it's not as bad as what you're showing. I only have a single socket for
my test machine, so same L3 cache for all CPUs. Let's try a dmapool for
every compute NUMA node for each device.