Why NVMe MSIx vectors affinity set across NUMA nodes?

[Keith Busch]

On Wed, Jan 24, 2018 at 09:39:02PM +0200, Sagi Grimberg wrote:
> 
> > > application uses libnuma to align to numa locality.
> > > here the driver is breaking the affinity.
> > > certainly having affinity with remote node cpu will add latency to
> > > interrupt response time.
> > > here it is for some NVMe queues.
> > 
> > I bet you can't come up with an IRQ CPU affinity map that performs better
> > than the current setup.
> 
> :)
> 
> While I agree that managed affinity will probably get the optimal
> affinitization in 99% of the cases, this is the second complaint we've
> had that managed affinity breaks an existing user interface (even though
> it was a sure way to allow userspace to screw up for years).

Well, this the only complaint for NVMe, and it doesn't seem aware of
how this work. If libnuma is used to run on a specific node, interrupts
will occur only on that node. An interrupt sent to a remote node means
you submitted a command from there, and handling the interrupt there is
cheaper than bouncing hot cache lines across nodes.