Why NVMe MSIx vectors affinity set across NUMA nodes?

[Ganapatrao Kulkarni]

On Tue, Jan 23, 2018 at 7:00 PM, Sagi Grimberg <sagi at grimberg.me> wrote:
>
>>> AFAIK, usually drivers sets default affinity and it is likely be node
>>> affinity for NUMA systems.
>>> Later, it is the user-space(like irqbalance etc) which decides the
>>> affinity not the driver.
>>
>>
>> Relying on userspace to provide an optimal setting is a bad idea,
>> especially for NVMe where we have submission queue cpu affinity that
>> doesn't work very efficiently if the completion affinity doesn't match.
>
>
> I tend to agree, also application locality is equally as important as
> device locality. so spreading across numa nodes will help applications
> running on the far numa node as well.
>
> Also, a recent thread [1] related to PCI_IRQ_AFFINITY not allowing
> userspace to modify irq affinity suggested that maybe that can be
> supported, but not sure what happened to it.
>

application uses libnuma to align to numa locality.
here the driver is breaking the affinity.
certainly having affinity with remote node cpu will add latency to
interrupt response time.
here it is for some NVMe queues.

> [1] https://www.spinics.net/lists/netdev/msg464301.html

thanks
Ganapat