[PATCH v3 0/9] Introduce per-device completion queue pools

[Sagi Grimberg]

> I also prefer more the CQ pools per ULP approach (like we did with the 
> MR pools per QP) in the first stage.

Fair enough.

> For example, we saw a big 
> improvement in NVMEoF performance when we did CQ moderation (currently 
> local implementation in our labs). If we'll moderate shared CQ (iser + 
> nvmf CQ) we can ruin other ULP performance. ISER/SRP/NVMEoF/NFS has 
> different needs and different architectures, so even adaptive moderation 
> will not supply the best performance in that case.

Here I disagree. Using hard-coded or pre-configured adaptive moderation
is something we should move away from. I have a generic adaptive
moderation implementation for rdma and nvme in the works, and once that
does its job correctly, it should benefit everyone equally. Moreover,
IMO it even *supports* the notion of sharing CQs across ULPs because the
more consumers we have on a CQ, the better the adaptive moderation
works.

In fact, if it works well I will vote to turn it on by default and not
even allow ULPs to control it but only the user (via sysctl or
something) because if you think about it, ULPs can't really choose
better then the core.