[PATCH 0/5] pci: nvmet: support completion queue sharing by multiple submission queues

[Chaitanya Kulkarni]

On 4/23/25 22:13, Wilfred Mallawa wrote:
> From: Wilfred Mallawa<wilfred.mallawa at wdc.com>
>
> Hi all,
>
> For the NVMe PCI transport, the NVMe specification allows different
> submission queues (SQs) to share completion queues (CQs), however,
> this is not supported in the current NVMe target implementation.
> Until now, the nvmet target implementation enforced a 1:1 relationship
> between SQs and CQs, which is not specification compliant for the NVMe
> PCI transport.
>
> This patch series adds support for CQ sharing between multiple SQs in the
> NVMe target driver, in line with the NVMe PCI transport specification.
> This series implements reference counting for completion queues to ensure
> proper lifecycle management when shared across multiple submission queues.
> This ensures that we retain CQs until all referencing SQs are deleted
> first, thereby avoiding premature CQ deletions.
>
> Sample callchain with CQ refcounting for the PCI endpoint target
> (pci-epf)
> =================================================================
>
> i.   nvmet_execute_create_cq -> nvmet_pci_epf_create_cq -> nvmet_cq_create
>       -> nvmet_cq_init			[cq refcount = 1]
>
> ii.  nvmet_execute_create_sq -> nvmet_pci_epf_create_sq -> nvmet_sq_create
>       -> nvmet_sq_init -> nvmet_cq_get	[cq refcount = 2]
>
> iii. nvmet_execute_delete_sq -> nvmet_pci_epf_delete_sq ->
>       nvmet_sq_destroy -> nvmet_cq_put	[cq refcount = 1]
>
> iv.  nvmet_execute_delete_cq -> nvmet_pci_epf_delete_cq -> nvmet_cq_put
> 					[cq refcount = 0]
>
> For NVMe over fabrics, CQ sharing is not supported per specification,
> however, the fabrics drivers are updated to integrate the new
> API changes. No functional change is intended here.
>
> Testing
> =======
>
> Core functionality changes were tested with a Rockchip-based Rock5B PCIe
> endpoint setup using the pci-epf driver. The host kernel was modified to
> support queue sharing. In the test setup, this resulted in IO SQs 1 & 2
> using IO CQ 1 and IO SQ 3 & 4 using IO CQ 2.
>
> Testing methodology includes:
>
> For PCI:
>
> 1. Boot up host
> 2. Assert that the endpoint device is detected as an NVMe drive
>     (IO CQs/SQs are created)
> 3. Run FIOs
> 4. Unbind NVMe driver (IO SQs then CQs are deleted)
> 5. Rebind NVMe driver (IO SQs then CQs are created)
> 6. Run FIOs
>
> For NVMe over fabrics: Using NVMe loop driver:
>
> Note that there is no queue sharing supported for fabrics.
>
> 1. Connect command (IO queues are created)
> 2. Run FIOs
> 3. Disconnect command (IO queues are deleted)
>
> Thanks!

FWIW,

Reviewed-by: Chaitanya Kulkarni <kch at nvidia.com>

-ck