[PATCH 1/1] nvme-rdma: post the command capsule inline when it fits

[Andrei-Alexandru Bleortu]

nvme_rdma_post_send() posts the 64-byte command capsule by reference,
so the HCA has to DMA-read the capsule from host memory before it can
send the command. That read is on the latency path of every I/O.

Ask for sizeof(struct nvme_command) bytes of inline data when creating
the queue pair, and create it without inline data if the device refuses.
Post a capsule that has no in-capsule data with IB_SEND_INLINE when it
fits in the granted inline size. The CPU then copies the capsule into
the send WQE and the HCA no longer reads it.

Providers copy inline data from sg_list[].addr as a kernel virtual
address, but nvme-rdma keeps the capsule's DMA address there. On the
inline path the SGE therefore points at qe->data. The capsule stays
DMA-mapped for the by-reference path.

An earlier version of this patch was measured with fio doing 4 KiB
random reads at queue depth 1 against a null_blk namespace exported by
nvmet-rdma. Both host pairs ran a 6.8 kernel over RoCEv2 at 100 Gb/s.
The mean completion latency fell by 0.998 us (95% CI 0.943-1.049)
between two AMD EPYC 7452 hosts with ConnectX-5 Ex, and by 0.589 us
(0.558-0.618) between two Xeon Platinum 8360Y hosts with ConnectX-6 Dx.
IOPS rose by 3.3% on the EPYC pair. A run there that asked for inline
data but kept posting by reference showed no change. That version asked
for 220 bytes and had no fallback, but it inlined the same capsules.

Assisted-by: LLM
Signed-off-by: Andrei-Alexandru Bleortu <me at andrei-z.com>
---
 drivers/nvme/host/rdma.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 5b9b540693b5..517098ebffb1 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -106,6 +106,7 @@ struct nvme_rdma_queue {
 	bool			pi_support;
 	int			cq_size;
 	struct mutex		queue_lock;
+	u32			max_inline_data;
 };
 
 struct nvme_rdma_setup_ctx {
@@ -265,7 +266,7 @@ static int nvme_rdma_wait_for_cm(struct nvme_rdma_queue *queue)
 static int nvme_rdma_create_qp(struct nvme_rdma_queue *queue, const int factor)
 {
 	struct nvme_rdma_device *dev = queue->device;
-	struct ib_qp_init_attr init_attr;
+	struct ib_qp_init_attr init_attr, attr;
 	int ret;
 
 	memset(&init_attr, 0, sizeof(init_attr));
@@ -284,9 +285,21 @@ static int nvme_rdma_create_qp(struct nvme_rdma_queue *queue, const int factor)
 		init_attr.create_flags |= IB_QP_CREATE_INTEGRITY_EN;
 	init_attr.qp_context = queue;
 
-	ret = rdma_create_qp(queue->cm_id, dev->pd, &init_attr);
+	/*
+	 * Ask for enough inline data to carry the command capsule in the send
+	 * WQE (see nvme_rdma_post_send()).  If the device cannot provide it,
+	 * create the QP without inline data and post the capsule by reference.
+	 */
+	attr = init_attr;
+	attr.cap.max_inline_data = sizeof(struct nvme_command);
+	ret = rdma_create_qp(queue->cm_id, dev->pd, &attr);
+	if (ret) {
+		attr = init_attr;
+		ret = rdma_create_qp(queue->cm_id, dev->pd, &attr);
+	}
 
 	queue->qp = queue->cm_id->qp;
+	queue->max_inline_data = ret ? 0 : attr.cap.max_inline_data;
 	return ret;
 }
 
@@ -1641,6 +1654,17 @@ static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
 	wr.opcode     = IB_WR_SEND;
 	wr.send_flags = IB_SEND_SIGNALED;
 
+	/*
+	 * Post a capsule without in-capsule data inline if it fits in the
+	 * granted inline size, so the HCA does not have to DMA-read it from
+	 * host memory.  The CPU copies inline data from a kernel virtual
+	 * address, so point the SGE at qe->data.
+	 */
+	if (num_sge == 1 && sge->length <= queue->max_inline_data) {
+		sge->addr = (uintptr_t)qe->data;
+		wr.send_flags |= IB_SEND_INLINE;
+	}
+
 	if (first)
 		first->next = ≀
 	else
-- 
2.56.0