[RFC PATCH 1/1] block: charge passthrough requests to the submitter's cgroup

[Tao Cui]

From: Tao Cui <cuitao at kylinos.cn>

Passthrough requests (SG_IO, bsg, nvme passthrough ioctls and uring
commands) are dispatched via blk_execute_rq{,_nowait}() without ever
passing through submit_bio(), so the bio mapped by blk_rq_map_user()
carries no blkcg association: the transferred bytes never show up in
cgroup io.stat, and every rq_qos policy on the queue (iocost,
iolatency, wbt) is bypassed, as is blk-throttle, which hooks
submit_bio_noacct() directly rather than going through rq_qos.

A quick demonstration on a scsi_debug device with iocost enabled and
vrate pinned to its 1% floor: a direct fio writer was throttled ~10x
while the same cgroup issuing sg_dd writes ran at full device speed
with zero io.stat accounting.

Associate the mapped bio with the submitter's blkcg at dispatch time
and run the regular bio accounting (blk_cgroup_bio_start()) and
rq_qos throttle paths with it.  DRV_IN/DRV_OUT commands are mapped to
READ/WRITE so io.stat classifies their bytes normally; request
completion already pairs with the throttle through bio_endio() ->
rq_qos_done_bio().

The charge is gated by opcode (READ/WRITE/DRV_IN/DRV_OUT) and to
queues that already have a gendisk: commands issued during device
probing (SCSI INQUIRY etc.) have no gendisk yet and stay exempt,
following the same probe-exemption reasoning as the passthrough
iostats support.

RFC notes:
 - validated on linux-next with scsi_debug + sg_dd (SG_IO) and qemu
   emulated nvme + a NVME_IOCTL_IO64_CMD loop: in both cases the
   transferred bytes are fully accounted (wbytes/wios) in the issuing
   cgroup, and the issuing task is observed waiting on the iocost
   waitqueue with vrate clamped;
 - iolatency gains the same coverage for free, and so does wbt:
   passthrough writes now pass through wbt_wait() like bio-path
   writes, a behavior change worth calling out even though wbt
   targets buffered writeback and direct passthrough rarely hits it;
   blk-throttle does not: it hooks
   submit_bio_noacct() directly and is not an rq_qos policy, so
   io.max stays unenforced for passthrough (measured).  Its
   queue-and-resubmit throttling model would also need a synchronous
   variant for request-bound bios;
 - nvme uring commands share the same blk_execute_rq_nowait()
   dispatch as the validated ioctl path;
 - the charge runs in the submitter's context and may sleep in
   rq_qos throttling; all data-op callers found run in sleepable
   context, but this deserves reviewer attention.

Signed-off-by: Tao Cui <cuitao at kylinos.cn>
---
 block/blk-mq.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index a26a11c73ee3..2dd59f6df321 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -10,6 +10,7 @@
 #include <linux/backing-dev.h>
 #include <linux/bio.h>
 #include <linux/blkdev.h>
+#include "blk-cgroup.h"
 #include <linux/blk-integrity.h>
 #include <linux/kmemleak.h>
 #include <linux/mm.h>
@@ -1400,6 +1401,73 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
 	plug->rq_count++;
 }
 
+/*
+ * Passthrough bios are mapped directly onto requests via
+ * blk_rq_map_user() and never pass through submit_bio(), so they carry
+ * no blkcg association and are invisible to cgroup io.stat and to every
+ * rq_qos policy (iocost, blk-throttle, iolatency).  Charge the ones that
+ * carry data to the submitter's blkcg at dispatch time and run the
+ * regular bio accounting and rq_qos throttle paths with the associated
+ * bio.
+ *
+ * Gated by opcode (READ/WRITE/DRV_IN/DRV_OUT, the latter two mapped
+ * to READ/WRITE for io.stat classification) and to queues that
+ * already have a gendisk: commands issued during device probing (SCSI
+ * INQUIRY and friends) have no gendisk yet and stay exempt.  The
+ * request bios may carry a stale ->bi_blkg from the mempool; the
+ * association helper drops the old reference and re-associates.
+ */
+static void blk_mq_pt_charge(struct request *rq)
+{
+	struct bio *bio = rq->bio;
+	enum req_op op = req_op(rq);
+
+	if (!bio || !rq->q->disk)
+		return;
+
+	switch (op) {
+	case REQ_OP_READ:
+	case REQ_OP_WRITE:
+	case REQ_OP_DRV_IN:
+	case REQ_OP_DRV_OUT:
+		break;
+	default:
+		return;
+	}
+	if (op == REQ_OP_DRV_IN)
+		op = REQ_OP_READ;
+	else if (op == REQ_OP_DRV_OUT)
+		op = REQ_OP_WRITE;
+
+	if (!bio->bi_bdev)
+		bio->bi_bdev = rq->q->disk->part0;
+	bio->bi_opf &= ~REQ_OP_MASK;
+	bio->bi_opf |= op;
+#ifdef CONFIG_BLK_CGROUP
+	/*
+	 * Issued from kthreads the css is root and the charge is a
+	 * no-op through the root exemptions; data-op issuers that
+	 * matter run in the submitter's task context.
+	 */
+	{
+		struct cgroup_subsys_state *css;
+
+		rcu_read_lock();
+		css = task_css(current, io_cgrp_id);
+		bio_associate_blkg_from_css(bio, css);
+		rcu_read_unlock();
+	}
+#endif
+	blk_cgroup_bio_start(bio);
+
+	/*
+	 * The rq_qos throttle path may sleep on the waitqueues like any
+	 * bio submitter; all callers found (ioctl / uring_cmd submit,
+	 * target and error handling kthreads) run in sleepable context.
+	 */
+	rq_qos_throttle(rq->q, bio);
+}
+
 /**
  * blk_execute_rq_nowait - insert a request to I/O scheduler for execution
  * @rq:		request to insert
@@ -1412,6 +1480,7 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
  * Note:
  *    This function will invoke @done directly if the queue is dead.
  */
+
 void blk_execute_rq_nowait(struct request *rq, bool at_head)
 {
 	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
@@ -1419,6 +1488,7 @@ void blk_execute_rq_nowait(struct request *rq, bool at_head)
 	WARN_ON(irqs_disabled());
 	WARN_ON(!blk_rq_is_passthrough(rq));
 
+	blk_mq_pt_charge(rq);
 	blk_account_io_start(rq);
 
 	if (current->plug && !at_head) {
@@ -1484,6 +1554,8 @@ blk_status_t blk_execute_rq(struct request *rq, bool at_head)
 	WARN_ON(irqs_disabled());
 	WARN_ON(!blk_rq_is_passthrough(rq));
 
+	blk_mq_pt_charge(rq);
+
 	rq->end_io_data = &wait;
 	rq->end_io = blk_end_sync_rq;
 
-- 
2.43.0