[RFC PATCH 0/3] nvme-pci: dynamic interrupt moderation
Fengnan
changfengnan at bytedance.com
Mon Nov 2 04:22:23 PST 2026
在 29/10/26 10:14, changfengnan 写道:
>
>> From: "Keith Busch"<kbusch at kernel.org>
>> Date: Wed, Oct 28, 2026, 23:15
>> Subject: Re: [RFC PATCH 0/3] nvme-pci: dynamic interrupt moderation
>> To: "Christoph Hellwig"<hch at lst.de>
>> Cc: "Fengnan"<changfengnan at bytedance.com>, "Keith Busch"<kbusch at meta.com>, <linux-nvme at lists.infradead.org>, <linux-pci at vger.kernel.org>, <tglx at kernel.org>, <namjain at linux.microsoft.com>
>> On Wed, Oct 28, 2026 at 04:07:12PM +0100, Christoph Hellwig wrote:
>>> On Thu, Oct 22, 2026 at 05:43:54PM +0800, Fengnan wrote:
>>>> Hi Keith:
>>>>
>>>> I made some modifications to DIM and then implemented a solution
>>>> using the DIM framework.
>>>> Based on current testing, the results are now comparable to those of
>>>> the previous patch.
>>>> Here are the preliminary test results:
>>>> https://docs.google.com/spreadsheets/d/1e8LQPnaDhmv50XlAM3XxI0TMQkhbk0bTZwlJP1Hzzdk/edit?gid=188992110#gid=188992110
>>>>
>>>> If you agree with this approach, I’ll continue testing and optimizing.
>>>
>>> І can't speak for Keith, but I'd really love to see this going forward!
>>
>> Same! This is a great problem to solve, but I'm just sorry I don't have
>> more cycles to work on the cause. I can quickly respin the msix noflush
>> pre-patches to address the naming issues tglx mentioned, and we can take
>> nvme in whichever direction has the best performance.
>>
>
> That sounds good.
> I'm currently working on continuously optimizing and testing both
> approaches (using DIM and not using DIM).
> I plan to update the code and add new data again next Monday.
>
> Thanks.
Hi Keith and Christoph,
I've prepared two implementations of adaptive IRQ polling: one reusing
the DIM tuning state machine and one using an NVMe-specific policy.
I tested both with a 158-case fio matrix and additional latency and
CPU-contention tests. The results show broadly similar throughput gains
over their respective OFF baselines in the tested workloads.
Test data:
[1] Non-DIM version:
https://docs.google.com/spreadsheets/d/17kmXH9Zkr2jrN-19BwaInSx53HwOdRPJFrZ-Hjlk0_Y/edit?gid=300100#gid=300100
[2] DIM version:
https://docs.google.com/spreadsheets/d/18JADnBXWAGkffY_3LsPJ6ftA9DGH6qpNbKvP-lQWLnE/edit?gid=300200#gid=300200
Both implementations are opt-in and choose between IRQ completion and
timer-driven polling per I/O queue. They evaluate samples of at least
32,768 completions and use a target polling interval of 10 us. A polling
trial must improve the completion rate by more than 1%. Once successful,
polling can continue while its rate stays within roughly 1% of the saved
IRQ baseline, subject to a completion-progress guard.
Neither implementation uses I/O latency as a tuning metric.
The non-DIM implementation uses explicit rules inside the NVMe driver.
It admits polling trials at an IRQ completion rate of about 100k
completions/s per queue and applies a fixed completion-count backoff
after three consecutive failures.
The DIM implementation uses IRQ and POLL as two profiles. It explores
polling without that fixed admission threshold and requires a trial
to gain more than both 1% and one completion/ms. While parked on IRQ,
it increases the wait between trials from 1 to 2, 4, 8 and then 16 valid
IRQ samples. It also discards stale samples and stops the polling timer
when the queue drains.
During development, I felt that the version with `dim` required more
rules to achieve a similar effect. So, I still prefer the version
without `dim` because it’s simpler, easier to understand, and easier
to maintain.
Which approach would you prefer?
All the patches are in the attachment.
-------------- next part --------------
From ab88b22a6f0ced47fa7ced19b24a846520e449ef Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan at bytedance.com>
Date: Mon, 19 Oct 2026 11:03:45 +0000
Subject: [PATCH DIM 1/4] lib/dim: allow DIMLIB without CONFIG_NET
Only net_dim.c needs networking: it manages net_device moderation
profiles. dim.c and rdma_dim.c are generic, yet DIMLIB as a whole
depends on NET, so a block driver cannot select it on a NET=n kernel.
Drop the NET dependency from DIMLIB and build net_dim.o only when
CONFIG_NET is enabled. All net_dim users already depend on NET.
Signed-off-by: Fengnan Chang <changfengnan at bytedance.com>
---
lib/Kconfig | 1 -
lib/dim/Makefile | 3 ++-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/Kconfig b/lib/Kconfig
index 55748b68714e036ddec1f0b751617a052c9f7c33..af2fc199338f45f6cd485ec56e06f85d41e86ff5 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -475,7 +475,6 @@ config SIGNATURE
config DIMLIB
tristate
- depends on NET
help
Dynamic Interrupt Moderation library.
Implements an algorithm for dynamically changing CQ moderation values
diff --git a/lib/dim/Makefile b/lib/dim/Makefile
index 5b9bfaac7ac1fd9c6005ff284b4c874b958684ad..47a0d5fe69649e070a0636d95bafe1125f49443b 100644
--- a/lib/dim/Makefile
+++ b/lib/dim/Makefile
@@ -4,4 +4,5 @@
obj-$(CONFIG_DIMLIB) += dimlib.o
-dimlib-y := dim.o net_dim.o rdma_dim.o
+dimlib-y := dim.o rdma_dim.o
+dimlib-$(CONFIG_NET) += net_dim.o
-------------- next part --------------
From af0e78ec947217a0995cddc42fce6526b015c159 Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan at bytedance.com>
Date: Mon, 19 Oct 2026 11:03:46 +0000
Subject: [PATCH DIM 2/4] lib/dim: factor the tuning state machine out of net_dim
net_dim_decision() walks a profile table: it steps towards the better
profile, turns when a step does not help, parks on top, and parks tired
after too many steps. Only the statistics comparison and the number of
profiles are specific to networking.
Move the state machine into dim.c as dim_decide(), which takes the
comparison as a callback and the number of profiles as an argument, so
that other drivers can reuse the DIM search with their own measurements
and profile tables. The tired limit becomes twice the number of
profiles, which is unchanged for net_dim. net_dim() now calls
dim_decide() with net_dim_stats_compare() and NET_DIM_PARAMS_NUM_PROFILES;
no functional change.
Signed-off-by: Fengnan Chang <changfengnan at bytedance.com>
---
include/linux/dim.h | 21 +++++++++++
lib/dim/dim.c | 88 +++++++++++++++++++++++++++++++++++++++++++
lib/dim/net_dim.c | 91 ++-------------------------------------------
3 files changed, 112 insertions(+), 88 deletions(-)
diff --git a/include/linux/dim.h b/include/linux/dim.h
index 06543fd40fccbd674f6880ebaa6ee7cf23ecde0f..1a3f7cc1ddc5d3c54055a0744ffb4665f6e5f8de 100644
--- a/include/linux/dim.h
+++ b/include/linux/dim.h
@@ -341,6 +341,27 @@ void dim_park_on_top(struct dim *dim);
*/
void dim_park_tired(struct dim *dim);
+/**
+ * dim_decide - one tuning decision on a new statistics sample
+ * @dim: DIM context
+ * @curr_stats: statistics of the sample just measured
+ * @compare: callback returning %DIM_STATS_BETTER,
+ * %DIM_STATS_SAME or %DIM_STATS_WORSE, given @dim and both samples
+ * @nprofiles: number of profiles, from 1 to 127
+ *
+ * Run the DIM tuning state machine used by net_dim: step towards the
+ * better profile, turn when a step does not help, park on top, and park
+ * tired after 2 * @nprofiles steps without settling. The caller serializes
+ * access to @dim and applies the new profile. profile_ix must be below
+ * @nprofiles; a tired parked context must have a nonzero tired count.
+ *
+ * Return: true if profile_ix changed.
+ */
+bool dim_decide(struct dim *dim, struct dim_stats *curr_stats,
+ int (*compare)(struct dim *dim, struct dim_stats *curr,
+ struct dim_stats *prev),
+ u8 nprofiles);
+
/**
* dim_calc_stats - calculate the difference between two samples
* @start: start sample
diff --git a/lib/dim/dim.c b/lib/dim/dim.c
index 97c3d084ebf023031226faf869652ee65af59337..11b0a080c019aa1bb8802f8459ff7637fbe91b03 100644
--- a/lib/dim/dim.c
+++ b/lib/dim/dim.c
@@ -54,6 +54,94 @@ void dim_park_tired(struct dim *dim)
}
EXPORT_SYMBOL(dim_park_tired);
+static int dim_step(struct dim *dim, u8 nprofiles)
+{
+ if (dim->tired == nprofiles * 2)
+ return DIM_TOO_TIRED;
+
+ switch (dim->tune_state) {
+ case DIM_PARKING_ON_TOP:
+ case DIM_PARKING_TIRED:
+ break;
+ case DIM_GOING_RIGHT:
+ if (dim->profile_ix == nprofiles - 1)
+ return DIM_ON_EDGE;
+ dim->profile_ix++;
+ dim->steps_right++;
+ break;
+ case DIM_GOING_LEFT:
+ if (dim->profile_ix == 0)
+ return DIM_ON_EDGE;
+ dim->profile_ix--;
+ dim->steps_left++;
+ break;
+ }
+
+ dim->tired++;
+ return DIM_STEPPED;
+}
+
+static void dim_exit_parking(struct dim *dim, u8 nprofiles)
+{
+ dim->tune_state = dim->profile_ix ? DIM_GOING_LEFT : DIM_GOING_RIGHT;
+ dim_step(dim, nprofiles);
+}
+
+bool dim_decide(struct dim *dim, struct dim_stats *curr_stats,
+ int (*compare)(struct dim *dim, struct dim_stats *curr,
+ struct dim_stats *prev),
+ u8 nprofiles)
+{
+ int prev_state = dim->tune_state;
+ int prev_ix = dim->profile_ix;
+ int stats_res;
+ int step_res;
+
+ switch (dim->tune_state) {
+ case DIM_PARKING_ON_TOP:
+ stats_res = compare(dim, curr_stats, &dim->prev_stats);
+ if (stats_res != DIM_STATS_SAME)
+ dim_exit_parking(dim, nprofiles);
+ break;
+
+ case DIM_PARKING_TIRED:
+ dim->tired--;
+ if (!dim->tired)
+ dim_exit_parking(dim, nprofiles);
+ break;
+
+ case DIM_GOING_RIGHT:
+ case DIM_GOING_LEFT:
+ stats_res = compare(dim, curr_stats, &dim->prev_stats);
+ if (stats_res != DIM_STATS_BETTER)
+ dim_turn(dim);
+
+ if (dim_on_top(dim)) {
+ dim_park_on_top(dim);
+ break;
+ }
+
+ step_res = dim_step(dim, nprofiles);
+ switch (step_res) {
+ case DIM_ON_EDGE:
+ dim_park_on_top(dim);
+ break;
+ case DIM_TOO_TIRED:
+ dim_park_tired(dim);
+ break;
+ }
+
+ break;
+ }
+
+ if (prev_state != DIM_PARKING_ON_TOP ||
+ dim->tune_state != DIM_PARKING_ON_TOP)
+ dim->prev_stats = *curr_stats;
+
+ return dim->profile_ix != prev_ix;
+}
+EXPORT_SYMBOL(dim_decide);
+
bool dim_calc_stats(const struct dim_sample *start,
const struct dim_sample *end,
struct dim_stats *curr_stats)
diff --git a/lib/dim/net_dim.c b/lib/dim/net_dim.c
index d8d4f6553559674fca5f3a3c859fe5157d6ec063..9ab1a9ec65fd85900609e9a7952c9d481aeb11bb 100644
--- a/lib/dim/net_dim.c
+++ b/lib/dim/net_dim.c
@@ -233,40 +233,7 @@ void net_dim_set_tx_mode(struct net_device *dev, u8 tx_mode)
}
EXPORT_SYMBOL(net_dim_set_tx_mode);
-static int net_dim_step(struct dim *dim)
-{
- if (dim->tired == (NET_DIM_PARAMS_NUM_PROFILES * 2))
- return DIM_TOO_TIRED;
-
- switch (dim->tune_state) {
- case DIM_PARKING_ON_TOP:
- case DIM_PARKING_TIRED:
- break;
- case DIM_GOING_RIGHT:
- if (dim->profile_ix == (NET_DIM_PARAMS_NUM_PROFILES - 1))
- return DIM_ON_EDGE;
- dim->profile_ix++;
- dim->steps_right++;
- break;
- case DIM_GOING_LEFT:
- if (dim->profile_ix == 0)
- return DIM_ON_EDGE;
- dim->profile_ix--;
- dim->steps_left++;
- break;
- }
-
- dim->tired++;
- return DIM_STEPPED;
-}
-
-static void net_dim_exit_parking(struct dim *dim)
-{
- dim->tune_state = dim->profile_ix ? DIM_GOING_LEFT : DIM_GOING_RIGHT;
- net_dim_step(dim);
-}
-
-static int net_dim_stats_compare(struct dim_stats *curr,
+static int net_dim_stats_compare(struct dim *dim, struct dim_stats *curr,
struct dim_stats *prev)
{
if (!prev->bpms)
@@ -294,59 +261,6 @@ static int net_dim_stats_compare(struct dim_stats *curr,
return DIM_STATS_SAME;
}
-static bool net_dim_decision(struct dim_stats *curr_stats, struct dim *dim)
-{
- int prev_state = dim->tune_state;
- int prev_ix = dim->profile_ix;
- int stats_res;
- int step_res;
-
- switch (dim->tune_state) {
- case DIM_PARKING_ON_TOP:
- stats_res = net_dim_stats_compare(curr_stats,
- &dim->prev_stats);
- if (stats_res != DIM_STATS_SAME)
- net_dim_exit_parking(dim);
- break;
-
- case DIM_PARKING_TIRED:
- dim->tired--;
- if (!dim->tired)
- net_dim_exit_parking(dim);
- break;
-
- case DIM_GOING_RIGHT:
- case DIM_GOING_LEFT:
- stats_res = net_dim_stats_compare(curr_stats,
- &dim->prev_stats);
- if (stats_res != DIM_STATS_BETTER)
- dim_turn(dim);
-
- if (dim_on_top(dim)) {
- dim_park_on_top(dim);
- break;
- }
-
- step_res = net_dim_step(dim);
- switch (step_res) {
- case DIM_ON_EDGE:
- dim_park_on_top(dim);
- break;
- case DIM_TOO_TIRED:
- dim_park_tired(dim);
- break;
- }
-
- break;
- }
-
- if (prev_state != DIM_PARKING_ON_TOP ||
- dim->tune_state != DIM_PARKING_ON_TOP)
- dim->prev_stats = *curr_stats;
-
- return dim->profile_ix != prev_ix;
-}
-
void net_dim(struct dim *dim, const struct dim_sample *end_sample)
{
struct dim_stats curr_stats;
@@ -361,7 +275,8 @@ void net_dim(struct dim *dim, const struct dim_sample *end_sample)
break;
if (!dim_calc_stats(&dim->start_sample, end_sample, &curr_stats))
break;
- if (net_dim_decision(&curr_stats, dim)) {
+ if (dim_decide(dim, &curr_stats, net_dim_stats_compare,
+ NET_DIM_PARAMS_NUM_PROFILES)) {
dim->state = DIM_APPLY_NEW_PROFILE;
schedule_work(&dim->work);
break;
-------------- next part --------------
From db6ef734309963486a7d9640850bb71e4bd5168a Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan at bytedance.com>
Date: Wed, 19 Aug 2026 15:31:27 +0800
Subject: [PATCH DIM 3/4] nvme-pci: return completion count from nvme_poll_cq
Return the number of completions consumed by nvme_poll_cq() instead of
a boolean. Existing callers still treat zero vs non-zero as before;
nvme_poll() keeps returning 0/1 via a bool local.
Extract nvme_irq_complete_batch() from nvme_irq() so the adaptive
handler in the next patch can reuse the batch completion path.
Signed-off-by: Fengnan Chang <changfengnan at bytedance.com>
---
drivers/nvme/host/pci.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 69932d640b5379c262856f7534923ad1330d0ef0..620d0430601e3a280284a76c7c2342a15592b1c9 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1606,13 +1606,12 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
}
}
-static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
- struct io_comp_batch *iob)
+static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob)
{
- bool found = false;
+ unsigned int found = 0;
while (nvme_cqe_pending(nvmeq)) {
- found = true;
/*
* load-load control dependency between phase and the rest of
* the cqe requires a full read memory barrier
@@ -1620,6 +1619,7 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
dma_rmb();
nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
+ found++;
}
if (found)
@@ -1627,17 +1627,22 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
+static irqreturn_t nvme_irq_complete_batch(struct io_comp_batch *iob,
+ unsigned int completions)
+{
+ if (!completions)
+ return IRQ_NONE;
+ if (!rq_list_empty(&iob->req_list))
+ nvme_pci_complete_batch(iob);
+ return IRQ_HANDLED;
+}
+
static irqreturn_t nvme_irq(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
DEFINE_IO_COMP_BATCH(iob);
- if (nvme_poll_cq(nvmeq, &iob)) {
- if (!rq_list_empty(&iob.req_list))
- nvme_pci_complete_batch(&iob);
- return IRQ_HANDLED;
- }
- return IRQ_NONE;
+ return nvme_irq_complete_batch(&iob, nvme_poll_cq(nvmeq, &iob));
}
static irqreturn_t nvme_irq_check(int irq, void *data)
-------------- next part --------------
From e4d1e421995bb8bc00b06a324e831eaa911cfc85 Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan at bytedance.com>
Date: Fri, 30 Oct 2026 01:47:12 +0000
Subject: [PATCH DIM 4/4] nvme-pci: add DIM-based adaptive interrupt polling
Frequent completion interrupts compete with command submission for CPU
time at high IOPS. Use the common DIM search to let each eligible I/O
queue choose IRQ completion or periodic polling.
Compare 32768-CQE samples using two profiles. A trial must gain more
than 1% and more than one CQE/ms to replace its starting profile; ties
keep that profile. Keep a selected POLL profile unless its rate falls
more than 1% below the saved IRQ reference. Space IRQ-to-POLL trials by
1, 2, 4, 8 and then 16 valid IRQ samples. Allow 128 CQEs of cumulative
lag, discard samples older than one second, and reject POLL samples with
no completions reaped by the poller.
Use a pinned hard hrtimer and irq_poll with a 10 us period and a 64-CQE
budget. Poll only while commands remain, and join the current CQ owner
before changing policy or tearing down the queue. Keep the feature off
by default and limit it to non-threaded I/O queues with dedicated MSI-X
vectors, outside PREEMPT_RT and with adequate timer resolution. The
module parameter sets the initial policy; sysfs changes it while
namespace I/O is frozen and drained.
The linked 158-case matrix compares adaptive polling disabled (OFF) and
enabled (ON) on the same kernel build, using paired ABBA runs with
libaio and io_uring. Median per-case paired IOPS changes for QD16-128
per job are shown below (12 cases per row; multi-device tests use 16
drives):
multi-device 4k randread, numjobs=8 +92.3%
multi-device 4k mixed, numjobs=8 +49.1%
single-device 4k randwrite, numjobs=1 +40.1%
multi-device 4k mixed, numjobs=1 +4.3%
Link: https://docs.google.com/spreadsheets/d/18JADnBXWAGkffY_3LsPJ6ftA9DGH6qpNbKvP-lQWLnE/edit
Signed-off-by: Fengnan Chang <changfengnan at bytedance.com>
---
Documentation/ABI/testing/sysfs-nvme | 17 ++
drivers/nvme/host/Kconfig | 2 +
drivers/nvme/host/pci.c | 423 ++++++++++++++++++++++++++-
3 files changed, 435 insertions(+), 7 deletions(-)
diff --git a/Documentation/ABI/testing/sysfs-nvme b/Documentation/ABI/testing/sysfs-nvme
index 499d5f843cd43b2480980ada0500ac202dec1f6a..eb9df624a8a074eaf9476a5cd71ad58450d7a353 100644
--- a/Documentation/ABI/testing/sysfs-nvme
+++ b/Documentation/ABI/testing/sysfs-nvme
@@ -11,3 +11,20 @@ Description:
(REPLACETLSPSK) with the target. After a reauthentication
the value returned by tls_configured_key will be the new
serial.
+
+What: /sys/class/nvme/nvmeX/adaptive_irq_polling
+Date: August 2026
+KernelVersion: 7.3
+Contact: Linux NVMe mailing list <linux-nvme at lists.infradead.org>
+Description:
+ Set the adaptive IRQ polling policy (0 or 1) for eligible
+ I/O queues of one PCI NVMe controller. Reading returns the
+ policy, not whether a queue is currently polling. Changing it
+ freezes the namespace request queues and waits for outstanding
+ namespace I/O. Writes fail with EBUSY unless the controller is
+ live.
+
+ Eligible queues use non-threaded MSI-X with a dedicated vector.
+ The attribute is unavailable with threaded interrupts. The
+ module parameter supplies only the initial policy for new
+ controllers.
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 31974c7dd20c9174961eac91e3823e939d325daa..184e9ccfde8b22f4e177ecdde4e050e146067fa4 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -6,6 +6,8 @@ config BLK_DEV_NVME
tristate "NVM Express block device"
depends on PCI && BLOCK
select NVME_CORE
+ select DIMLIB
+ select IRQ_POLL
help
The NVM Express driver is for solid state drives directly
connected to the PCI or PCI Express bus. If you know you
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 620d0430601e3a280284a76c7c2342a15592b1c9..89f8620046adb74fcec4cf81470532620a8fa0a2 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -9,10 +9,13 @@
#include <linux/blkdev.h>
#include <linux/blk-mq-dma.h>
#include <linux/blk-integrity.h>
+#include <linux/dim.h>
#include <linux/dmi.h>
+#include <linux/hrtimer.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/io.h>
+#include <linux/irq_poll.h>
#include <linux/kstrtox.h>
#include <linux/memremap.h>
#include <linux/mm.h>
@@ -82,6 +85,11 @@ struct quirk_entry {
static int use_threaded_interrupts;
module_param(use_threaded_interrupts, int, 0444);
+static bool use_adaptive_irq_polling;
+module_param(use_adaptive_irq_polling, bool, 0444);
+MODULE_PARM_DESC(use_adaptive_irq_polling,
+ "Initial adaptive IRQ polling policy for new controllers");
+
static bool use_cmb_sqes = true;
module_param(use_cmb_sqes, bool, 0444);
MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
@@ -334,6 +342,7 @@ struct nvme_dev {
unsigned int nr_allocated_queues;
unsigned int nr_write_queues;
unsigned int nr_poll_queues;
+ bool adaptive_irq_polling;
struct nvme_descriptor_pools descriptor_pools[];
};
@@ -369,6 +378,7 @@ struct nvme_queue {
void *sq_cmds;
/* only used for poll queues: */
spinlock_t cq_poll_lock ____cacheline_aligned_in_smp;
+ int adaptive_left; /* CQEs left in the DIM sample; cq_head cache line */
struct nvme_completion *cqes;
dma_addr_t sq_dma_addr;
dma_addr_t cq_dma_addr;
@@ -386,11 +396,27 @@ struct nvme_queue {
#define NVMEQ_SQ_CMB 1
#define NVMEQ_DELETE_ERROR 2
#define NVMEQ_POLLED 3
+#define NVMEQ_ADAPTIVE 4 /* adaptive IRQ handler installed */
+#define NVMEQ_POLLING 5 /* owns the CQ and one IRQ disable depth */
+#define NVMEQ_ADAPTIVE_POLL 6 /* DIM selected the polling profile */
+#define NVMEQ_ADAPTIVE_ENABLED 7 /* controller policy enables adaptive polling */
__le32 *dbbuf_sq_db;
__le32 *dbbuf_cq_db;
__le32 *dbbuf_sq_ei;
__le32 *dbbuf_cq_ei;
struct completion delete_done;
+ /* Adaptive state is owned by the current CQ consumer. */
+ struct dim dim;
+ /* Samples counted while parked on IRQ; backoff grows up to 16. */
+ u16 adaptive_hold;
+ u8 adaptive_backoff_shift;
+ /* CQEs reaped by the poller in the current sample. */
+ u32 adaptive_polled;
+ /* Saved IRQ reference in nanoseconds per completion. */
+ u64 adaptive_irq_ns;
+ int irq;
+ struct irq_poll iop;
+ struct hrtimer poll_timer;
};
/* bits for iod->flags */
@@ -1606,8 +1632,8 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
}
}
-static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
- struct io_comp_batch *iob)
+static inline unsigned int nvme_poll_cq_bounded(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob, int budget)
{
unsigned int found = 0;
@@ -1620,6 +1646,8 @@ static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
found++;
+ if (budget && found == budget)
+ break;
}
if (found)
@@ -1627,6 +1655,12 @@ static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
+static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob)
+{
+ return nvme_poll_cq_bounded(nvmeq, iob, 0);
+}
+
static irqreturn_t nvme_irq_complete_batch(struct io_comp_batch *iob,
unsigned int completions)
{
@@ -1654,6 +1688,310 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
return IRQ_NONE;
}
+/*
+ * Compare IRQ and POLL over 32768-CQE samples, with a 1% rate margin.
+ * Keep a winning POLL profile; IRQ parking backs off between POLL trials.
+ */
+#define NVME_ADAPTIVE_PERIOD_NS (10U * NSEC_PER_USEC)
+#define NVME_ADAPTIVE_SAMPLE_CQES 32768U
+#define NVME_ADAPTIVE_RATE_MARGIN_PCT 1
+#define NVME_ADAPTIVE_MAX_HOLD_SAMPLES 16
+#define NVME_ADAPTIVE_POLL_BUDGET 64
+#define NVME_ADAPTIVE_LAG_CQES 128
+
+enum nvme_adaptive_profile {
+ NVME_ADAPTIVE_IRQ,
+ NVME_ADAPTIVE_POLL,
+ NVME_ADAPTIVE_NUM_PROFILES,
+};
+
+static void nvme_adaptive_reset(struct nvme_queue *nvmeq)
+{
+ memset(&nvmeq->dim, 0, sizeof(nvmeq->dim));
+ /* A fresh IRQ sample starts a search without a parked-profile hold. */
+ nvmeq->dim.tune_state = DIM_GOING_RIGHT;
+ nvmeq->adaptive_hold = 0;
+ nvmeq->adaptive_backoff_shift = 0;
+ nvmeq->adaptive_left = 0;
+ nvmeq->adaptive_polled = 0;
+ nvmeq->adaptive_irq_ns = 0;
+ assign_bit(NVMEQ_ADAPTIVE_POLL, &nvmeq->flags, false);
+}
+
+static noinline void nvme_adaptive_park_irq(struct nvme_queue *nvmeq)
+{
+ memset(&nvmeq->dim, 0, sizeof(nvmeq->dim));
+ dim_park_on_top(&nvmeq->dim);
+ nvmeq->adaptive_hold = 0;
+ nvmeq->adaptive_left = 0;
+ nvmeq->adaptive_polled = 0;
+ assign_bit(NVMEQ_ADAPTIVE_POLL, &nvmeq->flags, false);
+}
+
+static int nvme_adaptive_compare(struct dim *dim, struct dim_stats *curr,
+ struct dim_stats *prev)
+{
+ struct nvme_queue *nvmeq = container_of(dim, struct nvme_queue, dim);
+ u32 irq_cpms;
+ int delta, hold;
+
+ if (dim->tune_state == DIM_PARKING_ON_TOP) {
+ /* A parked POLL profile may lose up to 1% of the IRQ rate. */
+ if (dim->profile_ix == NVME_ADAPTIVE_POLL) {
+ irq_cpms = div_u64(NSEC_PER_MSEC, nvmeq->adaptive_irq_ns);
+ return (u64)curr->cpms * 100 >=
+ (u64)irq_cpms * (100 - NVME_ADAPTIVE_RATE_MARGIN_PCT) ?
+ DIM_STATS_SAME : DIM_STATS_WORSE;
+ }
+ hold = 1U << nvmeq->adaptive_backoff_shift;
+ if (++nvmeq->adaptive_hold < hold)
+ return DIM_STATS_SAME;
+ if (hold < NVME_ADAPTIVE_MAX_HOLD_SAMPLES)
+ nvmeq->adaptive_backoff_shift++;
+ return DIM_STATS_BETTER;
+ }
+ /* Report ties as worse so DIM returns to the trial's starting profile. */
+ delta = curr->cpms - prev->cpms;
+ if (abs(delta) <= 1 ||
+ abs(delta) * 100 <= prev->cpms * NVME_ADAPTIVE_RATE_MARGIN_PCT)
+ return DIM_STATS_WORSE;
+ return delta > 0 ? DIM_STATS_BETTER : DIM_STATS_WORSE;
+}
+
+static noinline void nvme_adaptive_sample(struct nvme_queue *nvmeq)
+{
+ struct dim *dim = &nvmeq->dim;
+ struct dim_sample end;
+ struct dim_stats stats;
+ u64 duration;
+ s64 age;
+ int before;
+
+ dim_update_sample_with_comps(0, 0, 0,
+ NVME_ADAPTIVE_SAMPLE_CQES -
+ nvmeq->adaptive_left, &end);
+ if (dim->state == DIM_START_MEASURE) {
+ dim->state = DIM_MEASURE_IN_PROGRESS;
+ goto start_sample;
+ }
+ age = ktime_us_delta(end.time, dim->start_sample.time);
+ /* Expired evidence starts over with a fresh IRQ reference. */
+ if (age > USEC_PER_SEC) {
+ nvme_adaptive_reset(nvmeq);
+ return;
+ }
+ if (nvmeq->adaptive_left > 0)
+ return;
+ if (dim->profile_ix == NVME_ADAPTIVE_POLL && !nvmeq->adaptive_polled) {
+ nvme_adaptive_park_irq(nvmeq);
+ return;
+ }
+ if (dim_calc_stats(&dim->start_sample, &end, &stats)) {
+ if (dim->profile_ix == NVME_ADAPTIVE_IRQ) {
+ duration = ktime_to_ns(ktime_sub(end.time,
+ dim->start_sample.time));
+ nvmeq->adaptive_irq_ns =
+ max_t(u64, 1, div_u64(duration, end.comp_ctr));
+ }
+ before = dim->tune_state;
+ dim_decide(dim, &stats, nvme_adaptive_compare,
+ NVME_ADAPTIVE_NUM_PROFILES);
+ /* An exhausted two-profile search is back where it started. */
+ if (dim->tune_state == DIM_PARKING_TIRED)
+ dim_park_on_top(dim);
+ if (before != DIM_PARKING_ON_TOP &&
+ dim->tune_state == DIM_PARKING_ON_TOP) {
+ nvmeq->adaptive_hold = 0;
+ if (dim->profile_ix == NVME_ADAPTIVE_POLL)
+ nvmeq->adaptive_backoff_shift = 0;
+ }
+ }
+ assign_bit(NVMEQ_ADAPTIVE_POLL, &nvmeq->flags,
+ dim->profile_ix == NVME_ADAPTIVE_POLL);
+start_sample:
+ end.comp_ctr = 0;
+ dim->start_sample = end;
+ nvmeq->adaptive_left = NVME_ADAPTIVE_SAMPLE_CQES;
+ nvmeq->adaptive_polled = 0;
+}
+
+static __always_inline void nvme_adaptive_account(struct nvme_queue *nvmeq,
+ unsigned int completions)
+{
+ /* Empty polls must still check the age of an unfinished sample. */
+ if (likely(completions)) {
+ nvmeq->adaptive_left -= completions;
+ if (nvmeq->adaptive_left > 0)
+ return;
+ }
+ nvme_adaptive_sample(nvmeq);
+}
+
+static enum hrtimer_restart nvme_adaptive_timer(struct hrtimer *timer)
+{
+ struct nvme_queue *nvmeq = container_of(timer, struct nvme_queue,
+ poll_timer);
+
+ /*
+ * Never return HRTIMER_NORESTART: only the poller's exit and stop
+ * cancel the timer, so it cannot race the next IRQ's hrtimer_start().
+ */
+ irq_poll_sched(&nvmeq->iop);
+ hrtimer_forward_now(timer, ns_to_ktime(NVME_ADAPTIVE_PERIOD_NS));
+ return HRTIMER_RESTART;
+}
+
+static int nvme_adaptive_poll(struct irq_poll *iop, int budget)
+{
+ struct nvme_queue *nvmeq = container_of(iop, struct nvme_queue, iop);
+ unsigned int found = 0;
+ u64 now, completed;
+ bool want_irq;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ if (!test_bit(NVMEQ_POLLING, &nvmeq->flags))
+ goto complete;
+ want_irq = !test_bit(NVMEQ_ADAPTIVE_POLL, &nvmeq->flags) ||
+ !test_bit(NVMEQ_ENABLED, &nvmeq->flags);
+ if (!want_irq) {
+ found = nvme_poll_cq_bounded(nvmeq, &iob, budget);
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ /* The entire batch, including overshoot, closes this sample. */
+ nvmeq->adaptive_polled += found;
+ nvme_adaptive_account(nvmeq, found);
+ }
+ /* A full-budget return leaves the list and SCHED owned by irq_poll. */
+ if (found == budget)
+ return found;
+ if (!want_irq && test_bit(NVMEQ_ADAPTIVE_POLL, &nvmeq->flags)) {
+ now = ktime_get_ns();
+ /* Bank all sample completions and leave when lag exceeds 128 CQEs. */
+ completed = max_t(s64, 0, (s64)NVME_ADAPTIVE_SAMPLE_CQES -
+ nvmeq->adaptive_left);
+ if (now - ktime_to_ns(nvmeq->dim.start_sample.time) >
+ (completed + NVME_ADAPTIVE_LAG_CQES) *
+ nvmeq->adaptive_irq_ns)
+ nvme_adaptive_park_irq(nvmeq);
+ }
+ if (want_irq || !test_bit(NVMEQ_ADAPTIVE_POLL, &nvmeq->flags) ||
+ nvmeq->cq_head == READ_ONCE(nvmeq->sq_tail)) {
+ /* Finish all CQ/DIM accesses before returning ownership to IRQ. */
+ smp_mb__before_atomic();
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
+ /* No restart may overlap the next IRQ's hrtimer_start(). */
+ hrtimer_cancel(&nvmeq->poll_timer);
+ enable_irq(nvmeq->irq);
+ }
+complete:
+ /* Release SCHED last: stop can then join without a CQ lock. */
+ irq_poll_complete(iop);
+ return found;
+}
+
+static irqreturn_t nvme_adaptive_irq(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ unsigned int found;
+ irqreturn_t ret;
+ u64 expires;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ if (!test_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags))
+ return nvme_irq_complete_batch(&iob, nvme_poll_cq(nvmeq, &iob));
+
+ found = nvme_poll_cq(nvmeq, &iob);
+ ret = nvme_irq_complete_batch(&iob, found);
+ /* A force-threaded handler keeps the native behaviour. */
+ if (!in_hardirq())
+ return ret;
+ nvme_adaptive_account(nvmeq, found);
+ /* Avoid a poll/IRQ round trip when the handler drains the queue. */
+ if (test_bit(NVMEQ_ADAPTIVE_POLL, &nvmeq->flags) &&
+ test_bit(NVMEQ_ENABLED, &nvmeq->flags) &&
+ nvmeq->cq_head != READ_ONCE(nvmeq->sq_tail) &&
+ !test_bit(IRQ_POLL_F_SCHED, &nvmeq->iop.state) &&
+ hrtimer_resolution <= NVME_ADAPTIVE_PERIOD_NS) {
+ expires = ktime_get_ns();
+ /* Positive IRQ batches can skip the sample-age slow path. */
+ if (expires -
+ ktime_to_ns(nvmeq->dim.start_sample.time) > NSEC_PER_SEC) {
+ nvme_adaptive_reset(nvmeq);
+ return ret;
+ }
+ expires = (div_u64(expires, NVME_ADAPTIVE_PERIOD_NS) + 1) *
+ NVME_ADAPTIVE_PERIOD_NS;
+ /* Publish CQ/DIM before the timer takes over; do not touch them. */
+ smp_mb__before_atomic();
+ set_bit(NVMEQ_POLLING, &nvmeq->flags);
+ disable_irq_nosync(irq);
+ hrtimer_start(&nvmeq->poll_timer, ns_to_ktime(expires),
+ HRTIMER_MODE_ABS_PINNED_HARD);
+ }
+ return ret;
+}
+
+/* Caller has disabled and synchronized the IRQ, excluding new admissions. */
+static void nvme_adaptive_stop(struct nvme_queue *nvmeq)
+{
+ if (!test_bit(NVMEQ_ADAPTIVE, &nvmeq->flags))
+ return;
+ irq_poll_disable(&nvmeq->iop);
+ hrtimer_cancel(&nvmeq->poll_timer);
+ /* A full-budget callback may have left the poll disable depth to us. */
+ if (test_and_clear_bit(NVMEQ_POLLING, &nvmeq->flags))
+ enable_irq(nvmeq->irq);
+ irq_poll_enable(&nvmeq->iop);
+}
+
+static void nvme_adaptive_set_queue(struct nvme_queue *nvmeq, bool enable)
+{
+ if (!test_bit(NVMEQ_ADAPTIVE, &nvmeq->flags) ||
+ !test_bit(NVMEQ_ENABLED, &nvmeq->flags))
+ return;
+
+ disable_irq(nvmeq->irq);
+ nvme_adaptive_stop(nvmeq);
+ nvme_adaptive_reset(nvmeq);
+ assign_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags, enable);
+ enable_irq(nvmeq->irq);
+}
+
+/*
+ * Freeze namespace I/O before switching completion mode. scan_lock keeps
+ * the namespace set stable; shutdown_lock prevents concurrent reset.
+ */
+static int nvme_adaptive_switch(struct nvme_dev *dev, bool enable)
+{
+ int qid, ret = 0;
+
+ mutex_lock(&dev->ctrl.scan_lock);
+ if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_LIVE) {
+ ret = -EBUSY;
+ goto out_unlock;
+ }
+ if (enable == READ_ONCE(dev->adaptive_irq_polling))
+ goto out_unlock;
+
+ nvme_start_freeze(&dev->ctrl);
+ nvme_wait_freeze(&dev->ctrl);
+
+ mutex_lock(&dev->shutdown_lock);
+ if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_LIVE) {
+ ret = -EBUSY;
+ } else {
+ for (qid = 1; qid < dev->ctrl.queue_count; qid++)
+ nvme_adaptive_set_queue(&dev->queues[qid], enable);
+ WRITE_ONCE(dev->adaptive_irq_polling, enable);
+ }
+ mutex_unlock(&dev->shutdown_lock);
+
+ nvme_unfreeze(&dev->ctrl);
+out_unlock:
+ mutex_unlock(&dev->ctrl.scan_lock);
+ return ret;
+}
+
/*
* Poll for completions for any interrupt driven queue
* Can be called from any context.
@@ -1667,6 +2005,7 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
irq = pci_irq_vector(pdev, nvmeq->cq_vector);
disable_irq(irq);
+ nvme_adaptive_stop(nvmeq);
spin_lock(&nvmeq->cq_poll_lock);
nvme_poll_cq(nvmeq, NULL);
spin_unlock(&nvmeq->cq_poll_lock);
@@ -2056,8 +2395,15 @@ static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
nvmeq->dev->online_queues--;
if (!nvmeq->qid && nvmeq->dev->ctrl.admin_q)
nvme_quiesce_admin_queue(&nvmeq->dev->ctrl);
- if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags))
+ if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags)) {
+ if (test_bit(NVMEQ_ADAPTIVE, &nvmeq->flags)) {
+ disable_irq(nvmeq->irq);
+ nvme_adaptive_stop(nvmeq);
+ enable_irq(nvmeq->irq);
+ }
pci_free_irq(to_pci_dev(dev->dev), nvmeq->cq_vector, nvmeq);
+ clear_bit(NVMEQ_ADAPTIVE, &nvmeq->flags);
+ }
}
static void nvme_suspend_io_queues(struct nvme_dev *dev)
@@ -2174,15 +2520,36 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth)
static int queue_request_irq(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
+ irq_handler_t handler = nvme_irq;
int nr = nvmeq->dev->ctrl.instance;
+ int ret;
- if (use_threaded_interrupts) {
+ clear_bit(NVMEQ_ADAPTIVE, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ if (use_threaded_interrupts)
return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- } else {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
- NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
+ if (nvmeq->qid &&
+ pdev->msix_enabled && nvmeq->dev->num_vecs > 1 &&
+ !IS_ENABLED(CONFIG_PREEMPT_RT)) {
+ nvmeq->irq = pci_irq_vector(pdev, nvmeq->cq_vector);
+ if (nvmeq->irq < 0)
+ return nvmeq->irq;
+ nvme_adaptive_reset(nvmeq);
+ irq_poll_init(&nvmeq->iop, NVME_ADAPTIVE_POLL_BUDGET,
+ nvme_adaptive_poll);
+ hrtimer_setup(&nvmeq->poll_timer, nvme_adaptive_timer,
+ CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED_HARD);
+ set_bit(NVMEQ_ADAPTIVE, &nvmeq->flags);
+ assign_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags,
+ nvmeq->dev->adaptive_irq_polling);
+ handler = nvme_adaptive_irq;
}
+ ret = pci_request_irq(pdev, nvmeq->cq_vector, handler, NULL, nvmeq,
+ "nvme%dq%d", nr, nvmeq->qid);
+ if (ret)
+ clear_bit(NVMEQ_ADAPTIVE, &nvmeq->flags);
+ return ret;
}
static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
@@ -2813,6 +3180,33 @@ static ssize_t hmb_store(struct device *dev, struct device_attribute *attr,
}
static DEVICE_ATTR_RW(hmb);
+static ssize_t adaptive_irq_polling_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct nvme_dev *ndev = to_nvme_dev(dev_get_drvdata(dev));
+
+ return sysfs_emit(buf, "%d\n", READ_ONCE(ndev->adaptive_irq_polling));
+}
+
+static ssize_t adaptive_irq_polling_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct nvme_dev *ndev = to_nvme_dev(dev_get_drvdata(dev));
+ bool enable;
+ int ret;
+
+ ret = kstrtobool(buf, &enable);
+ if (ret)
+ return ret;
+ ret = nvme_adaptive_switch(ndev, enable);
+ if (ret)
+ return ret;
+ return count;
+}
+static DEVICE_ATTR_RW(adaptive_irq_polling);
+
static umode_t nvme_pci_attrs_are_visible(struct kobject *kobj,
struct attribute *a, int n)
{
@@ -2828,6 +3222,8 @@ static umode_t nvme_pci_attrs_are_visible(struct kobject *kobj,
}
if (a == &dev_attr_hmb.attr && !ctrl->hmpre)
return 0;
+ if (a == &dev_attr_adaptive_irq_polling.attr && use_threaded_interrupts)
+ return 0;
return a->mode;
}
@@ -2845,9 +3241,21 @@ static const struct attribute_group nvme_pci_dev_attrs_group = {
.is_visible = nvme_pci_attrs_are_visible,
};
+/* Policy visibility is fixed; reset must not remove an active store. */
+static struct attribute *nvme_pci_policy_attrs[] = {
+ &dev_attr_adaptive_irq_polling.attr,
+ NULL,
+};
+
+static const struct attribute_group nvme_pci_policy_attrs_group = {
+ .attrs = nvme_pci_policy_attrs,
+ .is_visible = nvme_pci_attrs_are_visible,
+};
+
static const struct attribute_group *nvme_pci_dev_attr_groups[] = {
&nvme_dev_attrs_group,
&nvme_pci_dev_attrs_group,
+ &nvme_pci_policy_attrs_group,
&nvme_dev_diag_attrs_group,
NULL,
};
@@ -3691,6 +4099,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
INIT_WORK(&dev->ctrl.reset_work, nvme_reset_work);
mutex_init(&dev->shutdown_lock);
+ dev->adaptive_irq_polling = use_adaptive_irq_polling;
dev->nr_write_queues = write_queues;
dev->nr_poll_queues = poll_queues;
dev->nr_allocated_queues = nvme_max_io_queues(dev) + 1;
-------------- next part --------------
From 9e6aad1e528bf376b8113c2bf07445d4f5130e22 Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan at bytedance.com>
Date: Wed, 19 Aug 2026 15:31:27 +0800
Subject: [PATCH non-DIM 1/2] nvme-pci: return completion count from nvme_poll_cq
Return the number of completions consumed by nvme_poll_cq() instead of
a boolean. Existing callers still treat zero vs non-zero as before;
nvme_poll() keeps returning 0/1 via a bool local.
Extract nvme_irq_complete_batch() from nvme_irq() so the adaptive
handler in the next patch can reuse the batch completion path.
Signed-off-by: Fengnan Chang <changfengnan at bytedance.com>
---
drivers/nvme/host/pci.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 69932d640b5379c262856f7534923ad1330d0ef0..620d0430601e3a280284a76c7c2342a15592b1c9 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1606,13 +1606,12 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
}
}
-static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
- struct io_comp_batch *iob)
+static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob)
{
- bool found = false;
+ unsigned int found = 0;
while (nvme_cqe_pending(nvmeq)) {
- found = true;
/*
* load-load control dependency between phase and the rest of
* the cqe requires a full read memory barrier
@@ -1620,6 +1619,7 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
dma_rmb();
nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
+ found++;
}
if (found)
@@ -1627,17 +1627,22 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
+static irqreturn_t nvme_irq_complete_batch(struct io_comp_batch *iob,
+ unsigned int completions)
+{
+ if (!completions)
+ return IRQ_NONE;
+ if (!rq_list_empty(&iob->req_list))
+ nvme_pci_complete_batch(iob);
+ return IRQ_HANDLED;
+}
+
static irqreturn_t nvme_irq(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
DEFINE_IO_COMP_BATCH(iob);
- if (nvme_poll_cq(nvmeq, &iob)) {
- if (!rq_list_empty(&iob.req_list))
- nvme_pci_complete_batch(&iob);
- return IRQ_HANDLED;
- }
- return IRQ_NONE;
+ return nvme_irq_complete_batch(&iob, nvme_poll_cq(nvmeq, &iob));
}
static irqreturn_t nvme_irq_check(int irq, void *data)
-------------- next part --------------
From 09a7d5e606bcb24875de2cd18660e8ea68eef75d Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan at bytedance.com>
Date: Wed, 19 Aug 2026 15:34:52 +0800
Subject: [PATCH non-DIM 2/2] nvme-pci: add adaptive interrupt polling
Frequent completion interrupts compete with command submission for CPU
time at high IOPS. Let each eligible I/O queue try periodic polling and
keep it when the measured completion rate justifies it.
Sample at least 32768 IRQ completions and try polling when their average
interval is at most 10 us. Require more than 1% rate gain in the first
poll window; later windows may take up to 1% longer than the saved IRQ
baseline. Allow 128 completion intervals of cumulative lag within a
window. After three consecutive failures, wait 524288 IRQ completions
before measuring again.
Use a pinned hard hrtimer and irq_poll with a 10 us target period and a
64-CQE budget. Serialize CQ access and stop polling before policy
changes or teardown. Keep the feature off by default and limit it to
non-threaded I/O queues with dedicated MSI-X vectors. The module
parameter sets the initial controller policy; sysfs changes it while
namespace I/O is frozen and drained.
The linked 158-case matrix compares adaptive polling disabled (OFF) and
enabled (ON) on the same kernel build, using paired ABBA runs with
libaio and io_uring. Median per-case paired IOPS changes for QD16-128
per job are shown below (12 cases per row; multi-device tests use 16
drives):
multi-device 4k randread, numjobs=8 +92.0%
multi-device 4k mixed, numjobs=8 +49.1%
single-device 4k randwrite, numjobs=1 +38.4%
multi-device 4k mixed, numjobs=1 +3.5%
Link: https://docs.google.com/spreadsheets/d/17kmXH9Zkr2jrN-19BwaInSx53HwOdRPJFrZ-Hjlk0_Y/edit
Signed-off-by: Guzebing <guzebing at bytedance.com>
Signed-off-by: Fengnan Chang <changfengnan at bytedance.com>
---
Documentation/ABI/testing/sysfs-nvme | 17 +
drivers/nvme/host/Kconfig | 1 +
drivers/nvme/host/pci.c | 503 ++++++++++++++++++++++++++-
3 files changed, 506 insertions(+), 15 deletions(-)
diff --git a/Documentation/ABI/testing/sysfs-nvme b/Documentation/ABI/testing/sysfs-nvme
index 499d5f843cd43b2480980ada0500ac202dec1f6a..eb9df624a8a074eaf9476a5cd71ad58450d7a353 100644
--- a/Documentation/ABI/testing/sysfs-nvme
+++ b/Documentation/ABI/testing/sysfs-nvme
@@ -11,3 +11,20 @@ Description:
(REPLACETLSPSK) with the target. After a reauthentication
the value returned by tls_configured_key will be the new
serial.
+
+What: /sys/class/nvme/nvmeX/adaptive_irq_polling
+Date: August 2026
+KernelVersion: 7.3
+Contact: Linux NVMe mailing list <linux-nvme at lists.infradead.org>
+Description:
+ Set the adaptive IRQ polling policy (0 or 1) for eligible
+ I/O queues of one PCI NVMe controller. Reading returns the
+ policy, not whether a queue is currently polling. Changing it
+ freezes the namespace request queues and waits for outstanding
+ namespace I/O. Writes fail with EBUSY unless the controller is
+ live.
+
+ Eligible queues use non-threaded MSI-X with a dedicated vector.
+ The attribute is unavailable with threaded interrupts. The
+ module parameter supplies only the initial policy for new
+ controllers.
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 31974c7dd20c9174961eac91e3823e939d325daa..22164b901da8506f613429f0d3cf373bdd26a491 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -5,6 +5,7 @@ config NVME_CORE
config BLK_DEV_NVME
tristate "NVM Express block device"
depends on PCI && BLOCK
+ select IRQ_POLL
select NVME_CORE
help
The NVM Express driver is for solid state drives directly
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 620d0430601e3a280284a76c7c2342a15592b1c9..2c1282d50649b7894330c22daff93cd74edb9842 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -10,9 +10,12 @@
#include <linux/blk-mq-dma.h>
#include <linux/blk-integrity.h>
#include <linux/dmi.h>
+#include <linux/hrtimer.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/io.h>
+#include <linux/irq_poll.h>
+#include <linux/ktime.h>
#include <linux/kstrtox.h>
#include <linux/memremap.h>
#include <linux/mm.h>
@@ -82,6 +85,25 @@ struct quirk_entry {
static int use_threaded_interrupts;
module_param(use_threaded_interrupts, int, 0444);
+static bool use_adaptive_irq_polling;
+module_param(use_adaptive_irq_polling, bool, 0444);
+MODULE_PARM_DESC(use_adaptive_irq_polling,
+ "Initial adaptive IRQ polling policy for new controllers");
+
+/*
+ * Measure 32768-CQE windows. Try polling when IRQ completions average at
+ * most 10 us apart, with a 10 us target period. The first poll window needs
+ * more than 1% rate gain; later windows may take up to 1% longer than IRQ.
+ * Allow 128 completion intervals of lag within a window. After three
+ * consecutive failures, wait 16 windows in IRQ mode before sampling again.
+ */
+#define NVME_ADAPTIVE_POLL_PERIOD_NS (10U * NSEC_PER_USEC)
+#define NVME_ADAPTIVE_WINDOW_CQES 32768U
+#define NVME_ADAPTIVE_BACKOFF_CQES (16U * NVME_ADAPTIVE_WINDOW_CQES)
+#define NVME_ADAPTIVE_POLL_RETRIES 2U
+#define NVME_ADAPTIVE_MARGIN_PCT 1U
+#define NVME_ADAPTIVE_LAG_CQES 128U
+
static bool use_cmb_sqes = true;
module_param(use_cmb_sqes, bool, 0444);
MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
@@ -307,6 +329,7 @@ struct nvme_dev {
void __iomem *bar;
unsigned long bar_mapped_size;
struct mutex shutdown_lock;
+ bool adaptive_irq_polling;
bool subsystem;
u64 cmb_size;
bool cmb_use_sqes;
@@ -358,6 +381,22 @@ static inline struct nvme_dev *to_nvme_dev(struct nvme_ctrl *ctrl)
return container_of(ctrl, struct nvme_dev, ctrl);
}
+struct nvme_adaptive_poll {
+ struct hrtimer timer;
+ struct irq_poll iopoll;
+ struct nvme_queue *nvmeq;
+ /* Start of the current IRQ sample or polling window. */
+ u64 start_ns;
+ /* IRQ baseline: nanoseconds per completion. */
+ u32 irq_interval_ns;
+ /* Completions in the current polling window. */
+ u32 completions;
+ int irq;
+ /* Consecutive rejected polling trials. */
+ u8 poll_failures;
+ bool first_window;
+};
+
/*
* An NVM Express queue. Each device has at least two (one for admin
* commands and one for I/O commands).
@@ -367,10 +406,15 @@ struct nvme_queue {
struct nvme_descriptor_pools descriptor_pools;
spinlock_t sq_lock;
void *sq_cmds;
- /* only used for poll queues: */
+ dma_addr_t sq_dma_addr;
+ struct nvme_adaptive_poll *adaptive;
+ /* Used for poll queues and adaptive interrupt polling. */
spinlock_t cq_poll_lock ____cacheline_aligned_in_smp;
+ /* IRQ sample countdown; nonpositive values trigger sampling. */
+ int adaptive_left;
+ /* IRQ completions to skip before taking a new baseline. */
+ u32 adaptive_backoff;
struct nvme_completion *cqes;
- dma_addr_t sq_dma_addr;
dma_addr_t cq_dma_addr;
u32 __iomem *q_db;
u32 q_depth;
@@ -386,6 +430,12 @@ struct nvme_queue {
#define NVMEQ_SQ_CMB 1
#define NVMEQ_DELETE_ERROR 2
#define NVMEQ_POLLED 3
+/* currently in poll mode with this queue's IRQ disabled */
+#define NVMEQ_ADAPTIVE_POLLING 4
+/* adaptive policy selected for this queue; picks the interrupt handler */
+#define NVMEQ_ADAPTIVE_ENABLED 5
+/* swallow one stale IRQ latched while switching back from poll mode */
+#define NVMEQ_ADAPTIVE_STALE_IRQ 6
__le32 *dbbuf_sq_db;
__le32 *dbbuf_cq_db;
__le32 *dbbuf_sq_ei;
@@ -1627,6 +1677,28 @@ static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
+/*
+ * Honor the irq_poll budget without adding a limit check to the unbounded
+ * completion path used by the regular interrupt handler.
+ */
+static unsigned int nvme_poll_cq_bounded(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob,
+ unsigned int limit)
+{
+ unsigned int found = 0;
+
+ while (found < limit && nvme_cqe_pending(nvmeq)) {
+ dma_rmb();
+ nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
+ nvme_update_cq_head(nvmeq);
+ found++;
+ }
+
+ if (found)
+ nvme_ring_cq_doorbell(nvmeq);
+ return found;
+}
+
static irqreturn_t nvme_irq_complete_batch(struct io_comp_batch *iob,
unsigned int completions)
{
@@ -1654,6 +1726,216 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
return IRQ_NONE;
}
+/* Reset adaptive state to an uninitialized IRQ baseline. */
+static void nvme_adaptive_state_reset(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+
+ nvmeq->adaptive_left = 0;
+ nvmeq->adaptive_backoff = 0;
+ if (!adaptive)
+ return;
+ adaptive->start_ns = 0;
+ adaptive->irq_interval_ns = 0;
+ adaptive->completions = 0;
+ adaptive->poll_failures = 0;
+ adaptive->first_window = false;
+}
+
+/*
+ * Restore IRQ mode. Back off for 16 windows after the third consecutive
+ * rejection; otherwise rebaseline immediately.
+ */
+static void nvme_adaptive_poll_end(struct nvme_queue *nvmeq, bool backoff)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ u8 poll_failures = adaptive->poll_failures;
+
+ nvme_adaptive_state_reset(nvmeq);
+ if (backoff) {
+ if (poll_failures < NVME_ADAPTIVE_POLL_RETRIES)
+ adaptive->poll_failures = poll_failures + 1;
+ else
+ nvmeq->adaptive_backoff = NVME_ADAPTIVE_BACKOFF_CQES;
+ }
+ clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ set_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ enable_irq(adaptive->irq);
+}
+
+static void nvme_adaptive_window_start(struct nvme_queue *nvmeq, u64 now)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+
+ nvmeq->adaptive_left = NVME_ADAPTIVE_WINDOW_CQES;
+ adaptive->start_ns = now;
+ adaptive->completions = 0;
+}
+
+static void nvme_adaptive_arm(struct nvme_adaptive_poll *adaptive, u64 now)
+{
+ hrtimer_start(&adaptive->timer,
+ ns_to_ktime(now + NVME_ADAPTIVE_POLL_PERIOD_NS),
+ HRTIMER_MODE_ABS_PINNED_HARD);
+}
+
+static enum hrtimer_restart nvme_adaptive_poll_timer(struct hrtimer *timer)
+{
+ struct nvme_adaptive_poll *adaptive =
+ container_of(timer, struct nvme_adaptive_poll, timer);
+ struct nvme_queue *nvmeq = adaptive->nvmeq;
+
+ if (test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))
+ irq_poll_sched(&adaptive->iopoll);
+ return HRTIMER_NORESTART;
+}
+
+/* Drain a polling window and compare its progress with the IRQ baseline. */
+static int nvme_adaptive_irq_poll(struct irq_poll *iop, int budget)
+{
+ struct nvme_adaptive_poll *adaptive =
+ container_of(iop, struct nvme_adaptive_poll, iopoll);
+ struct nvme_queue *nvmeq = adaptive->nvmeq;
+ unsigned int completions, limit;
+ unsigned long flags;
+ bool too_slow;
+ u64 irq_ns, elapsed, now;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
+ if (unlikely(!test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))) {
+ completions = 0;
+ irq_poll_complete(iop);
+ goto out;
+ }
+ if (!test_bit(NVMEQ_ENABLED, &nvmeq->flags)) {
+ completions = 0;
+ irq_poll_complete(iop);
+ nvme_adaptive_poll_end(nvmeq, false);
+ goto out;
+ }
+
+ limit = min_t(unsigned int, budget,
+ NVME_ADAPTIVE_WINDOW_CQES - adaptive->completions);
+ completions = nvme_poll_cq_bounded(nvmeq, &iob, limit);
+ adaptive->completions += completions;
+
+ if (completions >= budget &&
+ adaptive->completions < NVME_ADAPTIVE_WINDOW_CQES)
+ goto out;
+ if (completions < budget)
+ irq_poll_complete(iop);
+
+ now = ktime_get_ns();
+ elapsed = now - adaptive->start_ns;
+ irq_ns = (u64)adaptive->completions * adaptive->irq_interval_ns;
+ if (adaptive->completions < NVME_ADAPTIVE_WINDOW_CQES) {
+ if (elapsed > irq_ns +
+ NVME_ADAPTIVE_LAG_CQES * adaptive->irq_interval_ns)
+ nvme_adaptive_poll_end(nvmeq, true);
+ else
+ nvme_adaptive_arm(adaptive, now);
+ goto out;
+ }
+ if (adaptive->first_window)
+ too_slow = elapsed * (100U + NVME_ADAPTIVE_MARGIN_PCT) >=
+ irq_ns * 100U;
+ else
+ too_slow = elapsed * 100U >
+ irq_ns * (100U + NVME_ADAPTIVE_MARGIN_PCT);
+ if (too_slow) {
+ nvme_adaptive_poll_end(nvmeq, true);
+ goto out;
+ }
+
+ adaptive->poll_failures = 0;
+ adaptive->first_window = false;
+ nvme_adaptive_window_start(nvmeq, now);
+ nvme_adaptive_arm(adaptive, now);
+out:
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ return completions;
+}
+
+/*
+ * Count down an IRQ backoff or sample at least one completion window.
+ * A polling attempt is permitted when the average interval is no
+ * greater than the poll period.
+ */
+static void nvme_adaptive_sample(struct nvme_queue *nvmeq,
+ unsigned int completions)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ u64 delta, interval, now;
+
+ if (nvmeq->adaptive_backoff) {
+ /* Consume the IRQ backoff before starting another sample. */
+ nvmeq->adaptive_backoff -= min(completions, nvmeq->adaptive_backoff);
+ return;
+ }
+ nvmeq->adaptive_left -= completions;
+ if (nvmeq->adaptive_left > 0)
+ return;
+ if (!adaptive->start_ns) {
+ nvme_adaptive_window_start(nvmeq, ktime_get_ns());
+ return;
+ }
+
+ now = ktime_get_ns();
+ delta = now - adaptive->start_ns;
+ /* Admission only; a full poll window decides whether polling wins. */
+ completions = NVME_ADAPTIVE_WINDOW_CQES - nvmeq->adaptive_left;
+ interval = div64_u64(delta, completions);
+ if (!interval || interval > NVME_ADAPTIVE_POLL_PERIOD_NS ||
+ !test_bit(NVMEQ_ENABLED, &nvmeq->flags)) {
+ nvme_adaptive_window_start(nvmeq, now);
+ return;
+ }
+
+ adaptive->irq_interval_ns = interval;
+ adaptive->first_window = true;
+ nvme_adaptive_window_start(nvmeq, now);
+ set_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ disable_irq_nosync(adaptive->irq);
+ nvme_adaptive_arm(adaptive, now);
+}
+
+static noinline irqreturn_t nvme_irq_adaptive_enabled(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ unsigned int completions;
+ unsigned long flags;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
+ if (unlikely(test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))) {
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ return IRQ_HANDLED;
+ }
+ completions = nvme_poll_cq(nvmeq, &iob);
+ if (completions)
+ nvme_adaptive_sample(nvmeq, completions);
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ return nvme_irq_complete_batch(&iob, completions);
+}
+
+static irqreturn_t nvme_irq_adaptive(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ irqreturn_t ret;
+
+ if (test_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags))
+ ret = nvme_irq_adaptive_enabled(irq, data);
+ else
+ ret = nvme_irq(irq, data);
+ if (ret == IRQ_NONE &&
+ test_and_clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags))
+ return IRQ_HANDLED;
+ return ret;
+}
+
/*
* Poll for completions for any interrupt driven queue
* Can be called from any context.
@@ -1661,30 +1943,36 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
+ unsigned long flags;
int irq;
WARN_ON_ONCE(test_bit(NVMEQ_POLLED, &nvmeq->flags));
irq = pci_irq_vector(pdev, nvmeq->cq_vector);
disable_irq(irq);
- spin_lock(&nvmeq->cq_poll_lock);
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
nvme_poll_cq(nvmeq, NULL);
- spin_unlock(&nvmeq->cq_poll_lock);
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
enable_irq(irq);
}
static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
{
struct nvme_queue *nvmeq = hctx->driver_data;
+ unsigned long flags;
bool found;
if (!test_bit(NVMEQ_POLLED, &nvmeq->flags) ||
!nvme_cqe_pending(nvmeq))
return 0;
- spin_lock(&nvmeq->cq_poll_lock);
+ /*
+ * cq_poll_lock is also taken from hardirq by the adaptive handler.
+ * Disable IRQs here so lockdep sees a consistent lock class.
+ */
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
found = nvme_poll_cq(nvmeq, iob);
- spin_unlock(&nvmeq->cq_poll_lock);
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
return found;
}
@@ -2022,8 +2310,7 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
(void *)nvmeq->cqes, nvmeq->cq_dma_addr);
if (!nvmeq->sq_cmds)
- return;
-
+ goto free_adaptive;
if (test_and_clear_bit(NVMEQ_SQ_CMB, &nvmeq->flags)) {
pci_free_p2pmem(to_pci_dev(nvmeq->dev->dev),
nvmeq->sq_cmds, SQ_SIZE(nvmeq));
@@ -2031,6 +2318,9 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
dma_free_coherent(nvmeq->dev->dev, SQ_SIZE(nvmeq),
nvmeq->sq_cmds, nvmeq->sq_dma_addr);
}
+free_adaptive:
+ kfree(nvmeq->adaptive);
+ nvmeq->adaptive = NULL;
}
static void nvme_free_queues(struct nvme_dev *dev, int lowest)
@@ -2043,9 +2333,96 @@ static void nvme_free_queues(struct nvme_dev *dev, int lowest)
}
}
+static int nvme_adaptive_suspend(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ unsigned long flags;
+ int irq;
+
+ if (!adaptive || adaptive->irq < 0)
+ return -1;
+ irq = adaptive->irq;
+ synchronize_irq(irq);
+ irq_poll_disable(&adaptive->iopoll);
+ /* irq_poll_complete() can run before the poll callback returns. */
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
+ if (test_and_clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags)) {
+ set_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ enable_irq(irq);
+ }
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ hrtimer_cancel(&adaptive->timer);
+ return irq;
+}
+
+static void nvme_adaptive_set_queue(struct nvme_queue *nvmeq, bool enable)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ unsigned long flags;
+
+ if (nvme_adaptive_suspend(nvmeq) < 0) {
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ return;
+ }
+
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
+ nvme_adaptive_state_reset(nvmeq);
+ if (enable)
+ set_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ else
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ irq_poll_enable(&adaptive->iopoll);
+}
+
+/*
+ * Freeze namespace I/O before switching completion mode. scan_lock keeps
+ * the namespace set stable; shutdown_lock prevents concurrent reset.
+ */
+static int nvme_adaptive_switch(struct nvme_dev *dev, bool enable)
+{
+ int qid, ret = 0;
+
+ mutex_lock(&dev->ctrl.scan_lock);
+ if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_LIVE) {
+ ret = -EBUSY;
+ goto out_unlock;
+ }
+ if (enable == READ_ONCE(dev->adaptive_irq_polling))
+ goto out_unlock;
+
+ nvme_start_freeze(&dev->ctrl);
+ nvme_wait_freeze(&dev->ctrl);
+
+ mutex_lock(&dev->shutdown_lock);
+ if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_LIVE) {
+ ret = -EBUSY;
+ } else {
+ for (qid = 1; qid < dev->ctrl.queue_count; qid++)
+ nvme_adaptive_set_queue(&dev->queues[qid], enable);
+ WRITE_ONCE(dev->adaptive_irq_polling, enable);
+ }
+ mutex_unlock(&dev->shutdown_lock);
+
+ nvme_unfreeze(&dev->ctrl);
+out_unlock:
+ mutex_unlock(&dev->ctrl.scan_lock);
+ return ret;
+}
+
+static void nvme_adaptive_suspend_done(struct nvme_queue *nvmeq, int irq)
+{
+ if (irq < 0)
+ return;
+ nvmeq->adaptive->irq = -1;
+ irq_poll_enable(&nvmeq->adaptive->iopoll);
+}
+
static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
{
struct nvme_queue *nvmeq = &dev->queues[qid];
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+ int irq;
if (!test_and_clear_bit(NVMEQ_ENABLED, &nvmeq->flags))
return;
@@ -2056,8 +2433,11 @@ static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
nvmeq->dev->online_queues--;
if (!nvmeq->qid && nvmeq->dev->ctrl.admin_q)
nvme_quiesce_admin_queue(&nvmeq->dev->ctrl);
- if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags))
- pci_free_irq(to_pci_dev(dev->dev), nvmeq->cq_vector, nvmeq);
+ if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags)) {
+ irq = nvme_adaptive_suspend(nvmeq);
+ pci_free_irq(pdev, nvmeq->cq_vector, nvmeq);
+ nvme_adaptive_suspend_done(nvmeq, irq);
+ }
}
static void nvme_suspend_io_queues(struct nvme_dev *dev)
@@ -2076,12 +2456,13 @@ static void nvme_suspend_io_queues(struct nvme_dev *dev)
*/
static void nvme_reap_pending_cqes(struct nvme_dev *dev)
{
+ unsigned long flags;
int i;
for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
- spin_lock(&dev->queues[i].cq_poll_lock);
+ spin_lock_irqsave(&dev->queues[i].cq_poll_lock, flags);
nvme_poll_cq(&dev->queues[i], NULL);
- spin_unlock(&dev->queues[i].cq_poll_lock);
+ spin_unlock_irqrestore(&dev->queues[i].cq_poll_lock, flags);
}
}
@@ -2171,18 +2552,78 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth)
return -ENOMEM;
}
+/*
+ * Allocate or re-arm adaptive state after reset. The caller has established
+ * MSI-X eligibility; return false if vector lookup or allocation fails.
+ */
+static bool nvme_adaptive_init(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ int irq = pci_irq_vector(to_pci_dev(nvmeq->dev->dev),
+ nvmeq->cq_vector);
+
+ if (irq < 0)
+ return false;
+ if (!adaptive) {
+ adaptive = kzalloc_node(sizeof(*adaptive), GFP_KERNEL,
+ dev_to_node(nvmeq->dev->dev));
+ if (!adaptive)
+ return false;
+ adaptive->nvmeq = nvmeq;
+ hrtimer_setup(&adaptive->timer, nvme_adaptive_poll_timer,
+ CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED_HARD);
+ irq_poll_init(&adaptive->iopoll, 64, nvme_adaptive_irq_poll);
+ adaptive->irq = irq;
+ WRITE_ONCE(nvmeq->adaptive, adaptive);
+ return true;
+ }
+ adaptive->irq = irq;
+ return true;
+}
+
static int queue_request_irq(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
int nr = nvmeq->dev->ctrl.instance;
+ bool adaptive_queue;
+ int ret;
if (use_threaded_interrupts) {
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- } else {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
- NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
}
+ /* Install the adaptive-capable handler only on eligible queues. */
+ adaptive_queue = nvmeq->qid && nvmeq->dev->num_vecs > 1 &&
+ pdev->msix_enabled;
+ if (adaptive_queue)
+ adaptive_queue = nvme_adaptive_init(nvmeq);
+ if (adaptive_queue && READ_ONCE(nvmeq->dev->adaptive_irq_polling))
+ set_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ else
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ ret = pci_request_irq(pdev, nvmeq->cq_vector,
+ adaptive_queue ? nvme_irq_adaptive : nvme_irq,
+ NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
+ if (!adaptive_queue || ret) {
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ if (ret && nvmeq->adaptive)
+ nvmeq->adaptive->irq = -1;
+ }
+ return ret;
+}
+
+static void nvme_adaptive_reset(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+
+ clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ nvme_adaptive_state_reset(nvmeq);
+ if (!adaptive)
+ return;
+ adaptive->irq = -1;
}
static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
@@ -2193,6 +2634,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
nvmeq->last_sq_tail = 0;
nvmeq->cq_head = 0;
nvmeq->cq_phase = 1;
+ nvme_adaptive_reset(nvmeq);
nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq));
nvme_dbbuf_init(dev, nvmeq, qid);
@@ -2813,6 +3255,33 @@ static ssize_t hmb_store(struct device *dev, struct device_attribute *attr,
}
static DEVICE_ATTR_RW(hmb);
+static ssize_t adaptive_irq_polling_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct nvme_dev *ndev = to_nvme_dev(dev_get_drvdata(dev));
+
+ return sysfs_emit(buf, "%d\n", READ_ONCE(ndev->adaptive_irq_polling));
+}
+
+static ssize_t adaptive_irq_polling_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct nvme_dev *ndev = to_nvme_dev(dev_get_drvdata(dev));
+ bool enable;
+ int ret;
+
+ ret = kstrtobool(buf, &enable);
+ if (ret)
+ return ret;
+ ret = nvme_adaptive_switch(ndev, enable);
+ if (ret)
+ return ret;
+ return count;
+}
+static DEVICE_ATTR_RW(adaptive_irq_polling);
+
static umode_t nvme_pci_attrs_are_visible(struct kobject *kobj,
struct attribute *a, int n)
{
@@ -2828,6 +3297,8 @@ static umode_t nvme_pci_attrs_are_visible(struct kobject *kobj,
}
if (a == &dev_attr_hmb.attr && !ctrl->hmpre)
return 0;
+ if (a == &dev_attr_adaptive_irq_polling.attr && use_threaded_interrupts)
+ return 0;
return a->mode;
}
@@ -2837,6 +3308,7 @@ static struct attribute *nvme_pci_attrs[] = {
&dev_attr_cmbloc.attr,
&dev_attr_cmbsz.attr,
&dev_attr_hmb.attr,
+ &dev_attr_adaptive_irq_polling.attr,
NULL,
};
@@ -3690,6 +4162,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
return ERR_PTR(-ENOMEM);
INIT_WORK(&dev->ctrl.reset_work, nvme_reset_work);
mutex_init(&dev->shutdown_lock);
+ dev->adaptive_irq_polling = use_adaptive_irq_polling;
dev->nr_write_queues = write_queues;
dev->nr_poll_queues = poll_queues;
More information about the Linux-nvme
mailing list