[PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Kamaljit Singh
Kamaljit.Singh1 at wdc.com
Fri Apr 18 11:55:43 PDT 2025
Sagi,
From: Sagi Grimberg <sagi at grimberg.me>
Date: Friday, April 18, 2025 at 03:51
>> Sagi,
>> I tried both of these patches but looks like #1 causes an infinite loop. dmesg was full of panics.
>> I had tried just #1 and later with #1+#2. Both failed the same way.
>That's no good :)
>Can you share the panics? This version should be identical to what
>Hannes introduced in:
Sorry, I'm posting ~200 lines at the bottom, starting from the beginning of the panic until the next "---cut here---". I can send the whole dmesg as attachment as well. Please let me know.
The patches I applied are these:
1: nvme-tcp: partial write tcp-q-wake-sender untested by Sagi
2: nvme-tcp: recv-side EAGAIN untested fix by Sagi
3: nvme-tcp: fix I/O stalls on congested sockets
4: nvme-tcp: sanitize request list handling
5: nvme-tcp: avoid inline sending when handling R2T PDUs
6: tls: use independent record sz for data sends <--- this is my internal/temp fix for a TLS send issue we found
You can probably ignore #6.
[1] nvme-tcp: partial write tcp-q-wake-sender untested by Sagi
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 1839d2110c0d..12a790b534aa 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags {
NVME_TCP_Q_LIVE = 1,
NVME_TCP_Q_POLLING = 2,
NVME_TCP_Q_IO_CPU_SET = 3,
+ NVME_TCP_Q_WAKE_SENDER = 4,
};
enum nvme_tcp_recv_state {
@@ -1074,6 +1075,7 @@ static void nvme_tcp_write_space(struct sock *sk)
queue = sk->sk_user_data;
if (likely(queue && sk_stream_is_writeable(sk))) {
clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+ set_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags);
queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
}
read_unlock_bh(&sk->sk_callback_lock);
@@ -1368,6 +1370,7 @@ static void nvme_tcp_io_work(struct work_struct *w)
container_of(w, struct nvme_tcp_queue, io_work);
unsigned long deadline = jiffies + msecs_to_jiffies(1);
+ clear_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags);
do {
bool pending = false;
int result;
@@ -1387,10 +1390,15 @@ static void nvme_tcp_io_work(struct work_struct *w)
else if (unlikely(result < 0) && result != -EAGAIN)
return;
- if (nvme_tcp_queue_has_pending(queue))
+ /* did we get some space after spending time in recv ? */
+ if (nvme_tcp_queue_has_pending(queue) &&
+ sk_stream_is_writeable(queue->sock->sk))
pending = true;
- if (!pending || !queue->rd_enabled)
+ if (!queue->rd_enabled)
+ return;
+
+ if (!pending && !test_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags))
return;
} while (!time_after(jiffies, deadline)); /* quota is exhausted */
[2] nvme-tcp: recv-side EAGAIN untested fix by Sagi
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index d3f12b17e68e..1839d2110c0d 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1359,7 +1359,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
queue->nr_cqe = 0;
consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb);
release_sock(sk);
- return consumed;
+ return consumed == -EAGAIN ? 0 : consumed;
}
static void nvme_tcp_io_work(struct work_struct *w)
[3] nvme-tcp: fix I/O stalls on congested sockets (by Hannes R.)
When the socket is busy processing nvme_tcp_try_recv() might
return -EAGAIN, but this doesn't automatically imply that
the sending side is blocked, too.
So check if there are pending requests once nvme_tcp_try_recv()
returns -EAGAIN and continue with the sending loop to avoid
I/O stalls.
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 4647d5d547f8..d3f12b17e68e 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1384,9 +1384,12 @@ static void nvme_tcp_io_work(struct work_struct *w)
result = nvme_tcp_try_recv(queue);
if (result > 0)
pending = true;
- else if (unlikely(result < 0))
+ else if (unlikely(result < 0) && result != -EAGAIN)
return;
+ if (nvme_tcp_queue_has_pending(queue))
+ pending = true;
+
if (!pending || !queue->rd_enabled)
return;
[4] nvme-tcp: sanitize request list handling
Validate the request in nvme_tcp_handle_r2t() to ensure it's not
part of any list, otherwise a malicious R2T PDU might inject a
loop in request list processing.
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 40dce5dc3ff5..4647d5d547f8 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -455,6 +455,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue)
}
list_del(&req->entry);
+ init_llist_node(&req->lentry);
return req;
}
@@ -562,6 +563,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set,
req->queue = queue;
nvme_req(rq)->ctrl = &ctrl->ctrl;
nvme_req(rq)->cmd = &pdu->cmd;
+ init_llist_node(&req->lentry);
+ INIT_LIST_HEAD(&req->entry);
return 0;
}
@@ -773,8 +776,12 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
nvme_tcp_setup_h2c_data_pdu(req);
+ WARN_ON(queue->request == req);
+ WARN_ON(llist_on_list(&req->lentry));
+ WARN_ON(!list_empty(&req->entry));
llist_add(&req->lentry, &queue->req_list);
- queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
+ if (list_empty(&queue->send_list))
+ queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
return 0;
}
@@ -2632,6 +2639,8 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg)
ctrl->async_req.offset = 0;
ctrl->async_req.curr_bio = NULL;
ctrl->async_req.data_len = 0;
+ init_llist_node(&ctrl->async_req.lentry);
+ INIT_LIST_HEAD(&ctrl->async_req.entry);
nvme_tcp_queue_request(&ctrl->async_req, true);
}
[5] nvme-tcp: avoid inline sending when handling R2T PDUs (by Hannes R.)
When handling an R2T PDU we should not attempt to send consecutive
PDUs as we are running from an softirq context, and sending PDUs
from the receive context will mess up latencies.
So just queue it and let the io_work workqueue function do the work.
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 14047b23a559..40dce5dc3ff5 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -405,7 +405,7 @@ static inline bool nvme_tcp_queue_more(struct nvme_tcp_queue *queue)
}
static inline void nvme_tcp_queue_request(struct nvme_tcp_request *req,
- bool sync, bool last)
+ bool last)
{
struct nvme_tcp_queue *queue = req->queue;
bool empty;
@@ -419,7 +419,7 @@ static inline void nvme_tcp_queue_request(struct nvme_tcp_request *req,
* are on the same cpu, so we don't introduce contention.
*/
if (queue->io_cpu == raw_smp_processor_id() &&
- sync && empty && mutex_trylock(&queue->send_mutex)) {
+ empty && mutex_trylock(&queue->send_mutex)) {
nvme_tcp_send_all(queue);
mutex_unlock(&queue->send_mutex);
}
@@ -772,7 +772,9 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
req->ttag = pdu->ttag;
nvme_tcp_setup_h2c_data_pdu(req);
- nvme_tcp_queue_request(req, false, true);
+
+ llist_add(&req->lentry, &queue->req_list);
+ queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
return 0;
}
@@ -2631,7 +2633,7 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg)
ctrl->async_req.curr_bio = NULL;
ctrl->async_req.data_len = 0;
- nvme_tcp_queue_request(&ctrl->async_req, true, true);
+ nvme_tcp_queue_request(&ctrl->async_req, true);
}
static void nvme_tcp_complete_timed_out(struct request *rq)
@@ -2783,7 +2785,7 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx,
nvme_start_request(rq);
- nvme_tcp_queue_request(req, true, bd->last);
+ nvme_tcp_queue_request(req, bd->last);
return BLK_STS_OK;
}
[6] tls: use independent record sz for data sends (by Kamaljit)
Author: Kamaljit Singh <kamaljit.singh1 at wdc.com>
Date: Wed Apr 9 15:11:54 2025 -0700
tls: use independent record sz for data sends
[Hack] send data to use a smaller default value of 4096 instead of the
common macro TLS_MAX_PAYLOAD_SIZE, which applies to both directions.
Add a new user configurable parameter max_send_record_bytes to allow
testing with different values other than 4096 bytes.
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index f672a62a9a52..968dde6848ad 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -55,6 +55,8 @@ static DEFINE_SPINLOCK(tls_device_lock);
static struct page *dummy_page;
+extern unsigned int max_send_record_bytes;
+
static void tls_device_free_ctx(struct tls_context *ctx)
{
if (ctx->tx_conf == TLS_HW)
@@ -459,7 +461,7 @@ static int tls_push_data(struct sock *sk,
/* TLS_HEADER_SIZE is not counted as part of the TLS record, and
* we need to leave room for an authentication tag.
*/
- max_open_record_len = TLS_MAX_PAYLOAD_SIZE +
+ max_open_record_len = max_send_record_bytes +
prot->prepend_size;
do {
rc = tls_do_allocation(sk, ctx, pfrag, prot->prepend_size);
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index cb86b0bf9a53..6f2c42e5b603 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -52,6 +52,11 @@ MODULE_DESCRIPTION("Transport Layer Security Support");
MODULE_LICENSE("Dual BSD/GPL");
MODULE_ALIAS_TCP_ULP("tls");
+unsigned int max_send_record_bytes = 4096;
+module_param(max_send_record_bytes, uint, 0644);
+MODULE_PARM_DESC(max_send_record_bytes , " max send record size in bytes (default 4096)");
+EXPORT_SYMBOL_GPL(max_send_record_bytes);
+
enum {
TLSV4,
TLSV6,
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 914d4e1516a3..45486773ad07 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -48,6 +48,8 @@
#include "tls.h"
+extern unsigned int max_send_record_bytes;
+
struct tls_decrypt_arg { struct_group(inargs,
bool zc;
@@ -1059,7 +1061,7 @@ static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg,
orig_size = msg_pl->sg.size;
full_record = false; try_to_copy = msg_data_left(msg);
- record_room = TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size;
+ record_room = max_send_record_bytes - msg_pl->sg.size;
if (try_to_copy >= record_room) {
try_to_copy = record_room;
full_record = true;
================================
Snapshot of panic messages:
...
[2025-04-16 22:55:32.992] [Wed Apr 16 22:55:34 2025] nvme nvme0: QID 0x0: traddr=10.10.10.223,trsvcid=4420,sport=0x8ac6,dport=0x1144
[2025-04-16 22:55:33.445] [Wed Apr 16 22:55:34 2025] nvme nvme0: D3 entry latency set to 16 seconds
[2025-04-16 22:55:33.898] [Wed Apr 16 22:55:35 2025] nvme nvme0: queue_size 128 > ctrl sqsize 17, clamping down
[2025-04-16 22:55:33.929] [Wed Apr 16 22:55:35 2025] nvme nvme0: creating 3 I/O queues.
[2025-04-16 22:55:35.398] [Wed Apr 16 22:55:36 2025] nvme nvme0: mapped 3/0/0 default/read/poll queues.
[2025-04-16 22:55:35.445] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID 0x1: traddr=10.10.10.223,trsvcid=4420,sport=0x8ac8,dport=0x1144
[2025-04-16 22:55:35.508] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID 0x2: traddr=10.10.10.223,trsvcid=4420,sport=0x8ad4,dport=0x1144
[2025-04-16 22:55:35.554] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID 0x3: traddr=10.10.10.223,trsvcid=4420,sport=0x8ad8,dport=0x1144
[2025-04-16 22:55:35.632] [Wed Apr 16 22:55:37 2025] nvme nvme0: new ctrl: NQN "nqn.2015-09.com.wdc:nvme.1", addr 10.10.10.223:4420, hostnqn: nqn.2014-08.org.nvmexpress:host.0
[2025-04-16 22:55:36.195] [Wed Apr 16 22:55:37 2025] nvme nvme0: Failed to get ANA log: 16386
[2025-04-16 22:55:43.023]
[2025-04-16 22:55:43.492]
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 22 PID: 134878 at tcp.c:782 nvme_tcp_recv_skb+0x115e/0x11c0 [nvme_tcp]
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] Modules linked in: nvme_tcp(OE) nvme_rdma(E) nvme_fabrics(E) nvme(E) nvme_core(E) nvme_keyring(E) nvme_auth(E) xt_tcpudp(E) nft_compat(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) cmac(E) nls_utf8(E) nfs(E) cifs(E) lockd(E) cifs_arc4(E) nls_ucs2_utils(E) grace(E) cifs_md4(E) nf_tables(E) netfs(E) cfg80211(E) ipmi_ssif(E) amd_atl(E) intel_rapl_msr(E) intel_rapl_common(E) rpcrdma(E) amd64_edac(E) rdma_ucm(E) edac_mce_amd(E) scsi_transport_iscsi(E) rdma_cm(E) ib_umad(E) ib_ipoib(E) iw_cm(E) kvm_amd(E) ib_cm(E) kvm(E) ftdi_sio(E) ast(E)
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] usbserial(E)
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] rapl(E) i2c_algo_bit(E) acpi_ipmi(E) ccp(E) ipmi_si(E) i2c_piix4(E) ipmi_devintf(E) k10temp(E) ptdma(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 44 PID: 1609 at tcp.c:782 nvme_tcp_recv_skb+0x115e/0x11c0 [nvme_tcp]
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] i2c_smbus(E) joydev(E) input_leds(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] Modules linked in:
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ipmi_msghandler(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_tcp(OE) nvme_rdma(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] mac_hid(E) bnxt_re(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_fabrics(E) nvme(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] sunrpc(E) binfmt_misc(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_core(E) nvme_keyring(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] sch_fq_codel(E) efi_pstore(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_auth(E) xt_tcpudp(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nfnetlink(E) dmi_sysfs(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nft_compat(E) rpcsec_gss_krb5(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ip_tables(E) x_tables(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] auth_rpcgss(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] autofs4(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nfsv4(E) cmac(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] btrfs(E) blake2b_generic(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nls_utf8(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] zstd_compress(E) raid10(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nfs(E) cifs(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] raid456(E) async_raid6_recov(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] lockd(E) cifs_arc4(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] async_memcpy(E) async_pq(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nls_ucs2_utils(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] async_xor(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] grace(E) cifs_md4(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] async_tx(E) xor(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nf_tables(E) netfs(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] raid6_pq(E) raid1(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] cfg80211(E) ipmi_ssif(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] raid0(E) mlx5_ib(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] amd_atl(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ib_uverbs(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] intel_rapl_msr(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] macsec(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] intel_rapl_common(E) rpcrdma(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ib_core(E) mlx5_core(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] amd64_edac(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] mlxfw(E) psample(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] rdma_ucm(E) edac_mce_amd(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] polyval_clmulni(E) polyval_generic(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] scsi_transport_iscsi(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ghash_clmulni_intel(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 0 PID: 832 at tcp.c:782 nvme_tcp_recv_skb+0x115e/0x11c0 [nvme_tcp]
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] sha256_ssse3(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] rdma_cm(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] Modules linked in:
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ib_umad(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] sha1_ssse3(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_tcp(OE)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ahci(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_rdma(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ib_ipoib(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] bnxt_en(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_fabrics(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] iw_cm(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] tls(E) pci_hyperv_intf(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] nvme_core(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] kvm_amd(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] ib_cm(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] nvme_keyring(E) nvme_auth(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] kvm(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] libahci(E) hid_generic(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] ftdi_sio(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] xt_tcpudp(E) nft_compat(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] ast(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] usbhid(E) hid(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] usbserial(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] rpcsec_gss_krb5(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] rapl(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] aesni_intel(E)
[2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] auth_rpcgss(E) nfsv4(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] crypto_simd(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] cmac(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] i2c_algo_bit(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] acpi_ipmi(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] nls_utf8(E) nfs(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] cryptd(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] cifs(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] [last unloaded: nvme_tcp(E)]
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] ccp(E) ipmi_si(E)
[2025-04-16 22:56:24.820]
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] i2c_piix4(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] lockd(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] ipmi_devintf(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] cifs_arc4(E) nls_ucs2_utils(E)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] CPU: 22 UID: 0 PID: 134878 Comm: kworker/22:0H Tainted: G OE 6.14.0-qp-sdprt-adc-49-50-51-52-ktls+ #1 PREEMPT(voluntary)
[2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] k10temp(E) ptdma(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] grace(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] i2c_smbus(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] cifs_md4(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] Hardware name: Supermicro AS -1114S-WTRT/H12SSW-NT, BIOS 1.0b 11/15/2019
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] joydev(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] nf_tables(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] input_leds(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] ipmi_msghandler(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] Workqueue: nvme_tcp_wq nvme_tcp_io_work [nvme_tcp]
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] netfs(E) cfg80211(E)
[2025-04-16 22:56:24.835]
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] mac_hid(E) bnxt_re(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] ipmi_ssif(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] RIP: 0010:nvme_tcp_recv_skb+0x115e/0x11c0 [nvme_tcp]
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] amd_atl(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] sunrpc(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] Code: 00 41 8b 57 20 44 89 d1 4d 89 c8 48 c7 c6 88 d1 7a c1 e8 a5 b6 2f c9 e9 cc fb ff ff 0f 0b e9 28 fb ff ff 0f 0b e9 35 fb ff ff <0f> 0b e9 45 fb ff ff 49 8b 87 28 01 00 00 89 d1 89 f2 48 c7 c6 10
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] binfmt_misc(E) sch_fq_codel(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] intel_rapl_msr(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] RSP: 0018:ffffcdccf87afc80 EFLAGS: 00010293
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] intel_rapl_common(E)
[2025-04-16 22:56:24.851]
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] efi_pstore(E) nfnetlink(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] rpcrdma(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] RAX: dead000000000100 RBX: ffff8dbda9193680 RCX: 0000000000000000
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] amd64_edac(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8dbd986b14a8
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] dmi_sysfs(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] rdma_ucm(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] ip_tables(E) x_tables(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] edac_mce_amd(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] RBP: ffffcdccf87afd08 R08: 0000000000008000 R09: 0000000000000000
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] scsi_transport_iscsi(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] R10: 0000000000000000 R11: 000000000000001c R12: ffff8dbe027c8290
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] autofs4(E) btrfs(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] R13: 0000000000000005 R14: 000000000000001c R15: ffff8dbd986b1498
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] rdma_cm(E) ib_umad(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] FS: 0000000000000000(0000) GS:ffff8e3b3f1b4000(0000) knlGS:0000000000000000
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] blake2b_generic(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] ib_ipoib(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] zstd_compress(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] raid10(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] iw_cm(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] kvm_amd(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] CR2: 00005e9c7424be98 CR3: 00000002357ec000 CR4: 0000000000350ef0
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] raid456(E) async_raid6_recov(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] Call Trace:
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] ib_cm(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] async_memcpy(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] <TASK>
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] async_pq(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] kvm(E) ftdi_sio(E) ast(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] async_xor(E) async_tx(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] usbserial(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] xor(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] tls_sw_read_sock+0x12c/0x4a0 [tls]
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] rapl(E) i2c_algo_bit(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] raid6_pq(E) raid1(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] acpi_ipmi(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] ? __pfx_nvme_tcp_recv_skb+0x10/0x10 [nvme_tcp]
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] ccp(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] raid0(E) mlx5_ib(E) ib_uverbs(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] ipmi_si(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] nvme_tcp_try_recv+0x78/0xc0 [nvme_tcp]
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] i2c_piix4(E)
[2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] macsec(E) ib_core(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] ipmi_devintf(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] nvme_tcp_io_work+0xc1/0x1e0 [nvme_tcp]
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] k10temp(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] mlx5_core(E) mlxfw(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] ptdma(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] process_one_work+0x191/0x3e0
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] i2c_smbus(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] psample(E) polyval_clmulni(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] joydev(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] worker_thread+0x2e3/0x420
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] polyval_generic(E) ghash_clmulni_intel(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] input_leds(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] sha256_ssse3(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] ? srso_return_thunk+0x5/0x5f
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] sha1_ssse3(E)
[2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] ipmi_msghandler(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ? __pfx_worker_thread+0x10/0x10
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] mac_hid(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ahci(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] kthread+0x10d/0x230
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] bnxt_re(E) sunrpc(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] bnxt_en(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ? __pfx_kthread+0x10/0x10
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] tls(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] binfmt_misc(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] pci_hyperv_intf(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ret_from_fork+0x47/0x70
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] sch_fq_codel(E) efi_pstore(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] libahci(E)
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ? __pfx_kthread+0x10/0x10
[2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] hid_generic(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] nfnetlink(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] ret_from_fork_asm+0x1a/0x30
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] usbhid(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] dmi_sysfs(E) ip_tables(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] hid(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] </TASK>
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] aesni_intel(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] x_tables(E) autofs4(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] crypto_simd(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] btrfs(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] blake2b_generic(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] cryptd(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] ---[ end trace 0000000000000000 ]---
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] [last unloaded: nvme_tcp(E)]
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] zstd_compress(E)
[2025-04-16 22:56:24.929]
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] raid10(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] CPU: 44 UID: 0 PID: 1609 Comm: kworker/44:1H Tainted: G W OE 6.14.0-qp-sdprt-adc-49-50-51-52-ktls+ #1 PREEMPT(voluntary)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] async_pq(E) async_xor(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------
...
...
More information about the Linux-nvme
mailing list