blktests failures with v7.0-rc1 kernel
Shinichiro Kawasaki
shinichiro.kawasaki at wdc.com
Thu Feb 26 00:09:10 PST 2026
Hi all,
I ran the latest blktests (git hash: f14914d04256) with the v7.0-rc1 kernel. I
observed 8 failures listed below. Comparing with the previous report with the
v6.19 kernel [1], 1 failure was resolved (zbd/013) and 3 failures are newly
observed (blktrace/002, nvme/060 and nvme/061 fc transport). Your actions to
fix the failures will be welcomed as always. Especially, nvme/058 and nvme/061
are causing test run hangs for fc transport. Fixing of the hangs will be highly
appreciated.
[1] https://lore.kernel.org/linux-block/aY7ZBfMjVIhe_wh3@shinmob/
List of failures
================
#1: blktrace/002 (new)
#2: nvme/005,063 (tcp transport)
#3: nvme/058 (fc transport)(new symptoms, kmemleak and hang)
#4: nvme/060 (new)
#5: nvme/061 (rdma transport, siw driver)(kmemleak)
#6: nvme/061 (new)(fc transport)(hang)
#7: nbd/002
#8: zbd/009 (kmemleak)
Failure description
===================
#1: blktrace/002
The test case blktrace/002 failed due to "BUG: using __this_cpu_read() in
preemptible" and a following WARN [2].
#2: nvme/005,063 (tcp transport)
The test case nvme/005 and 063 fail for tcp transport due to the lockdep
WARN related to the three locks q->q_usage_counter, q->elevator_lock and
set->srcu. Refer to the nvme/063 failure report for v6.16-rc1 kernel [3].
[3] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
#3: nvme/058 (fc transport)
When the test case nvme/058 is repeated for fc transport about 50 times, it
fails or hangs. Until the kernel v6.19, this test case falied due to various
WARNs [1], but now it shows different symptoms with v7.0-rc1 kernel.
When the test case fails, it reports kmemleak failure [4]. When the test
case hangs, the kernel reports Oops and KASAN null-ptr-deref [5].
#4: nvme/060 (rdma transport)
When the test case is repeated for rdma transports around 50 times, the test
case fails. The failure is very rare, and it is observed with v6.19 kernel
also. Then I don't think this is a new kernel problem.
There are two failure symptoms. Both symptoms do not look like kernel side
problem to me. I suspect this test case needs some more care for the nvme
connect failure and the state file existence check.
[symptom 1]
nvme/060 (tr=rdma) (test nvme fabrics target reset) [failed]
runtime ... 87.444s
--- tests/nvme/060.out 2026-02-20 12:15:11.066947841 +0000
+++ /home/fedora/blktests/results/nodev_tr_rdma/nvme/060.out.bad 2026-02-20 15:06:44.552705787 +0000
@@ -1,2 +1,3 @@
Running nvme/060
+FAIL: nvme connect return error code
Test complete
[symptom 2]
nvme/060 (tr=rdma) (test nvme fabrics target reset) [failed]
runtime ... 22.545s
--- tests/nvme/060.out 2025-08-26 21:28:52.798847739 +0900
+++ /home/shin/Blktests/blktests/results/nodev_tr_rdma/nvme/060.out.bad 2026-02-26 15:20:36.973686247 +0900
@@ -1,2 +1,3 @@
Running nvme/060
+_: line 1: /sys/kernel/debug/nvmet/blktests-subsystem-1/ctrl1/state: No such file or directory
Test complete
#5: nvme/061 (rdma transport, siw driver)
When the test case nvme/061 is repeated twice for the rdma transport and the
siw driver on the kernel v6.19 with CONFIG_DEBUG_KMEMLEAK enabled, it fails
with a kmemleak message. Refer to the nvme/061 failure report for v6.19
kernel [1].
#6: nvme/061 (fc transport)
When the test case nvme/061 is repeated around 50 times for the fc
transport, the test process hangs after Oops and KASAN null-ptr-deref [6].
This symptom looks similar as the hang observed with nvme/058, but call
traces look different between the two test cases.
#7: nbd/002
The test case nbd/002 fails due to the lockdep WARN related to
mm->mmap_lock, sk_lock-AF_INET6 and fs_reclaim. Refer to the nbd/002 failure
report for v6.18-rc1 kernel [7].
[7] https://lore.kernel.org/linux-block/ynmi72x5wt5ooljjafebhcarit3pvu6axkslqenikb2p5txe57@ldytqa2t4i2x/
#8: zbd/009
When the test case zbd/009 is repeated twice on the kernel v6.19 with
CONFIG_DEBUG_KMEMLEAK enabled, it fails with a kmemleak message. Memory leak
in btrfs is suspected. Refer to the zbd/009 failure report for v6.19 kernel
[1].
[2] BUG during blktrace/002
[ 73.902736] [ T1212] run blktests blktrace/002 at 2026-02-26 09:14:35
[ 73.983894] [ T1212] null_blk: disk nullb1 created
[ 74.200040] [ T1234] check_preemption_disabled: 3 callbacks suppressed
[ 74.200045] [ T1234] BUG: using __this_cpu_read() in preemptible [00000000] code: dd/1234
[ 74.202064] [ T1234] caller is tracing_record_cmdline+0x10/0x40
[ 74.202964] [ T1234] CPU: 2 UID: 0 PID: 1234 Comm: dd Not tainted 7.0.0-rc1 #581 PREEMPT(full)
[ 74.202983] [ T1234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
[ 74.202988] [ T1234] Call Trace:
[ 74.202990] [ T1234] <TASK>
[ 74.202992] [ T1234] dump_stack_lvl+0x6a/0x90
[ 74.203016] [ T1234] check_preemption_disabled+0xe5/0xf0
[ 74.203020] [ T1234] tracing_record_cmdline+0x10/0x40
[ 74.203024] [ T1234] __blk_add_trace+0x45d/0xa20
[ 74.203033] [ T1234] blk_add_trace_bio+0x1be/0x360
[ 74.203039] [ T1234] submit_bio_noacct_nocheck+0x314/0xb40
[ 74.203045] [ T1234] ? __pfx___might_resched+0x10/0x10
[ 74.203063] [ T1234] ? __pfx_submit_bio_noacct_nocheck+0x10/0x10
[ 74.203067] [ T1234] ? should_fail_bio+0xb3/0x150
[ 74.203070] [ T1234] ? submit_bio_noacct+0x86f/0x1ee0
[ 74.203077] [ T1234] submit_bio_wait+0x106/0x200
[ 74.203080] [ T1234] ? __pfx_submit_bio_wait+0x10/0x10
[ 74.203089] [ T1234] ? bio_init+0x3f2/0x6a0
[ 74.203093] [ T1234] __blkdev_direct_IO_simple+0x3f5/0x860
[ 74.203098] [ T1234] ? __pfx___blkdev_direct_IO_simple+0x10/0x10
[ 74.203104] [ T1234] ? __pfx_submit_bio_wait_endio+0x10/0x10
[ 74.203110] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.203118] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.203122] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.203128] [ T1234] blkdev_direct_IO+0xa38/0x1f50
[ 74.203130] [ T1234] ? reacquire_held_locks+0xcc/0x1e0
[ 74.203133] [ T1234] ? __mark_inode_dirty+0xb2d/0x1150
[ 74.203138] [ T1234] ? do_raw_spin_lock+0x124/0x260
[ 74.203140] [ T1234] ? lock_acquire+0x1be/0x340
[ 74.203143] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.203148] [ T1234] ? __pfx_blkdev_direct_IO+0x10/0x10
[ 74.203150] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.203153] [ T1234] ? lock_is_held_type+0xd5/0x140
[ 74.203158] [ T1234] ? rcu_read_lock_any_held+0x66/0xa0
[ 74.203161] [ T1234] ? touch_atime+0x2c2/0x620
[ 74.203168] [ T1234] blkdev_read_iter+0x204/0x410
[ 74.203172] [ T1234] vfs_read+0x6b4/0xb10
[ 74.203179] [ T1234] ? __pfx_vfs_read+0x10/0x10
[ 74.203181] [ T1234] ? lock_acquire+0x1ae/0x340
[ 74.203185] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.203192] [ T1234] ? rcu_read_unlock+0x17/0x60
[ 74.203195] [ T1234] ? lock_release+0x1a7/0x320
[ 74.203200] [ T1234] ksys_read+0xfb/0x200
[ 74.203204] [ T1234] ? __pfx_ksys_read+0x10/0x10
[ 74.203207] [ T1234] ? lock_release+0x1a7/0x320
[ 74.203212] [ T1234] do_syscall_64+0x136/0x1540
[ 74.203221] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.203225] [ T1234] ? __pfx_css_rstat_updated+0x10/0x10
[ 74.203231] [ T1234] ? lock_acquire+0x1ae/0x340
[ 74.203254] [ T1234] ? count_memcg_events_mm.constprop.0+0x22/0x130
[ 74.203257] [ T1234] ? rcu_is_watching+0x11/0xb0
[ 74.203261] [ T1234] ? count_memcg_events+0x106/0x4a0
[ 74.203263] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.203289] [ T1234] ? rcu_read_unlock+0x17/0x60
[ 74.203291] [ T1234] ? lock_release+0x1a7/0x320
[ 74.203295] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.203314] [ T1234] ? exc_page_fault+0x83/0x110
[ 74.203316] [ T1234] ? lock_release+0x1a7/0x320
[ 74.203337] [ T1234] ? rcu_is_watching+0x11/0xb0
[ 74.203339] [ T1234] ? irqentry_exit+0xdd/0x6a0
[ 74.203342] [ T1234] ? lockdep_hardirqs_on_prepare+0xce/0x1b0
[ 74.203345] [ T1234] ? irqentry_exit+0xe2/0x6a0
[ 74.203349] [ T1234] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 74.203352] [ T1234] RIP: 0033:0x7fa05c56577e
[ 74.203355] [ T1234] Code: 4d 89 d8 e8 d4 bc 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 13 ff ff ff 0f 1f 00 f3 0f 1e fa
[ 74.203360] [ T1234] RSP: 002b:00007ffe274681f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
[ 74.203365] [ T1234] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa05c56577e
[ 74.203367] [ T1234] RDX: 0000000000001000 RSI: 0000556479668000 RDI: 0000000000000000
[ 74.203369] [ T1234] RBP: 00007ffe27468200 R08: 0000000000000000 R09: 0000000000000000
[ 74.203370] [ T1234] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000001000
[ 74.203371] [ T1234] R13: 0000556479668000 R14: 0000000000000000 R15: 0000000000000000
[ 74.203380] [ T1234] </TASK>
[ 74.203395] [ T1234] BUG: using __this_cpu_read() in preemptible [00000000] code: dd/1234
[ 74.267244] [ T1234] caller is tracing_record_cmdline+0x10/0x40
[ 74.268426] [ T1234] CPU: 2 UID: 0 PID: 1234 Comm: dd Not tainted 7.0.0-rc1 #581 PREEMPT(full)
[ 74.268430] [ T1234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
[ 74.268431] [ T1234] Call Trace:
[ 74.268433] [ T1234] <TASK>
[ 74.268435] [ T1234] dump_stack_lvl+0x6a/0x90
[ 74.268441] [ T1234] check_preemption_disabled+0xe5/0xf0
[ 74.268446] [ T1234] tracing_record_cmdline+0x10/0x40
[ 74.268449] [ T1234] __blk_add_trace+0x45d/0xa20
[ 74.268458] [ T1234] blk_add_trace_bio+0x1be/0x360
[ 74.268464] [ T1234] blk_mq_submit_bio+0x1453/0x26d0
[ 74.268471] [ T1234] ? __pfx_blk_mq_submit_bio+0x10/0x10
[ 74.268473] [ T1234] ? __pfx_ring_buffer_lock_reserve+0x10/0x10
[ 74.268480] [ T1234] ? rb_commit+0xdc/0x950
[ 74.268483] [ T1234] ? trace_hardirqs_on+0x15/0x1a0
[ 74.268489] [ T1234] __submit_bio+0x1b0/0x5f0
[ 74.268494] [ T1234] ? __pfx___submit_bio+0x10/0x10
[ 74.268499] [ T1234] ? lock_release+0x1a7/0x320
[ 74.268508] [ T1234] ? submit_bio_noacct_nocheck+0x422/0xb40
[ 74.268511] [ T1234] submit_bio_noacct_nocheck+0x422/0xb40
[ 74.268516] [ T1234] ? __pfx_submit_bio_noacct_nocheck+0x10/0x10
[ 74.268522] [ T1234] ? submit_bio_noacct+0x86f/0x1ee0
[ 74.268528] [ T1234] submit_bio_wait+0x106/0x200
[ 74.268531] [ T1234] ? __pfx_submit_bio_wait+0x10/0x10
[ 74.268540] [ T1234] ? bio_init+0x3f2/0x6a0
[ 74.268544] [ T1234] __blkdev_direct_IO_simple+0x3f5/0x860
[ 74.268550] [ T1234] ? __pfx___blkdev_direct_IO_simple+0x10/0x10
[ 74.268555] [ T1234] ? __pfx_submit_bio_wait_endio+0x10/0x10
[ 74.268562] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.268569] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.268573] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.268578] [ T1234] blkdev_direct_IO+0xa38/0x1f50
[ 74.268581] [ T1234] ? reacquire_held_locks+0xcc/0x1e0
[ 74.268584] [ T1234] ? __mark_inode_dirty+0xb2d/0x1150
[ 74.268589] [ T1234] ? do_raw_spin_lock+0x124/0x260
[ 74.268591] [ T1234] ? lock_acquire+0x1be/0x340
[ 74.268594] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.268599] [ T1234] ? __pfx_blkdev_direct_IO+0x10/0x10
[ 74.268601] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.268604] [ T1234] ? lock_is_held_type+0xd5/0x140
[ 74.268624] [ T1234] ? rcu_read_lock_any_held+0x66/0xa0
[ 74.268627] [ T1234] ? touch_atime+0x2c2/0x620
[ 74.268634] [ T1234] blkdev_read_iter+0x204/0x410
[ 74.268641] [ T1234] vfs_read+0x6b4/0xb10
[ 74.268647] [ T1234] ? __pfx_vfs_read+0x10/0x10
[ 74.268649] [ T1234] ? lock_acquire+0x1ae/0x340
[ 74.268654] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.268675] [ T1234] ? rcu_read_unlock+0x17/0x60
[ 74.268678] [ T1234] ? lock_release+0x1a7/0x320
[ 74.268683] [ T1234] ksys_read+0xfb/0x200
[ 74.268687] [ T1234] ? __pfx_ksys_read+0x10/0x10
[ 74.268690] [ T1234] ? lock_release+0x1a7/0x320
[ 74.268695] [ T1234] do_syscall_64+0x136/0x1540
[ 74.268704] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.268709] [ T1234] ? __pfx_css_rstat_updated+0x10/0x10
[ 74.268714] [ T1234] ? lock_acquire+0x1ae/0x340
[ 74.268717] [ T1234] ? count_memcg_events_mm.constprop.0+0x22/0x130
[ 74.268720] [ T1234] ? rcu_is_watching+0x11/0xb0
[ 74.268723] [ T1234] ? count_memcg_events+0x106/0x4a0
[ 74.268725] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.268729] [ T1234] ? rcu_read_unlock+0x17/0x60
[ 74.268731] [ T1234] ? lock_release+0x1a7/0x320
[ 74.268735] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.268739] [ T1234] ? exc_page_fault+0x83/0x110
[ 74.268741] [ T1234] ? lock_release+0x1a7/0x320
[ 74.268746] [ T1234] ? rcu_is_watching+0x11/0xb0
[ 74.268748] [ T1234] ? irqentry_exit+0xdd/0x6a0
[ 74.268751] [ T1234] ? lockdep_hardirqs_on_prepare+0xce/0x1b0
[ 74.268754] [ T1234] ? irqentry_exit+0xe2/0x6a0
[ 74.268758] [ T1234] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 74.268761] [ T1234] RIP: 0033:0x7fa05c56577e
[ 74.268765] [ T1234] Code: 4d 89 d8 e8 d4 bc 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 13 ff ff ff 0f 1f 00 f3 0f 1e fa
[ 74.268767] [ T1234] RSP: 002b:00007ffe274681f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
[ 74.268770] [ T1234] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa05c56577e
[ 74.268772] [ T1234] RDX: 0000000000001000 RSI: 0000556479668000 RDI: 0000000000000000
[ 74.268773] [ T1234] RBP: 00007ffe27468200 R08: 0000000000000000 R09: 0000000000000000
[ 74.268774] [ T1234] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000001000
[ 74.268776] [ T1234] R13: 0000556479668000 R14: 0000000000000000 R15: 0000000000000000
[ 74.268785] [ T1234] </TASK>
[ 74.347663] [ T1234] ------------[ cut here ]------------
[ 74.348681] [ T1234] WARNING: kernel/trace/trace_sched_switch.c:263 at trace_save_cmdline+0x2a5/0x400, CPU#2: dd/1234
[ 74.350349] [ T1234] Modules linked in: null_blk nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr 9pnet_virtio 9pnet pcspkr netfs i2c_piix4 i2c_smbus fuse loop vsock_loopback vmw_vsock_virtio_transport_common zram vsock xfs bochs drm_client_lib drm_shmem_helper drm_kms_helper nvme floppy nvme_core drm e1000 sym53c8xx nvme_keyring nvme_auth scsi_transport_spi hkdf serio_raw ata_generic pata_acpi btrfs xor zstd_compress raid6_pq sunrpc dm_multipath i2c_dev nfnetlink qemu_fw_cfg
[ 74.358449] [ T1234] CPU: 2 UID: 0 PID: 1234 Comm: dd Not tainted 7.0.0-rc1 #581 PREEMPT(full)
[ 74.360126] [ T1234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
[ 74.361962] [ T1234] RIP: 0010:trace_save_cmdline+0x2a5/0x400
[ 74.363254] [ T1234] Code: 65 8b 05 7e ae 3d 05 65 8b 15 cf 02 3e 05 25 ff ff ff 7f 09 d0 0f 85 2c fe ff ff 65 8b 05 5f ff 3d 05 85 c0 0f 84 1d fe ff ff <0f> 0b e9 16 fe ff ff 48 8d b9 14 00 02 00 48 b8 00 00 00 00 00 fc
[ 74.367039] [ T1234] RSP: 0018:ffff88810fd37188 EFLAGS: 00010202
[ 74.368422] [ T1234] RAX: 0000000000000001 RBX: ffff88810e3bb380 RCX: 1ffffffff25cf8c0
[ 74.370080] [ T1234] RDX: 0000000000000000 RSI: ffffffff91e86019 RDI: ffff88810e3bb380
[ 74.371752] [ T1234] RBP: 1ffff11021fa6e34 R08: 0000000000000000 R09: 0000000000000001
[ 74.373440] [ T1234] R10: 0000000000000002 R11: 0000000000000001 R12: ffff88810e3bbd70
[ 74.375138] [ T1234] R13: 00000000000004d2 R14: 0000000000000000 R15: 0000000000000002
[ 74.376817] [ T1234] FS: 00007fa05c4f4740(0000) GS:ffff88841a1b2000(0000) knlGS:0000000000000000
[ 74.378658] [ T1234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 74.380148] [ T1234] CR2: 0000556479669ac8 CR3: 000000014c606000 CR4: 00000000000006f0
[ 74.381850] [ T1234] Call Trace:
[ 74.382868] [ T1234] <TASK>
[ 74.383833] [ T1234] ? dump_stack_lvl+0x7f/0x90
[ 74.385049] [ T1234] ? __pfx_trace_save_cmdline+0x10/0x10
[ 74.386400] [ T1234] ? dump_stack_lvl+0x85/0x90
[ 74.387663] [ T1234] tracing_record_cmdline+0x22/0x40
[ 74.388960] [ T1234] __blk_add_trace+0x45d/0xa20
[ 74.390193] [ T1234] blk_add_trace_bio+0x1be/0x360
[ 74.391475] [ T1234] blk_mq_submit_bio+0x1453/0x26d0
[ 74.392798] [ T1234] ? __pfx_blk_mq_submit_bio+0x10/0x10
[ 74.394130] [ T1234] ? __pfx_ring_buffer_lock_reserve+0x10/0x10
[ 74.395564] [ T1234] ? rb_commit+0xdc/0x950
[ 74.396758] [ T1234] ? trace_hardirqs_on+0x15/0x1a0
[ 74.398069] [ T1234] __submit_bio+0x1b0/0x5f0
[ 74.399274] [ T1234] ? __pfx___submit_bio+0x10/0x10
[ 74.400567] [ T1234] ? lock_release+0x1a7/0x320
[ 74.401820] [ T1234] ? submit_bio_noacct_nocheck+0x422/0xb40
[ 74.403210] [ T1234] submit_bio_noacct_nocheck+0x422/0xb40
[ 74.404607] [ T1234] ? __pfx_submit_bio_noacct_nocheck+0x10/0x10
[ 74.406070] [ T1234] ? submit_bio_noacct+0x86f/0x1ee0
[ 74.407388] [ T1234] submit_bio_wait+0x106/0x200
[ 74.408636] [ T1234] ? __pfx_submit_bio_wait+0x10/0x10
[ 74.409931] [ T1234] ? bio_init+0x3f2/0x6a0
[ 74.411064] [ T1234] __blkdev_direct_IO_simple+0x3f5/0x860
[ 74.412399] [ T1234] ? __pfx___blkdev_direct_IO_simple+0x10/0x10
[ 74.413820] [ T1234] ? __pfx_submit_bio_wait_endio+0x10/0x10
[ 74.415162] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.416393] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.417620] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.418830] [ T1234] blkdev_direct_IO+0xa38/0x1f50
[ 74.419999] [ T1234] ? reacquire_held_locks+0xcc/0x1e0
[ 74.421112] [ T1234] ? __mark_inode_dirty+0xb2d/0x1150
[ 74.422224] [ T1234] ? do_raw_spin_lock+0x124/0x260
[ 74.423278] [ T1234] ? lock_acquire+0x1be/0x340
[ 74.424267] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.425227] [ T1234] ? __pfx_blkdev_direct_IO+0x10/0x10
[ 74.426272] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.427228] [ T1234] ? lock_is_held_type+0xd5/0x140
[ 74.428224] [ T1234] ? rcu_read_lock_any_held+0x66/0xa0
[ 74.429239] [ T1234] ? touch_atime+0x2c2/0x620
[ 74.430136] [ T1234] blkdev_read_iter+0x204/0x410
[ 74.431056] [ T1234] vfs_read+0x6b4/0xb10
[ 74.431891] [ T1234] ? __pfx_vfs_read+0x10/0x10
[ 74.432783] [ T1234] ? lock_acquire+0x1ae/0x340
[ 74.433686] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.434567] [ T1234] ? rcu_read_unlock+0x17/0x60
[ 74.435473] [ T1234] ? lock_release+0x1a7/0x320
[ 74.436363] [ T1234] ksys_read+0xfb/0x200
[ 74.437177] [ T1234] ? __pfx_ksys_read+0x10/0x10
[ 74.438054] [ T1234] ? lock_release+0x1a7/0x320
[ 74.438916] [ T1234] do_syscall_64+0x136/0x1540
[ 74.439785] [ T1234] ? __lock_acquire+0x497/0x2230
[ 74.440693] [ T1234] ? __pfx_css_rstat_updated+0x10/0x10
[ 74.441656] [ T1234] ? lock_acquire+0x1ae/0x340
[ 74.442530] [ T1234] ? count_memcg_events_mm.constprop.0+0x22/0x130
[ 74.443618] [ T1234] ? rcu_is_watching+0x11/0xb0
[ 74.444487] [ T1234] ? count_memcg_events+0x106/0x4a0
[ 74.445424] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.446316] [ T1234] ? rcu_read_unlock+0x17/0x60
[ 74.447195] [ T1234] ? lock_release+0x1a7/0x320
[ 74.448060] [ T1234] ? find_held_lock+0x2b/0x80
[ 74.448923] [ T1234] ? exc_page_fault+0x83/0x110
[ 74.449794] [ T1234] ? lock_release+0x1a7/0x320
[ 74.450669] [ T1234] ? rcu_is_watching+0x11/0xb0
[ 74.451533] [ T1234] ? irqentry_exit+0xdd/0x6a0
[ 74.452401] [ T1234] ? lockdep_hardirqs_on_prepare+0xce/0x1b0
[ 74.453416] [ T1234] ? irqentry_exit+0xe2/0x6a0
[ 74.454290] [ T1234] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 74.455305] [ T1234] RIP: 0033:0x7fa05c56577e
[ 74.456139] [ T1234] Code: 4d 89 d8 e8 d4 bc 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 13 ff ff ff 0f 1f 00 f3 0f 1e fa
[ 74.459113] [ T1234] RSP: 002b:00007ffe274681f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
[ 74.460475] [ T1234] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa05c56577e
[ 74.461787] [ T1234] RDX: 0000000000001000 RSI: 0000556479668000 RDI: 0000000000000000
[ 74.463093] [ T1234] RBP: 00007ffe27468200 R08: 0000000000000000 R09: 0000000000000000
[ 74.464403] [ T1234] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000001000
[ 74.465718] [ T1234] R13: 0000556479668000 R14: 0000000000000000 R15: 0000000000000000
[ 74.467032] [ T1234] </TASK>
[ 74.467760] [ T1234] irq event stamp: 7545
[ 74.468628] [ T1234] hardirqs last enabled at (7557): [<ffffffff8e6e8d1e>] __up_console_sem+0x5e/0x70
[ 74.470111] [ T1234] hardirqs last disabled at (7568): [<ffffffff8e6e8d03>] __up_console_sem+0x43/0x70
[ 74.471609] [ T1234] softirqs last enabled at (7408): [<ffffffff8e523036>] __irq_exit_rcu+0x126/0x240
[ 74.473113] [ T1234] softirqs last disabled at (7399): [<ffffffff8e523036>] __irq_exit_rcu+0x126/0x240
[ 74.474615] [ T1234] ---[ end trace 0000000000000000 ]---
...
[4] kmemleak messages during nvme/058 with fc transport
The reported kmemleak backtraces differs for each recreation. Here I share 3
kmemleak logs for 3 failures.
[failure 1]
unreferenced object 0xffff888144883800 (size 512):
comm "kworker/1:2H", pid 33161, jiffies 4299855935
hex dump (first 32 bytes):
40 55 73 28 81 88 ff ff 00 00 00 00 00 00 00 00 @Us(............
00 00 00 00 ad 4e ad de ff ff ff ff 00 00 00 00 .....N..........
backtrace (crc 70dc004c):
__kmalloc_cache_noprof+0x52b/0x6f0
0xffffffffc1bc9438
0xffffffffc13f52ee
blk_mq_dispatch_rq_list+0x39b/0x2340
__blk_mq_sched_dispatch_requests+0x1dd/0x1510
blk_mq_sched_dispatch_requests+0xa8/0x150
blk_mq_run_work_fn+0x1c5/0x2c0
process_one_work+0x868/0x1590
worker_thread+0x5ee/0xfd0
kthread+0x368/0x460
ret_from_fork+0x66a/0x9e0
ret_from_fork_asm+0x1a/0x30
[failure 2]
unreferenced object 0xffff8881164b3000 (size 512):
comm "kworker/u16:13", pid 15464, jiffies 4300014387
hex dump (first 32 bytes):
40 c1 9d 7b 81 88 ff ff 00 00 00 00 00 00 00 00 @..{............
00 00 00 00 ad 4e ad de ff ff ff ff 00 00 00 00 .....N..........
backtrace (crc cba4ad6d):
__kmalloc_cache_noprof+0x52b/0x6f0
0xffffffffc0bf8438
0xffffffffc0bee2ee
blk_mq_dispatch_rq_list+0x39b/0x2340
__blk_mq_sched_dispatch_requests+0x1dd/0x1510
blk_mq_sched_dispatch_requests+0xa8/0x150
blk_mq_run_hw_queue+0x28c/0x520
blk_execute_rq+0x166/0x380
__nvme_submit_sync_cmd+0x104/0x320 [nvme_core]
nvme_start_ctrl+0x33d/0x620 [nvme_core]
0xffffffffc0bf52b0
process_one_work+0x868/0x1590
worker_thread+0x5ee/0xfd0
kthread+0x368/0x460
ret_from_fork+0x66a/0x9e0
ret_from_fork_asm+0x1a/0x30
[failure 3]
unreferenced object 0xffff888163d4d000 (size 512):
comm "kworker/u16:7", pid 29836, jiffies 4300535674
hex dump (first 32 bytes):
40 d5 02 42 81 88 ff ff 00 00 00 00 00 00 00 00 @..B............
00 00 00 00 ad 4e ad de ff ff ff ff 00 00 00 00 .....N..........
backtrace (crc 885849ba):
__kmalloc_cache_noprof+0x52b/0x6f0
0xffffffffc1bc9438
0xffffffffc13f52ee
blk_mq_dispatch_rq_list+0x39b/0x2340
__blk_mq_sched_dispatch_requests+0x1dd/0x1510
blk_mq_sched_dispatch_requests+0xa8/0x150
blk_mq_run_hw_queue+0x28c/0x520
blk_execute_rq+0x166/0x380
__nvme_submit_sync_cmd+0x104/0x320 [nvme_core]
nvmf_reg_read32+0xd5/0x190 [nvme_fabrics]
nvme_wait_ready+0x154/0x2c0 [nvme_core]
nvme_enable_ctrl+0x366/0x490 [nvme_core]
0xffffffffc13f29d0
process_one_work+0x868/0x1590
worker_thread+0x5ee/0xfd0
kthread+0x368/0x460
[5] Oops and KASAN null-ptr-deref during nvme/058 with fc transport
...
Feb 24 19:13:57 testnode2 kernel: nvme nvme6: Removing ctrl: NQN "blktests-subsystem-1"
Feb 24 19:13:58 testnode2 kernel: nvmet_fc: {0:2}: Association freed
Feb 24 19:13:58 testnode2 kernel: (NULL device *): Disconnect LS failed: No Association
Feb 24 19:13:58 testnode2 kernel: nvmet_fc: {1:0}: Association deleted
Feb 24 19:13:58 testnode2 kernel: nvme nvme7: Removing ctrl: NQN "blktests-subsystem-1"
Feb 24 19:13:58 testnode2 kernel: Oops: general protection fault, probably for non-canonical address 0xdffffc0000000002: 0000 [#1] SMP KASAN PTI
Feb 24 19:13:58 testnode2 kernel: KASAN: null-ptr-deref in range [0x0000000000000010-0x0000000000000017]
Feb 24 19:13:58 testnode2 kernel: CPU: 2 UID: 0 PID: 97800 Comm: kworker/u16:13 Not tainted 7.0.0-rc1 #581 PREEMPT(full)
Feb 24 19:13:58 testnode2 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
Feb 24 19:13:58 testnode2 kernel: Workqueue: nvmet-wq fcloop_tgt_fcprqst_done_work [nvme_fcloop]
Feb 24 19:13:58 testnode2 kernel: RIP: 0010:do_raw_spin_lock+0x6b/0x260
Feb 24 19:13:58 testnode2 kernel: Code: fe 6d 9b 41 c7 04 04 f1 f1 f1 f1 42 c7 44 20 04 04 f3 f3 f3 65 48 8b 15 1b 61 69 05 48 89 54 24 60 31 d2 48 89 fa 48 c1 ea 03 <0f> b6 14 02 48 89 f8 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 af
Feb 24 19:13:58 testnode2 kernel: RSP: 0018:ffff8881318f7b88 EFLAGS: 00010203
Feb 24 19:13:58 testnode2 kernel: RAX: dffffc0000000000 RBX: 0000000000000010 RCX: 0000000000000000
Feb 24 19:13:58 testnode2 kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000014
Feb 24 19:13:58 testnode2 kernel: RBP: ffff8881389885f8 R08: 0000000000000001 R09: 0000000000000000
Feb 24 19:13:58 testnode2 kernel: R10: 0000000000000002 R11: fefefefefefefeff R12: 1ffff1102631ef72
Feb 24 19:13:58 testnode2 kernel: R13: 0000000000000000 R14: 0000000000000006 R15: ffff8881328ec700
Feb 24 19:13:58 testnode2 kernel: FS: 0000000000000000(0000) GS:ffff88840d1b2000(0000) knlGS:0000000000000000
Feb 24 19:13:58 testnode2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 24 19:13:58 testnode2 kernel: CR2: 00007f2c9ab5a7d4 CR3: 0000000150130000 CR4: 00000000000006f0
Feb 24 19:13:58 testnode2 kernel: Call Trace:
Feb 24 19:13:58 testnode2 kernel: <TASK>
Feb 24 19:13:58 testnode2 kernel: ? lock_acquire+0x2ed/0x340
Feb 24 19:13:58 testnode2 kernel: ? __pfx_do_raw_spin_lock+0x10/0x10
Feb 24 19:13:58 testnode2 kernel: ? _raw_spin_unlock+0x29/0x50
Feb 24 19:13:58 testnode2 kernel: ? lock_release+0x27d/0x320
Feb 24 19:13:58 testnode2 kernel: fcloop_tgt_fcprqst_done_work+0x101/0x200 [nvme_fcloop]
Feb 24 19:13:58 testnode2 kernel: process_one_work+0x868/0x1590
Feb 24 19:13:58 testnode2 kernel: ? __pfx_process_one_work+0x10/0x10
Feb 24 19:13:58 testnode2 kernel: ? __pfx_do_raw_spin_lock+0x10/0x10
Feb 24 19:13:58 testnode2 kernel: worker_thread+0x5ee/0xfd0
Feb 24 19:13:58 testnode2 kernel: ? __pfx_worker_thread+0x10/0x10
Feb 24 19:13:58 testnode2 kernel: kthread+0x368/0x460
Feb 24 19:13:58 testnode2 kernel: ? _raw_spin_unlock_irq+0x24/0x50
Feb 24 19:13:58 testnode2 kernel: ? __pfx_kthread+0x10/0x10
Feb 24 19:13:58 testnode2 kernel: ret_from_fork+0x66a/0x9e0
Feb 24 19:13:58 testnode2 kernel: ? __pfx_ret_from_fork+0x10/0x10
Feb 24 19:13:58 testnode2 kernel: ? __switch_to+0x108/0xe10
Feb 24 19:13:58 testnode2 kernel: ? __switch_to_asm+0x33/0x70
Feb 24 19:13:58 testnode2 kernel: ? __pfx_kthread+0x10/0x10
Feb 24 19:13:58 testnode2 kernel: ret_from_fork_asm+0x1a/0x30
Feb 24 19:13:58 testnode2 kernel: </TASK>
Feb 24 19:13:58 testnode2 kernel: Modules linked in: nvme_fcloop nvmet_fc nvmet nvme_fc nvme_fabrics chacha chacha20poly1305 tls iw_cm ib_cm ib_core nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr 9pnet_virtio 9pnet pcspkr netfs i2c_piix4 i2c_smbus fuse loop vsock_loopback vmw_vsock_virtio_transport_common zram vsock xfs bochs drm_client_lib drm_shmem_helper drm_kms_helper nvme nvme_core nvme_keyring drm floppy sym53c8xx nvme_auth e1000 hkdf scsi_transport_spi serio_raw ata_generic pata_acpi btrfs xor zstd_compress raid6_pq sunrpc dm_multipath i2c_dev nfnetlink qemu_fw_cfg [last unloaded: nvmet]
Feb 24 19:13:58 testnode2 kernel: ---[ end trace 0000000000000000 ]---
Feb 24 19:13:58 testnode2 kernel: RIP: 0010:do_raw_spin_lock+0x6b/0x260
Feb 24 19:13:58 testnode2 kernel: Code: fe 6d 9b 41 c7 04 04 f1 f1 f1 f1 42 c7 44 20 04 04 f3 f3 f3 65 48 8b 15 1b 61 69 05 48 89 54 24 60 31 d2 48 89 fa 48 c1 ea 03 <0f> b6 14 02 48 89 f8 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 af
Feb 24 19:13:58 testnode2 kernel: RSP: 0018:ffff8881318f7b88 EFLAGS: 00010203
Feb 24 19:13:58 testnode2 kernel: RAX: dffffc0000000000 RBX: 0000000000000010 RCX: 0000000000000000
Feb 24 19:13:58 testnode2 kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000014
Feb 24 19:13:58 testnode2 kernel: RBP: ffff8881389885f8 R08: 0000000000000001 R09: 0000000000000000
Feb 24 19:13:58 testnode2 kernel: R10: 0000000000000002 R11: fefefefefefefeff R12: 1ffff1102631ef72
Feb 24 19:13:58 testnode2 kernel: R13: 0000000000000000 R14: 0000000000000006 R15: ffff8881328ec700
Feb 24 19:13:58 testnode2 kernel: FS: 0000000000000000(0000) GS:ffff88840d1b2000(0000) knlGS:0000000000000000
Feb 24 19:13:58 testnode2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 24 19:13:58 testnode2 kernel: CR2: 00007f2c9ab5a7d4 CR3: 0000000150130000 CR4: 00000000000006f0
Feb 24 19:13:58 testnode2 kernel: note: kworker/u16:13[97800] exited with preempt_count 1
Feb 24 19:13:58 testnode2 kernel: (NULL device *): Disconnect LS failed: No Association
Feb 24 19:13:58 testnode2 kernel: nvmet_fc: {1:0}: Association freed
Feb 24 19:13:58 testnode2 kernel: nvmet_fc: {2:1}: Association deleted
Feb 24 19:13:58 testnode2 kernel: nvme nvme8: Removing ctrl: NQN "blktests-subsystem-1"
Feb 24 19:13:59 testnode2 kernel: nvmet_fc: {2:1}: Association freed
Feb 24 19:13:59 testnode2 kernel: (NULL device *): Disconnect LS failed: No Association
Feb 24 19:13:59 testnode2 kernel: nvmet_fc: {3:0}: Association deleted
Feb 24 19:13:59 testnode2 kernel: nvme nvme9: Removing ctrl: NQN "blktests-subsystem-1"
Feb 24 19:13:59 testnode2 kernel: nvmet_fc: {3:0}: Association freed
Feb 24 19:13:59 testnode2 kernel: (NULL device *): Disconnect LS failed: No Association
Feb 24 19:14:00 testnode2 kernel: nvmet_fc: {4:1}: Association deleted
Feb 24 19:14:00 testnode2 kernel: nvmet_fc: {4:1}: Association freed
Feb 24 19:14:00 testnode2 kernel: (NULL device *): Disconnect LS failed: No Association
Feb 24 19:14:00 testnode2 kernel: nvme nvme5: NVME-FC{0}: create association : host wwpn 0x20001100aa000001 rport wwpn 0x20001100ab000001: NQN "nqn.2014-08.org.nvmexpress.discovery"
Feb 24 19:14:00 testnode2 kernel: (NULL device *): queue 0 connect admin queue failed (-111).
Feb 24 19:14:00 testnode2 kernel: nvme nvme5: NVME-FC{0}: reset: Reconnect attempt failed (-111)
Feb 24 19:14:00 testnode2 kernel: nvme nvme5: NVME-FC{0}: Reconnect attempt in 2 seconds
Feb 24 19:14:00 testnode2 kernel: nvme nvme5: NVME-FC{0}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", hostnqn: nqn.2014-08.org.nvmexpress:uuid:35d40691-eac6-4da7-a3b0-bada395faa22
Feb 24 19:14:02 testnode2 kernel: nvme nvme5: NVME-FC{0}: create association : host wwpn 0x20001100aa000001 rport wwpn 0x20001100ab000001: NQN "nqn.2014-08.org.nvmexpress.discovery"
Feb 24 19:14:02 testnode2 kernel: (NULL device *): queue 0 connect admin queue failed (-111).
Feb 24 19:14:02 testnode2 kernel: nvme nvme5: NVME-FC{0}: reset: Reconnect attempt failed (-111)
Feb 24 19:14:02 testnode2 kernel: nvme nvme5: NVME-FC{0}: Reconnect attempt in 2 seconds
...
[6] Oops and KASAN null-ptr-deref during nvme/061 with fc transport
...
[ 4751.498800][T89843] nvme nvme5: NVME-FC{0}: io failed due to lldd error -107
[ 4751.498810][T89843] nvme nvme5: NVME-FC{0}: io failed due to lldd error -107
[ 4751.498816][T56591] I/O error, dev nvme5c5n1, sector 734384 op 0x0:(READ) flags 0x2000800 phys_seg 1 prio class 2
[ 4751.498945][T85516] I/O error, dev nvme5c5n1, sector 1372600 op 0x0:(READ) flags 0x2000800 phys_seg 1 prio class 2
[ 4751.506995][T100934] Oops: general protection fault, probably for non-canonical address 0xdffffc0000000003: 0000 [#1] SMP KASAN PTI
[ 4751.547065][T100934] KASAN: null-ptr-deref in range [0x0000000000000018-0x000000000000001f]
[ 4751.547660][T100934] CPU: 0 UID: 0 PID: 100934 Comm: fio Not tainted 7.0.0-rc1 #13 PREEMPT(full)
[ 4751.548332][T100934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
[ 4751.549044][T100934] RIP: 0010:fcloop_fcp_req+0x17a/0x620 [nvme_fcloop]
[ 4751.549588][T100934] Code: fc ff df 48 c1 ea 03 80 3c 02 00 0f 85 4d 04 00 00 48 b8 00 00 00 00 00 fc ff df 49 8b 6d 08 48 8d 7d 18 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 20 04 00 00 48 89 da 48 8b 6d 18 48 b8 00 00 00
[ 4751.551099][T100934] RSP: 0018:ffff88810d7af038 EFLAGS: 00010206
[ 4751.551655][T100934] RAX: dffffc0000000000 RBX: ffff888199806c00 RCX: 0000000000000000
[ 4751.552362][T100934] RDX: 0000000000000003 RSI: 0000000000000000 RDI: 0000000000000018
[ 4751.553061][T100934] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 4751.553781][T100934] R10: 0000000000000000 R11: 0000000000000003 R12: ffff88823a459438
[ 4751.554463][T100934] R13: ffff88814e785930 R14: ffff88823a4594b8 R15: ffff888133dd4000
[ 4751.555167][T100934] FS: 00007fbf896a5540(0000) GS:ffff88840e2b5000(0000) knlGS:0000000000000000
[ 4751.555916][T100934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4751.556586][T100934] CR2: 00007ffe9c796920 CR3: 000000012e3e0000 CR4: 00000000000006f0
[ 4751.557319][T100934] Call Trace:
[ 4751.557789][T100934] <TASK>
[ 4751.558251][T100934] nvme_fc_start_fcp_op+0xb1e/0x1900 [nvme_fc]
[ 4751.558897][T100934] ? __pfx_nvme_fc_start_fcp_op+0x10/0x10 [nvme_fc]
[ 4751.559580][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.560250][T100934] ? nvme_fc_queue_rq+0x1ae/0x520 [nvme_fc]
[ 4751.560920][T100934] __blk_mq_issue_directly+0xd4/0x1d0
[ 4751.561546][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.562190][T100934] ? __pfx___blk_mq_issue_directly+0x10/0x10
[ 4751.562880][T100934] ? blk_mq_request_issue_directly+0xc3/0x190
[ 4751.563582][T100934] blk_mq_issue_direct+0x16d/0xa00
[ 4751.564260][T100934] ? lock_acquire+0x2ed/0x340
[ 4751.564874][T100934] blk_mq_dispatch_queue_requests+0x44b/0x7c0
[ 4751.565571][T100934] ? trace_hardirqs_on+0x15/0x1a0
[ 4751.566237][T100934] blk_mq_flush_plug_list+0x183/0x680
[ 4751.566902][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.567573][T100934] ? blk_add_rq_to_plug+0x382/0x7a0
[ 4751.568287][T100934] ? blk_account_io_start+0x42a/0x8a0
[ 4751.569001][T100934] ? __pfx_blk_mq_flush_plug_list+0x10/0x10
[ 4751.569748][T100934] ? blk_mq_submit_bio+0x16ac/0x26d0
[ 4751.570452][T100934] __blk_flush_plug+0x26a/0x4e0
[ 4751.571132][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.571813][T100934] ? __pfx___blk_flush_plug+0x10/0x10
[ 4751.572529][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.573255][T100934] ? lock_release+0x27d/0x320
[ 4751.573945][T100934] __submit_bio+0x492/0x5f0
[ 4751.574695][T100934] ? __pfx___submit_bio+0x10/0x10
[ 4751.575463][T100934] ? __pfx_blk_cgroup_bio_start+0x10/0x10
[ 4751.576246][T100934] ? submit_bio_noacct_nocheck+0x5b7/0xb40
[ 4751.577043][T100934] submit_bio_noacct_nocheck+0x5b7/0xb40
[ 4751.577883][T100934] ? __pfx_submit_bio_noacct_nocheck+0x10/0x10
[ 4751.578684][T100934] ? submit_bio_noacct+0x86f/0x1ee0
[ 4751.579470][T100934] submit_bio_wait+0x106/0x200
[ 4751.580206][T100934] ? __pfx_submit_bio_wait+0x10/0x10
[ 4751.580943][T100934] ? bio_init+0x3f2/0x6a0
[ 4751.581615][T100934] __blkdev_direct_IO_simple+0x3f5/0x860
[ 4751.582370][T100934] ? rcu_nocb_bypass_lock+0xc1/0xf0
[ 4751.583101][T100934] ? __pfx___blkdev_direct_IO_simple+0x10/0x10
[ 4751.583858][T100934] ? __pfx_submit_bio_wait_endio+0x10/0x10
[ 4751.584584][T100934] ? inode_to_bdi+0xa2/0x150
[ 4751.585319][T100934] ? filemap_writeback+0xe7/0x280
[ 4751.586079][T100934] ? __pfx___filemap_fdatawait_range+0x10/0x10
[ 4751.586923][T100934] blkdev_direct_IO+0xa38/0x1f50
[ 4751.587605][T100934] ? __pfx_blkdev_direct_IO+0x10/0x10
[ 4751.588325][T100934] ? __pfx_blkdev_direct_IO+0x10/0x10
[ 4751.589019][T100934] ? filemap_check_errors+0x56/0xf0
[ 4751.589694][T100934] blkdev_read_iter+0x204/0x410
[ 4751.590391][T100934] vfs_read+0x6b4/0xb10
[ 4751.591008][T100934] ? __pfx_vfs_read+0x10/0x10
[ 4751.591638][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.592316][T100934] ? vfs_read+0x3b4/0xb10
[ 4751.592941][T100934] __x64_sys_pread64+0x18d/0x250
[ 4751.593579][T100934] ? __pfx___x64_sys_pread64+0x10/0x10
[ 4751.594280][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.594909][T100934] ? do_syscall_64+0x57/0x1540
[ 4751.595587][T100934] do_syscall_64+0x136/0x1540
[ 4751.596241][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.596858][T100934] ? do_syscall_64+0x25a/0x1540
[ 4751.597534][T100934] ? trace_hardirqs_on_prepare+0x14f/0x1a0
[ 4751.598245][T100934] ? do_syscall_64+0x278/0x1540
[ 4751.598877][T100934] ? __pfx___x64_sys_pread64+0x10/0x10
[ 4751.599598][T100934] ? vfs_read+0x3b4/0xb10
[ 4751.600224][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.600834][T100934] ? do_syscall_64+0x25a/0x1540
[ 4751.601575][T100934] ? trace_hardirqs_on_prepare+0x14f/0x1a0
[ 4751.602295][T100934] ? do_syscall_64+0x278/0x1540
[ 4751.602933][T100934] ? __pfx___x64_sys_pread64+0x10/0x10
[ 4751.603688][T100934] ? trace_hardirqs_on_prepare+0x14f/0x1a0
[ 4751.604392][T100934] ? __x64_sys_pread64+0x18d/0x250
[ 4751.605034][T100934] ? __pfx___x64_sys_pread64+0x10/0x10
[ 4751.605728][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.606376][T100934] ? do_syscall_64+0x25a/0x1540
[ 4751.607012][T100934] ? trace_hardirqs_on_prepare+0x14f/0x1a0
[ 4751.607722][T100934] ? do_syscall_64+0x278/0x1540
[ 4751.608382][T100934] ? irqentry_exit+0xe2/0x6a0
[ 4751.609004][T100934] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 4751.609695][T100934] RIP: 0033:0x7fbf89720462
[ 4751.610415][T100934] Code: 08 0f 85 61 42 ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 66
[ 4751.612341][T100934] RSP: 002b:00007ffddb547148 EFLAGS: 00000246 ORIG_RAX: 0000000000000011
[ 4751.613225][T100934] RAX: ffffffffffffffda RBX: 000000001df0f800 RCX: 00007fbf89720462
[ 4751.614072][T100934] RDX: 0000000000001000 RSI: 000000001df10000 RDI: 0000000000000007
[ 4751.614937][T100934] RBP: 00007ffddb547170 R08: 0000000000000000 R09: 0000000000000000
[ 4751.615842][T100934] R10: 000000003d8d9000 R11: 0000000000000246 R12: 0000000000000000
[ 4751.616720][T100934] R13: 0000000000001000 R14: 0000000000000000 R15: 00007fbf812cfc10
[ 4751.617617][T100934] </TASK>
[ 4751.618227][T100934] Modules linked in: nvme_fcloop nvmet_fc nvmet nvme_fc nvme_fabrics chacha chacha20poly1305 tls iw_cm ib_cm ib_core nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr sunrpc 9pnet_virtio 9pnet netfs i2c_piix4 pcspkr i2c_smbus fuse dm_multipath loop nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock zram xfs bochs drm_client_lib nvme drm_shmem_helper drm_kms_helper nvme_core drm nvme_keyring e1000 sym53c8xx nvme_auth scsi_transport_spi hkdf floppy serio_raw ata_generic pata_acpi i2c_dev qemu_fw_cfg [last unloaded: nvmet]
[ 4751.623881][T85516] I/O error, dev nvme5c5n1, sector 1491056 op 0x0:(READ) flags 0x2000800 phys_seg 1 prio class 2
[ 4751.623885][T100934] ---[ end trace 0000000000000000 ]---
[ 4751.625730][T89797] nvme nvme5: NVME-FC{0}: resetting controller
[ 4751.626880][T100934] RIP: 0010:fcloop_fcp_req+0x17a/0x620 [nvme_fcloop]
[ 4751.627781][T100934] Code: fc ff df 48 c1 ea 03 80 3c 02 00 0f 85 4d 04 00 00 48 b8 00 00 00 00 00 fc ff df 49 8b 6d 08 48 8d 7d 18 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 20 04 00 00 48 89 da 48 8b 6d 18 48 b8 00 00 00
[ 4751.630389][T100934] RSP: 0018:ffff88810d7af038 EFLAGS: 00010206
[ 4751.631257][T100934] RAX: dffffc0000000000 RBX: ffff888199806c00 RCX: 0000000000000000
[ 4751.632210][T100934] RDX: 0000000000000003 RSI: 0000000000000000 RDI: 0000000000000018
[ 4751.633140][T100934] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 4751.634008][T100934] R10: 0000000000000000 R11: 0000000000000003 R12: ffff88823a459438
[ 4751.634897][T100934] R13: ffff88814e785930 R14: ffff88823a4594b8 R15: ffff888133dd4000
[ 4751.635979][T100934] FS: 00007fbf896a5540(0000) GS:ffff88840e435000(0000) knlGS:0000000000000000
[ 4751.636893][T100934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4751.637664][T100934] CR2: 000000000060d7a4 CR3: 000000012e3e0000 CR4: 00000000000006f0
[ 4751.638585][T100934] ------------[ cut here ]------------
[ 4751.639475][T100934] WARNING: kernel/exit.c:902 at do_exit+0x1670/0x2530, CPU#3: fio/100934
[ 4751.640442][T100934] Modules linked in: nvme_fcloop nvmet_fc nvmet nvme_fc nvme_fabrics chacha chacha20poly1305 tls iw_cm ib_cm ib_core nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr sunrpc 9pnet_virtio 9pnet netfs i2c_piix4 pcspkr i2c_smbus fuse dm_multipath loop nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock zram xfs bochs drm_client_lib nvme drm_shmem_helper drm_kms_helper nvme_core drm nvme_keyring e1000 sym53c8xx nvme_auth scsi_transport_spi hkdf floppy serio_raw ata_generic pata_acpi i2c_dev qemu_fw_cfg [last unloaded: nvmet]
[ 4751.647118][T100934] CPU: 3 UID: 0 PID: 100934 Comm: fio Tainted: G D 7.0.0-rc1 #13 PREEMPT(full)
[ 4751.648290][T100934] Tainted: [D]=DIE
[ 4751.649023][T100934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
[ 4751.650041][T100934] RIP: 0010:do_exit+0x1670/0x2530
[ 4751.650821][T100934] Code: af 0d 00 00 8b 83 80 1e 00 00 65 01 05 4d e6 86 05 e9 db fd ff ff 48 89 df e8 bc 3e 3b 00 e9 65 ee ff ff 0f 0b e9 0d ea ff ff <0f> 0b e9 36 ea ff ff 4c 8d 64 24 58 31 c0 b9 04 00 00 00 4c 89 e7
[ 4751.653342][T100934] RSP: 0018:ffff88810d7afe58 EFLAGS: 00010282
[ 4751.654331][T100934] RAX: dffffc0000000000 RBX: ffff888100de3380 RCX: 0000000000000001
[ 4751.655418][T100934] RDX: 1ffff110201bc946 RSI: 0000000000000004 RDI: ffff888100de4a30
[ 4751.656540][T100934] RBP: 000000000000000b R08: ffffffff9a51bb75 R09: fffffbfff3fc1dcc
[ 4751.657505][T100934] R10: 0000000000000003 R11: 00000000000c5e50 R12: 0000000000002710
[ 4751.658469][T100934] R13: 000000000000000b R14: 0000000000000000 R15: 0000000000000000
[ 4751.659605][T100934] FS: 00007fbf896a5540(0000) GS:ffff88840e435000(0000) knlGS:0000000000000000
[ 4751.660675][T100934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4751.661553][T100934] CR2: 000000000060d7a4 CR3: 000000012e3e0000 CR4: 00000000000006f0
[ 4751.662518][T100934] Call Trace:
[ 4751.663300][T100934] <TASK>
[ 4751.663961][T100934] ? __x64_sys_pread64+0x18d/0x250
[ 4751.664726][T100934] ? __pfx_do_exit+0x10/0x10
[ 4751.665496][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.666424][T100934] ? do_syscall_64+0x25a/0x1540
[ 4751.667272][T100934] ? trace_hardirqs_on_prepare+0x14f/0x1a0
[ 4751.668119][T100934] make_task_dead+0xf3/0x110
[ 4751.668872][T100934] rewind_stack_and_make_dead+0x16/0x20
[ 4751.669655][T100934] RIP: 0033:0x7fbf89720462
[ 4751.670540][T100934] Code: 08 0f 85 61 42 ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 66
[ 4751.672738][T100934] RSP: 002b:00007ffddb547148 EFLAGS: 00000246 ORIG_RAX: 0000000000000011
[ 4751.673735][T100934] RAX: ffffffffffffffda RBX: 000000001df0f800 RCX: 00007fbf89720462
[ 4751.674642][T100934] RDX: 0000000000001000 RSI: 000000001df10000 RDI: 0000000000000007
[ 4751.675540][T100934] RBP: 00007ffddb547170 R08: 0000000000000000 R09: 0000000000000000
[ 4751.676478][T100934] R10: 000000003d8d9000 R11: 0000000000000246 R12: 0000000000000000
[ 4751.677553][T100934] R13: 0000000000001000 R14: 0000000000000000 R15: 00007fbf812cfc10
[ 4751.678445][T100934] </TASK>
[ 4751.679084][T100934] irq event stamp: 0
[ 4751.679739][T100934] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 4751.680574][T100934] hardirqs last disabled at (0): [<ffffffff9a4ff238>] copy_process+0x1c78/0x6be0
[ 4751.681526][T100934] softirqs last enabled at (0): [<ffffffff9a4ff28c>] copy_process+0x1ccc/0x6be0
[ 4751.682468][T100934] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 4751.683456][T100934] ---[ end trace 0000000000000000 ]---
[ 4751.684413][T100934] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:591
[ 4751.685457][T100934] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 100934, name: fio
[ 4751.686382][T100934] preempt_count: 0, expected: 0
[ 4751.687133][T100934] RCU nest depth: 1, expected: 0
[ 4751.687847][T100934] INFO: lockdep is turned off.
[ 4751.688546][T100934] CPU: 1 UID: 0 PID: 100934 Comm: fio Tainted: G D W 7.0.0-rc1 #13 PREEMPT(full)
[ 4751.688549][T100934] Tainted: [D]=DIE, [W]=WARN
[ 4751.688550][T100934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
[ 4751.688551][T100934] Call Trace:
[ 4751.688553][T100934] <TASK>
[ 4751.688554][T100934] dump_stack_lvl+0x6a/0x90
[ 4751.688560][T100934] __might_resched.cold+0x213/0x29d
[ 4751.688564][T100934] ? __pfx___might_resched+0x10/0x10
[ 4751.688568][T100934] __mutex_lock+0x118/0x25f0
[ 4751.688572][T100934] ? do_exit+0x1670/0x2530
[ 4751.688574][T100934] ? sched_mm_cid_exit+0x93/0x4e0
[ 4751.688575][T100934] ? preempt_schedule_notrace_thunk+0x16/0x30
[ 4751.688578][T100934] ? preempt_schedule_notrace_thunk+0x16/0x30
[ 4751.688580][T100934] ? __pfx___mutex_lock+0x10/0x10
[ 4751.688581][T100934] ? report_bug+0x13f/0x190
[ 4751.688584][T100934] ? exc_invalid_op+0x14/0x50
[ 4751.688586][T100934] ? asm_exc_invalid_op+0x16/0x20
[ 4751.688587][T100934] ? do_raw_spin_lock+0x124/0x260
[ 4751.688590][T100934] ? sched_mm_cid_exit+0x93/0x4e0
[ 4751.688591][T100934] sched_mm_cid_exit+0x93/0x4e0
[ 4751.688593][T100934] ? _raw_spin_unlock_irq+0x24/0x50
[ 4751.688595][T100934] ? trace_hardirqs_on+0x15/0x1a0
[ 4751.688598][T100934] do_exit+0x25e/0x2530
[ 4751.688599][T100934] ? __x64_sys_pread64+0x18d/0x250
[ 4751.688602][T100934] ? __pfx_do_exit+0x10/0x10
[ 4751.688603][T100934] ? rcu_is_watching+0x11/0xb0
[ 4751.688606][T100934] ? do_syscall_64+0x25a/0x1540
[ 4751.688608][T100934] ? trace_hardirqs_on_prepare+0x14f/0x1a0
[ 4751.688609][T100934] make_task_dead+0xf3/0x110
[ 4751.688610][T100934] rewind_stack_and_make_dead+0x16/0x20
[ 4751.688612][T100934] RIP: 0033:0x7fbf89720462
[ 4751.688614][T100934] Code: 08 0f 85 61 42 ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 66
[ 4751.688616][T100934] RSP: 002b:00007ffddb547148 EFLAGS: 00000246 ORIG_RAX: 0000000000000011
[ 4751.688618][T100934] RAX: ffffffffffffffda RBX: 000000001df0f800 RCX: 00007fbf89720462
[ 4751.688619][T100934] RDX: 0000000000001000 RSI: 000000001df10000 RDI: 0000000000000007
[ 4751.688619][T100934] RBP: 00007ffddb547170 R08: 0000000000000000 R09: 0000000000000000
[ 4751.688620][T100934] R10: 000000003d8d9000 R11: 0000000000000246 R12: 0000000000000000
[ 4751.688621][T100934] R13: 0000000000001000 R14: 0000000000000000 R15: 00007fbf812cfc10
[ 4751.688624][T100934] </TASK>
[ 4751.725012][T89797] ==================================================================
[ 4751.725043][T85511] nvmet_fc: {0:0}: Association deleted
[ 4751.725459][T89797] BUG: KASAN: wild-memory-access in percpu_ref_put_many.constprop.0+0x139/0x1d0
[ 4751.727636][T89797] Write of size 8 at addr da8948110cb60f03 by task kworker/u16:33/89797
[ 4751.728638][T89797]
[ 4751.729219][T89797] CPU: 0 UID: 0 PID: 89797 Comm: kworker/u16:33 Tainted: G D W 7.0.0-rc1 #13 PREEMPT(full)
[ 4751.729222][T89797] Tainted: [D]=DIE, [W]=WARN
[ 4751.729223][T89797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
[ 4751.729226][T89797] Workqueue: nvme-reset-wq nvme_fc_reset_ctrl_work [nvme_fc]
[ 4751.729235][T89797] Call Trace:
[ 4751.729237][T89797] <TASK>
[ 4751.729238][T89797] dump_stack_lvl+0x6a/0x90
[ 4751.729243][T89797] ? rcu_is_watching+0x49/0xb0
[ 4751.729246][T89797] ? percpu_ref_put_many.constprop.0+0x139/0x1d0
[ 4751.729248][T89797] kasan_report+0xb0/0x190
[ 4751.729253][T89797] ? percpu_ref_put_many.constprop.0+0x139/0x1d0
[ 4751.729256][T89797] ? rcu_is_watching+0x49/0xb0
[ 4751.729257][T89797] kasan_check_range+0x115/0x1f0
[ 4751.729259][T89797] ? rcu_is_watching+0x49/0xb0
[ 4751.729261][T89797] percpu_ref_put_many.constprop.0+0x139/0x1d0
[ 4751.729263][T89797] bio_associate_blkg_from_css+0x4a/0xf20
[ 4751.729266][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.729267][T89797] ? lock_acquire+0x2ed/0x340
[ 4751.729271][T89797] bio_associate_blkg+0xd8/0x1f0
[ 4751.729274][T89797] nvme_failover_req+0x297/0x530 [nvme_core]
[ 4751.729292][T89797] nvme_fc_complete_rq+0x240/0x350 [nvme_fc]
[ 4751.729297][T89797] nvme_fc_fcpio_done+0x42b/0xda0 [nvme_fc]
[ 4751.729304][T89797] ? __pfx_nvme_fc_fcpio_done+0x10/0x10 [nvme_fc]
[ 4751.729307][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.729309][T89797] ? lock_release+0x27d/0x320
[ 4751.729312][T89797] fcloop_fcp_abort+0x2e5/0x4b0 [nvme_fcloop]
[ 4751.729317][T89797] ? __pfx_fcloop_fcp_abort+0x10/0x10 [nvme_fcloop]
[ 4751.729319][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.729320][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.729322][T89797] ? lock_release+0x27d/0x320
[ 4751.729324][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.729325][T89797] ? _raw_spin_unlock_irqrestore+0x4c/0x60
[ 4751.729329][T89797] ? __pfx_fcloop_fcp_abort+0x10/0x10 [nvme_fcloop]
[ 4751.729332][T89797] __nvme_fc_abort_op+0x17f/0x300 [nvme_fc]
[ 4751.729336][T89797] ? lock_acquire+0x2ed/0x340
[ 4751.729338][T89797] nvme_fc_terminate_exchange+0x56/0x90 [nvme_fc]
[ 4751.729341][T89797] ? blk_mq_find_and_get_req.isra.0+0x87/0x130
[ 4751.729346][T89797] ? __pfx_nvme_fc_terminate_exchange+0x10/0x10 [nvme_fc]
[ 4751.729349][T89797] bt_tags_iter+0x15d/0x350
[ 4751.729352][T89797] blk_mq_tagset_busy_iter+0x4ea/0xc10
[ 4751.729354][T89797] ? __pfx_nvme_fc_terminate_exchange+0x10/0x10 [nvme_fc]
[ 4751.729359][T89797] ? __pfx_blk_mq_tagset_busy_iter+0x10/0x10
[ 4751.729361][T89797] ? __pfx___might_resched+0x10/0x10
[ 4751.729364][T89797] ? __pfx_nvme_fc_terminate_exchange+0x10/0x10 [nvme_fc]
[ 4751.729368][T89797] __nvme_fc_abort_outstanding_ios+0x23f/0x320 [nvme_fc]
[ 4751.729372][T89797] nvme_fc_delete_association+0x143/0x700 [nvme_fc]
[ 4751.729376][T89797] ? __pfx_nvme_fc_delete_association+0x10/0x10 [nvme_fc]
[ 4751.729380][T89797] ? __pfx_enable_work+0x10/0x10
[ 4751.729383][T89797] ? __pfx___might_resched+0x10/0x10
[ 4751.729385][T89797] ? __pfx___might_resched+0x10/0x10
[ 4751.729386][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.729389][T89797] nvme_fc_reset_ctrl_work+0x27/0x110 [nvme_fc]
[ 4751.729392][T89797] process_one_work+0x868/0x1590
[ 4751.729395][T89797] ? trace_hardirqs_on+0x15/0x1a0
[ 4751.729398][T89797] ? __pfx_process_one_work+0x10/0x10
[ 4751.729400][T89797] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 4751.729403][T89797] worker_thread+0x5ee/0xfd0
[ 4751.729406][T89797] ? __pfx_worker_thread+0x10/0x10
[ 4751.729407][T89797] kthread+0x368/0x460
[ 4751.729410][T89797] ? _raw_spin_unlock_irq+0x24/0x50
[ 4751.729412][T89797] ? __pfx_kthread+0x10/0x10
[ 4751.729414][T89797] ret_from_fork+0x66a/0x9e0
[ 4751.729418][T89797] ? __pfx_ret_from_fork+0x10/0x10
[ 4751.729420][T89797] ? __switch_to+0x108/0xe10
[ 4751.729423][T89797] ? __switch_to_asm+0x33/0x70
[ 4751.729425][T89797] ? __pfx_kthread+0x10/0x10
[ 4751.729427][T89797] ret_from_fork_asm+0x1a/0x30
[ 4751.729431][T89797] </TASK>
[ 4751.729432][T89797] ==================================================================
[ 4751.779844][T89797] Oops: general protection fault, probably for non-canonical address 0xda8948110cb60f03: 0000 [#2] SMP KASAN PTI
[ 4751.780936][T89797] CPU: 0 UID: 0 PID: 89797 Comm: kworker/u16:33 Tainted: G B D W 7.0.0-rc1 #13 PREEMPT(full)
[ 4751.782027][T89797] Tainted: [B]=BAD_PAGE, [D]=DIE, [W]=WARN
[ 4751.782748][T89797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025
[ 4751.783688][T89797] Workqueue: nvme-reset-wq nvme_fc_reset_ctrl_work [nvme_fc]
[ 4751.784507][T89797] RIP: 0010:percpu_ref_put_many.constprop.0+0x13d/0x1d0
[ 4751.785293][T89797] Code: fc ff df 48 8d 53 08 48 c1 ea 03 80 3c 02 00 75 79 48 8b 53 08 be 08 00 00 00 48 89 d7 48 89 14 24 e8 a7 08 73 ff 48 8b 14 24 <f0> 48 83 2a 01 0f 85 1e ff ff ff 48 b8 00 00 00 00 00 fc ff df 48
[ 4751.787159][T89797] RSP: 0018:ffff88811d28f650 EFLAGS: 00010046
[ 4751.787908][T89797] RAX: 0000000000000000 RBX: ffffffff9a743a89 RCX: 0000000000000000
[ 4751.788783][T89797] RDX: da8948110cb60f03 RSI: 0000000000000008 RDI: ffffffff9fe0e880
[ 4751.789658][T89797] RBP: ffffed10241ee009 R08: ffffffff9a506e16 R09: fffffbfff3fc1d10
[ 4751.790564][T89797] R10: fffffbfff3fc1d11 R11: 00000000000c9788 R12: 9df92120c51c0348
[ 4751.791437][T89797] R13: ffff88810d7af790 R14: ffff88810d7af790 R15: ffff88810d7af7e0
[ 4751.792420][T89797] FS: 0000000000000000(0000) GS:ffff88840e2b5000(0000) knlGS:0000000000000000
[ 4751.793356][T89797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4751.794152][T89797] CR2: 000000001df1f840 CR3: 000000018e09e000 CR4: 00000000000006f0
[ 4751.795026][T89797] Call Trace:
[ 4751.795661][T89797] <TASK>
[ 4751.796286][T89797] bio_associate_blkg_from_css+0x4a/0xf20
[ 4751.797054][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.797756][T89797] ? lock_acquire+0x2ed/0x340
[ 4751.798475][T89797] bio_associate_blkg+0xd8/0x1f0
[ 4751.799286][T89797] nvme_failover_req+0x297/0x530 [nvme_core]
[ 4751.800065][T89797] nvme_fc_complete_rq+0x240/0x350 [nvme_fc]
[ 4751.800824][T89797] nvme_fc_fcpio_done+0x42b/0xda0 [nvme_fc]
[ 4751.801588][T89797] ? __pfx_nvme_fc_fcpio_done+0x10/0x10 [nvme_fc]
[ 4751.802401][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.803158][T89797] ? lock_release+0x27d/0x320
[ 4751.803851][T89797] fcloop_fcp_abort+0x2e5/0x4b0 [nvme_fcloop]
[ 4751.804633][T89797] ? __pfx_fcloop_fcp_abort+0x10/0x10 [nvme_fcloop]
[ 4751.805578][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.806372][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.807071][T89797] ? lock_release+0x27d/0x320
[ 4751.807766][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.808475][T89797] ? _raw_spin_unlock_irqrestore+0x4c/0x60
[ 4751.809253][T89797] ? __pfx_fcloop_fcp_abort+0x10/0x10 [nvme_fcloop]
[ 4751.810052][T89797] __nvme_fc_abort_op+0x17f/0x300 [nvme_fc]
[ 4751.810831][T89797] ? lock_acquire+0x2ed/0x340
[ 4751.811580][T89797] nvme_fc_terminate_exchange+0x56/0x90 [nvme_fc]
[ 4751.812404][T89797] ? blk_mq_find_and_get_req.isra.0+0x87/0x130
[ 4751.813201][T89797] ? __pfx_nvme_fc_terminate_exchange+0x10/0x10 [nvme_fc]
[ 4751.814054][T89797] bt_tags_iter+0x15d/0x350
[ 4751.814778][T89797] blk_mq_tagset_busy_iter+0x4ea/0xc10
[ 4751.815546][T89797] ? __pfx_nvme_fc_terminate_exchange+0x10/0x10 [nvme_fc]
[ 4751.816683][T89797] ? __pfx_blk_mq_tagset_busy_iter+0x10/0x10
[ 4751.817526][T89797] ? __pfx___might_resched+0x10/0x10
[ 4751.818305][T89797] ? __pfx_nvme_fc_terminate_exchange+0x10/0x10 [nvme_fc]
[ 4751.819213][T89797] __nvme_fc_abort_outstanding_ios+0x23f/0x320 [nvme_fc]
[ 4751.820064][T89797] nvme_fc_delete_association+0x143/0x700 [nvme_fc]
[ 4751.820885][T89797] ? __pfx_nvme_fc_delete_association+0x10/0x10 [nvme_fc]
[ 4751.821722][T89797] ? __pfx_enable_work+0x10/0x10
[ 4751.822468][T89797] ? __pfx___might_resched+0x10/0x10
[ 4751.823200][T89797] ? __pfx___might_resched+0x10/0x10
[ 4751.823920][T89797] ? rcu_is_watching+0x11/0xb0
[ 4751.824615][T89797] nvme_fc_reset_ctrl_work+0x27/0x110 [nvme_fc]
[ 4751.825391][T89797] process_one_work+0x868/0x1590
[ 4751.826068][T89797] ? trace_hardirqs_on+0x15/0x1a0
[ 4751.826752][T89797] ? __pfx_process_one_work+0x10/0x10
[ 4751.827570][T89797] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 4751.828283][T89797] worker_thread+0x5ee/0xfd0
[ 4751.828928][T89797] ? __pfx_worker_thread+0x10/0x10
[ 4751.829599][T89797] kthread+0x368/0x460
[ 4751.830229][T89797] ? _raw_spin_unlock_irq+0x24/0x50
[ 4751.830891][T89797] ? __pfx_kthread+0x10/0x10
[ 4751.831522][T89797] ret_from_fork+0x66a/0x9e0
[ 4751.832169][T89797] ? __pfx_ret_from_fork+0x10/0x10
[ 4751.832826][T89797] ? __switch_to+0x108/0xe10
[ 4751.833452][T89797] ? __switch_to_asm+0x33/0x70
[ 4751.834083][T89797] ? __pfx_kthread+0x10/0x10
[ 4751.834702][T89797] ret_from_fork_asm+0x1a/0x30
[ 4751.835352][T89797] </TASK>
[ 4751.835887][T89797] Modules linked in: nvme_fcloop nvmet_fc nvmet nvme_fc nvme_fabrics chacha chacha20poly1305 tls iw_cm ib_cm ib_core nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr sunrpc 9pnet_virtio 9pnet netfs i2c_piix4 pcspkr i2c_smbus fuse dm_multipath loop nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock zram xfs bochs drm_client_lib nvme drm_shmem_helper drm_kms_helper nvme_core drm nvme_keyring e1000 sym53c8xx nvme_auth scsi_transport_spi hkdf floppy serio_raw ata_generic pata_acpi i2c_dev qemu_fw_cfg [last unloaded: nvmet]
[ 4751.841355][T89797] ---[ end trace 0000000000000000 ]---
[ 4751.842097][T89797] RIP: 0010:fcloop_fcp_req+0x17a/0x620 [nvme_fcloop]
[ 4751.842905][T89797] Code: fc ff df 48 c1 ea 03 80 3c 02 00 0f 85 4d 04 00 00 48 b8 00 00 00 00 00 fc ff df 49 8b 6d 08 48 8d 7d 18 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 20 04 00 00 48 89 da 48 8b 6d 18 48 b8 00 00 00
[ 4751.844886][T89797] RSP: 0018:ffff88810d7af038 EFLAGS: 00010206
[ 4751.845751][T89797] RAX: dffffc0000000000 RBX: ffff888199806c00 RCX: 0000000000000000
[ 4751.846676][T89797] RDX: 0000000000000003 RSI: 0000000000000000 RDI: 0000000000000018
[ 4751.847666][T89797] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 4751.848612][T89797] R10: 0000000000000000 R11: 0000000000000003 R12: ffff88823a459438
[ 4751.849598][T89797] R13: ffff88814e785930 R14: ffff88823a4594b8 R15: ffff888133dd4000
[ 4751.850528][T89797] FS: 0000000000000000(0000) GS:ffff88840e2b5000(0000) knlGS:0000000000000000
[ 4751.851517][T89797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4751.852386][T89797] CR2: 000000001df1f840 CR3: 000000018e09e000 CR4: 00000000000006f0
[ 4751.853336][T89797] note: kworker/u16:33[89797] exited with irqs disabled
[ 4751.854269][T89797] note: kworker/u16:33[89797] exited with preempt_count 1
[ 4751.877997][T68669] nvme nvme5: long keepalive RTT (4452952 ms)
[ 4751.878754][T68669] nvme nvme5: failed nvme_keep_alive_end_io error=4
[ 4751.886418][T68669] nvme nvme5: NVME-FC{0}: io failed due to lldd error 6
[ 4751.887272][T85118] nvme nvme6: NVME-FC{1}: create association : host wwpn 0x20001100aa000001 rport wwpn 0x20001100ab000001: NQN "nqn.2014-08.org.nvmexpress.discovery"
[ 4751.889247][T85118] (NULL device *): queue 0 connect admin queue failed (-111).
[ 4751.890471][T85118] nvme nvme6: NVME-FC{1}: reset: Reconnect attempt failed (-111)
[ 4751.891125][T85511] nvmet_fc: {0:0}: Association freed
[ 4751.891872][T85118] nvme nvme6: NVME-FC{1}: Reconnect attempt in 2 seconds
[ 4751.894011][T101048] nvme nvme6: NVME-FC{1}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", hostnqn: nqn.2014-08.org.nvmexpress:uuid:db1ce8ea-678a-4714-adaa-11e7f7e6517d
[ 4751.896800][T100818] nvme nvme5: NVME-FC{0}: controller connectivity lost. Awaiting Reconnect
[ 4751.897940][T100818] nvme nvme6: NVME-FC{1}: controller connectivity lost. Awaiting Reconnect
[ 4758.154005][T101131] nvme nvme5: Removing ctrl: NQN "blktests-subsystem-1"
[ 4811.925080][T85511] nvme nvme6: NVME-FC{1}: dev_loss_tmo (60) expired while waiting for remoteport connectivity.
[ 4811.926953][T85511] nvme nvme6: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
More information about the Linux-nvme
mailing list