[bug report] kmemleak osberved during blktests nvme/fc test
Yi Zhang
yi.zhang at redhat.com
Wed Jun 4 06:28:05 PDT 2025
On Mon, Jun 2, 2025 at 11:10 PM Daniel Wagner <dwagner at suse.de> wrote:
>
> Hi,
>
> On Sat, May 31, 2025 at 10:08:40AM +0800, Yi Zhang wrote:
> > Hello
> > I found the following kmemleak issue during blktests nvme/fc with the
> > latest linux-block/for-next, please help check it and let me know if you
> > need any test/info for it, thanks.
>
> I've tried to reproduce the kmemleaks by enabling
> CONFIG_DEBUG_KMEMLEAK and
>
> echo scan > /sys/kernel/debug/kmemleak
> echo check > /sys/kernel/debug/kmemleak
>
> but I don't get any reports. I've used 3bc1b5c03cbd ("Merge branch
> 'io_uring-6.16' into for-next")
>
> > unreferenced object 0xffff88816cd58400 (size 192):
> > comm "check", pid 24201, jiffies 4302692431
> > hex dump (first 32 bytes):
> > 00 00 00 00 00 00 00 00 01 00 00 aa 00 11 00 10 ................
> > 01 00 00 aa 00 11 00 20 b8 84 d5 6c 81 88 ff ff ....... ...l....
> > backtrace (crc 9556a114):
> > __kmalloc_noprof+0x442/0x5e0
> > trace_raw_output_nvmet_req_init+0x3b3/0x450 [nvmet]
>
> So that would be something in nvmet_req_init:
>
> trace_nvmet_req_init(req, req->cmd);
>
> though I can't spot it.
>
> > fcloop_fcp_copy_data+0x6b5/0x770 [nvme_fcloop]
> > kernfs_fop_write_iter+0x357/0x530
>
> this looks really strange. No idea how we can end up from
> kernfd_fop_write_iter to fcloop_fcp_copy_data.
>
> > vfs_write+0x9bc/0xf60
> > ksys_write+0xf3/0x1d0
> > do_syscall_64+0x8c/0x3d0
> > entry_SYSCALL_64_after_hwframe+0x76/0x7e
>
>
> > unreferenced object 0xffff888224353800 (size 1024):
> > comm "check", pid 24479, jiffies 4302692654
> > hex dump (first 32 bytes):
> > 00 00 00 00 60 00 00 00 01 00 00 ab 00 11 00 10 ....`...........
> > 01 00 00 ab 00 11 00 20 00 84 d5 6c 81 88 ff ff ....... ...l....
> > backtrace (crc 9aae07c2):
> > __kmalloc_noprof+0x442/0x5e0
> > nvmet_ns_device_uuid_store+0x4c/0xf0 [nvmet]
> > 0xffffffffc33aca5a
> > kernfs_fop_write_iter+0x357/0x530
> > vfs_write+0x9bc/0xf60
> > ksys_write+0xf3/0x1d0
> > do_syscall_64+0x8c/0x3d0
> > entry_SYSCALL_64_after_hwframe+0x76/0x7e
>
>
> I don't see any allocation happening in nvmet_ns_device_uuid_store, the
> code is just parsing the string into a fixed sized buffer.
>
> > unreferenced object 0xffff88811e90ed00 (size 64):
> > comm "kworker/u67:8", pid 24542, jiffies 4303041902
> > hex dump (first 32 bytes):
> > 00 38 35 24 82 88 ff ff 08 8d 53 40 81 88 ff ff .85$......S at ....
> > 00 a8 8f e1 82 88 ff ff 00 19 5f b3 82 88 ff ff .........._.....
> > backtrace (crc 45d8e045):
> > __kmalloc_cache_noprof+0x411/0x4e0
> > nvmet_ns_device_path_store+0x12/0x180 [nvmet]
> > fcloop_create_remote_port+0x5d3/0x640 [nvme_fcloop]
> > nvme_fc_handle_ls_rqst_work+0x77c/0xfe0 [nvme_fc]
> > nvme_fc_fcpio_done+0x715/0xfb0 [nvme_fc]
> > __nvme_fc_send_ls_req+0x750/0x1010 [nvme_fc]
> > process_one_work+0x8cd/0x1950
> > worker_thread+0x58d/0xcf0
> > kthread+0x3d8/0x7a0
> > ret_from_fork+0x406/0x510
> > ret_from_fork_asm+0x1a/0x30
>
> nvmet_ns_device_patch_store is not called from
> fcloop_create_remote_port.
>
> > unreferenced object 0xffff88846c4ac700 (size 128):
> > comm "kworker/u65:101", pid 4587, jiffies 4296544937
> > hex dump (first 32 bytes):
> > 01 00 00 00 00 00 00 20 00 00 00 01 00 00 00 08 ....... ........
> > 05 00 00 00 00 00 00 00 00 00 00 02 00 00 00 08 ................
> > backtrace (crc d98f69c0):
> > __kmalloc_cache_noprof+0x411/0x4e0
> > nvme_fc_rcv_ls_req+0x370/0x1080 [nvme_fc]
> > fcloop_t2h_ls_req+0x173/0x3f0 [nvme_fcloop]
> > __nvmet_fc_send_ls_req.constprop.0+0x5dc/0xee0 [nvmet_fc]
> > nvmet_fc_target_assoc_free+0x675/0xab0 [nvmet_fc]
> > nvmet_fc_delete_assoc_work+0x300/0x400 [nvmet_fc]
> > process_one_work+0x8cd/0x1950
> > worker_thread+0x58d/0xcf0
> > kthread+0x3d8/0x7a0
> > ret_from_fork+0x406/0x510
> > ret_from_fork_asm+0x1a/0x30
>
> This stack trace looks more reasonable and I thought I fixed this one
> already. I'll double check.
>
> > unreferenced object 0xffff888454806400 (size 192):
> > comm "check", pid 2795, jiffies 4295104867
> > hex dump (first 32 bytes):
> > 00 00 00 00 00 00 00 00 01 00 00 aa 00 11 00 10 ................
> > 01 00 00 aa 00 11 00 20 b8 64 80 54 84 88 ff ff ....... .d.T....
> > backtrace (crc 47ccaf54):
> > __kmalloc_noprof+0x442/0x5e0
> > nvme_fc_register_localport+0x453/0x1440 [nvme_fc]
> > fcloop_create_local_port+0x265/0x520 [nvme_fcloop]
> > kernfs_fop_write_iter+0x357/0x530
> > vfs_write+0x9bc/0xf60
> > ksys_write+0xf3/0x1d0
> > do_syscall_64+0x8c/0x3d0
> > entry_SYSCALL_64_after_hwframe+0x76/0x7e
>
> This looks like a localport is leaked and it should be fairly simple to
> reproduce it for me...
>
> > unreferenced object 0xffff8883f6377800 (size 1024):
> > comm "check", pid 3079, jiffies 4295105085
> > hex dump (first 32 bytes):
> > 00 00 00 00 60 00 00 00 01 00 00 ab 00 11 00 10 ....`...........
> > 01 00 00 ab 00 11 00 20 00 64 80 54 84 88 ff ff ....... .d.T....
> > backtrace (crc 8e00bf47):
> > __kmalloc_noprof+0x442/0x5e0
> > nvme_fc_register_remoteport+0x2ec/0x1360 [nvme_fc]
> > fcloop_create_remote_port+0x19a/0x640 [nvme_fcloop]
> > kernfs_fop_write_iter+0x357/0x530
> > vfs_write+0x9bc/0xf60
> > ksys_write+0xf3/0x1d0
> > do_syscall_64+0x8c/0x3d0
> > entry_SYSCALL_64_after_hwframe+0x76/0x7e
>
> Same here.
>
Hi Daniel
Sorry for the late response, I need to set up another environment to
reproduce it. And now I can only reproduce below kmemleak:
unreferenced object 0xffff888392808e00 (size 192):
comm "check", pid 11567, jiffies 4296030049
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 01 00 00 aa 00 11 00 10 ................
01 00 00 aa 00 11 00 20 b8 8e 80 92 83 88 ff ff ....... ........
backtrace (crc bc6eb8d):
__kmalloc_noprof+0x427/0x5e0
nvme_fc_register_localport+0x428/0x12a0 [nvme_fc]
fcloop_create_local_port+0x29a/0x550 [nvme_fcloop]
kernfs_fop_write_iter+0x3a3/0x5a0
vfs_write+0x525/0xea0
ksys_write+0xf9/0x1d0
do_syscall_64+0x94/0x8d0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
unreferenced object 0xffff88815c6d5000 (size 1024):
comm "check", pid 11629, jiffies 4296030320
hex dump (first 32 bytes):
00 00 00 00 60 00 00 00 01 00 00 ab 00 11 00 10 ....`...........
01 00 00 ab 00 11 00 20 00 8e 80 92 83 88 ff ff ....... ........
backtrace (crc 2f920dee):
__kmalloc_noprof+0x427/0x5e0
nvme_fc_register_remoteport+0x27f/0x1340 [nvme_fc]
fcloop_create_remote_port+0x1c6/0x670 [nvme_fcloop]
kernfs_fop_write_iter+0x3a3/0x5a0
vfs_write+0x525/0xea0
ksys_write+0xf9/0x1d0
do_syscall_64+0x94/0x8d0
entry_SYSCALL_64_after_hwframe+0x76/0x7eunreferenced object
0xffff888155f1f000 (size 64):
comm "kworker/u67:4", pid 11101, jiffies 4296040752
hex dump (first 32 bytes):
00 50 6d 5c 81 88 ff ff d8 6d 66 72 82 88 ff ff .Pm\.....mfr....
00 78 d9 34 83 88 ff ff 00 81 75 76 82 88 ff ff .x.4......uv....
backtrace (crc db67a5df):
__kmalloc_cache_noprof+0x3b1/0x4d0
nvme_fc_rcv_ls_req+0x1cd/0xcf0 [nvme_fc]
fcloop_t2h_ls_req+0x177/0x3f0 [nvme_fcloop]
__nvmet_fc_send_ls_req.constprop.0+0x5c6/0xee0 [nvmet_fc]
nvmet_fc_target_assoc_free+0x66f/0x900 [nvmet_fc]
nvmet_fc_delete_assoc_work+0xc8/0x200 [nvmet_fc]
process_one_work+0xd8b/0x1320
worker_thread+0x5f3/0xfe0
kthread+0x3b4/0x770
ret_from_fork+0x393/0x480
ret_from_fork_asm+0x1a/0x30
unreferenced object 0xffff888334d97800 (size 1024):
comm "kworker/u67:4", pid 11101, jiffies 4296040752
hex dump (first 32 bytes):
05 00 00 00 00 00 00 28 00 00 00 07 00 00 00 08 .......(........
9f be 1f 39 ac a1 00 00 00 00 00 05 00 00 00 10 ...9............
backtrace (crc 50742219):
__kmalloc_cache_noprof+0x3b1/0x4d0
nvme_fc_rcv_ls_req+0x20e/0xcf0 [nvme_fc]
fcloop_t2h_ls_req+0x177/0x3f0 [nvme_fcloop]
__nvmet_fc_send_ls_req.constprop.0+0x5c6/0xee0 [nvmet_fc]
nvmet_fc_target_assoc_free+0x66f/0x900 [nvmet_fc]
nvmet_fc_delete_assoc_work+0xc8/0x200 [nvmet_fc]
process_one_work+0xd8b/0x1320
worker_thread+0x5f3/0xfe0
kthread+0x3b4/0x770
ret_from_fork+0x393/0x480
ret_from_fork_asm+0x1a/0x30
unreferenced object 0xffff888276758100 (size 128):
comm "kworker/u67:4", pid 11101, jiffies 4296040752
hex dump (first 32 bytes):
01 00 00 00 00 00 00 20 00 00 00 01 00 00 00 08 ....... ........
05 00 00 00 00 00 00 00 00 00 00 02 00 00 00 08 ................
backtrace (crc d98f69c0):
__kmalloc_cache_noprof+0x3b1/0x4d0
nvme_fc_rcv_ls_req+0x267/0xcf0 [nvme_fc]
fcloop_t2h_ls_req+0x177/0x3f0 [nvme_fcloop]
__nvmet_fc_send_ls_req.constprop.0+0x5c6/0xee0 [nvmet_fc]
nvmet_fc_target_assoc_free+0x66f/0x900 [nvmet_fc]
nvmet_fc_delete_assoc_work+0xc8/0x200 [nvmet_fc]
process_one_work+0xd8b/0x1320
worker_thread+0x5f3/0xfe0
kthread+0x3b4/0x770
ret_from_fork+0x393/0x480
ret_from_fork_asm+0x1a/0x30
>
> Can you share a bit more how to you test for mem leaks?
I only run the blktests nvme/fc with the debug kernel, I also attached
the config in case you want to try it.
# nvme_trtype=fc ./check nvme/
>
> Thanks,
> Daniel
>
--
Best Regards,
Yi Zhang
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config
Type: application/octet-stream
Size: 258074 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20250604/c111c894/attachment-0001.obj>
More information about the Linux-nvme
mailing list