[bug report] kmemleak osberved during blktests nvme/fc test

[Daniel Wagner]

Hi,

On Sat, May 31, 2025 at 10:08:40AM +0800, Yi Zhang wrote:
> Hello
> I found the following kmemleak issue during blktests nvme/fc with the
> latest linux-block/for-next, please help check it and let me know if you
> need any test/info for it, thanks.

I've tried to reproduce the kmemleaks by enabling
CONFIG_DEBUG_KMEMLEAK and

echo scan > /sys/kernel/debug/kmemleak
echo check > /sys/kernel/debug/kmemleak

but I don't get any reports. I've used 3bc1b5c03cbd ("Merge branch
'io_uring-6.16' into for-next")

> unreferenced object 0xffff88816cd58400 (size 192):
>   comm "check", pid 24201, jiffies 4302692431
>   hex dump (first 32 bytes):
>     00 00 00 00 00 00 00 00 01 00 00 aa 00 11 00 10  ................
>     01 00 00 aa 00 11 00 20 b8 84 d5 6c 81 88 ff ff  ....... ...l....
>   backtrace (crc 9556a114):
>     __kmalloc_noprof+0x442/0x5e0
>     trace_raw_output_nvmet_req_init+0x3b3/0x450 [nvmet]

So that would be something in nvmet_req_init:

   trace_nvmet_req_init(req, req->cmd);

though I can't spot it.

>     fcloop_fcp_copy_data+0x6b5/0x770 [nvme_fcloop]
>     kernfs_fop_write_iter+0x357/0x530

this looks really strange. No idea how we can end up from
kernfd_fop_write_iter to fcloop_fcp_copy_data.

>     vfs_write+0x9bc/0xf60
>     ksys_write+0xf3/0x1d0
>     do_syscall_64+0x8c/0x3d0
>     entry_SYSCALL_64_after_hwframe+0x76/0x7e


> unreferenced object 0xffff888224353800 (size 1024):
>   comm "check", pid 24479, jiffies 4302692654
>   hex dump (first 32 bytes):
>     00 00 00 00 60 00 00 00 01 00 00 ab 00 11 00 10  ....`...........
>     01 00 00 ab 00 11 00 20 00 84 d5 6c 81 88 ff ff  ....... ...l....
>   backtrace (crc 9aae07c2):
>     __kmalloc_noprof+0x442/0x5e0
>     nvmet_ns_device_uuid_store+0x4c/0xf0 [nvmet]
>     0xffffffffc33aca5a
>     kernfs_fop_write_iter+0x357/0x530
>     vfs_write+0x9bc/0xf60
>     ksys_write+0xf3/0x1d0
>     do_syscall_64+0x8c/0x3d0
>     entry_SYSCALL_64_after_hwframe+0x76/0x7e


I don't see any allocation happening in nvmet_ns_device_uuid_store, the
code is just parsing the string into a fixed sized buffer.

> unreferenced object 0xffff88811e90ed00 (size 64):
>   comm "kworker/u67:8", pid 24542, jiffies 4303041902
>   hex dump (first 32 bytes):
>     00 38 35 24 82 88 ff ff 08 8d 53 40 81 88 ff ff  .85$......S at ....
>     00 a8 8f e1 82 88 ff ff 00 19 5f b3 82 88 ff ff  .........._.....
>   backtrace (crc 45d8e045):
>     __kmalloc_cache_noprof+0x411/0x4e0
>     nvmet_ns_device_path_store+0x12/0x180 [nvmet]
>     fcloop_create_remote_port+0x5d3/0x640 [nvme_fcloop]
>     nvme_fc_handle_ls_rqst_work+0x77c/0xfe0 [nvme_fc]
>     nvme_fc_fcpio_done+0x715/0xfb0 [nvme_fc]
>     __nvme_fc_send_ls_req+0x750/0x1010 [nvme_fc]
>     process_one_work+0x8cd/0x1950
>     worker_thread+0x58d/0xcf0
>     kthread+0x3d8/0x7a0
>     ret_from_fork+0x406/0x510
>     ret_from_fork_asm+0x1a/0x30

nvmet_ns_device_patch_store is not called from
fcloop_create_remote_port.

> unreferenced object 0xffff88846c4ac700 (size 128):
>   comm "kworker/u65:101", pid 4587, jiffies 4296544937
>   hex dump (first 32 bytes):
>     01 00 00 00 00 00 00 20 00 00 00 01 00 00 00 08  ....... ........
>     05 00 00 00 00 00 00 00 00 00 00 02 00 00 00 08  ................
>   backtrace (crc d98f69c0):
>     __kmalloc_cache_noprof+0x411/0x4e0
>     nvme_fc_rcv_ls_req+0x370/0x1080 [nvme_fc]
>     fcloop_t2h_ls_req+0x173/0x3f0 [nvme_fcloop]
>     __nvmet_fc_send_ls_req.constprop.0+0x5dc/0xee0 [nvmet_fc]
>     nvmet_fc_target_assoc_free+0x675/0xab0 [nvmet_fc]
>     nvmet_fc_delete_assoc_work+0x300/0x400 [nvmet_fc]
>     process_one_work+0x8cd/0x1950
>     worker_thread+0x58d/0xcf0
>     kthread+0x3d8/0x7a0
>     ret_from_fork+0x406/0x510
>     ret_from_fork_asm+0x1a/0x30

This stack trace looks more reasonable and I thought I fixed this one
already. I'll double check.

> unreferenced object 0xffff888454806400 (size 192):
>   comm "check", pid 2795, jiffies 4295104867
>   hex dump (first 32 bytes):
>     00 00 00 00 00 00 00 00 01 00 00 aa 00 11 00 10  ................
>     01 00 00 aa 00 11 00 20 b8 64 80 54 84 88 ff ff  ....... .d.T....
>   backtrace (crc 47ccaf54):
>     __kmalloc_noprof+0x442/0x5e0
>     nvme_fc_register_localport+0x453/0x1440 [nvme_fc]
>     fcloop_create_local_port+0x265/0x520 [nvme_fcloop]
>     kernfs_fop_write_iter+0x357/0x530
>     vfs_write+0x9bc/0xf60
>     ksys_write+0xf3/0x1d0
>     do_syscall_64+0x8c/0x3d0
>     entry_SYSCALL_64_after_hwframe+0x76/0x7e

This looks like a localport is leaked and it should be fairly simple to
reproduce it for me...

> unreferenced object 0xffff8883f6377800 (size 1024):
>   comm "check", pid 3079, jiffies 4295105085
>   hex dump (first 32 bytes):
>     00 00 00 00 60 00 00 00 01 00 00 ab 00 11 00 10  ....`...........
>     01 00 00 ab 00 11 00 20 00 64 80 54 84 88 ff ff  ....... .d.T....
>   backtrace (crc 8e00bf47):
>     __kmalloc_noprof+0x442/0x5e0
>     nvme_fc_register_remoteport+0x2ec/0x1360 [nvme_fc]
>     fcloop_create_remote_port+0x19a/0x640 [nvme_fcloop]
>     kernfs_fop_write_iter+0x357/0x530
>     vfs_write+0x9bc/0xf60
>     ksys_write+0xf3/0x1d0
>     do_syscall_64+0x8c/0x3d0
>     entry_SYSCALL_64_after_hwframe+0x76/0x7e

Same here.

Can you share a bit more how to you test for mem leaks?

Thanks,
Daniel