crash on device removal
Steve Wise
swise at opengridcomputing.com
Tue Jul 12 09:38:58 PDT 2016
>
> Hey Christoph,
>
> I see a crash when shutting down a nvme host node via 'reboot' that has 1
> target device attached. The shutdown causes iw_cxgb4 to be removed which
> triggers the device removal logic in the nvmf rdma transport. The crash
> is here:
>
> (gdb) list *nvme_rdma_free_qe+0x18
> 0x1e8 is in nvme_rdma_free_qe (drivers/nvme/host/rdma.c:196).
> 191 }
> 192
> 193 static void nvme_rdma_free_qe(struct ib_device *ibdev, struct
> nvme_rdma_qe *qe,
> 194 size_t capsule_size, enum dma_data_direction dir)
> 195 {
> 196 ib_dma_unmap_single(ibdev, qe->dma, capsule_size, dir);
> 197 kfree(qe->data);
> 198 }
> 199
> 200 static int nvme_rdma_alloc_qe(struct ib_device *ibdev, struct
> nvme_rdma_qe *qe,
>
> Apparently qe is NULL.
>
> Looking at the device removal path, the logic appears correct (see
> nvme_rdma_device_unplug() and the nice function comment :) ). I'm
> wondering if concurrently to the host device removal path cleaning up
> queues, the target is disconnecting all of its queues due to the first
> disconnect event from the host causing some cleanup race on the host side?
> Although since the removal path executing in the cma event handler upcall,
> I don't think another thread would be handling a disconnect event. Maybe
> the qp async event handler flow?
>
I see the async event handler, nvme_rdma_qp_event() does nothing but a
pr_debug(), so no race with the cm event handler thread and the async event
handler thread...
> Thoughts?
>
> Here is the Oops:
>
> [ 710.929451] iw_cxgb4:0000:83:00.4: Detach
> [ 711.242989] iw_cxgb4:0000:82:00.4: Detach
> [ 711.247039] nvme nvme1: Got rdma device removal event, deleting ctrl
> [ 711.298244] BUG: unable to handle kernel NULL pointer dereference at
> 0000000000000010
> [ 711.306162] IP: [<ffffffffa039a1e8>] nvme_rdma_free_qe+0x18/0x80
> [nvme_rdma]
> [ 711.313286] PGD 0
> [ 711.315348] Oops: 0000 [#1] SMP
> [ 711.318519] Modules linked in: nvme_rdma nvme_fabrics brd iw_cxgb4
> cxgb4 ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
> nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
> nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT nf_reject_ipv4 xt_CHECKSUM
> iptable_mangle iptable_filter ip_tables bridge 8021q mrp garp stp llc
> cachefiles fscache rdma_ucm rdma_cm iw_cm ib_ipoib ib_cm ib_uverbs ib_umad
> ocrdma be2net iw_nes libcrc32c iw_cxgb3 cxgb3 mdio ib_qib rdmavt mlx5_ib
> mlx5_core mlx4_en ib_mthca binfmt_misc dm_mirror dm_region_hash dm_log
> vhost_net macvtap macvlan vhost tun kvm irqbypass uinput iTCO_wdt
> iTCO_vendor_support mxm_wmi pcspkr mlx4_ib ib_core mlx4_core dm_mod
> i2c_i801 sg ipmi_ssif ipmi_si ipmi_msghandler nvme nvme_core lpc_ich
> mfd_core mei_me mei igb dca ptp pps_core wmi ext4(E) mbcache(E) jbd2(E)
> sd_mod(E) ahci(E) libahci(E) libata(E) mgag200(E) ttm(E) drm_kms_helper(E)
> drm(E) fb_sys_fops(E) sysimgblt(E) sysfillrect(E) syscopyarea(E)
> i2c_algo_bit(E) i2c_core(E) [last unloaded: cxgb4]
> [ 711.412158] CPU: 0 PID: 4213 Comm: reboot Tainted: G E
> 4.7.0-rc2-block-for-next+ #77
> [ 711.421064] Hardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 3.2a
> 07/09/2015
> [ 711.428058] task: ffff881033b495c0 ti: ffff88100fc24000 task.ti:
> ffff88100fc24000
> [ 711.435563] RIP: 0010:[<ffffffffa039a1e8>] [<ffffffffa039a1e8>]
> nvme_rdma_free_qe+0x18/0x80 [nvme_rdma]
> [ 711.445104] RSP: 0018:ffff88100fc279a8 EFLAGS: 00010292
> [ 711.450442] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> 0000000000000002
> [ 711.457608] RDX: 0000000000000010 RSI: 0000000000000000 RDI:
> ffff881034168000
> [ 711.464775] RBP: ffff88100fc279b8 R08: 0000000000000001 R09:
> ffffea0001e51d10
> [ 711.471943] R10: ffffea0001e51d18 R11: 0000000000000000 R12:
> 0000000000000000
> [ 711.479112] R13: 0000000000000020 R14: ffff881034168000 R15:
> ffff8810345b8140
> [ 711.486285] FS: 00007feac7042700(0000) GS:ffff88103ee00000(0000)
> knlGS:0000000000000000
> [ 711.494405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 711.500175] CR2: 0000000000000010 CR3: 00000010229d7000 CR4:
> 00000000000406f0
> [ 711.507341] Stack:
> [ 711.509367] ffff881034285000 0000000000000001 ffff88100fc279f8
> ffffffffa039adcf
> [ 711.516868] ffff88100fc279d8 ffff881034285000 ffff881037f9f000
> ffff881034272c00
> [ 711.524384] ffff88100fc27b18 ffff881034272dd8 ffff88100fc27a88
> ffffffffa039c8f5
> [ 711.531897] Call Trace:
> [ 711.534371] [<ffffffffa039adcf>] nvme_rdma_destroy_queue_ib+0x5f/0x90
> [nvme_rdma]
> [ 711.541972] [<ffffffffa039c8f5>] nvme_rdma_cm_handler+0x2c5/0x340
> [nvme_rdma]
> [ 711.549228] [<ffffffff811ff71d>] ? kmem_cache_free+0x1dd/0x200
> [ 711.555177] [<ffffffffa070e669>] ? cma_comp+0x49/0x60 [rdma_cm]
> [ 711.561217] [<ffffffffa071310f>] cma_remove_id_dev+0x8f/0xa0 [rdma_cm]
> [ 711.567860] [<ffffffffa07131d7>] cma_process_remove+0xb7/0x100
> [rdma_cm]
> [ 711.574678] [<ffffffff812a4de4>] ? __kernfs_remove+0x114/0x1d0
> [ 711.580626] [<ffffffffa071325e>] cma_remove_one+0x3e/0x60 [rdma_cm]
> [ 711.587015] [<ffffffffa03b8ca0>] ib_unregister_device+0xb0/0x150
> [ib_core]
> [ 711.595252] [<ffffffffa0816034>] c4iw_unregister_device+0x64/0x90
> [iw_cxgb4]
> [ 711.603648] [<ffffffffa0809357>] c4iw_remove+0x27/0x60 [iw_cxgb4]
> [ 711.611069] [<ffffffffa080a061>] c4iw_uld_state_change+0x111/0x250
> [iw_cxgb4]
> [ 711.619532] [<ffffffff816da18d>] ? _cond_resched+0x1d/0x30
> [ 711.626317] [<ffffffff81371971>] ? list_del+0x11/0x40
> [ 711.632678] [<ffffffffa07ce71a>] detach_ulds+0x4a/0xf0 [cxgb4]
> [ 711.639822] [<ffffffffa07ce94d>] remove_one+0x18d/0x1b0 [cxgb4]
> [ 711.647060] [<ffffffff81397c21>] pci_device_shutdown+0x41/0x90
> [ 711.654189] [<ffffffff814861f5>] device_shutdown+0x45/0x1b0
> [ 711.661051] [<ffffffff810ac746>] kernel_restart_prepare+0x36/0x40
> [ 711.668414] [<ffffffff810ac8c6>] kernel_restart+0x16/0x60
> [ 711.675084] [<ffffffff810acb15>] SYSC_reboot+0x1a5/0x230
> [ 711.681645] [<ffffffff81245ad1>] ? mntput+0x21/0x30
> [ 711.687738] [<ffffffff812267a7>] ? __fput+0x177/0x240
> [ 711.693964] [<ffffffff8122691e>] ? ____fput+0xe/0x10
> [ 711.700097] [<ffffffff81003476>] ? do_audit_syscall_entry+0x66/0x70
> [ 711.707481] [<ffffffff81003578>] ?
> syscall_trace_enter_phase1+0xf8/0x120
> [ 711.715273] [<ffffffff81003344>] ? exit_to_usermode_loop+0x74/0xf0
> [ 711.722514] [<ffffffff810acbae>] SyS_reboot+0xe/0x10
> [ 711.728517] [<ffffffff81003f08>] do_syscall_64+0x78/0x1d0
> [ 711.734931] [<ffffffff8106e327>] ? do_page_fault+0x37/0x90
> [ 711.741410] [<ffffffff816ddee1>] entry_SYSCALL64_slow_path+0x25/0x25
> [ 711.748731] Code: 01 00 00 c9 c3 0f 0b eb fe 66 2e 0f 1f 84 00 00 00 00
> 00 55 48 89 e5 53 48 83 ec 08 66 66 66 66 90 48 8b 87 f0 02 00 00 48 89 f3
> <48> 8b 76 10 48 85 c0 74 13 ff 50 10 48 8b 7b 08 e8 93 4d e6 e0
> [ 711.770832] RIP [<ffffffffa039a1e8>] nvme_rdma_free_qe+0x18/0x80
> [nvme_rdma]
> [ 711.778904] RSP <ffff88100fc279a8>
> [ 711.783290] CR2: 0000000000000010
More information about the Linux-nvme
mailing list