way to unbind a bad nvme device/controller without powering off system
James Puthukattukaran
james.puthukattukaran at oracle.com
Mon Oct 24 14:40:30 PDT 2022
Hi -
I'm seeing a scenario where what seems to be a non-functioning nvme controller/drive where the IO transactions are timing out and the controller is not responding to any controller commands. The controller seems to be disabled (nvme_dev_disable called via the nvme_timeout) but we're still seeing the nvme_reset_work thread blocked and not making progress. I tried to remove the controller via the HP sysfs interface and that also hangs behind the reset thread waiting for it to complete.
I thought the the disable controller path does not talk to the controller and simply unblocks the queues and cleans them out before unbinding the controller from the device. Not sure why the reset thread is still stuck then? Does the reset thread have to finish its course even though the controller has been disabled? trying to understand the flow here.
I guess what I'm really looking for is a way to simply unbind the device from the driver, kill any threads and allow the device to be powered of via the hotplug interface (trying to avoid rebooting the system to remove the device).
thanks,
James
systemd[1]:info: Removed slice User Slice of root.
kernel:warning: [10416583.556274] nvme nvme3: I/O 527 QID 1 timeout, aborting
kernel:warning: [10416583.556287] nvme nvme3: I/O 603 QID 1 timeout, aborting
kernel:warning: [10416583.556291] nvme nvme3: I/O 607 QID 1 timeout, aborting
kernel:warning: [10416583.556301] nvme nvme3: I/O 765 QID 1 timeout, aborting
kernel:warning: [10416583.556274] nvme nvme3: I/O 527 QID 1 timeout, aborting
kernel:warning: [10416583.556287] nvme nvme3: I/O 603 QID 1 timeout, aborting
kernel:warning: [10416583.556291] nvme nvme3: I/O 607 QID 1 timeout, aborting
kernel:warning: [10416583.556301] nvme nvme3: I/O 765 QID 1 timeout, aborting
kernel:warning: [10416583.557700] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557705] nvme nvme3: I/O 894 QID 22 timeout, aborting
kernel:warning: [10416583.557711] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557700] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557705] nvme nvme3: I/O 894 QID 22 timeout, aborting
kernel:warning: [10416583.557711] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557713] nvme nvme3: I/O 897 QID 22 timeout, aborting
kernel:warning: [10416583.557714] nvme nvme3: Abort status: 0x0
....
....
kernel:warning: [10416594.580023] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588219] nvme nvme3: I/O 191 QID 1 timeout, aborting
kernel:warning: [10416595.588228] nvme nvme3: I/O 192 QID 1 timeout, aborting
kernel:warning: [10416595.588233] nvme nvme3: I/O 195 QID 1 timeout, aborting
kernel:warning: [10416595.588237] nvme nvme3: I/O 196 QID 1 timeout, aborting
kernel:warning: [10416595.588244] nvme nvme3: I/O 209 QID 1 timeout, reset controller
kernel:warning: [10416595.588402] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588219] nvme nvme3: I/O 191 QID 1 timeout, aborting
kernel:warning: [10416595.588228] nvme nvme3: I/O 192 QID 1 timeout, aborting
kernel:warning: [10416595.588233] nvme nvme3: I/O 195 QID 1 timeout, aborting
kernel:warning: [10416595.588237] nvme nvme3: I/O 196 QID 1 timeout, aborting
kernel:warning: [10416595.588244] nvme nvme3: I/O 209 QID 1 timeout, reset controller
kernel:warning: [10416595.588402] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588449] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588449] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588513] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588561] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588513] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588561] nvme nvme3: Abort status: 0x0
kernel:info: [10416598.164310] nvme nvme3: Shutdown timeout set to 10 seconds
kernel:info: [10416598.164310] nvme nvme3: Shutdown timeout set to 10 seconds
systemd[1]:info: Removed slice User Slice of root.
kernel:warning: [10416583.556274] nvme nvme3: I/O 527 QID 1 timeout, aborting
kernel:warning: [10416583.556287] nvme nvme3: I/O 603 QID 1 timeout, aborting
kernel:warning: [10416583.556291] nvme nvme3: I/O 607 QID 1 timeout, aborting
kernel:warning: [10416583.556301] nvme nvme3: I/O 765 QID 1 timeout, aborting
kernel:warning: [10416583.556274] nvme nvme3: I/O 527 QID 1 timeout, aborting
kernel:warning: [10416583.556287] nvme nvme3: I/O 603 QID 1 timeout, aborting
kernel:warning: [10416583.556291] nvme nvme3: I/O 607 QID 1 timeout, aborting
kernel:warning: [10416583.556301] nvme nvme3: I/O 765 QID 1 timeout, aborting
kernel:warning: [10416583.557700] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557705] nvme nvme3: I/O 894 QID 22 timeout, aborting
kernel:warning: [10416583.557711] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557700] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557705] nvme nvme3: I/O 894 QID 22 timeout, aborting
kernel:warning: [10416583.557711] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557713] nvme nvme3: I/O 897 QID 22 timeout, aborting
kernel:warning: [10416583.557714] nvme nvme3: Abort status: 0x0
....
....
kernel:warning: [10416594.580023] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588219] nvme nvme3: I/O 191 QID 1 timeout, aborting
kernel:warning: [10416595.588228] nvme nvme3: I/O 192 QID 1 timeout, aborting
kernel:warning: [10416595.588233] nvme nvme3: I/O 195 QID 1 timeout, aborting
kernel:warning: [10416595.588237] nvme nvme3: I/O 196 QID 1 timeout, aborting
kernel:warning: [10416595.588244] nvme nvme3: I/O 209 QID 1 timeout, reset controller
kernel:warning: [10416595.588402] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588219] nvme nvme3: I/O 191 QID 1 timeout, aborting
kernel:warning: [10416595.588228] nvme nvme3: I/O 192 QID 1 timeout, aborting
kernel:warning: [10416595.588233] nvme nvme3: I/O 195 QID 1 timeout, aborting
kernel:warning: [10416595.588237] nvme nvme3: I/O 196 QID 1 timeout, aborting
kernel:warning: [10416595.588244] nvme nvme3: I/O 209 QID 1 timeout, reset controller
kernel:warning: [10416595.588402] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588449] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588449] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588513] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588561] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588513] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588561] nvme nvme3: Abort status: 0x0
kernel:info: [10416598.164310] nvme nvme3: Shutdown timeout set to 10 seconds
kernel:info: [10416598.164310] nvme nvme3: Shutdown timeout set to 10 seconds
systemd[1]:info: Removed slice User Slice of root.
kernel:warning: [10416583.556274] nvme nvme3: I/O 527 QID 1 timeout, aborting
kernel:warning: [10416583.556287] nvme nvme3: I/O 603 QID 1 timeout, aborting
kernel:warning: [10416583.556291] nvme nvme3: I/O 607 QID 1 timeout, aborting
kernel:warning: [10416583.556301] nvme nvme3: I/O 765 QID 1 timeout, aborting
kernel:warning: [10416583.556274] nvme nvme3: I/O 527 QID 1 timeout, aborting
kernel:warning: [10416583.556287] nvme nvme3: I/O 603 QID 1 timeout, aborting
kernel:warning: [10416583.556291] nvme nvme3: I/O 607 QID 1 timeout, aborting
kernel:warning: [10416583.556301] nvme nvme3: I/O 765 QID 1 timeout, aborting
kernel:warning: [10416583.557700] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557705] nvme nvme3: I/O 894 QID 22 timeout, aborting
kernel:warning: [10416583.557711] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557700] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557705] nvme nvme3: I/O 894 QID 22 timeout, aborting
kernel:warning: [10416583.557711] nvme nvme3: Abort status: 0x0
kernel:warning: [10416583.557713] nvme nvme3: I/O 897 QID 22 timeout, aborting
kernel:warning: [10416583.557714] nvme nvme3: Abort status: 0x0
....
....
kernel:warning: [10416594.580023] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588219] nvme nvme3: I/O 191 QID 1 timeout, aborting
kernel:warning: [10416595.588228] nvme nvme3: I/O 192 QID 1 timeout, aborting
kernel:warning: [10416595.588233] nvme nvme3: I/O 195 QID 1 timeout, aborting
kernel:warning: [10416595.588237] nvme nvme3: I/O 196 QID 1 timeout, aborting
kernel:warning: [10416595.588244] nvme nvme3: I/O 209 QID 1 timeout, reset controller
kernel:warning: [10416595.588402] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588219] nvme nvme3: I/O 191 QID 1 timeout, aborting
kernel:warning: [10416595.588228] nvme nvme3: I/O 192 QID 1 timeout, aborting
kernel:warning: [10416595.588233] nvme nvme3: I/O 195 QID 1 timeout, aborting
kernel:warning: [10416595.588237] nvme nvme3: I/O 196 QID 1 timeout, aborting
kernel:warning: [10416595.588244] nvme nvme3: I/O 209 QID 1 timeout, reset controller
kernel:warning: [10416595.588402] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588449] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588449] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588513] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588561] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588513] nvme nvme3: Abort status: 0x0
kernel:warning: [10416595.588561] nvme nvme3: Abort status: 0x0
kernel:info: [10416598.164310] nvme nvme3: Shutdown timeout set to 10 seconds
kernel:info: [10416598.164310] nvme nvme3: Shutdown timeout set to 10 seconds
kernel:warning: [10416608.580157] nvme nvme3: I/O 209 QID 1 timeout, disable controller
kernel:warning: [10416608.580157] nvme nvme3: I/O 209 QID 1 timeout, disable controller
...
...
kernel:info: [10419813.132341] Workqueue: nvme-reset-wq nvme_reset_work [nvme]
kernel:warning: [10419813.132342] Call Trace:
kernel:warning: [10419813.132345] __schedule+0x2bc/0x89b
kernel:warning: [10419813.132348] schedule+0x36/0x7c
kernel:warning: [10419813.132351] blk_mq_freeze_queue_wait+0x4b/0xaa
kernel:warning: [10419813.132353] ? remove_wait_queue+0x60/0x60
kernel:warning: [10419813.132359] nvme_wait_freeze+0x33/0x50 [nvme_core]
kernel:warning: [10419813.132362] nvme_reset_work+0x802/0xd84 [nvme]
kernel:warning: [10419813.132364] ? __switch_to_asm+0x40/0x62
kernel:warning: [10419813.132365] ? __switch_to_asm+0x34/0x62
kernel:warning: [10419813.132367] ? __switch_to+0x9b/0x505
kernel:warning: [10419813.132368] ? __switch_to_asm+0x40/0x62
kernel:warning: [10419813.132370] ? __switch_to_asm+0x40/0x62
kernel:warning: [10419813.132372] process_one_work+0x169/0x399
kernel:warning: [10419813.132374] worker_thread+0x4d/0x3e5
kernel:warning: [10419813.132377] kthread+0x105/0x138
kernel:warning: [10419813.132379] ? rescuer_thread+0x380/0x375
kernel:warning: [10419813.132380] ? kthread_bind+0x20/0x15
kernel:warning: [10419813.132382] ret_from_fork+0x24/0x49
...
...
kernel:warning: [10419813.158116] __schedule+0x2bc/0x89b
kernel:warning: [10419813.158119] schedule+0x36/0x7c
kernel:warning: [10419813.158122] schedule_timeout+0x1f6/0x31f
kernel:warning: [10419813.158124] ? sched_clock_cpu+0x11/0xa5
kernel:warning: [10419813.158126] ? try_to_wake_up+0x59/0x505
kernel:warning: [10419813.158130] wait_for_completion+0x12b/0x18a
kernel:warning: [10419813.158132] ? wake_up_q+0x80/0x73
kernel:warning: [10419813.158134] flush_work+0x122/0x1a7
kernel:warning: [10419813.158137] ? wake_up_worker+0x30/0x2b
kernel:warning: [10419813.158141] nvme_remove+0x71/0x100 [nvme]
kernel:warning: [10419813.158146] pci_device_remove+0x3e/0xb6
kernel:warning: [10419813.158149] device_release_driver_internal+0x134/0x1eb
kernel:warning: [10419813.158151] device_release_driver+0x12/0x14
kernel:warning: [10419813.158155] pci_stop_bus_device+0x7c/0x96
kernel:warning: [10419813.158158] pci_stop_bus_device+0x39/0x96
kernel:warning: [10419813.158164] pci_stop_and_remove_bus_device+0x12/0x1d
kernel:warning: [10419813.158167] pciehp_unconfigure_device+0x7a/0x1d7
kernel:warning: [10419813.158169] pciehp_disable_slot+0x52/0xca
kernel:warning: [10419813.158171] pciehp_sysfs_disable_slot+0x67/0x112
kernel:warning: [10419813.158174] disable_slot+0x12/0x14
kernel:warning: [10419813.158175] power_write_file+0x6e/0xf8
kernel:warning: [10419813.158179] pci_slot_attr_store+0x24/0x2e
kernel:warning: [10419813.158180] sysfs_kf_write+0x3f/0x46
kernel:warning: [10419813.158182] kernfs_fop_write+0x124/0x1a3
kernel:warning: [10419813.158184] __vfs_write+0x3a/0x16d
kernel:warning: [10419813.158187] ? audit_filter_syscall+0x33/0xce
kernel:warning: [10419813.158189] vfs_write+0xb2/0x1a1
...
...
kernel:warning: [10419813.158507] Call Trace:
kernel:warning: [10419813.158513] __schedule+0x2bc/0x89b
kernel:warning: [10419813.158521] schedule+0x36/0x7c
kernel:warning: [10419813.158527] schedule_timeout+0x189/0x31f
kernel:warning: [10419813.158533] ? __next_timer_interrupt+0xe0/0xd4
kernel:warning: [10419813.158539] ? blk_rq_append_bio+0x96/0xa3
kernel:warning: [10419813.158544] ? __blk_mq_insert_request+0x76/0x104
kernel:warning: [10419813.158549] io_schedule_timeout+0x1e/0x43
kernel:warning: [10419813.158554] wait_for_completion_io_timeout+0x13b/0x19d
kernel:warning: [10419813.158562] ? wake_up_q+0x80/0x73
kernel:warning: [10419813.158568] blk_execute_rq+0x6e/0x9a
kernel:warning: [10419813.158577] nvme_submit_user_cmd+0xc4/0x320 [nvme_core]
kernel:warning: [10419813.158583] nvme_user_cmd+0x21d/0x3c0 [nvme_core]
kernel:warning: [10419813.158591] nvme_ioctl+0x175/0x220 [nvme_core]
kernel:warning: [10419813.158598] blkdev_ioctl+0x878/0x912
kernel:warning: [10419813.158605] block_ioctl+0x41/0x45
kernel:warning: [10419813.158611] do_vfs_ioctl+0xaa/0x602
kernel:warning: [10419813.158616] ? __audit_syscall_entry+0xac/0xef
kernel:warning: [10419813.158621] ? syscall_trace_enter+0x1ce/0x2b8
kernel:warning: [10419813.158627] SyS_ioctl+0x79/0x84
More information about the Linux-nvme
mailing list