Bug report - drivers/nvme/host/pci.c?
Fangjian (Turing)
f.fangjian at huawei.com
Tue Jan 2 22:31:51 PST 2018
Hi,
Linux version 4.15.0-rc1.
I use fio tool to test my NVME card performace, I found kernel will dump
call trace and error info.
test log:
$ fio -name=write -numjobs=10 -filename=/dev/nvme0n1 -rw=write
-iodepth=128 -direct=1 -sync=0 -norandommap -group_reporting -runtime=30
-time
_based -bs=1M
write: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=sync, iodepth=128
...
fio-2.1.11
Starting 10 processes
[330929.298425] WARNING: CPU: 27 PID: 2516 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.298815] WARNING: CPU: 28 PID: 2520 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
fio: pid=2520, err=5/file:engines[330929.298818] Modules linked in:
/sync.c:67, func=xfer, error=Inp[330929.298825] CPU: 28 PID: 2520 Comm:
fio Tainted: G W 4.15.0-rc1-g1d739bd #18
ut/output error
[330929.298826] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.298827] task: ffff8413e6734880 task.stack: ffff00001b6a0000
[330929.298829] pstate: 20000005 (nzCv daif -PAN -UAO)
fio: pid=2513, err=5/file:engines[330929.298831] pc :
nvme_queue_rq+0x660/0x9f8
/sync.c:67, func=xfer, error=Inp[330929.298833] lr :
nvme_queue_rq+0x59c/0x9f8
ut/output error
[330929.298834] sp : ffff00001b6a3800
[330929.298835] x29: ffff00001b6a3800 x28: ffff8017e6f3d720
[330929.298838] x27: ffff8017e6f3d5c0 x26: 0000000000000000
[330929.298842] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.298845] x23: 0000000000000001 x22: ffff8013e2808000
[330929.298848] x21: ffff8013e3760000 x20: ffff8017e917ba00
[330929.298850] x19: 0000000000000001 x18: 0000000000000001
[330929.298854] x17: 000000000047c678 x16: ffff00000821d848
[330929.298857] x15: 0000000000000008 x14: 0000ffff7e235c1c
[330929.298860] x13: 000000005f9702c0 x12: 0000000000000000
[330929.298864] x11: 0000820000000000 x10: 0000000000000000
[330929.298867] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.298870] x7 : 0000000000000010 x6 : 000000000000c000
[330929.298873] x5 : 0000000000000003 x4 : ffff8413e0615000
[330929.298877] x3 : ffff8413e07d3d90 x2 : ffff7e005f604000
[330929.298880] x1 : 000000000000000a x0 : ffff8413e07d3d90
[330929.298884] Call trace:
[330929.298886] nvme_queue_rq+0x660/0x9f8
[330929.298892] __blk_mq_try_issue_directly+0x178/0x1d8
[330929.298894] blk_mq_try_issue_directly+0x88/0xa0
[330929.298896] blk_mq_make_request+0x294/0x488
[330929.298900] generic_make_request+0xe4/0x258
[330929.298902] submit_bio+0xa4/0x1b8
[330929.298906] __blkdev_direct_IO_simple+0x120/0x298
[330929.298908] blkdev_direct_IO+0x3b4/0x3b8
[330929.298911] generic_file_direct_write+0xe8/0x188
[330929.298913] __generic_file_write_iter+0xac/0x1b0
[330929.298915] blkdev_write_iter+0x84/0xf8
[330929.298919] __vfs_write+0xb0/0x118
[330929.298920] vfs_write+0xa0/0x1b0
[330929.298922] SyS_write+0x44/0xa0
[330929.298925] el0_svc_naked+0x20/0x24
[330929.298927] ---[ end trace 7626222de6819d61 ]---
[330929.299262] WARNING: CPU: 28 PID: 2502 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.299263] Modules linked in:
[330929.299267] CPU: 28 PID: 2502 Comm: kworker/28:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.299269] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.299273] Workqueue: kblockd blk_mq_requeue_work
[330929.299275] task: ffff8017e6101d00 task.stack: ffff00001b458000
[330929.299277] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.299279] pc : nvme_queue_rq+0x660/0x9f8
[330929.299282] lr : nvme_queue_rq+0x59c/0x9f8
[330929.299283] sp : ffff00001b45bad0
[330929.299284] x29: ffff00001b45bad0 x28: ffff8017e6f3d720
[330929.299288] x27: ffff8017e6f3d5c0 x26: 0000000000000000
[330929.299292] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.299296] x23: 0000000000000001 x22: ffff8013e2808000
[330929.299299] x21: ffff8013e3760000 x20: ffff8017e917ba00
[330929.299302] x19: 0000000000000001 x18: 0000000000000400
[330929.299305] x17: 000000000047c678 x16: ffff00000821d848
[330929.299309] x15: 0000000000000000 x14: 0000000000000400
[330929.299313] x13: 0000000000000400 x12: 0000000000000000
[330929.299316] x11: 0000000000000000 x10: 0000000000000000
[330929.299320] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.299325] x7 : 0000000000000010 x6 : 000000000000c000
[330929.299328] x5 : 00000017d810c000 x4 : ffff8413e0615000
[330929.299331] x3 : ffff8413e07d3d90 x2 : ffff7e005f604000
[330929.299335] x1 : 000000000000000a x0 : ffff8413e07d3d90
[330929.299339] Call trace:
[330929.299341] nvme_queue_rq+0x660/0x9f8
[330929.299343] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.299346] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.299349] __blk_mq_run_hw_queue+0xec/0x100
[330929.299351] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.299353] blk_mq_run_hw_queue+0x3c/0x98
[330929.299355] blk_mq_run_hw_queues+0x50/0x70
[330929.299357] blk_mq_requeue_work+0x140/0x158
[330929.299361] process_one_work+0x138/0x2f0
[330929.299363] worker_thread+0x144/0x408
[330929.299365] kthread+0xf8/0x128
[330929.299368] ret_from_fork+0x10/0x18
[330929.299370] ---[ end trace 7626222de6819d62 ]---
[330929.299772] WARNING: CPU: 28 PID: 2502 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.299774] Modules linked in:
[330929.299777] CPU: 28 PID: 2502 Comm: kworker/28:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.299778] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.299781] Workqueue: kblockd blk_mq_requeue_work
[330929.299784] task: ffff8017e6101d00 task.stack: ffff00001b458000
[330929.299787] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.299789] pc : nvme_queue_rq+0x660/0x9f8
[330929.299791] lr : nvme_queue_rq+0x59c/0x9f8
[330929.299792] sp : ffff00001b45bad0
[330929.299793] x29: ffff00001b45bad0 x28: ffff8017e6f3d720
[330929.299797] x27: ffff8017e6f3d5c0 x26: 0000000000000000
[330929.299801] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.299805] x23: 0000000000000001 x22: ffff8013e2808000
[330929.299809] x21: ffff8013e3760000 x20: ffff8017e917ba00
[330929.299812] x19: 0000000000000001 x18: 0000000000000400
[330929.299815] x17: 000000000047c678 x16: ffff00000821d848
[330929.299818] x15: 0000000000000000 x14: 0000000000000400
[330929.299821] x13: 0000000000000400 x12: 0000000000000001
[330929.299826] x11: 0000000000d4b5b1 x10: 0000000000000000
[330929.299830] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.299833] x7 : 0000000000000010 x6 : 000000000000c000
[330929.299837] x5 : 00000017d810c000 x4 : ffff8413e0615000
[330929.299840] x3 : ffff8413e07d3d90 x2 : ffff7e005f604000
[330929.299844] x1 : 000000000000000a x0 : ffff8413e07d3d90
[330929.299848] Call trace:
[330929.299850] nvme_queue_rq+0x660/0x9f8
[330929.299852] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.299856] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.299859] __blk_mq_run_hw_queue+0xec/0x100
[330929.299860] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.299863] blk_mq_run_hw_queue+0x3c/0x98
[330929.299866] blk_mq_run_hw_queues+0x50/0x70
[330929.299868] blk_mq_requeue_work+0x140/0x158
[330929.299870] process_one_work+0x138/0x2f0
[330929.299872] worker_thread+0x144/0x408
[330929.299875] kthread+0xf8/0x128
[330929.299878] ret_from_fork+0x10/0x18
[330929.299879] ---[ end trace 7626222de6819d63 ]---
[330929.300186] WARNING: CPU: 28 PID: 2502 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.300187] Modules linked in:
[330929.300190] CPU: 28 PID: 2502 Comm: kworker/28:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.300193] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.300197] Workqueue: kblockd blk_mq_requeue_work
[330929.300199] task: ffff8017e6101d00 task.stack: ffff00001b458000
[330929.300201] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.300205] pc : nvme_queue_rq+0x660/0x9f8
[330929.300208] lr : nvme_queue_rq+0x59c/0x9f8
[330929.300209] sp : ffff00001b45bad0
[330929.300210] x29: ffff00001b45bad0 x28: ffff8017e6f3d720
[330929.300213] x27: ffff8017e6f3d5c0 x26: 0000000000000000
[330929.300217] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.300220] x23: 0000000000000001 x22: ffff8013e2808000
[330929.300223] x21: ffff8013e3760000 x20: ffff8017e917ba00
[330929.300226] x19: 0000000000000001 x18: 0000000000000400
[330929.300230] x17: 000000000047c678 x16: ffff00000821d848
[330929.300234] x15: 0000000000000000 x14: 0000000000000400
[330929.300237] x13: 0000000000000400 x12: 0000000000000001
[330929.300240] x11: 00000000014be4e4 x10: 0000000000000000
[330929.300243] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.300247] x7 : 0000000000000010 x6 : 000000000000c000
[330929.300251] x5 : 00000017d810c000 x4 : ffff8413e0615000
[330929.300255] x3 : ffff8413e07d3d90 x2 : ffff7e005f604000
[330929.300259] x1 : 000000000000000a x0 : ffff8413e07d3d90
[330929.300262] Call trace:
[330929.300265] nvme_queue_rq+0x660/0x9f8
[330929.300269] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.300270] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.300273] __blk_mq_run_hw_queue+0xec/0x100
[330929.300276] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.300278] blk_mq_run_hw_queue+0x3c/0x98
[330929.300281] blk_mq_run_hw_queues+0x50/0x70
[330929.300282] blk_mq_requeue_work+0x140/0x158
[330929.300285] process_one_work+0x138/0x2f0
[330929.300288] worker_thread+0x144/0x408
[330929.300290] kthread+0xf8/0x128
[330929.300292] ret_from_fork+0x10/0x18
[330929.300294] ---[ end trace 7626222de6819d64 ]---
Jobs: 8 (f=8)
[330929.300599] WARNING: CPU: 28 PID: 2502 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.300600] Modules linked in:
[330929.300603] CPU: 28 PID: 2502 Comm: kworker/28:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.300605] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.300608] Workqueue: kblockd blk_mq_requeue_work
[330929.300610] task: ffff8017e6101d00 task.stack: ffff00001b458000
[330929.300612] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.300615] pc : nvme_queue_rq+0x660/0x9f8
[330929.300617] lr : nvme_queue_rq+0x59c/0x9f8
[330929.300617] sp : ffff00001b45bad0
[330929.300619] x29: ffff00001b45bad0 x28: ffff8017e6f3d720
[330929.300622] x27: ffff8017e6f3d5c0 x26: 0000000000000000
[330929.300624] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.300628] x23: 0000000000000001 x22: ffff8013e2808000
[330929.300632] x21: ffff8013e3760000 x20: ffff8017e917ba00
[330929.300635] x19: 0000000000000001 x18: 0000000000000400
[330929.300639] x17: 000000000047c678 x16: ffff00000821d848
[330929.300642] x15: 0000000000000000 x14: 0000000000000400
[330929.300647] x13: 0000000000000400 x12: 0000000000000001
[330929.300650] x11: 0000000001c46f35 x10: 0000000000000000
[330929.300653] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.300656] x7 : 0000000000000010 x6 : 000000000000c000
[330929.300661] x5 : 00000017d810c000 x4 : ffff8413e0615000
[330929.300664] x3 : ffff8413e07d3d90 x2 : ffff7e005f604000
[330929.300667] x1 : 000000000000000a x0 : ffff8413e07d3d90
[330929.300671] Call trace:
[330929.300673] nvme_queue_rq+0x660/0x9f8
[330929.300675] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.300677] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.300681] __blk_mq_run_hw_queue+0xec/0x100
[330929.300683] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.300686] blk_mq_run_hw_queue+0x3c/0x98
[330929.300687] blk_mq_run_hw_queues+0x50/0x70
[330929.300690] blk_mq_requeue_work+0x140/0x158
[330929.300692] process_one_work+0x138/0x2f0
[330929.300695] worker_thread+0x144/0x408
[330929.300696] kthread+0xf8/0x128
[330929.300699] ret_from_fork+0x10/0x18
[330929.300700] ---[ end trace 7626222de6819d65 ]---
[330929.301006] WARNING: CPU: 28 PID: 2502 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.301008] Modules linked in:
[330929.301011] CPU: 28 PID: 2502 Comm: kworker/28:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.301012] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.301016] Workqueue: kblockd blk_mq_requeue_work
[330929.301021] task: ffff8017e6101d00 task.stack: ffff00001b458000
[330929.301023] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.301025] pc : nvme_queue_rq+0x660/0x9f8
[330929.301028] lr : nvme_queue_rq+0x59c/0x9f8
[330929.301031] sp : ffff00001b45bad0
[330929.301032] x29: ffff00001b45bad0 x28: ffff8017e6f3d720
[330929.301036] x27: ffff8017e6f3d5c0 x26: 0000000000000000
[330929.301039] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.301043] x23: 0000000000000001 x22: ffff8013e2808000
[330929.301048] x21: ffff8013e3760000 x20: ffff8017e917ba00
[330929.301052] x19: 0000000000000001 x18: 0000000000000400
[330929.301057] x17: 000000000047c678 x16: ffff00000821d848
[330929.301062] x15: 0000000000000000 x14: 0000000000000400
[330929.301065] x13: 0000000000000400 x12: 0000000000000001
[330929.301069] x11: 000000000234d930 x10: 0000000000000000
[330929.301074] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.301078] x7 : 0000000000000010 x6 : 000000000000c000
[330929.301083] x5 : 00000017d810c000 x4 : ffff8413e0615000
[330929.301086] x3 : ffff8413e07d3d90 x2 : ffff7e005f604000
[330929.301089] x1 : 000000000000000a x0 : ffff8413e07d3d90
[330929.301093] Call trace:
[330929.301095] nvme_queue_rq+0x660/0x9f8
[330929.301097] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.301100] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.301103] __blk_mq_run_hw_queue+0xec/0x100
[330929.301104] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.301108] blk_mq_run_hw_queue+0x3c/0x98
[330929.301110] blk_mq_run_hw_queues+0x50/0x70
[330929.301112] blk_mq_requeue_work+0x140/0x158
[330929.301114] process_one_work+0x138/0x2f0
[330929.301117] worker_thread+0x144/0x408
[330929.301119] kthread+0xf8/0x128
[330929.301121] ret_from_fork+0x10/0x18
[330929.301124] ---[ end trace 7626222de6819d66 ]---
[330929.301385] print_req_error: I/O error, dev nvme0n1, sector 0
[330929.312726] WARNING: CPU: 13 PID: 2513 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.312729] Modules linked in:
[330929.312733] CPU: 13 PID: 2513 Comm: fio Tainted: G W
4.15.0-rc1-g1d739bd #18
[330929.312735] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.312736] task: ffff8413e65a8e80 task.stack: ffff000011c08000
[330929.312738] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.312740] pc : nvme_queue_rq+0x660/0x9f8
[330929.312742] lr : nvme_queue_rq+0x59c/0x9f8
[330929.312743] sp : ffff000011c0b800
[330929.312744] x29: ffff000011c0b800 x28: ffff8013e288d4e0
[330929.312748] x27: ffff8013e288d380 x26: 0000000000000000
[330929.312751] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.312754] x23: 0000000000000001 x22: ffff8013e2808000
[330929.312758] x21: ffff8013e3760000 x20: ffff8013e6734400
[330929.312760] x19: 0000000000000001 x18: 0000000000000001
[330929.312764] x17: 000000000047c678 x16: ffff00000821d848
[330929.312767] x15: 0000000000000008 x14: 0000ffff7e235c1c
[330929.312770] x13: 000000004ed809c0 x12: 0000000000000000
[330929.312773] x11: 0000820000000000 x10: 0000000000000000
[330929.312776] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.312780] x7 : 0000000000000010 x6 : 0000000000009000
[330929.312783] x5 : 0000000000000003 x4 : ffff8413e0615000
[330929.312786] x3 : ffff8413e0b73d68 x2 : ffff7e004ed9a9c0
[330929.312790] x1 : 0000000000000009 x0 : ffff8413e0b73d68
[330929.312793] Call trace:
[330929.312795] nvme_queue_rq+0x660/0x9f8
[330929.312798] __blk_mq_try_issue_directly+0x178/0x1d8
[330929.312800] blk_mq_try_issue_directly+0x88/0xa0
[330929.312803] blk_mq_make_request+0x294/0x488
[330929.312806] generic_make_request+0xe4/0x258
[330929.312808] submit_bio+0xa4/0x1b8
[330929.312810] __blkdev_direct_IO_simple+0x120/0x298
[330929.312812] blkdev_direct_IO+0x3b4/0x3b8
[330929.312815] generic_file_direct_write+0xe8/0x188
[330929.312817] __generic_file_write_iter+0xac/0x1b0
[330929.312819] blkdev_write_iter+0x84/0xf8
[330929.312821] __vfs_write+0xb0/0x118
[330929.312823] vfs_write+0xa0/0x1b0
[330929.312825] SyS_write+0x44/0xa0
[330929.312827] el0_svc_naked+0x20/0x24
[330929.312828] ---[ end trace 7626222de6819d67 ]---
[330929.313418] WARNING: CPU: 12 PID: 2440 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.313419] Modules linked in:
[330929.313423] CPU: 12 PID: 2440 Comm: kworker/12:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.313424] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.313429] Workqueue: kblockd blk_mq_requeue_work
[330929.313432] task: ffff8013e6b78e80 task.stack: ffff000011b30000
[330929.313434] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.313437] pc : nvme_queue_rq+0x660/0x9f8
[330929.313441] lr : nvme_queue_rq+0x59c/0x9f8
[330929.313442] sp : ffff000011b33ad0
[330929.313443] x29: ffff000011b33ad0 x28: ffff8013e288d4e0
[330929.313448] x27: ffff8013e288d380 x26: 0000000000000000
[330929.313451] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.313454] x23: 0000000000000001 x22: ffff8013e2808000
[330929.313458] x21: ffff8013e3760000 x20: ffff8013e6734400
[330929.313461] x19: 0000000000000001 x18: 0000000000000400
[330929.313465] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330929.313469] x15: 0000000000000000 x14: 0000000000000000
[330929.313473] x13: 0000000000000007 x12: 0000000000000000
[330929.313477] x11: ffff000008a31880 x10: 0000000000000000
[330929.313479] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.313482] x7 : 0000000000000010 x6 : 0000000000009000
[330929.313486] x5 : 00000013b66b0000 x4 : ffff8413e0615000
[330929.313489] x3 : ffff8013e4ee8368 x2 : ffff7e004ed9a9c0
[330929.313493] x1 : 0000000000000009 x0 : ffff8013e4ee8368
[330929.313496] Call trace:
[330929.313498] nvme_queue_rq+0x660/0x9f8
[330929.313500] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.313504] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.313506] __blk_mq_run_hw_queue+0xec/0x100
[330929.313509] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.313512] blk_mq_run_hw_queue+0x3c/0x98
[330929.313514] blk_mq_run_hw_queues+0x50/0x70
[330929.313516] blk_mq_requeue_work+0x140/0x158
[330929.313520] process_one_work+0x138/0x2f0
[330929.313523] worker_thread+0x144/0x408
[330929.313524] kthread+0xf8/0x128
[330929.313528] ret_from_fork+0x10/0x18
[330929.313529] ---[ end trace 7626222de6819d68 ]---
[330929.314452] WARNING: CPU: 12 PID: 2440 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.314453] Modules linked in:
[330929.314456] CPU: 12 PID: 2440 Comm: kworker/12:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.314458] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.314460] Workqueue: kblockd blk_mq_requeue_work
[330929.314462] task: ffff8013e6b78e80 task.stack: ffff000011b30000
[330929.314465] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.314467] pc : nvme_queue_rq+0x660/0x9f8
[330929.314469] lr : nvme_queue_rq+0x59c/0x9f8
[330929.314470] sp : ffff000011b33ad0
[330929.314471] x29: ffff000011b33ad0 x28: ffff8013e288d4e0
[330929.314474] x27: ffff8013e288d380 x26: 0000000000000000
[330929.314477] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.314480] x23: 0000000000000001 x22: ffff8013e2808000
[330929.314482] x21: ffff8013e3760000 x20: ffff8013e6734400
[330929.314485] x19: 0000000000000001 x18: 0000000000000400
[330929.314488] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330929.314491] x15: 0000000000000000 x14: 0000000000000400
[330929.314494] x13: 0000000000000400 x12: 0000000000000001
[330929.314497] x11: 0000000000820560 x10: 0000000000000000
[330929.314500] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.314503] x7 : 0000000000000010 x6 : 0000000000009000
[330929.314506] x5 : 00000013b66b0000 x4 : ffff8413e0615000
[330929.314509] x3 : ffff8013e4ee8368 x2 : ffff7e004ed9a9c0
[330929.314511] x1 : 0000000000000009 x0 : ffff8013e4ee8368
[330929.314515] Call trace:
[330929.314516] nvme_queue_rq+0x660/0x9f8
[330929.314519] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.314520] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.314522] __blk_mq_run_hw_queue+0xec/0x100
[330929.314524] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.314527] blk_mq_run_hw_queue+0x3c/0x98
[330929.314528] blk_mq_run_hw_queues+0x50/0x70
[330929.314530] blk_mq_requeue_work+0x140/0x158
[330929.314532] process_one_work+0x138/0x2f0
[330929.314534] worker_thread+0x144/0x408
[330929.314536] kthread+0xf8/0x128
[330929.314537] ret_from_fork+0x10/0x18
[330929.314538] ---[ end trace 7626222de6819d69 ]---
[330929.315364] WARNING: CPU: 12 PID: 2440 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.315366] Modules linked in:
[330929.315370] CPU: 12 PID: 2440 Comm: kworker/12:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.315373] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
Jobs: 8 (f=8): [X(1),W(6),X(1),W([330929.315375] Workqueue: kblockd
blk_mq_requeue_work
2)] [9.7% done] [0KB/2035MB/0KB /[330929.315378] task: ffff8013e6b78e80
task.stack: ffff000011b30000
s] [0/2035/0 iops] [eta 00m:28s][330929.315382] pstate: 20000005 (nzCv
daif -PAN -UAO)
[330929.315385] pc : nvme_queue_rq+0x660/0x9f8
[330929.315389] lr : nvme_queue_rq+0x59c/0x9f8
[330929.315390] sp : ffff000011b33ad0
[330929.315391] x29: ffff000011b33ad0 x28: ffff8013e288d4e0
[330929.315394] x27: ffff8013e288d380 x26: 0000000000000000
[330929.315398] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.315401] x23: 0000000000000001 x22: ffff8013e2808000
[330929.315404] x21: ffff8013e3760000 x20: ffff8013e6734400
[330929.315408] x19: 0000000000000001 x18: 0000000000000400
[330929.315411] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330929.315414] x15: 0000000000000000 x14: 0000000000000400
[330929.315419] x13: 0000000000000400 x12: 0000000000000001
[330929.315422] x11: 0000000000dcb000 x10: 0000000000000000
[330929.315426] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.315429] x7 : 0000000000000010 x6 : 0000000000009000
[330929.315432] x5 : 00000013b66b0000 x4 : ffff8413e0615000
[330929.315436] x3 : ffff8013e4ee8368 x2 : ffff7e004ed9a9c0
[330929.315439] x1 : 0000000000000009 x0 : ffff8013e4ee8368
[330929.315442] Call trace:
[330929.315446] nvme_queue_rq+0x660/0x9f8
[330929.315448] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.315450] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.315453] __blk_mq_run_hw_queue+0xec/0x100
[330929.315456] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.315458] blk_mq_run_hw_queue+0x3c/0x98
[330929.315460] blk_mq_run_hw_queues+0x50/0x70
[330929.315462] blk_mq_requeue_work+0x140/0x158
[330929.315465] process_one_work+0x138/0x2f0
[330929.315467] worker_thread+0x144/0x408
[330929.315469] kthread+0xf8/0x128
[330929.315471] ret_from_fork+0x10/0x18
[330929.315473] ---[ end trace 7626222de6819d6a ]---
[330929.316402] WARNING: CPU: 12 PID: 2440 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.316403] Modules linked in:
[330929.316407] CPU: 12 PID: 2440 Comm: kworker/12:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.316407] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.316410] Workqueue: kblockd blk_mq_requeue_work
[330929.316413] task: ffff8013e6b78e80 task.stack: ffff000011b30000
[330929.316415] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.316417] pc : nvme_queue_rq+0x660/0x9f8
[330929.316418] lr : nvme_queue_rq+0x59c/0x9f8
[330929.316419] sp : ffff000011b33ad0
[330929.316420] x29: ffff000011b33ad0 x28: ffff8013e288d4e0
[330929.316423] x27: ffff8013e288d380 x26: 0000000000000000
[330929.316426] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.316429] x23: 0000000000000001 x22: ffff8013e2808000
[330929.316434] x21: ffff8013e3760000 x20: ffff8013e6734400
[330929.316437] x19: 0000000000000001 x18: 0000000000000400
[330929.316441] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330929.316444] x15: 0000000000000000 x14: 0000000000000400
[330929.316448] x13: 0000000000000400 x12: 0000000000000001
[330929.316451] x11: 000000000157efc3 x10: 0000000000000000
[330929.316454] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.316457] x7 : 0000000000000010 x6 : 0000000000009000
[330929.316460] x5 : 00000013b66b0000 x4 : ffff8413e0615000
[330929.316463] x3 : ffff8013e4ee8368 x2 : ffff7e004ed9a9c0
[330929.316467] x1 : 0000000000000009 x0 : ffff8013e4ee8368
[330929.316471] Call trace:
[330929.316472] nvme_queue_rq+0x660/0x9f8
[330929.316474] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.316478] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.316482] __blk_mq_run_hw_queue+0xec/0x100
[330929.316486] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.316487] blk_mq_run_hw_queue+0x3c/0x98
[330929.316489] blk_mq_run_hw_queues+0x50/0x70
[330929.316491] blk_mq_requeue_work+0x140/0x158
[330929.316495] process_one_work+0x138/0x2f0
[330929.316496] worker_thread+0x144/0x408
[330929.316499] kthread+0xf8/0x128
[330929.316501] ret_from_fork+0x10/0x18
[330929.316502] ---[ end trace 7626222de6819d6b ]---
[330929.317400] WARNING: CPU: 12 PID: 2440 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330929.317401] Modules linked in:
[330929.317404] CPU: 12 PID: 2440 Comm: kworker/12:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330929.317405] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330929.317408] Workqueue: kblockd blk_mq_requeue_work
[330929.317410] task: ffff8013e6b78e80 task.stack: ffff000011b30000
[330929.317411] pstate: 20000005 (nzCv daif -PAN -UAO)
[330929.317414] pc : nvme_queue_rq+0x660/0x9f8
[330929.317416] lr : nvme_queue_rq+0x59c/0x9f8
[330929.317417] sp : ffff000011b33ad0
[330929.317418] x29: ffff000011b33ad0 x28: ffff8013e288d4e0
[330929.317421] x27: ffff8013e288d380 x26: 0000000000000000
[330929.317423] x25: ffff8413e0615a00 x24: 0000000000000010
[330929.317426] x23: 0000000000000001 x22: ffff8013e2808000
[330929.317429] x21: ffff8013e3760000 x20: ffff8013e6734400
[330929.317432] x19: 0000000000000001 x18: 0000000000000400
[330929.317434] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330929.317437] x15: 0000000000000000 x14: 0000000000000001
[330929.317440] x13: 0000000000000000 x12: 0000000000000001
[330929.317443] x11: ffff000008a31880 x10: 0000000000000000
[330929.317446] x9 : 0000000000000000 x8 : 00600000000008c4
[330929.317448] x7 : 0000000000000010 x6 : 0000000000009000
[330929.317451] x5 : 00000013b66b0000 x4 : ffff8413e0615000
[330929.317455] x3 : ffff8013e4ee8368 x2 : ffff7e004ed9a9c0
[330929.317457] x1 : 0000000000000009 x0 : ffff8013e4ee8368
[330929.317460] Call trace:
[330929.317462] nvme_queue_rq+0x660/0x9f8
[330929.317464] blk_mq_dispatch_rq_list+0x1a8/0x548
[330929.317465] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330929.317467] __blk_mq_run_hw_queue+0xec/0x100
[330929.317470] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330929.317471] blk_mq_run_hw_queue+0x3c/0x98
[330929.317473] blk_mq_run_hw_queues+0x50/0x70
[330929.317475] blk_mq_requeue_work+0x140/0x158
[330929.317477] process_one_work+0x138/0x2f0
[330929.317479] worker_thread+0x144/0x408
[330929.317481] kthread+0xf8/0x128
[330929.317483] ret_from_fork+0x10/0x18
[330929.317484] ---[ end trace 7626222de6819d6c ]---
[330929.318364] print_req_error: I/O error, dev nvme0n1, sector 0
[330931.711078] Modules linked in:
[330931.714212] CPU: 27 PID: 2516 Comm: fio Tainted: G W
4.15.0-rc1-g1d739bd #18
[330931.722549] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330931.731754] task: ffff8413e65ad700 task.stack: ffff00001b680000
[330931.737748] pstate: 20000005 (nzCv daif -PAN -UAO)
[330931.742614] pc : nvme_queue_rq+0x660/0x9f8
[330931.746786] lr : nvme_queue_rq+0x59c/0x9f8
[330931.750956] sp : ffff00001b683800
[330931.754345] x29: ffff00001b683800 x28: ffff8017e6ef95e0
[330931.759735] x27: ffff8017e6ef9480 x26: 0000000000000000
[330931.765125] x25: ffff8413e0615500 x24: 0000000000000030
[330931.770515] x23: 0000000000000001 x22: ffff8013e2808000
[330931.775904] x21: ffff8013e3760000 x20: ffff8017e917bb00
[330931.781293] x19: 0000000000000003 x18: 0000000000000001
[330931.786681] x17: 000000000047c678 x16: ffff00000821d848
[330931.792070] x15: 0000000000000008 x14: 0000ffff7e235c1c
[330931.797459] x13: 000000005f9dc90 x12: 0000000000000000
[330931.802848] x11: 0000820000000000 x10: 0000000000000000
[330931.808237] x9 : 0000000000000000 x8 : 00600000000008c4
[330931.813627] x7 : 0000000000000010 x6 : 0000000000009000
[330931.819016] x5 : 0000000000000003 x4 : ffff8413e0615000
[330931.824405] x3 : ffff8413e04bbd68 x2 : ffff7e005f970c00
[330931.829794] x1 : 0000000000000009 x0 : ffff8413e04bbd68
[330931.835183] Call trace:
[330931.837705] nvme_queue_rq+0x660/0x9f8
[330931.841532] __blk_mq_try_issue_directly+0x178/0x1d8
[330931.846572] blk_mq_try_issue_directly+0x88/0xa0
[330931.851265] blk_mq_make_request+0x294/0x488
[330931.855611] generic_make_request+0xe4/0x258
[330931.859955] submit_bio+0xa4/0x1b8
[330931.863433] __blkdev_direct_IO_simple+0x120/0x298
[330931.868298] blkdev_direct_IO+0x3b4/0x3b8
[330931.872383] generic_file_direct_write+0xe8/0x188
[330931.877162] __generic_file_write_iter+0xac/0x1b0
[330931.881941] blkdev_write_iter+0x84/0xf8
[330931.885940] __vfs_write+0xb0/0x118
[330931.889504] vfs_write+0xa0/0x1b0
[330931.892894] SyS_write+0x44/0xa0
[330931.896197] el0_svc_naked+0x20/0x24
[330931.899847] ---[ end trace 7626222de6819d6d ]---
[330931.904652] WARNING: CPU: 25 PID: 2443 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330931.913424] Modules linked in:
[330931.916555] CPU: 25 PID: 2443 Comm: kworker/25:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330931.925760] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330931.934966] Workqueue: kblockd blk_mq_run_work_fn
[330931.939747] task: ffff8017e62d0000 task.stack: ffff00001b508000
[330931.945740] pstate: 20000005 (nzCv daif -PAN -UAO)
[330931.950606] pc : nvme_queue_rq+0x660/0x9f8
[330931.954777] lr : nvme_queue_rq+0x59c/0x9f8
[330931.958947] sp : ffff00001b50bb90
[330931.962336] x29: ffff00001b50bb90 x28: ffff8017e6ef9820
[330931.967724] x27: ffff8017e6ef96c0 x26: 0000000000000000
[330931.973113] x25: ffff8413e0615900 x24: 0000000000000010
[330931.978501] x23: 0000000000000001 x22: ffff8013e2808000
[330931.983889] x21: ffff8013e3760000 x20: ffff8017e917bb00
[330931.989277] x19: 0000000000000001 x18: 0000000000000400
[330931.994666] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330932.000055] x15: 0000000000000000 x14: 0000000000000400
[330932.005443] x13: 0000000000000400 x12: 0000000000000000
[330932.010831] x11: 000000000004102b x10: 0000000000000000
[330932.016219] x9 : 0000000000000000 x8 : 00600000000008c4
[330932.021608] x7 : 0000000000000010 x6 : 0000000000005000
[330932.026996] x5 : 0000000000000003 x4 : ffff8413e0615000
[330932.032385] x3 : ffff8017e89d15e0 x2 : ffff7e005f9dc800
[330932.037773] x1 : 000000000000000c x0 : ffff8017e89d15e0
[330932.043161] Call trace:
[330932.045683] nvme_queue_rq+0x660/0x9f8
[330932.049507] blk_mq_dispatch_rq_list+0x1a8/0x548
[330932.054198] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330932.059498] __blk_mq_run_hw_queue+0xec/0x100
[330932.063929] blk_mq_run_work_fn+0x50/0x68
[330932.068014] process_one_work+0x138/0x2f0
[330932.072099] worker_thread+0x144/0x408
[330932.075922] kthread+0xf8/0x128
[330932.079138] ret_from_fork+0x10/0x18
[330932.082787] ---[ end trace 7626222de6819d6e ]---
[330932.087685] WARNING: CPU: 25 PID: 2443 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330932.096455] Modules linked in:
[330932.099585] CPU: 25 PID: 2443 Comm: kworker/25:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330932.108790] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330932.117996] Workqueue: kblockd blk_mq_requeue_work
[330932.122862] task: ffff8017e62d0000 task.stack: ffff00001b508000
[330932.128856] pstate: 20000005 (nzCv daif -PAN -UAO)
[330932.133721] pc : nvme_queue_rq+0x660/0x9f8
[330932.137892] lr : nvme_queue_rq+0x59c/0x9f8
[330932.142062] sp : ffff00001b50bad0
[330932.145450] x29: ffff00001b50bad0 x28: ffff8017e6ef95e0
[330932.150838] x27: ffff8017e6ef9480 x26: 0000000000000000
Jobs: 8 (f=8): [X(1),W(6),X(1),W([330932.156226] x25: ffff8413e0615500
x24: 0000000000000030
2)] [12.9% done] [0KB/2041MB/0KB [330932.164473] x23: 0000000000000001
x22: ffff8013e2808000
/s] [0/2041/0 iops] [eta 00m:27s[330932.172721] x21: ffff8013e3760000
x20: ffff8017e917bb00
]
[330932.180880] x19: 0000000000000003 x18: 0000000000000400
[330932.186437] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330932.191825] x15: 0000000000000000 x14: 0000000000000400
[330932.197213] x13: 0000000000000400 x12: 0000000000000000
[330932.202601] x11: 000000000004102b x10: 0000000000000000
[330932.207989] x9 : 0000000000000000 x8 : 00600000000008c4
[330932.213376] x7 : 0000000000000010 x6 : 0000000000009000
[330932.218764] x5 : 00000017d5fb9000 x4 : ffff8413e0615000
[330932.224152] x3 : ffff8017e89c8568 x2 : ffff7e005f970c00
[330932.229539] x1 : 0000000000000009 x0 : ffff8017e89c8568
[330932.234927] Call trace:
[330932.237448] nvme_queue_rq+0x660/0x9f8
[330932.241272] blk_mq_dispatch_rq_list+0x1a8/0x548
[330932.245962] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330932.251261] __blk_mq_run_hw_queue+0xec/0x100
[330932.255693] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330932.260644] blk_mq_run_hw_queue+0x3c/0x98
[330932.264815] blk_mq_run_hw_queues+0x50/0x70
[330932.269073] blk_mq_requeue_work+0x140/0x158
[330932.273416] process_one_work+0x138/0x2f0
[330932.277500] worker_thread+0x144/0x408
[330932.281323] kthread+0xf8/0x128
[330932.284539] ret_from_fork+0x10/0x18
[330932.288188] ---[ end trace 7626222de6819d6f ]---
[330932.295917] WARNING: CPU: 25 PID: 2443 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330932.304687] Modules linked in:
[330932.307818] CPU: 25 PID: 2443 Comm: kworker/25:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330932.317023] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330932.326229] Workqueue: kblockd blk_mq_requeue_work
[330932.331095] task: ffff8017e62d0000 task.stack: ffff00001b508000
[330932.337088] pstate: 20000005 (nzCv daif -PAN -UAO)
[330932.341953] pc : nvme_queue_rq+0x660/0x9f8
[330932.346123] lr : nvme_queue_rq+0x59c/0x9f8
[330932.350292] sp : ffff00001b50bad0
[330932.353681] x29: ffff00001b50bad0 x28: ffff8017e6ef95e0
[330932.359069] x27: ffff8017e6ef9480 x26: 0000000000000000
[330932.364456] x25: ffff8413e0615500 x24: 0000000000000030
[330932.369844] x23: 0000000000000001 x22: ffff8013e2808000
[330932.375231] x21: ffff8013e3760000 x20: ffff8017e917bb00
[330932.380618] x19: 0000000000000003 x18: 0000000000000400
[330932.386005] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330932.391392] x15: 0000000000000000 x14: 0000000000000000
[330932.396780] x13: 0000000000000000 x12: 0000000000000000
[330932.402169] x11: ffff000008a31880 x10: 0000000000000000
[330932.407556] x9 : 0000000000000000 x8 : 00600000000008c4
[330932.412944] x7 : 0000000000000010 x6 : 0000000000009000
[330932.418332] x5 : 00000017d5fb9000 x4 : ffff8413e0615000
[330932.423719] x3 : ffff8017e89c8568 x2 : ffff7e005f970c00
[330932.429107] x1 : 0000000000000009 x0 : ffff8017e89c8568
[330932.434493] Call trace:
[330932.437015] nvme_queue_rq+0x660/0x9f8
[330932.440838] blk_mq_dispatch_rq_list+0x1a8/0x548
[330932.445529] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330932.450829] __blk_mq_run_hw_queue+0xec/0x100
[330932.455260] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330932.460211] blk_mq_run_hw_queue+0x3c/0x98
[330932.464382] blk_mq_run_hw_queues+0x50/0x70
[330932.468639] blk_mq_requeue_work+0x140/0x158
[330932.472983] process_one_work+0x138/0x2f0
[330932.477067] worker_thread+0x144/0x408
[330932.480889] kthread+0xf8/0x128
[330932.484105] ret_from_fork+0x10/0x18
[330932.487754] ---[ end trace 7626222de6819d70 ]---
[330932.495186] WARNING: CPU: 25 PID: 2443 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330932.503957] Modules linked in:
[330932.507087] CPU: 25 PID: 2443 Comm: kworker/25:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330932.516291] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330932.525497] Workqueue: kblockd blk_mq_requeue_work
[330932.530364] task: ffff8017e62d0000 task.stack: ffff00001b508000
[330932.536356] pstate: 20000005 (nzCv daif -PAN -UAO)
[330932.541221] pc : nvme_queue_rq+0x660/0x9f8
[330932.545392] lr : nvme_queue_rq+0x59c/0x9f8
[330932.549563] sp : ffff00001b50bad0
[330932.552951] x29: ffff00001b50bad0 x28: ffff8017e6ef95e0
[330932.558338] x27: ffff8017e6ef9480 x26: 0000000000000000
[330932.563725] x25: ffff8413e0615500 x24: 0000000000000030
[330932.569113] x23: 0000000000000001 x22: ffff8013e2808000
[330932.574501] x21: ffff8013e3760000 x20: ffff8017e917bb00
[330932.579888] x19: 0000000000000003 x18: 0000000000000400
[330932.585275] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330932.590663] x15: 0000000000000000 x14: 0000000000000002
[330932.596051] x13: 0000000000000000 x12: 0000000000000002
[330932.601438] x11: ffff000008a31880 x10: 0000000000000000
[330932.606826] x9 : 0000000000000000 x8 : 00600000000008c4
[330932.612214] x7 : 0000000000000010 x6 : 0000000000009000
[330932.617601] x5 : 00000017d5fb9000 x4 : ffff8413e0615000
[330932.622988] x3 : ffff8017e89c8568 x2 : ffff7e005f970c00
[330932.628376] x1 : 0000000000000009 x0 : ffff8017e89c8568
[330932.633764] Call trace:
[330932.636285] nvme_queue_rq+0x660/0x9f8
[330932.640109] blk_mq_dispatch_rq_list+0x1a8/0x548
[330932.644800] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330932.650099] __blk_mq_run_hw_queue+0xec/0x100
[330932.654530] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330932.659482] blk_mq_run_hw_queue+0x3c/0x98
[330932.663652] blk_mq_run_hw_queues+0x50/0x70
[330932.667909] blk_mq_requeue_work+0x140/0x158
[330932.672254] process_one_work+0x138/0x2f0
[330932.676337] worker_thread+0x144/0x408
[330932.680160] kthread+0xf8/0x128
[330932.683375] ret_from_fork+0x10/0x18
[330932.687024] ---[ end trace 7626222de6819d71 ]---
[330932.695046] WARNING: CPU: 25 PID: 2443 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330932.703816] Modules linked in:
[330932.706947] CPU: 25 PID: 2443 Comm: kworker/25:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330932.716151] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330932.725357] Workqueue: kblockd blk_mq_requeue_work
[330932.730222] task: ffff8017e62d0000 task.stack: ffff00001b508000
[330932.736215] pstate: 20000005 (nzCv daif -PAN -UAO)
[330932.741080] pc : nvme_queue_rq+0x660/0x9f8
[330932.745251] lr : nvme_queue_rq+0x59c/0x9f8
[330932.749420] sp : ffff00001b50bad0
[330932.752809] x29: ffff00001b50bad0 x28: ffff8017e6ef95e0
[330932.758196] x27: ffff8017e6ef9480 x26: 0000000000000000
[330932.763583] x25: ffff8413e0615500 x24: 0000000000000030
[330932.768970] x23: 0000000000000001 x22: ffff8013e2808000
[330932.774358] x21: ffff8013e3760000 x20: ffff8017e917bb00
[330932.779745] x19: 0000000000000003 x18: 0000000000000400
[330932.785132] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330932.790520] x15: 0000000000000000 x14: 0000000000000002
[330932.795908] x13: 0000000000000000 x12: 0000000000000002
[330932.801296] x11: ffff000008a31880 x10: 0000000000000000
[330932.806683] x9 : 0000000000000000 x8 : 00600000000008c4
[330932.812070] x7 : 0000000000000010 x6 : 0000000000009000
[330932.817458] x5 : 00000017d5fb9000 x4 : ffff8413e0615000
[330932.822846] x3 : ffff8017e89c8568 x2 : ffff7e005f970c00
[330932.828233] x1 : 0000000000000009 x0 : ffff8017e89c8568
[330932.833620] Call trace:
[330932.836141] nvme_queue_rq+0x660/0x9f8
[330932.839964] blk_mq_dispatch_rq_list+0x1a8/0x548
[330932.844656] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330932.849956] __blk_mq_run_hw_queue+0xec/0x100
[330932.854386] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330932.859339] blk_mq_run_hw_queue+0x3c/0x98
[330932.863509] blk_mq_run_hw_queues+0x50/0x70
[330932.867766] blk_mq_requeue_work+0x140/0x158
[330932.872111] process_one_work+0x138/0x2f0
[330932.876194] worker_thread+0x144/0x408
[330932.880017] kthread+0xf8/0x128
[330932.883233] ret_from_fork+0x10/0x18
[330932.886882] ---[ end trace 7626222de6819d72 ]---
[330932.894867] WARNING: CPU: 25 PID: 2443 at
drivers/nvme/host/pci.c:769 nvme_queue_rq+0x660/0x9f8
[330932.903638] Modules linked in:
[330932.906768] CPU: 25 PID: 2443 Comm: kworker/25:1H Tainted: G
W 4.15.0-rc1-g1d739bd #18
[330932.915972] Hardware name: Huawei TaiShan 2280 /D05, BIOS Hisilicon
D05 IT17 Nemo 2.0 RC0 11/29/2017
[330932.925178] Workqueue: kblockd blk_mq_requeue_work
[330932.930044] task: ffff8017e62d0000 task.stack: ffff00001b508000
[330932.936036] pstate: 20000005 (nzCv daif -PAN -UAO)
[330932.940901] pc : nvme_queue_rq+0x660/0x9f8
[330932.945072] lr : nvme_queue_rq+0x59c/0x9f8
[330932.949241] sp : ffff00001b50bad0
[330932.952630] x29: ffff00001b50bad0 x28: ffff8017e6ef95e0
[330932.958017] x27: ffff8017e6ef9480 x26: 0000000000000000
[330932.963406] x25: ffff8413e0615500 x24: 0000000000000030
[330932.968793] x23: 0000000000000001 x22: ffff8013e2808000
[330932.974180] x21: ffff8013e3760000 x20: ffff8017e917bb00
[330932.979567] x19: 0000000000000003 x18: 0000000000000400
[330932.984955] x17: 000000000047c2a0 x16: ffff0000080d07b8
[330932.990343] x15: 0000000000000000 x14: 0000000000000002
[330932.995730] x13: 0000000000000000 x12: 0000000000000002
[330933.001118] x11: ffff000008a31880 x10: 0000000000000000
[330933.006506] x9 : 0000000000000000 x8 : 00600000000008c4
[330933.011893] x7 : 0000000000000010 x6 : 0000000000009000
[330933.017281] x5 : 00000017d5fb9000 x4 : ffff8413e0615000
[330933.022669] x3 : ffff8017e89c8568 x2 : ffff7e005f970c00
[330933.028056] x1 : 0000000000000009 x0 : ffff8017e89c8568
[330933.033444] Call trace:
[330933.035965] nvme_queue_rq+0x660/0x9f8
[330933.039788] blk_mq_dispatch_rq_list+0x1a8/0x548
[330933.044480] blk_mq_sched_dispatch_requests+0x1ac/0x1b0
[330933.049778] __blk_mq_run_hw_queue+0xec/0x100
[330933.054209] __blk_mq_delay_run_hw_queue+0xd8/0x130
[330933.059161] blk_mq_run_hw_queue+0x3c/0x98
[330933.063332] blk_mq_run_hw_queues+0x50/0x70
[330933.067589] blk_mq_requeue_work+0x140/0x158
[330933.071933] process_one_work+0x138/0x2f0
[330933.076016] worker_thread+0x144/0x408
[330933.079838] kthread+0xf8/0x128
[330933.083054] ret_from_fork+0x10/0x18
[330933.086703] ---[ end trace 7626222de6819d73 ]---
[330933.094683] print_req_error: I/O error, dev nvme0n1, sector 0
fio: pid=2516, err=5/file:engines/sync.c:67, func=xfer,
error=Input/output error
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [14.7% done]
[0KB/2032MB/0KB /s] [0/2032/0 iops] [eta 00m:29s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [17.6% done]
[0KB/2053MB/0KB /s] [0/2053/0 iops] [eta 00m:28s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [20.6% done]
[0KB/2015MB/0KB /s] [0/2015/0 iops] [eta 00m:27s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [23.5% done]
[0KB/1982MB/0KB /s] [0/1982/0 iops] [eta 00m:26s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [26.5% done]
[0KB/1993MB/0KB /s] [0/1993/0 iops] [eta 00m:25s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [29.4% done]
[0KB/1985MB/0KB /s] [0/1985/0 iops] [eta 00m:24s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [32.4% done]
[0KB/1982MB/0KB /s] [0/1982/0 iops] [eta 00m:23s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [35.3% done]
[0KB/1996MB/0KB /s] [0/1996/0 iops] [eta 00m:22s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [38.2% done]
[0KB/1987MB/0KB /s] [0/1987/0 iops] [eta 00m:21s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [41.2% done]
[0KB/1996MB/0KB /s] [0/1996/0 iops] [eta 00m:20s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [44.1% done]
[0KB/1983MB/0KB /s] [0/1983/0 iops] [eta 00m:19s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [47.1% done]
[0KB/1993MB/0KB /s] [0/1993/0 iops] [eta 00m:18s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [50.0% done]
[0KB/1974MB/0KB /s] [0/1974/0 iops] [eta 00m:17s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [52.9% done]
[0KB/1991MB/0KB /s] [0/1991/0 iops] [eta 00m:16s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [55.9% done]
[0KB/2006MB/0KB /s] [0/2006/0 iops] [eta 00m:15s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [58.8% done]
[0KB/1980MB/0KB /s] [0/1980/0 iops] [eta 00m:14s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [61.8% done]
[0KB/1991MB/0KB /s] [0/1991/0 iops] [eta 00m:13s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [64.7% done]
[0KB/1985MB/0KB /s] [0/1985/0 iops] [eta 00m:12s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [67.6% done]
[0KB/1979MB/0KB /s] [0/1979/0 iops] [eta 00m:11s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [70.6% done]
[0KB/1981MB/0KB /s] [0/1981/0 iops] [eta 00m:10s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [73.5% done]
[0KB/1975MB/0KB /s] [0/1975/0 iops] [eta 00m:09s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [76.5% done]
[0KB/1981MB/0KB /s] [0/1981/0 iops] [eta 00m:08s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [79.4% done]
[0KB/1981MB/0KB /s] [0/1981/0 iops] [eta 00m:07s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [82.4% done]
[0KB/1983MB/0KB /s] [0/1983/0 iops] [eta 00m:06s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [85.3% done]
[0KB/1978MB/0KB /s] [0/1978/0 iops] [eta 00m:05s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [88.2% done]
[0KB/1971MB/0KB /s] [0/1971/0 iops] [eta 00m:04s]
Jobs: 7 (f=7): [X(1),W(2),X(1),W(3),X(1),W(2)] [91.2% done]
[0KB/1984MB/0KB /s] [0/1984/0 iops] [eta 00m:03s]
write: (groupid=0, jobs=10): err= 5 (file:engines/sync.c:67, func=xfer,
error=Input/output error): pid=2513: Tue May 17 07:49:49 2016
write: io=59914MB, bw=1996.9MB/s, iops=1996, runt= 30004msec
clat (usec): min=484, max=18385, avg=3368.54, stdev=792.34
lat (usec): min=513, max=19692, avg=3502.80, stdev=793.56
clat percentiles (usec):
| 1.00th=[ 2640], 5.00th=[ 2736], 10.00th=[ 2736], 20.00th=[ 2864],
| 30.00th=[ 2960], 40.00th=[ 3088], 50.00th=[ 3216], 60.00th=[ 3344],
| 70.00th=[ 3472], 80.00th=[ 3696], 90.00th=[ 4016], 95.00th=[ 4448],
| 99.00th=[ 7456], 99.50th=[ 8384], 99.90th=[ 9792], 99.95th=[10048],
| 99.99th=[10688]
bw (KB /s): min=281498, max=323584, per=14.28%, avg=292075.33,
stdev=4460.87
lat (usec) : 500=0.01%, 750=0.09%, 1000=0.02%
lat (msec) : 2=0.09%, 4=89.24%, 10=10.51%, 20=0.05%
cpu : usr=2.61%, sys=1.68%, ctx=59965, majf=0, minf=730
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=59917/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: io=59914MB, aggrb=1996.9MB/s, minb=1996.9MB/s,
maxb=1996.9MB/s, mint=30004msec, maxt=30004msec
Disk stats (read/write):
nvme0n1: ios=0/119093, merge=0/0, ticks=0/389736, in_queue=355852,
util=99.44%
Thanks
FangJian
.
More information about the Linux-nvme
mailing list