[ 3350.987102] systemd-journald[660]: Sent WATCHDOG=1 notification.
[ 3388.701949] nvme nvme1: resetting controller
[ 3389.349332] nvme nvme1: creating 1 I/O queues.
[ 3390.465439] nvme nvme1: resetting controller
[ 3391.123563] nvme nvme1: creating 1 I/O queues.
[ 3392.233235] nvme nvme1: resetting controller
[ 3392.894457] nvme nvme1: creating 1 I/O queues.
[ 3394.010517] nvme nvme1: resetting controller
[ 3394.668662] nvme nvme1: creating 1 I/O queues.
[ 3395.771915] nvme nvme1: resetting controller
[ 3396.427127] nvme nvme1: creating 1 I/O queues.
[ 3397.543184] nvme nvme1: resetting controller
[ 3398.087327] nvme nvme1: Identify Controller failed (16386)
[ 3398.100646] nvme nvme1: Reconnecting in 10 seconds...
[ 3408.783020] nvme nvme1: creating 1 I/O queues.
[ 3408.876071] nvme nvme1: Successfully reconnected (2 attempts)
[ 3409.219786] nvme nvme1: resetting controller
[ 3409.871732] nvme nvme1: creating 1 I/O queues.
[ 3410.987901] nvme nvme1: resetting controller
[ 3411.643449] nvme nvme1: creating 1 I/O queues.
[ 3412.771824] nvme nvme1: resetting controller
[ 3413.417846] nvme nvme1: creating 1 I/O queues.
[ 3414.521296] nvme nvme1: resetting controller
[ 3415.169188] nvme nvme1: creating 1 I/O queues.
[ 3416.272676] nvme nvme1: resetting controller
[ 3416.929756] nvme nvme1: creating 1 I/O queues.
[ 3418.045403] nvme nvme1: resetting controller
[ 3418.690977] nvme nvme1: creating 1 I/O queues.
[ 3419.794368] nvme nvme1: resetting controller
[ 3420.442395] nvme nvme1: creating 1 I/O queues.
[ 3421.557885] nvme nvme1: resetting controller
[ 3422.217794] nvme nvme1: creating 1 I/O queues.
[ 3423.321255] nvme nvme1: resetting controller
[ 3423.336537] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2041): DESTROY_MKEY(0x202) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3423.338812] mlx5_1:clean_mr:1501:(pid 2041): failed to destroy mkey 0x18a644 (-22)
[ 3423.421866] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2041): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3423.425407] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2041): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3423.427903] ------------[ cut here ]------------
[ 3423.429175] failed to drain recv queue: -22
[ 3423.430467] WARNING: CPU: 10 PID: 2041 at drivers/infiniband/core/verbs.c:2199 __ib_drain_rq+0x15e/0x190 [ib_core]
[ 3423.431738] Modules linked in: fuse ib_iser iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ses sr_mod enclosure cdrom vfat fat intel_rapl sb_edac ftdi_sio cp210x x86_pkg_temp_thermal intel_powerclamp coretemp mei_me lpc_ich ioatdma shpchp mei sg ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler kvm_intel kvm acpi_power_meter acpi_pad irqbypass netconsole nvme_rdma nvme_fabrics nvme nvme_core ib_umad ib_ucm rdma_ucm ib_uverbs rdma_cm iw_cm ib_cm nfsd auth_rpcgss mlx5_ib nfs_acl lockd ib_core grace sunrpc ext4 mbcache jbd2 btrfs zstd_decompress zstd_compress xxhash raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear uas usb_storage sd_mod crct10dif_pclmul ast crc32_pclmul ttm crc32c_intel drm_kms_helper ghash_clmulni_intel syscopyarea
[ 3423.440531] pcbc sysfillrect sysimgblt fb_sys_fops mlx5_core igb ahci aesni_intel devlink dca drm libahci mpt3sas crypto_simd glue_helper i2c_algo_bit ptp raid_class cryptd wmi libata pps_core scsi_transport_sas i2c_core sha512_ssse3(E) sha512_generic(E)
[ 3423.444625] CPU: 10 PID: 2041 Comm: kworker/u40:3 Tainted: G E 4.15.1+ #3
[ 3423.446045] Hardware name: Supermicro SSG-5028R-E1CR12L-CE010/X10SRH-CLN4F, BIOS 1.0c 10/02/2015
[ 3423.447494] Workqueue: nvme-wq nvme_rdma_reset_ctrl_work [nvme_rdma]
[ 3423.448982] RIP: 0010:__ib_drain_rq+0x15e/0x190 [ib_core]
[ 3423.450446] RSP: 0018:ffffa3d1876a3d48 EFLAGS: 00010282
[ 3423.451918] RAX: 0000000000000000 RBX: ffff8d2b2a7c0400 RCX: 0000000000000000
[ 3423.453406] RDX: ffff8d2b7f29e6d8 RSI: ffff8d2b7f2968f8 RDI: ffff8d2b7f2968f8
[ 3423.454901] RBP: ffffa3d1876a3d70 R08: 00000000000005a0 R09: 0000000000000000
[ 3423.456406] R10: ffffa3d1876a3b80 R11: 000000002b300059 R12: ffff8d2b2a83dc00
[ 3423.457912] R13: 0000000000000000 R14: ffff8d2b6d20c658 R15: ffff8d2b6d20c660
[ 3423.459418] FS: 0000000000000000(0000) GS:ffff8d2b7f280000(0000) knlGS:0000000000000000
[ 3423.460935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3423.462454] CR2: 00007f7e009472a0 CR3: 000000089e00a002 CR4: 00000000001606e0
[ 3423.464021] Call Trace:
[ 3423.465572] ? ib_sg_to_pages+0x1a0/0x1a0 [ib_core]
[ 3423.467126] nvme_rdma_destroy_admin_queue+0x14/0x70 [nvme_rdma]
[ 3423.468698] nvme_rdma_reset_ctrl_work+0x2c/0xb0 [nvme_rdma]
[ 3423.470280] process_one_work+0x198/0x370
[ 3423.471858] worker_thread+0x1cd/0x390
[ 3423.473456] ? process_one_work+0x370/0x370
[ 3423.475045] kthread+0x111/0x130
[ 3423.476623] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3423.478221] ret_from_fork+0x35/0x40
[ 3423.479814] Code: eb ad 48 8d 7d 08 e8 22 72 f0 f8 eb a2 80 3d 96 05 03 00 00 75 99 89 c6 48 c7 c7 98 37 86 c0 c6 05 84 05 03 00 01 e8 b2 51 84 f8 <0f> ff eb 80 89 c6 48 c7 c7 98 37 86 c0 c6 05 6a 05 03 00 01 e8
[ 3423.483173] ---[ end trace 7732566bc069addf ]---
[ 3423.486978] systemd-journald[660]: Compressed data object 805 -> 580 using XZ
[ 3424.049765] nvme nvme1: creating 1 I/O queues.
[ 3425.177447] nvme nvme1: resetting controller
[ 3425.193591] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 299): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3425.279695] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 299): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3425.284337] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 299): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3425.843436] nvme nvme1: creating 1 I/O queues.
[ 3426.948104] nvme nvme1: resetting controller
[ 3426.967016] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 299): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3427.052577] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 299): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3427.057209] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 299): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3427.615625] nvme nvme1: creating 1 I/O queues.
[ 3428.718647] nvme nvme1: resetting controller
[ 3428.737964] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2237): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3428.823503] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2237): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3428.828470] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2237): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3429.190471] systemd-journald[660]: Sent WATCHDOG=1 notification.
[ 3429.392519] nvme nvme1: creating 1 I/O queues.
[ 3430.508357] nvme nvme1: resetting controller
[ 3430.525687] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2237): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3430.613384] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2237): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3430.618657] mlx5_core 0000:08:00.1: mlx5_cmd_check:710:(pid 2237): 2ERR_QP(0x507) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5)
[ 3431.176931] nvme nvme1: creating 1 I/O queues.
[ 3432.280515] nvme nvme1: resetting controller
[ 3432.299578] BUG: unable to handle kernel paging request at 00000000001f400c
[ 3432.301746] IP: kmem_cache_alloc_trace+0x99/0x1a0
[ 3432.303886] PGD 0 P4D 0
[ 3432.306011] Oops: 0000 [#1] SMP PTI
[ 3432.308138] Modules linked in: fuse ib_iser iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ses sr_mod enclosure cdrom vfat fat intel_rapl sb_edac ftdi_sio cp210x x86_pkg_temp_thermal intel_powerclamp coretemp mei_me lpc_ich ioatdma shpchp mei sg ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler kvm_intel kvm acpi_power_meter acpi_pad irqbypass netconsole nvme_rdma nvme_fabrics nvme nvme_core ib_umad ib_ucm rdma_ucm ib_uverbs rdma_cm iw_cm ib_cm nfsd auth_rpcgss mlx5_ib nfs_acl lockd ib_core grace sunrpc ext4 mbcache jbd2 btrfs zstd_decompress zstd_compress xxhash raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear uas usb_storage sd_mod crct10dif_pclmul ast crc32_pclmul ttm crc32c_intel drm_kms_helper ghash_clmulni_intel syscopyarea
[ 3432.324252] pcbc sysfillrect sysimgblt fb_sys_fops mlx5_core igb ahci aesni_intel devlink dca drm libahci mpt3sas crypto_simd glue_helper i2c_algo_bit ptp raid_class cryptd wmi libata pps_core scsi_transport_sas i2c_core sha512_ssse3(E) sha512_generic(E)
[ 3432.331483] CPU: 16 PID: 2237 Comm: kworker/u40:4 Tainted: G W E 4.15.1+ #3
[ 3432.333910] Hardware name: Supermicro SSG-5028R-E1CR12L-CE010/X10SRH-CLN4F, BIOS 1.0c 10/02/2015
[ 3432.336327] Workqueue: nvme-wq nvme_rdma_reset_ctrl_work [nvme_rdma]
[ 3432.338713] RIP: 0010:kmem_cache_alloc_trace+0x99/0x1a0
[ 3432.341055] RSP: 0018:ffffa3d187fefa90 EFLAGS: 00010206
[ 3432.343355] RAX: 0000000000000000 RBX: 00000000001f400c RCX: 00000000000011a8
[ 3432.345637] RDX: 00000000000011a7 RSI: 00000000014080c0 RDI: 0000000000025800
[ 3432.347908] RBP: ffff8d2b66071b80 R08: ffff8d2b7f425800 R09: 0000000000000000
[ 3432.350187] R10: 0000000000000000 R11: 0000000000000006 R12: 00000000014080c0
[ 3432.352435] R13: 0000000000000038 R14: ffff8d0d07c07780 R15: ffff8d0d07c07780
[ 3432.354644] FS: 0000000000000000(0000) GS:ffff8d2b7f400000(0000) knlGS:0000000000000000
[ 3432.356828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3432.358973] CR2: 00000000001f400c CR3: 000000089e00a005 CR4: 00000000001606e0
[ 3432.361097] Call Trace:
[ 3432.363191] ? mlx5_alloc_cmd_msg+0x45/0x210 [mlx5_core]
[ 3432.365292] mlx5_alloc_cmd_msg+0x45/0x210 [mlx5_core]
[ 3432.367389] ? kfree+0x95/0x180
[ 3432.369449] cmd_exec+0x31c/0x950 [mlx5_core]
[ 3432.371465] ? pick_next_task_fair+0x14f/0x5f0
[ 3432.373447] mlx5_cmd_exec+0x1e/0x40 [mlx5_core]
[ 3432.375394] mlx5_core_qp_modify+0xf6/0x2e0 [mlx5_core]
[ 3432.377291] ? _cond_resched+0x15/0x40
[ 3432.379147] __mlx5_ib_modify_qp+0x98b/0xa00 [mlx5_ib]
[ 3432.381001] mlx5_ib_modify_qp+0x23b/0x430 [mlx5_ib]
[ 3432.382850] ib_modify_qp_with_udata+0x32/0x80 [ib_core]
[ 3432.384709] __ib_drain_rq+0xb5/0x190 [ib_core]
[ 3432.386525] ? ib_sg_to_pages+0x1a0/0x1a0 [ib_core]
[ 3432.388295] nvme_rdma_destroy_io_queues+0x3b/0xc0 [nvme_rdma]
[ 3432.390036] nvme_rdma_shutdown_ctrl+0x60/0xc0 [nvme_rdma]
[ 3432.391736] nvme_rdma_reset_ctrl_work+0x2c/0xb0 [nvme_rdma]
[ 3432.393395] process_one_work+0x198/0x370
[ 3432.395009] worker_thread+0x1cd/0x390
[ 3432.396609] ? process_one_work+0x370/0x370
[ 3432.398217] kthread+0x111/0x130
[ 3432.399804] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3432.401389] ? do_group_exit+0x3a/0xa0
[ 3432.402975] ret_from_fork+0x35/0x40
[ 3432.404552] Code: 00 00 00 49 63 47 20 49 8b 3f 48 8d 4a 01 48 8b 5c 05 00 48 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 ba 48 85 db 74 0b 49 63 47 20 <48> 8b 04 03 0f 18 08 41 f7 c4 00 80 00 00 0f 85 d4 00 00 00 e9
[ 3432.407791] RIP: kmem_cache_alloc_trace+0x99/0x1a0 RSP: ffffa3d187fefa90
[ 3432.409352] CR2: 00000000001f400c
[ 3432.410879] ---[ end trace 7732566bc069ade0 ]---
[ 3432.418508] Kernel panic - not syncing: Fatal exception
[ 3432.419905] Kernel Offset: 0x38000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 3432.427151] ---[ end Kernel panic - not syncing: Fatal exception
[ 3432.428526] WARNING: CPU: 16 PID: 2237 at kernel/sched/core.c:1188 set_task_cpu+0x180/0x190
[ 3432.429844] Modules linked in: fuse ib_iser iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ses sr_mod enclosure cdrom vfat fat intel_rapl sb_edac ftdi_sio cp210x x86_pkg_temp_thermal intel_powerclamp coretemp mei_me lpc_ich ioatdma shpchp mei sg ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler kvm_intel kvm acpi_power_meter acpi_pad irqbypass netconsole nvme_rdma nvme_fabrics nvme nvme_core ib_umad ib_ucm rdma_ucm ib_uverbs rdma_cm iw_cm ib_cm nfsd auth_rpcgss mlx5_ib nfs_acl lockd ib_core grace sunrpc ext4 mbcache jbd2 btrfs zstd_decompress zstd_compress xxhash raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear uas usb_storage sd_mod crct10dif_pclmul ast crc32_pclmul ttm crc32c_intel drm_kms_helper ghash_clmulni_intel syscopyarea
[ 3432.439619] pcbc sysfillrect sysimgblt fb_sys_fops mlx5_core igb ahci aesni_intel devlink dca drm libahci mpt3sas crypto_simd glue_helper i2c_algo_bit ptp raid_class cryptd wmi libata pps_core scsi_transport_sas i2c_core sha512_ssse3(E) sha512_generic(E)
[ 3432.444228] CPU: 16 PID: 2237 Comm: kworker/u40:4 Tainted: G D W E 4.15.1+ #3
[ 3432.445815] Hardware name: Supermicro SSG-5028R-E1CR12L-CE010/X10SRH-CLN4F, BIOS 1.0c 10/02/2015
[ 3432.447417] Workqueue: nvme-wq nvme_rdma_reset_ctrl_work [nvme_rdma]
[ 3432.449037] RIP: 0010:set_task_cpu+0x180/0x190
[ 3432.450619] RSP: 0018:ffff8d2b7f403d68 EFLAGS: 00010006
[ 3432.452199] RAX: 0000000000000200 RBX: ffff8d2afa7a1740 RCX: 0000000000000000
[ 3432.453787] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8d2afa7a1740
[ 3432.455375] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000000fffff
[ 3432.456960] R10: 0000000000000010 R11: 0000000000000000 R12: ffff8d2afa7a22dc
[ 3432.458541] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000022280
[ 3432.460138] FS: 0000000000000000(0000) GS:ffff8d2b7f400000(0000) knlGS:0000000000000000
[ 3432.461730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3432.463326] CR2: 00000000001f400c CR3: 000000089e00a005 CR4: 00000000001606e0
[ 3432.464930] Call Trace:
[ 3432.466523]
[ 3432.468103] try_to_wake_up+0x157/0x470
[ 3432.469687] autoremove_wake_function+0x11/0x50
[ 3432.471290] __wake_up_common+0x71/0x170
[ 3432.472869] __wake_up_common_lock+0x7c/0xc0
[ 3432.474437] irq_work_run_list+0x47/0x70
[ 3432.475996] ? tick_sched_do_timer+0x60/0x60
[ 3432.477554] update_process_times+0x3b/0x50
[ 3432.479104] tick_sched_handle+0x22/0x70
[ 3432.480676] tick_sched_timer+0x34/0x70
[ 3432.482219] __hrtimer_run_queues+0xeb/0x230
[ 3432.483771] hrtimer_interrupt+0xa6/0x1f0
[ 3432.485315] smp_apic_timer_interrupt+0x56/0x110
[ 3432.486855] apic_timer_interrupt+0xa2/0xb0
[ 3432.488397]
[ 3432.489897] RIP: 0010:panic+0x1f2/0x232
[ 3432.491370] RSP: 0018:ffffa3d187fef840 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff11
[ 3432.492879] RAX: 0000000000000034 RBX: 0000000000000000 RCX: 0000000000000006
[ 3432.494362] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff8d2b7f4168f0
[ 3432.495841] RBP: ffffa3d187fef8b8 R08: 0000000000000606 R09: 0000000000000000
[ 3432.497319] R10: 0000000000000008 R11: 000000002b30006e R12: 0000000000000000
[ 3432.498772] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000001
[ 3432.500187] oops_end+0xb8/0xc0
[ 3432.501598] no_context+0x180/0x400
[ 3432.502945] ? account_entity_dequeue+0xa4/0xd0
[ 3432.504262] __do_page_fault+0xbe/0x4c0
[ 3432.505540] ? debug_object_assert_init+0xce/0x180
[ 3432.506782] do_page_fault+0x32/0x110
[ 3432.507985] page_fault+0x2c/0x60
[ 3432.509137] RIP: 0010:kmem_cache_alloc_trace+0x99/0x1a0
[ 3432.510265] RSP: 0018:ffffa3d187fefa90 EFLAGS: 00010206
[ 3432.511354] RAX: 0000000000000000 RBX: 00000000001f400c RCX: 00000000000011a8
[ 3432.512446] RDX: 00000000000011a7 RSI: 00000000014080c0 RDI: 0000000000025800
[ 3432.513473] RBP: ffff8d2b66071b80 R08: ffff8d2b7f425800 R09: 0000000000000000
[ 3432.514488] R10: 0000000000000000 R11: 0000000000000006 R12: 00000000014080c0
[ 3432.515486] R13: 0000000000000038 R14: ffff8d0d07c07780 R15: ffff8d0d07c07780
[ 3432.516481] ? mlx5_alloc_cmd_msg+0x45/0x210 [mlx5_core]
[ 3432.517467] mlx5_alloc_cmd_msg+0x45/0x210 [mlx5_core]
[ 3432.518450] ? kfree+0x95/0x180
[ 3432.519422] cmd_exec+0x31c/0x950 [mlx5_core]
[ 3432.520386] ? pick_next_task_fair+0x14f/0x5f0
[ 3432.521363] mlx5_cmd_exec+0x1e/0x40 [mlx5_core]
[ 3432.522373] mlx5_core_qp_modify+0xf6/0x2e0 [mlx5_core]
[ 3432.523354] ? _cond_resched+0x15/0x40
[ 3432.524327] __mlx5_ib_modify_qp+0x98b/0xa00 [mlx5_ib]
[ 3432.525293] mlx5_ib_modify_qp+0x23b/0x430 [mlx5_ib]
[ 3432.526258] ib_modify_qp_with_udata+0x32/0x80 [ib_core]
[ 3432.527208] __ib_drain_rq+0xb5/0x190 [ib_core]
[ 3432.528168] ? ib_sg_to_pages+0x1a0/0x1a0 [ib_core]
[ 3432.529129] nvme_rdma_destroy_io_queues+0x3b/0xc0 [nvme_rdma]
[ 3432.530103] nvme_rdma_shutdown_ctrl+0x60/0xc0 [nvme_rdma]
[ 3432.531074] nvme_rdma_reset_ctrl_work+0x2c/0xb0 [nvme_rdma]
[ 3432.532059] process_one_work+0x198/0x370
[ 3432.533041] worker_thread+0x1cd/0x390
[ 3432.534042] ? process_one_work+0x370/0x370
[ 3432.535022] kthread+0x111/0x130
[ 3432.535996] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3432.536990] ? do_group_exit+0x3a/0xa0
[ 3432.537980] ret_from_fork+0x35/0x40
[ 3432.538964] Code: ff 80 8b 9c 08 00 00 04 e9 2a ff ff ff 0f ff e9 cb fe ff ff f7 83 84 00 00 00 fd ff ff ff 0f 84 d5 fe ff ff 0f ff e9 ce fe ff ff <0f> ff e9 d8 fe ff ff 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
[ 3432.541115] ---[ end trace 7732566bc069ade1 ]---
[ 3432.542190] ------------[ cut here ]------------
[ 3432.543264] sched: Unexpected reschedule of offline CPU#0!
[ 3432.544364] WARNING: CPU: 16 PID: 2237 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x31/0x40
[ 3432.545466] Modules linked in: fuse ib_iser iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ses sr_mod enclosure cdrom vfat fat intel_rapl sb_edac ftdi_sio cp210x x86_pkg_temp_thermal intel_powerclamp coretemp mei_me lpc_ich ioatdma shpchp mei sg ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler kvm_intel kvm acpi_power_meter acpi_pad irqbypass netconsole nvme_rdma nvme_fabrics nvme nvme_core ib_umad ib_ucm rdma_ucm ib_uverbs rdma_cm iw_cm ib_cm nfsd auth_rpcgss mlx5_ib nfs_acl lockd ib_core grace sunrpc ext4 mbcache jbd2 btrfs zstd_decompress zstd_compress xxhash raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear uas usb_storage sd_mod crct10dif_pclmul ast crc32_pclmul ttm crc32c_intel drm_kms_helper ghash_clmulni_intel syscopyarea
[ 3432.554288] pcbc sysfillrect sysimgblt fb_sys_fops mlx5_core igb ahci aesni_intel devlink dca drm libahci mpt3sas crypto_simd glue_helper i2c_algo_bit ptp raid_class cryptd wmi libata pps_core scsi_transport_sas i2c_core sha512_ssse3(E) sha512_generic(E)
[ 3432.558615] CPU: 16 PID: 2237 Comm: kworker/u40:4 Tainted: G D W E 4.15.1+ #3
[ 3432.560111] Hardware name: Supermicro SSG-5028R-E1CR12L-CE010/X10SRH-CLN4F, BIOS 1.0c 10/02/2015
[ 3432.561629] Workqueue: nvme-wq nvme_rdma_reset_ctrl_work [nvme_rdma]
[ 3432.563159] RIP: 0010:native_smp_send_reschedule+0x31/0x40
[ 3432.564702] RSP: 0018:ffff8d2b7f403d50 EFLAGS: 00010086
[ 3432.566229] RAX: 0000000000000000 RBX: ffff8d2b7f022280 RCX: 0000000000000006
[ 3432.567771] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff8d2b7f4168f0
[ 3432.569314] RBP: ffff8d2b7f022280 R08: 0000000000000656 R09: 0000000000000000
[ 3432.570857] R10: afb504000afb5041 R11: 000000002b300068 R12: ffff8d2afa7a1740
[ 3432.572393] R13: ffff8d2b7f403da0 R14: 0000000000000046 R15: 0000000000022280
[ 3432.573930] FS: 0000000000000000(0000) GS:ffff8d2b7f400000(0000) knlGS:0000000000000000
[ 3432.575600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3432.577178] CR2: 00000000001f400c CR3: 000000089e00a005 CR4: 00000000001606e0
[ 3432.578752] Call Trace:
[ 3432.580316]
[ 3432.581869] check_preempt_curr+0x78/0x90
[ 3432.583426] ttwu_do_wakeup+0x19/0x140
[ 3432.584978] try_to_wake_up+0x1d4/0x470
[ 3432.586541] autoremove_wake_function+0x11/0x50
[ 3432.588088] __wake_up_common+0x71/0x170
[ 3432.589627] __wake_up_common_lock+0x7c/0xc0
[ 3432.591158] irq_work_run_list+0x47/0x70
[ 3432.592680] ? tick_sched_do_timer+0x60/0x60
[ 3432.594196] update_process_times+0x3b/0x50
[ 3432.595711] tick_sched_handle+0x22/0x70
[ 3432.597238] tick_sched_timer+0x34/0x70
[ 3432.598738] __hrtimer_run_queues+0xeb/0x230
[ 3432.600238] hrtimer_interrupt+0xa6/0x1f0
[ 3432.601738] smp_apic_timer_interrupt+0x56/0x110
[ 3432.603239] apic_timer_interrupt+0xa2/0xb0
[ 3432.604733]
[ 3432.606221] RIP: 0010:panic+0x1f2/0x232
[ 3432.607703] RSP: 0018:ffffa3d187fef840 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff11
[ 3432.609168] RAX: 0000000000000034 RBX: 0000000000000000 RCX: 0000000000000006
[ 3432.610620] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff8d2b7f4168f0
[ 3432.612073] RBP: ffffa3d187fef8b8 R08: 0000000000000606 R09: 0000000000000000
[ 3432.613527] R10: 0000000000000008 R11: 000000002b30006e R12: 0000000000000000
[ 3432.614952] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000001
[ 3432.616341] oops_end+0xb8/0xc0
[ 3432.617679] no_context+0x180/0x400
[ 3432.618997] ? account_entity_dequeue+0xa4/0xd0
[ 3432.620260] __do_page_fault+0xbe/0x4c0
[ 3432.621483] ? debug_object_assert_init+0xce/0x180
[ 3432.622674] do_page_fault+0x32/0x110
[ 3432.623822] page_fault+0x2c/0x60
[ 3432.624927] RIP: 0010:kmem_cache_alloc_trace+0x99/0x1a0
[ 3432.626002] RSP: 0018:ffffa3d187fefa90 EFLAGS: 00010206
[ 3432.627041] RAX: 0000000000000000 RBX: 00000000001f400c RCX: 00000000000011a8
[ 3432.628105] RDX: 00000000000011a7 RSI: 00000000014080c0 RDI: 0000000000025800
[ 3432.629121] RBP: ffff8d2b66071b80 R08: ffff8d2b7f425800 R09: 0000000000000000
[ 3432.630120] R10: 0000000000000000 R11: 0000000000000006 R12: 00000000014080c0
[ 3432.631117] R13: 0000000000000038 R14: ffff8d0d07c07780 R15: ffff8d0d07c07780
[ 3432.632130] ? mlx5_alloc_cmd_msg+0x45/0x210 [mlx5_core]
[ 3432.633133] mlx5_alloc_cmd_msg+0x45/0x210 [mlx5_core]
[ 3432.634127] ? kfree+0x95/0x180
[ 3432.635118] cmd_exec+0x31c/0x950 [mlx5_core]
[ 3432.636098] ? pick_next_task_fair+0x14f/0x5f0
[ 3432.637082] mlx5_cmd_exec+0x1e/0x40 [mlx5_core]
[ 3432.638053] mlx5_core_qp_modify+0xf6/0x2e0 [mlx5_core]
[ 3432.639022] ? _cond_resched+0x15/0x40
[ 3432.639959] __mlx5_ib_modify_qp+0x98b/0xa00 [mlx5_ib]
[ 3432.640891] mlx5_ib_modify_qp+0x23b/0x430 [mlx5_ib]
[ 3432.641838] ib_modify_qp_with_udata+0x32/0x80 [ib_core]
[ 3432.642791] __ib_drain_rq+0xb5/0x190 [ib_core]
[ 3432.643747] ? ib_sg_to_pages+0x1a0/0x1a0 [ib_core]
[ 3432.644710] nvme_rdma_destroy_io_queues+0x3b/0xc0 [nvme_rdma]
[ 3432.645682] nvme_rdma_shutdown_ctrl+0x60/0xc0 [nvme_rdma]
[ 3432.646655] nvme_rdma_reset_ctrl_work+0x2c/0xb0 [nvme_rdma]
[ 3432.647638] process_one_work+0x198/0x370
[ 3432.648618] worker_thread+0x1cd/0x390
[ 3432.649620] ? process_one_work+0x370/0x370
[ 3432.650599] kthread+0x111/0x130
[ 3432.651576] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3432.652572] ? do_group_exit+0x3a/0xa0
[ 3432.653566] ret_from_fork+0x35/0x40
[ 3432.654553] Code: f8 48 0f a3 05 61 91 1c 01 73 12 48 8b 05 d8 89 eb 00 be fd 00 00 00 48 8b 40 30 ff e0 89 fe 48 c7 c7 b8 bc e4 b9 e8 0f ae 03 00 <0f> ff c3 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53
[ 3432.656712] ---[ end trace 7732566bc069ade2 ]---