BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 37868055...

Tue Nov 11 11:40:12 PST 2025

On Tue, Nov 11, 2025, at 1:25 PM, Justin Piszcz wrote:

> Absolutely correct!!  Luckily, I do have remote syslog enabled and
> captured the errors that were not logged to persistent storage, do the
> errors below point to an NVME firmware issue?
>
> "2025-11-10 01:42:51","notice","user","","machine1","[4043499.402974]
> BTRFS info (device nvme1n1p2): scrub: started on devid 2"
> "2025-11-10 01:42:51","notice","user","","machine1","[4043499.403000]
> BTRFS info (device nvme1n1p2): scrub: started on devid 1"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.683686]
> nvme nvme0: I/O tag 2 (a002) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.684742]
> nvme nvme0: I/O tag 3 (c003) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.685778]
> nvme nvme0: I/O tag 4 (1004) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.686802]
> nvme nvme0: I/O tag 5 (c005) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.712274]
> nvme nvme0: I/O tag 6 (0006) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.712483]
> nvme nvme0: I/O tag 7 (4007) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.712698]
> nvme nvme0: I/O tag 8 (3008) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.712906]
> nvme nvme0: I/O tag 9 (5009) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:53","notice","user","","machine1","[4043621.419061]
> nvme nvme0: I/O tag 2 (a002) opcode 0x2 (I/O Cmd) QID 2 timeout, reset
> controller"

All of those read requests were dropped by the nvme controller and then the kernel nvme driver reset the conroller which appears to not work...

> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.821452]
> nvme nvme0: Device not ready; aborting reset, CSTS=0x1"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.850059]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.850351]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.850630]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.850903]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.851173]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.851440]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.851705]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.851971]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:34","notice","user","","machine1","[4043722.877036]
> nvme nvme0: Device not ready; aborting reset, CSTS=0x1"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.941289]
> pcieport 0000:00:1b.4: AER: Multiple Uncorrectable (Non-Fatal) error
> message received from 0000:06:00.0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.942254]
> nvme nvme0: Disabling device after reset failure: -19"

And now it's going to drop writes too because the device is disabled.

> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969046]
> I/O error, dev nvme0n1, sector 86788096 op 0x1:(WRITE) flags 0x100000
> phys_seg 1 prio class 2"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969083]
> I/O error, dev nvme0n1, sector 464581888 op 0x0:(READ) flags 0x4000
> phys_seg 3 prio class 3"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969128]
> I/O error, dev nvme0n1, sector 45027984 op 0x1:(WRITE) flags 0x4000800
> phys_seg 1 prio class 2"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969127]
> I/O error, dev nvme0n1, sector 117882144 op 0x1:(WRITE) flags 0x1800
> phys_seg 8 prio class 2"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969146]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 2, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969146]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 2, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969163]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 4, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969160]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 3, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969167]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 5, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969167]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 6, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969179]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 7, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969179]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 8, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969184]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 9, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969187]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 10, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.972280]
> BTRFS warning (device nvme1n1p2): lost super block write due to IO
> error on /dev/nvme0n1p2 (-5)"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.972938]
> BTRFS error (device nvme1n1p2): fixed up error at logical 782844493824
> on dev /dev/nvme0n1p2 physical 237354287104"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.972957]
> BTRFS error (device nvme1n1p2): fixed up error at logical 782844821504
> on dev /dev/nvme0n1p2 physical 237354614784"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.972960]
> BTRFS error (device nvme1n1p2): fixed up error at logical 782844821504
> on dev /dev/nvme0n1p2 physical 237354614784"

> Got it, I was running 4B2QJXD7 firmware on both drives when this issue
> occurred.  Recently, Samsung released a new NVME F/W update 7B2QJXD7,
> which they note "address(es) an intermittent non-recognition and blue
> screen issue."  I've flashed the drives to 7B2QJXD7 and will see if
> this issue recurs.

That's probably the proper fix. The only other thing I can think of is there's a quirk for this drive that the kernel doesn't yet have, and therefore see if kernel 6.18-rc5 behaves better. 

-- 
Chris Murphy