[BUG] I/O timeouts and system freezes on Kingston A2000 NVME with BCACHEFS
Mia Kanashi
chad at redpilled.dev
Fri Jan 19 04:25:58 PST 2024
This issue was originally reported here:
https://github.com/koverstreet/bcachefs/issues/628
Transferring large amounts of files to the bcachefs from the btrfs
causes I/O timeouts and freezes the whole system. This doesn't seem to
be related to the btrfs, but rather to the heavy I/O on the drive, as it
happens without btrfs being mounted. Transferring the files to the HDD,
and then from it to the bcachefs on the NVME sometimes doesn't make the
problem occur.
The problem only happens on the bcachefs, not on btrfs or ext4. It
doesn't happen on the HDD, I can't test with other NVME drives sadly.
The behaviour when it is frozen is like this: all drive accesses can't
process, when not cached in ram, so every app that is loaded in the ram,
continues to function, but at the moment it tries to access the drive it
freezes, until the drive is reset and those abort status messages appear
in the dmesg, after that system is unfrozen for a moment, if you keep
copying the files then the problem reoccurs once again.
This drive is known to have problems with the power management in the
past:
https://wiki.archlinux.org/title/Solid_state_drive/NVMe#Troubleshooting
But those problems where since fixed with kernel workarounds / firmware
updates.
This issue is may be related, perhaps bcachefs does something different
from the other filesystems, and workarounds don't apply, which causes
the bug to occur only on it. It may be a problem in the nvme subsystem,
or just some edge case in the bcachefs too, who knows.
I tried to disable ASPM and setting latency to 0 like was suggested, it
didn't fix the problem, so I don't know.
If this is indeed related to that specific drive it would be hard to
reproduce.
---
Errors:
```
! dmesg
[ 34.890981] bcachefs (nvme0n1p3): mounting version 1.3:
rebalance_work
[ 34.890988] bcachefs (nvme0n1p3): recovering from clean shutdown,
journal seq 1782
[ 34.899111] bcachefs (nvme0n1p3): alloc_read... done
[ 34.899130] bcachefs (nvme0n1p3): stripes_read... done
[ 34.899132] bcachefs (nvme0n1p3): snapshots_read... done
[ 34.906883] bcachefs (nvme0n1p3): journal_replay... done
[ 34.906887] bcachefs (nvme0n1p3): resume_logged_ops... done
[ 34.907482] bcachefs (nvme0n1p3): going read-write
[ 92.196122] nvme nvme0: I/O 512 (I/O Cmd) QID 1 timeout, aborting
[ 92.196134] nvme nvme0: I/O 513 (I/O Cmd) QID 1 timeout, aborting
[ 92.196138] nvme nvme0: I/O 514 (I/O Cmd) QID 1 timeout, aborting
[ 92.196141] nvme nvme0: I/O 515 (I/O Cmd) QID 1 timeout, aborting
[ 92.196145] nvme nvme0: I/O 516 (I/O Cmd) QID 1 timeout, aborting
[ 122.405176] nvme nvme0: I/O 512 QID 1 timeout, reset controller
[ 185.384762] nvme0n1: I/O Cmd(0x2) @ LBA 105272408, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384768] I/O error, dev nvme0n1, sector 105272408 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384772] nvme0n1: I/O Cmd(0x2) @ LBA 105272664, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384774] I/O error, dev nvme0n1, sector 105272664 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384775] nvme0n1: I/O Cmd(0x2) @ LBA 105272920, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384776] I/O error, dev nvme0n1, sector 105272920 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384778] nvme0n1: I/O Cmd(0x2) @ LBA 105273176, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384779] I/O error, dev nvme0n1, sector 105273176 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384780] nvme0n1: I/O Cmd(0x2) @ LBA 105273432, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384781] I/O error, dev nvme0n1, sector 105273432 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384782] nvme0n1: I/O Cmd(0x2) @ LBA 105273688, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384783] I/O error, dev nvme0n1, sector 105273688 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384784] nvme0n1: I/O Cmd(0x2) @ LBA 105273944, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384785] I/O error, dev nvme0n1, sector 105273944 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384786] nvme0n1: I/O Cmd(0x2) @ LBA 105274200, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384787] I/O error, dev nvme0n1, sector 105274200 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384788] nvme0n1: I/O Cmd(0x2) @ LBA 105274456, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384789] I/O error, dev nvme0n1, sector 105274456 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384790] nvme0n1: I/O Cmd(0x2) @ LBA 105274712, 256 blocks, I/O
Error (sct 0x3 / sc 0x71)
[ 185.384791] I/O error, dev nvme0n1, sector 105274712 op 0x0:(READ)
flags 0x84700 phys_seg 1 prio class 2
[ 185.384834] nvme nvme0: Abort status: 0x371
[ 185.384836] nvme nvme0: Abort status: 0x371
[ 185.384837] nvme nvme0: Abort status: 0x371
[ 185.384839] nvme nvme0: Abort status: 0x371
[ 185.384840] nvme nvme0: Abort status: 0x371
[ 185.388439] nvme nvme0: 8/0/0 default/read/poll queues
```
---
System info:
```
› uname -a
Linux hp-laptop 6.7.0 #1-NixOS SMP PREEMPT_DYNAMIC Sun Jan 7 20:18:38
UTC 2024 x86_64 GNU/Linux
```
```
› rg -z -i bcachefs /proc/config.gz
10478:CONFIG_BCACHEFS_FS=m
10479:CONFIG_BCACHEFS_QUOTA=y
10480:# CONFIG_BCACHEFS_ERASURE_CODING is not set
10481:CONFIG_BCACHEFS_POSIX_ACL=y
10482:# CONFIG_BCACHEFS_DEBUG_TRANSACTIONS is not set
10483:# CONFIG_BCACHEFS_DEBUG is not set
10484:# CONFIG_BCACHEFS_TESTS is not set
10485:# CONFIG_BCACHEFS_LOCK_TIME_STATS is not set
10486:# CONFIG_BCACHEFS_NO_LATENCY_ACCT is not set
```
```
! nvme list
Node Generic Model Namespace Usage
Format FW Rev
-------------- ----------- ----------------------- ----------
-------------------------- ---------------- --------
/dev/nvme0n1 /dev/ng0n1 KINGSTON SA2000M8500G 0x1 348.70 GB
/ 500.11 GB 512 B + 0 B S5Z42109
```
```
› lsblk -f
NAME FSTYPE FSVER LABEL UUID
FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 ext4 1.0 storage 7ad8dd91-b675-4411-81bc-301e72af3ddb
├─sda2
└─sda3 ntfs System EAEAA44FEAA419B9
zram0
[SWAP]
nvme0n1
├─nvme0n1p1 vfat FAT32 boot 145B-7C42
402.4M 19% /boot
├─nvme0n1p2 btrfs iris 0501be49-5d61-483d-b95e-8879cecd0f12
50G 66% /home
│
/nix/store
│
/nix
│
/
└─nvme0n1p3 bcachefs 1027 irene 85599249-65d6-47dc-b17c-635dc7407581
133.8G 2% /mnt
```
---
This is when it happens on my machine:
```
! mkfs -t bcachefs -f /dev/nvme0n1p3
/dev/nvme0n1p3 contains a bcachefs filesystem
External UUID:
5bb48a77-c303-4b98-aa7d-ec3d01443fc6
Internal UUID:
c8348431-90e8-4cb1-b31d-170fdfe00522
Device index: 0
Label:
Version: 1.3: rebalance_work
Version upgrade complete: 0.0: (unknown version)
Oldest version on disk: 1.3: rebalance_work
Created: Wed Jan 10 10:26:51 2024
Sequence number: 0
Superblock size: 952
Clean: 0
Devices: 1
Sections: members_v1,members_v2
Features:
new_siphash,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:
Options:
block_size: 512 B
btree_node_size: 256 KiB
errors: continue [ro] panic
metadata_replicas: 1
data_replicas: 1
metadata_replicas_required: 1
data_replicas_required: 1
encoded_extent_max: 64.0 KiB
metadata_checksum: none [crc32c] crc64 xxhash
data_checksum: none [crc32c] crc64 xxhash
compression: none
background_compression: none
str_hash: crc32c crc64 [siphash]
metadata_target: none
foreground_target: none
background_target: none
promote_target: none
erasure_code: 0
inodes_32bit: 1
shard_inode_numbers: 1
inodes_use_key_cache: 1
gc_reserve_percent: 8
gc_reserve_bytes: 0 B
root_reserve_percent: 0
wide_macs: 0
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
journal_flush_delay: 1000
journal_flush_disabled: 0
journal_reclaim_delay: 100
journal_transaction_names: 1
version_upgrade: [compatible] incompatible
none
nocow: 0
members_v2 (size 136):
Device: 0
Label: (none)
UUID:
109a3a6c-bf69-435d-b2cc-c6b92dab1a22
Size: 153 GiB
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 256 KiB
First bucket: 0
Buckets: 625088
Last mount: (never)
State: rw
Data allowed: journal,btree,user
Has data: (none)
Durability: 2
Discard: 0
Freespace initialized: 0
mounting version 1.3: rebalance_work
initializing new filesystem
going read-write
initializing freespace
! mount -t bcachefs
/dev/disk/by-partuuid/dab50f50-ff2e-4a54-8d59-6d267cb31148 /mnt
! cp -ax /home /mnt/
```
---
Please tell as to what other info do you need and how to provide it.
More information about the Linux-nvme
mailing list