nvme nvme0: I/O 0 (I/O Cmd) QID 1 timeout, aborting, source drive corruption observed

[J. Hart]

I have done an fsck check on the /dev/nvme0n1p3 file system after the 
rsync invocation referenced earlier.  In the first run I found errors 
which fsck should have repaired if I understand correctly.  In repeating 
the fsck invocation immediately afterwards, I found errors again each time.
This was done using the replacement Samsung nvme ssd (Samsung 970 EVO 
Plus 500G).

Here is the output :

-bash-3.2# fsck -f -C /dev/nvme0n1p3
fsck from util-linux 2.34
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure 

Pass 3: Checking directory connectivity 

Pass 4: Checking reference counts
Pass 5: Checking group summary information 

Block bitmap differences:  -87163220 -87163384 -87187960 -122079572 
-122079736
Fix<y>? yes
 

/dev/nvme0n1p3: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p3: 699563/30523392 files (0.1% non-contiguous), 
13841458/122079744 blocks
-bash-3.2# fsck -f -C /dev/nvme0n1p3
fsck from util-linux 2.34
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure 

Pass 3: Checking directory connectivity 

Pass 4: Checking reference counts 

Pass 5: Checking group summary information 

Block bitmap differences:  -87163220 -87163384 -122079736 

Fix<y>? yes
 

/dev/nvme0n1p3: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p3: 699563/30523392 files (0.1% non-contiguous), 
13841458/122079744 blocks
-bash-3.2# fsck -f -C /dev/nvme0n1p3
fsck from util-linux 2.34
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure 

Pass 3: Checking directory connectivity 

Pass 4: Checking reference counts
Pass 5: Checking group summary information 

Block bitmap differences:  -87163220 -87163384 -122079572 -122079736 

Fix<y>? yes
 

/dev/nvme0n1p3: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p3: 699563/30523392 files (0.1% non-contiguous), 
13841458/122079744 blocks


I also tried loading a smaller partition on the Samsung card at 
/dev/nvme0n1p3.  The copy stopped with a "no space left on device" 
error, which should not have been possible as the source device is a 
32MB partition, and the destination partition on the nvme ssd is a 64MB 
partition.  The two files to be transferred were very small and could 
not have accounted for this as they totaled less than 5MB. I found file 
system damage on the nvme destination partition in this case as well. 
It also occurred repeatedly. I am still investigating this last case.

In no instance did I note any otherwise unusual log messages or errors 
from the nvme driver.

I do not yet know if there has been any damage to any other filesystems, 
but I will check.

J. Hart