aarch64: ext4 metadata integrity regression in kernels >= 5.5 ?

Sun Jul 12 05:22:31 EDT 2020

Hi,

Some will know that during the last six months, I've been seeing
problems on the LX2160A rev 1 with corrupted checksums on a EXT4
FS on a NVMe recently.  I'm not certain exactly which kernels are
affected, but I know that 5.1 seems to be fine, and 5.5, possibly
5.4 onwards seem affected, maybe earlier.

The symptom is that the kernel will run for some random amount of
time (between a few days and a few months) and then EXT4 will
complain with "iget: checksum invalid" on the root filesystem either
during a logrotate or a mandb rebuild.

Upon investigation with debugfs and hexdump, it appeared that a single
EXT4 inode in one sector contained an invalid 32-bit checksum.  EXT4
splits the 32-bit checksum into two 16-bit halves and stores them in
separate locations in the inode, consequently any read or update of
the checksum requires two separate reads or writes.

The problem initially seemed to correlate with powering the platform
down as the trigger, and it was suggested that the NVMe was at fault.
However, a recent case disproved that theory when the problem appeared
to self-correct itself after using "hdparm -f" on the drive, and the
problem going away - e2fsck found no errors on the filesystem, and I
could remount the filesystem in read/write mode.  "hdparm -f" syncs
the device and flushes the kernel cache, which it also does when you
use "hdparm -t" to measure disk performance.

My next question was whether it was being caused by PCIe ordering
issues.  I've since upgraded the machine to a LX2160A rev 2, which has
yet to show any symptoms of this.

However, the reason for this email is a troubling development with this
problem:

[7478798.720368] EXT4-fs error (device mmcblk0p1): ext4_lookup:1707: inode #157096: comm mandb: iget: checksum invalid
[7478798.729925] Aborting journal on device mmcblk0p1-8.
[7478798.734070] EXT4-fs (mmcblk0p1): Remounting filesystem read-only
[7478798.734589] EXT4-fs error (device mmcblk0p1): ext4_journal_check_start:84: Detected aborted journal

Running "e2fsck -n" on the system without having done anything gives:

Inode 13755 passes checks, but checksum does not match inode.  Fix? no
Inode 157096 passes checks, but checksum does not match inode.  Fix? no

amongst other errors, which are expected for a filesystem that is
normally "in-use".  Using "hdparm -f" does not make these errors go
away.

The offending inodes found by e2fsck corresponds with:
  /usr/share/man/nl/man1/apt-transport-mirror.1.gz
  /lib/firmware/rtl_bt/rtl8723a_fw.bin

However, just like all the other instances, these would not have changed
recently except for atime updates.

There are a couple of important differences here:
- It is an Armada 8040 system - Clearfog GT-8K running a 5.6 kernel,
  rather than the LX2160A.
- Its rootfs is on eMMC, not NVMe.

That seems to rule out the NVMe being a cause of the problem, and any
PCIe issues of the LX2160A rev 1.

Another data point is that I'm also running an Armada 8040 system as a
VM host, which has over a year uptime, so is on an older kernel (5.1).
This uses EXT4 for its rootfs as well, but is on SATA SSD, and has not
shown any issues.  The VMs it runs are a later kernel (5.6) also with
EXT4, and have yet to display any symptoms.

The similarities are - the kernel is the same or similar binary on the
failing systems (I've been running the same kernel config on both.)
Both are a Cortex-A72, but slightly different revisions.

So, it's starting to feel like an aarch64 problem, potentially a
locking or ordering issue.  Due to how rare this issue is,
investigating it is likely very difficult.  However, it seems to be
very real, as the symptoms have now been observed on two rather
different aarch64 platforms.

Due to the amount of time required to test, it very difficult to do any
kind of bisection, or test alternative kernels - it would take months
of runtime for a single test.

I'm chucking this out there so that if anyone else is seeing this
behaviour, they can shout and maybe confirm what I'm seeing.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!