UBIFS Read only after corruption

Tue Apr 19 08:29:33 PDT 2022

Hi,
I'm using a 2 GB Micron Nand Flash on my Arm v7 SOC.
OS is Ubuntu 20.04 on a 5.4 Mainline Kernel.

My rootfs is booted from mtd2.
During runtime I'm only writing just some log files to my flash, so
not much data.
>From my 2 GB storage, 1,5 GB is used.

My problem is that after some hours, while my system is running, my
root filesystem is mounted read only and i get some ubifs error, see
below.
After booting the same error persists.

If I make a backup with a corrupted ubifs and I flash that to another
SOC, I get that same error on that new module.
So I guess it's not a hardware problem.

At the moment I create a backup from a working installation with

dd if=/dev/mtd2 of=/image.ubi bs=4M

and I flash that image to a new SOC with

ubiformat /dev/mtd2 -y -f image.ubi

while my system is booted from a USB boot stick.

I don't know if that "backup strategy" implements some errors.

Have you got any idea, how I can repair my corrupted ubifs?

Thanks

Here is a boot log from JTAG

Attempting flash boot...
ubi0: default fastmap pool size: 256
ubi0: default fastmap WL pool size: 128
ubi0: attaching mtd1
ubi0: scanning is finished
ubi0: attached mtd1 (name "mtd=2", size 2031 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 16248, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 67/18, WL threshold: 4096, image
sequence number: 1218135936
ubi0: available PEBs: 0, total reserved PEBs: 16248, PEBs reserved for
bad PEB handling: 320

[   10.085119] ubi0: scanning is finished
[   10.121500] ubi0: attached mtd2 (name "rootfs", size 2031 MiB)
[   10.127374] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[   10.134270] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[   10.141078] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[   10.148058] ubi0: good PEBs: 16248, bad PEBs: 0, corrupted PEBs: 0
[   10.154255] ubi0: user volume: 1, internal volumes: 1, max. volumes
count: 128
[   10.161494] ubi0: max/mean erase counter: 67/18, WL threshold:
4096, image sequence number: 1218135936
[   10.170815] ubi0: available PEBs: 0, total reserved PEBs: 16248,
PEBs reserved for bad PEB handling: 320
[   10.180338] ubi0: background thread "ubi_bgt0d" started, PID 201
[   10.187931] rtc-ds1672 0-0068: setting system clock to
2022-04-19T07:54:30 UTC (1650354870)
[   10.197308] ALSA device list:
[   10.200289]   No soundcards found.
[   10.204062] md: Waiting for all devices to be available before autodetect
[   10.210901] md: If you don't use raid, use raid=noautodetect
[   10.217349] md: Autodetecting RAID arrays.
[   10.221475] md: autorun ...
[   10.224271] md: ... autorun DONE.
[   10.228393] UBIFS (ubi0:0): Mounting in unauthenticated mode
[   10.234298] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 203
[   10.252031] UBIFS (ubi0:0): recovery needed
[   10.418302] UBIFS (ubi0:0): recovery completed
[   10.422909] UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0,
name "rootfs"
[   10.430337] UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB),
min./max. I/O unit sizes: 2048 bytes/2048 bytes
[   10.440270] UBIFS (ubi0:0): FS size: 2019934208 bytes (1926 MiB,
15908 LEBs), journal size 9023488 bytes (8 MiB, 72 LEBs)
[   10.451243] UBIFS (ubi0:0): reserved for root: 0 bytes (0 KiB)
[   10.457096] UBIFS (ubi0:0): media format: w4/r0 (latest is w5/r0),
UUID 3DEDAF38-D1D7-4590-8089-41424C883F66, small LPT model
[   10.469407] VFS: Mounted root (ubifs filesystem) on device 0:20.
[   10.476111] devtmpfs: mounted

[ 12.824030] NET: Registered protocol family 38
[   12.825304] UBIFS: parse errors=remount-ro
[   12.832760] UBIFS error (ubi0:0 pid 234):
ubifs_parse_options.constprop.8: unrecognized mount option
"errors=remount-ro" or missing value
[   12.834029] systemd[1]: Mounted Kernel Debug File System.
[   12.845183] UBIFS error (ubi0:0 pid 234): ubifs_remount_fs: invalid
or unknown remount parameter

[   13.495527] UBIFS error (ubi0:0 pid 241): ubifs_check_node: bad
CRC: calculated 0x96790159, read 0x847e0f2f
[   13.505400] UBIFS error (ubi0:0 pid 241): ubifs_check_node: bad
node at LEB 13872:16312
[   13.505409]  magic          0x6101831
[   13.517137]  crc            0x847e0f2f
[   13.520917]         node_type      0 (inode node)Mounting /tmp...
[   13.525812]  group_type     2 (last of node group)
[   13.533114]  sqnum          11173456
[   13.536870]  len            160
[   13.540061]  key            (2125, inode)
[   13.544161]  creat_sqnum    54577
[   13.544169]  size           71584
[   13.550867]  nlink          4294967295
[   13.554667]  atime          1632961583.4294967295Starting udev Kernel Dev
[   13.560175]  mtime          -1.4294967295 ice Manager...
[   13.567069]  ctime          1646802804.4294967295
[   13.573509]  uid            4294967295
[   13.577331]  gid            4294967295
[   13.577340]  mode           4294967295
[   13.584882]  flags          0xffffffff
[   13.588664]  xattr_cnt      4294967295
[  OK  ]
[   13.592446]         xattr_size     4294967295 Mounted /tmp.
[   13.597930]     xattr_names    4294967295
[   13.603953]  compr_type     0xffff
[   13.607548]  data len       4294967295
[  OK  ]
[   13.611764] UBIFS error (ubi0:0 pid 241): ubifs_scan: bad node
] Reached target Local F
[   13.618976] UBIFS error (ubi0:0 pid 241): ubifs_scanned_corruption:
corruption at LEB 13872:16312 ile Systems.
[   13.630722] UBIFS error (ubi0:0 pid 241): ubifs_scanned_corruption:
first 8192 bytes from LEB 13872:16312
[   13.644139] UBIFS error (ubi0:0 pid 241): ubifs_scan: LEB 13872
scanning failed
[   13.651883] UBIFS error (ubi0:0 pid 222): make_reservation: cannot
reserve 445 bytes in jhead 2, error -30
[   13.651892] UBIFS error (ubi0:0 pid 241): make_reservation: cannot
reserve 416 bytes in jhead 1, error -117
[   13.671444] UBIFS error (ubi0:0 pid 222): do_writepage: cannot
write page 0 of inode 259235, error -30

[ 4565.631586] systemd-journald[219]: Failed to rotate
/var/log/journal/cc446b961f474416970159a09ad171db/system.journal:
Read-only file system
[ 4565.648903] systemd-journald[219]: Failed to write entry (9 items,
248 bytes) despite vacuuming, ignoring: Input/output error
[ 4565.660461] systemd-journald[219]: Failed to rotate
/var/log/journal/cc446b961f474416970159a09ad171db/system.journal:
Read-only file system
[ 4565.675811] systemd-journald[219]: Failed to write entry (9 items,
308 bytes), ignoring: Input/output error
[ 4565.685765] systemd-journald[219]: Failed to rotate
/var/log/journal/cc446b961f474416970159a09ad171db/system.journal:
Read-only file system
[ 4565.701106] systemd-journald[219]: Failed to write entry (9 items,
280 bytes), ignoring: Input/output error