UBIFS panic during brown-out

[Kristof Havasi]

Dear Richard,

we have been in contact regarding another (similar) kernel panic a year ago:
http://lists.infradead.org/pipermail/linux-mtd/2020-September/082175.html

Now we are stress testing our board for brown-out stability, and we can
see reproducible file system corruptions.
The culprit was narrowed down to a pending SQLite operation during brown-out,
in the business-logic.

As far as I understood, UBIFS is extremely robust in such cases, so I would
expect a corrupted file as the worst case scenario, not an unbootable system.

Am I too "optimistic" about UBIFS's brown-out stability?

Does the Auth+Encryption combo increase the chances for corrupting the FS
during brown-out, due to the extra operations?

Would you suggest turning on any of the chk_* knobs in the debugfs?
I am not sure that they are helpful, or will just modify the behaviour of the
timings of the system too much.

Would it be a last resort in case the brown-out is detected to switch the FS
into ro mode? Is there any API/ABI apart from the debugfs's ro_error knob to
switch the FS into read-only mode and so trying to prevent file-system
corruption?

Any pointers are welcomed!

We are on Kernel v5.4.150
We use _both_ UBIFS Authentication and Encryption.
The board is a SAMA5D3 powered embedded device with 1GB NAND flash,
which is not even nearly full (10% used).

Kernel panic after the last brown-out:
ubi0: scanning is finished
ubi0: attached mtd1 (name "ubivols", size 1023 MiB)
ubi0: PEB size: 262144 bytes (256 KiB), LEB size: 253952 bytes
ubi0: min./max. I/O unit sizes: 4096/4096, sub-page size 4096
ubi0: VID header offset: 4096 (aligned 4096), data offset: 8192
ubi0: good PEBs: 4082, bad PEBs: 13, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 118/11, WL threshold: 4096, image
sequence number: 921361235
>ubi0: available PEBs: 0, total reserved PEBs: 4082, PEBs reserved for bad PEB handling: 67
ubi0: background thread "ubi_bgt0d" started, PID 617
UBIFS (ubi0:4): Mounting in authenticated mode
UBIFS (ubi0:4): recovery needed
>VFS: Cannot open root device "ubi0:rootfs" or ubi0:rootfs: error -1
Please append a correct "root=" boot option; here are the available partitions:
Kernel panic - not syncing: VFS: Unable to mount root fs on ubi0:rootfs
CPU: 0 PID: 1 Comm: swapper Not tainted 5.4.109-00033-ga88943c1e68 #1
Hardware name: Atmel SAMA5
[<c010babc>] (unwind_backtrace) from [<c010a8d0>] (show_stack+0x10/0x14)
[<c010a8d0>] (show_stack) from [<c0569970>] (panic+0xfc/0x2e8)
[<c0569970>] (panic) from [<c0801598>] (mount_block_root+0x268/0x2a4)
[<c0801598>] (mount_block_root) from [<c0801674>] (prepare_namespace+0x9c/0x184)
[<c0801674>] (prepare_namespace) from [<c056b774>] (kernel_init+0x8/0x108)
[<c056b774>] (kernel_init) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
Exception stack(0xc7829fb0 to 0xc7829ff8)
9fa0:                                     00000000 00000000 00000000 00000000
9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on
ubi0:rootfs ]---

Best regards,
Kristóf Havasi