UBIFS panic during brown-out

[Richard Weinberger]

Kristóf,

----- Ursprüngliche Mail -----
> Von: "Kristof Havasi" <havasiefr at gmail.com>
> An: "richard" <richard at nod.at>
> CC: "linux-mtd" <linux-mtd at lists.infradead.org>
> Gesendet: Montag, 11. Oktober 2021 19:16:01
> Betreff: UBIFS panic during brown-out

> Dear Richard,
> 
> we have been in contact regarding another (similar) kernel panic a year ago:
> http://lists.infradead.org/pipermail/linux-mtd/2020-September/082175.html
> 
> Now we are stress testing our board for brown-out stability, and we can
> see reproducible file system corruptions.
> The culprit was narrowed down to a pending SQLite operation during brown-out,
> in the business-logic.
> 
> As far as I understood, UBIFS is extremely robust in such cases, so I would
> expect a corrupted file as the worst case scenario, not an unbootable system.
> 
> Am I too "optimistic" about UBIFS's brown-out stability?

A brown-out is something very bad. Electronic components show undefined behavior
in this phase. UBIFS (and Linux) can nothing do there.
But both UBI and UBIFS try to be robust against sudden power loss (power-cut).
E.g. an interrupted erase or program operation. 

So, are you really talking about brown-out? If so, better talk to you hardware guys.

> Does the Auth+Encryption combo increase the chances for corrupting the FS
> during brown-out, due to the extra operations?

Assuming you actually meant power-cut.
Both features are rather new, so there can be still bugs.
 
> Would you suggest turning on any of the chk_* knobs in the debugfs?
> I am not sure that they are helpful, or will just modify the behaviour of the
> timings of the system too much.

Let's start with logs first.
 
> Would it be a last resort in case the brown-out is detected to switch the FS
> into ro mode? Is there any API/ABI apart from the debugfs's ro_error knob to
> switch the FS into read-only mode and so trying to prevent file-system
> corruption?

Not really. You need to make sure that the current NAND command is finished
and no new one will be scheduled. Depending on your hardware design this can
be a challenge.

> Any pointers are welcomed!
> 
> We are on Kernel v5.4.150
> We use _both_ UBIFS Authentication and Encryption.
> The board is a SAMA5D3 powered embedded device with 1GB NAND flash,
> which is not even nearly full (10% used).
> 
> Kernel panic after the last brown-out:
> ubi0: scanning is finished
> ubi0: attached mtd1 (name "ubivols", size 1023 MiB)
> ubi0: PEB size: 262144 bytes (256 KiB), LEB size: 253952 bytes
> ubi0: min./max. I/O unit sizes: 4096/4096, sub-page size 4096
> ubi0: VID header offset: 4096 (aligned 4096), data offset: 8192
> ubi0: good PEBs: 4082, bad PEBs: 13, corrupted PEBs: 0
> ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
> ubi0: max/mean erase counter: 118/11, WL threshold: 4096, image
> sequence number: 921361235
>>ubi0: available PEBs: 0, total reserved PEBs: 4082, PEBs reserved for bad PEB
>>handling: 67
> ubi0: background thread "ubi_bgt0d" started, PID 617
> UBIFS (ubi0:4): Mounting in authenticated mode
> UBIFS (ubi0:4): recovery needed
>>VFS: Cannot open root device "ubi0:rootfs" or ubi0:rootfs: error -1

There is a lot of context missing. Can you please share the full logs?
Usually UBIFS prints in detail what went wrong.

> Please append a correct "root=" boot option; here are the available partitions:
> Kernel panic - not syncing: VFS: Unable to mount root fs on ubi0:rootfs
> CPU: 0 PID: 1 Comm: swapper Not tainted 5.4.109-00033-ga88943c1e68 #1

5.4 is not fresh. Can you use a more recent kernel?

Thanks,
//richard