Ubifs mounting error causes kernel panic
yusuf.kaya andasis.com
yusuf.kaya at andasis.com
Wed Oct 18 07:11:17 PDT 2023
Hi,
I have MT29F4G08ABBDAH4-IT:D Micron NAND flash on my board. It is 512MiB size. I have LS1088A cpu on that board. I use Linux 5.4.3. I run u-boot, kernel, rootfs and several other partitions on the NAND. I created the partitions in device tree as follows:
/* DTS snippet begins */
&ifc {
ranges = <0 0 0x5 0x30000000 0x00010000
2 0 0x5 0x20000000 0x00010000>;
status = "okay";
nand at 0,0 {
compatible = "fsl,ifc-nand";
reg = <0x0 0x0 0x10000>;
#address-cells = <1>;
#size-cells = <1>;
partition at 0 {
label = "NAND.kernel";
reg = <0x00000000 0x02D00000>;
};
partition at 1 {
label = "NAND.file-system.config";
reg = <0x02D00000 0x02000000>;
};
partition at 2 {
label = "NAND.file-system.backup";
reg = <0x04D00000 0x0C800000>;
};
partition at 3 {
label = "NAND.file-system.log";
reg = <0x11500000 0x01000000>;
};
partition at 4 {
label = "NAND.file-system";
reg = <0x12500000 0x0DB00000>;
};
};
};
/* DTS snippet ends */
And my u-boot parameters as follows:
/* U-Boot snippet begins */
#define MTDIDS_DEFAULT "nand0=nand.0"
#define MTDPARTS_DEFAULT "mtdparts=nand.0:" \
"45m(NAND.kernel)," \
"32m(NAND.file-system.config)," \
"200m(NAND.file-system.backup)," \
"16m(NAND.file-system.log)," \
"-(NAND.file-system)"
#define NANDARGS \
"mtdids=" MTDIDS_DEFAULT "\0" \
"mtdparts=" MTDPARTS_DEFAULT "\0" \
"nandargs=setenv bootargs " \
"console=${console} " \
"fdtfile=${fdtfile} " \
"nandsize=${nand_size} " \
"mtdparts=${mtdparts} " \
"ubi.mtd=NAND.file-system,2048 " \
"root=${nandroot} " \
"rootfstype=${nandrootfstype} " \
"earlyprintk debug=3 rootwait\0" \
"nandroot=ubi0:NAND.file-system rw rootflags=sync\0" \
"nandrootfstype=ubifs\0" \
"nandboot="\
"run mtdparts; " \
"run nandargs; " \
"if ubi part NAND.kernel; then " \
"if ubifsmount ubi0:NAND.kernel; then " \
"echo Booting from NAND.kernel ...; " \
"if ubifsload ${mc_addr_r} ${mc_file}; then " \
"if ubifsload ${dpc_addr_r} ${dpc_file}; then " \
"fsl_mc start mc ${mc_addr_r} ${dpc_addr_r};" \
"fi;"\
"fi;" \
"if ubifsload ${dpl_addr_r} ${dpl_file}; then " \
"fsl_mc lazyapply dpl ${dpl_addr_r};" \
"fi;" \
"if ubifsload ${kernel_addr_r} ${bootfile}; then " \
"if ubifsload ${fdt_addr_r} ${fdtfile}; then " \
"booti ${kernel_addr_r} - ${fdt_addr_r};" \
/* U-Boot snippet ends */
And I created the ubi files one by one as follows:
/* Script snippet for rootfs ubi begins */
build_root_ubi() {
pushd "$RELEASE_DIR" > /dev/null
cat <<- EOF > ubinize.cfg
[rootfs]
# Volume mode (other option is static)
mode=ubi
# Source image
image=root.ubifs
# Volume ID in UBI image
vol_id=0
# Allow for dynamic resize
vol_size=200MiB
vol_type=dynamic
# Volume name
vol_name=NAND.file-system
# Autoresize volume at first mount
vol_flags=autoresize
EOF
mkfs.ubifs -r filesystem_ubi/ -F -m 2048 -e 124KiB -c 4096 -o root.ubifs
ubinize -o filesystem_ubi/root.ubi -m 2048 -p 128KiB -s 2048 -O 2048 ubinize.cfg
cp filesystem_ubi/root.ubi "$RELEASE_DIR/update/"
cp filesystem_ubi/root.ubi "$RELEASE_DIR/filesystem/"
cp filesystem_ubi/root.ubi "$RELEASE_DIR/filesystem/usr/and/backup/"
rm {ubinize.cfg,root.ubifs}
popd > /dev/null
}
/* Script snippet for rootfs ubi ends */
And I writing the ubi files to NAND as follows:
/* Script snippet for rootfs ubi begins */
# create flash-root.sh script
sudo tee $RELEASE_DIR/filesystem/root/.local/bin/flash-root.sh > /dev/null <<- EOF
#!/bin/sh
set -e
ubidetach /dev/ubi_ctrl -m 4 || true
ubiformat /dev/mtd4 -f /root.ubi -s 2048
EOF
sudo chmod u+x $RELEASE_DIR/filesystem/root/.local/bin/flash-root.sh
/* Script snippet for rootfs ubi ends */
So after that informations do you see anything wron in my configuration? After that section first I write the board code to an SD card and run the device from it. After opening kernel I write the ubi files to NAND as below and reboot the board and run it from NAND. At first mostly there is no error when kernel and rootfs is opening. There is ubifs logs for kernel and rootfs as follows:
/* Kernel and rootfs mounting logs begin*/
ubi0: attaching mtd1
ubi0: scanning is finished
ubi0: attached mtd1 (name "NAND.kernel", size 45 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 360, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 3/2, WL threshold: 4096, image sequence number: 419214874
ubi0: available PEBs: 0, total reserved PEBs: 360, PEBs reserved for bad PEB handling: 80
[ 3.905750] ubi0: attaching mtd4
[ 3.984291] random: fast init done
[ 5.345916] ubi0: scanning is finished
[ 5.362184] ubi0: attached mtd4 (name "NAND.file-system", size 219 MiB)
[ 5.368824] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[ 5.375717] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[ 5.382523] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[ 5.389504] ubi0: good PEBs: 1748, bad PEBs: 4, corrupted PEBs: 0
[ 5.395614] ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
[ 5.402855] ubi0: max/mean erase counter: 2/0, WL threshold: 4096, image sequence number: 1086286734
[ 5.412013] ubi0: available PEBs: 0, total reserved PEBs: 1748, PEBs reserved for bad PEB handling: 76
[ 5.476431] UBIFS: parse sync
[ 5.480400] UBIFS (ubi0:0): Mounting in unauthenticated mode
[ 5.486182] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 241
[ 5.536650] UBIFS (ubi0:0): recovery needed
[ 5.666057] UBIFS (ubi0:0): recovery completed
[ 5.670584] UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "NAND.file-system"
[ 5.678870] UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[ 5.688811] UBIFS (ubi0:0): FS size: 210399232 bytes (200 MiB, 1657 LEBs), journal size 9023488 bytes (8 MiB, 72 LEBs)
[ 5.699531] UBIFS (ubi0:0): reserved for root: 0 bytes (0 KiB)
[ 5.705379] UBIFS (ubi0:0): media format: w4/r0 (latest is w5/r0), UUID A820488C-5A5A-496A-8898-D990B0325EDC, small LPT model
[ 5.718319] VFS: Mounted root (ubifs filesystem) on device 0:20.
[ 5.727175] devtmpfs: mounted
/* Kernel and rootfs mounting logs end*/
So as far as here I have one question: The rootfs partition says "Fixing free space" at the first time after I write it to NAND, it is normal. But after the first time it is always says "recovery needed" although I am soft reboot it or not. I cannot understand that. Lets continue, After that normally there is no error unless I install a openwrt ipk package that holds middleware level of my board. After the installation of package with "opkg install --force-reinstall xxx.ipk" I reboot the device and sometimes (sometimes at first, sometimes at 10th try of installation of ipk package) at the mounting of rootfs it says:
/* Rootfs mounting logs begin*/
[ 3.905928] ubi0: attaching mtd4
[ 3.982237] random: fast init done
[ 5.277840] ubi0: scanning is finished
[ 5.293738] ubi0: attached mtd4 (name "NAND.file-system", size 219 MiB)
[ 5.300379] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[ 5.307275] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[ 5.314080] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[ 5.321058] ubi0: good PEBs: 1748, bad PEBs: 4, corrupted PEBs: 0
[ 5.327166] ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
[ 5.334407] ubi0: max/mean erase counter: 2/0, WL threshold: 4096, image sequence number: 1086286734
[ 5.343562] ubi0: available PEBs: 0, total reserved PEBs: 1748, PEBs reserved for bad PEB handling: 76
[ 5.352926] printk: console [netcon0] enabled
[ 5.352976] ubi0: background thread "ubi_bgt0d" started, PID 238
[ 5.408066] UBIFS: parse sync
[ 5.411946] UBIFS (ubi0:0): Mounting in unauthenticated mode
[ 5.417697] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 242
[ 5.465889] UBIFS (ubi0:0): recovery needed
[ 5.633511] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
[ 5.639918] VFS: Cannot open root device "ubi0:NAND.file-system" or unknown-block(0,0): error -2
[ 5.648732] Please append a correct "root=" boot option; here are the available partitions:
[ 5.657108] 0100 4096 ram0
[ 5.657110] (driver?)
[ 5.663228] 0101 4096 ram1
[ 5.663229] (driver?)
[ 5.669347] 0102 4096 ram2
[ 5.669348] (driver?)
[ 5.675466] 0103 4096 ram3
[ 5.675467] (driver?)
[ 5.681584] 0104 4096 ram4
[ 5.681586] (driver?)
[ 5.687704] 0105 4096 ram5
[ 5.687705] (driver?)
[ 5.693823] 0106 4096 ram6
[ 5.693824] (driver?)
[ 5.699943] 0107 4096 ram7
[ 5.699944] (driver?)
[ 5.706062] 0108 4096 ram8
[ 5.706063] (driver?)
[ 5.712181] 0109 4096 ram9
[ 5.712182] (driver?)
[ 5.718300] 010a 4096 ram10
[ 5.718301] (driver?)
[ 5.724506] 010b 4096 ram11
[ 5.724507] (driver?)
[ 5.730713] 010c 4096 ram12
[ 5.730714] (driver?)
[ 5.736920] 010d 4096 ram13
[ 5.736921] (driver?)
[ 5.743126] 010e 4096 ram14
[ 5.743127] (driver?)
[ 5.749333] 010f 4096 ram15
[ 5.749334] (driver?)
[ 5.755545] 1f00 46080 mtdblock0
[ 5.755546] (driver?)
[ 5.762099] 1f01 32768 mtdblock1
[ 5.762100] (driver?)
[ 5.768655] 1f02 204800 mtdblock2
[ 5.768656] (driver?)
[ 5.775211] 1f03 16384 mtdblock3
[ 5.775212] (driver?)
[ 5.781766] 1f04 224256 mtdblock4
[ 5.781767] (driver?)
[ 5.788321] 1f05 65536 mtdblock5
[ 5.788322] (driver?)
[ 5.794876] b300 15558144 mmcblk0
[ 5.794878] driver: mmcblk
[ 5.801693] b301 102400 mmcblk0p1 186fc0ac-01
[ 5.801694]
[ 5.808508] b302 1048576 mmcblk0p2 186fc0ac-02
[ 5.808509]
[ 5.815323] b303 6291456 mmcblk0p3 186fc0ac-03
[ 5.815324]
[ 5.822137] b304 8043008 mmcblk0p4 186fc0ac-04
[ 5.822138]
[ 5.828954] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 5.837234] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.4.3 #4
[ 5.843072] Hardware name: LS1088A RDB Board (DT)
[ 5.847778] Call trace:
[ 5.850229] dump_backtrace+0x0/0x180
[ 5.853893] show_stack+0x14/0x20
[ 5.857211] dump_stack+0xbc/0x100
[ 5.860614] panic+0x15c/0x370
[ 5.863670] mount_block_root+0x1b0/0x2a8
[ 5.867682] prepare_namespace+0x98/0x1d0
[ 5.871693] kernel_init_freeable+0x264/0x2b0
[ 5.876055] kernel_init+0x10/0x108
[ 5.879545] ret_from_fork+0x10/0x18
[ 5.883127] SMP: stopping secondary CPUs
[ 5.887067] Kernel Offset: 0x257599200000 from 0xffff800010000000
[ 5.893166] PHYS_OFFSET: 0xffffe54e00000000
[ 5.897350] CPU features: 0x0002,2080200c
[ 5.901360] Memory Limit: none
[ 5.904419] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---
/* Rootfs mounting logs end*/
As you can see, after the recovery needed, the backgroung thread of ubifs stops due to error:-2, and because of this the rootfs partition cannot found and it cause a kernel panic of course. The only way to recover from that is rewrite the rootfs to NAND with ubiformat command.
I trace it with printks and the error is from: ubifs_tnc_lookup() function that called in do_kill_orphans() function that called from kill_orphans() function that called from ubifs_mount_orphans() function that called from mount_ubifs() function. ubifs_tnc_lookup() returns -2 and then because of that the background process killed. What do you think about it? I have also several other openwrt packages and none of them causes this when installing the ipks.
And worth mention that sometimes it is occuring after first writing of ubi to NAND with ubiformat. So it is possible that it is caused by openwrt package only.
Thank you for any support.
Best regards.
Jozephka99
More information about the linux-mtd
mailing list