[PATCH mtd-utils v2 RESEND 089/102] fsck.ubifs: Add README to describe fsck
Zhihao Cheng
chengzhihao1 at huawei.com
Mon Nov 11 01:08:17 PST 2024
Add documents to describe fsck, which includes introductions, designment,
advantage and limitations.
Signed-off-by: Zhihao Cheng <chengzhihao1 at huawei.com>
Signed-off-by: Zhang Yi <yi.zhang at huawei.com>
---
ubifs-utils/fsck.ubifs/README.txt | 388 ++++++++++++++++++++++++++++++++++++++
1 file changed, 388 insertions(+)
create mode 100644 ubifs-utils/fsck.ubifs/README.txt
diff --git a/ubifs-utils/fsck.ubifs/README.txt b/ubifs-utils/fsck.ubifs/README.txt
new file mode 100644
index 00000000..a4daae74
--- /dev/null
+++ b/ubifs-utils/fsck.ubifs/README.txt
@@ -0,0 +1,388 @@
+ fsck.ubifs
+ ==========
+ The fsck.ubifs can check and repair the UBIFS image on a given UBI volume, it
+ could fix inconsistent UBIFS image(which is corrupted by hardware exceptions
+ or UBIFS realization bugs) and makes filesystem become consistent.
+
+
+ Manuals
+ -------
+ There are four modes for fsck.ubifs:
+ 1. normal mode(no options): Check the filesystem, ask user whether to fix the
+ problem as long as inconsistent data is found during fs checking.
+ 2. safe mode(-a option): Check and automatic safely repair the filesystem, if
+ there are any data dropping operations needed by fixing, fsck will fail.
+ 3. danger mode(-y option): Answer 'yes' to all questions. There are two sub
+ modes:
+ a) Default submode(no options): Check and automatic repair the filesystem
+ according to TNC, data dropping will be reported. If TNC/master/log is
+ corrupted, fsck will fail.
+ b) rebuild submode(-b option): Check and automatic forcibly repair the
+ filesystem, turns to rebuild filesystem if TNC/master/log is corrupted.
+ Always make fsck successful.
+ 4. check mode(-n option): Make no changes to the filesystem, only check the
+ filesystem. This mode doesn't check space, because unclean LEBs cannot be
+ rewritten in read-only mode.
+
+ The exit code returned by fsck.ubifs is compatible with FSCK(8), which is the
+ sum of the following conditions:
+ 0 - No errors
+ 1 - File system errors corrected
+ 2 - System should be rebooted
+ 4 - File system errors left uncorrected
+ 8 - Operational error
+ 16 - Usage or syntax error
+ 32 - Fsck canceled by user request
+ 128 - Shared library error
+
+
+ Designment
+ ----------
+ There are 2 working modes for fsck: rebuild mode and non-rebuild mode. The main
+ idea is that construct all files by scanning the entire filesystem, then check
+ the consistency of metadata(file meta information, space statistics, etc.)
+ according to the files. The file(xattr is treated as a file) is organized as:
+ file tree(rbtree, inum indexed)
+ / \
+ file1 file2
+ / \
+ file3 file4
+ file {
+ inode node // each file has 1 inode node
+ dentry (sub rb_tree, sqnum indexed)
+ // '/' has no dentries, otherwise at least 1 dentry is required.
+ trun node // the newest one truncation node
+ data (sub rb_tree, block number indexed)
+ // Each file may have 0 or many data nodes
+ xattrs (sub rb_tree, inum indexed)
+ // Each file may have 0 or many xattr files
+ }
+
+ Step 0. Both two modes need to read the superblock firstly, fsck fails if
+ superblock is corrupted, because fsck has no idea about the location
+ of each area(master, log, main, etc.) when the layout is lost.
+
+ A. Rebuild mode:
+ Step 1. Scan nodes(inode node/dentry node/data node/truncation node) from all
+ LEBs.
+ a) Corrupted LEBs(eg. garbage data, corrupted empty space) are dropped
+ during scanning.
+ b) Corrupted nodes(eg. incorrect crc, bad inode size, bad dentry name
+ length, etc.) are dropped during scanning.
+ c) Valid inode nodes(nlink > 0) and dentry nodes(inum != 0) are put
+ into two valid trees(valid_inos & valid_dents) separately.
+ d) Deleted inode nodes (nlink is 0) and deleted dentry nodes(inum is 0)
+ are put into two deleted trees(del_inos & del_dents) separately.
+ e) Other nodes(data nodes/truncation node) are put into corresponding
+ file, if the file doesn't exist, insert a new file into the file
+ tree.
+ Step 2. Traverse nodes from deleted trees, remove inode nodes and dentry nodes
+ with smaller sqnum from valid trees. valid_inos - del_inos = left_inos,
+ valid_dents - del_dents = left_dents.
+ This step handles the deleting case, for example, file A is deleted,
+ deleted inode node and deleted dentry node are written, if we ignore
+ the deleted nodes, file A can be recovered after rebuilding because
+ undeleted inode node and undeleted dentry node can be scanned. There's
+ an exception, if deleted inode node and deleted dentry node are
+ reclaimed(by gc) after deletion, file A is recovered. So deleted data
+ or files could be recovered by rebuild mode.
+ Step 3. Traverse left_inos and left_dents, insert inode node and dentry nodes
+ into the corresponding file.
+ Step 4. Traverse all files, drop invalid files, move xattr files into the
+ corresponding host file's subtree. Invalid files such as:
+ a) File has no inode node or inode nlink is zero
+ b) Non-consistent file types between inode node and dentry nodes
+ c) File has no dentry nodes(excepts '/')
+ d) Encrypted file has no xattr information
+ e) Non regular file has data nodes
+ f) Directory/xattr file has more than one dentries
+ g) Xattr file has no host inode, or the host inode is a xattr
+ h) Non-xattr file's parent is not a directory
+ i) etc.
+ Step 5. Extract reachable directory entries tree. Make sure that all files can
+ be searched from '/', unreachable file is deleted. Since all xattr
+ files are attached to the corresponding host file, only non-xattr
+ files should be checked. Luckily, directory file only has one dentry,
+ the reachable checking of a dentry becomes easy. Traverse all
+ dentries for each file, check whether the dentry is reachable, if not,
+ remove dentry from the file. If the file has no dentries, the file is
+ unreachable.
+ Step 6. Correct the file information. Traverse all files and calculate
+ information(nlink, size, xattr_cnt, etc.) for each file just like
+ check_leaf(in linux kernel) does, correct the inode node based on the
+ calculated information.
+ Step 7. Record used LEBs. Traverse all files'(including effective nodes from
+ deletion trees in step 2) position, after this step fsck knows which
+ LEB is empty.
+ Step 8. Re-write data. Read data from LEB and write back data, make sure that
+ all LEB is ended with empty data(0xFF). It will prevent failed gc
+ scanning in the next mounting.
+ Step 9. Build TNC. Construct TNC according to all files' nodes, just like mkfs
+ does(refer to add_to_index in mkfs), then write TNC(refer to
+ write_index in mkfs) on flash. (If there are no files, create a new
+ root dir file.)
+ Step 10.Build LPT. Construct LPT according to all nodes' position and length,
+ just like mkfs does, then write LPT(refer to write_lpt) on flash.
+ Step 11.Clean up log area and orphan area. Log area and orphan area can be
+ erased.
+ Step 12.Write master node. Since all meta areas are ready, master node can be
+ updated.
+
+ B. Non-rebuild mode:
+ Step 1. Read master & init lpt.
+ a) Scan master nodes failed or master node is invalid (which is not
+ caused by invalid space statistics), danger mode with rebuild_fs and
+ normal mode with 'yes' answer will turn to rebuild mode, other modes
+ will exit. Fsck cannot find the right TNC/LPT if the master node is
+ invalid, which affects subsequent steps, so this problem must be
+ fixed.
+ b) Invalid space statistics in master node, set %FR_LPT_INCORRECT for
+ for lpt status and ignore the error.
+ c) LPT node is corrupted, set %FR_LPT_CORRUPTED for lpt status and
+ ignore the error.
+ Step 2. Replay journal.
+ I. Scan log LEBs to get all buds.
+ a) Nodes in log LEBs are invalid/corrupted, danger mode with
+ rebuild_fs and normal mode with 'yes' answer will turn to rebuild
+ mode, other modes will exit. Corrupted log LEB could fail
+ ubifs_consolidate_log, which may lead to commit failure by out of
+ space in the log area, so this problem must be fixed.
+ II. Scan bud LEBs to get all nodes.
+ a) Nodes in bud LEBs are invalid/corrupted, danger mode and normal
+ mode with 'yes' answer will drop bud LEB and set
+ %FR_LPT_INCORRECT for lpt status, other modes will exit.
+ Corrupted LEB will make gc failed, so this problem must be
+ fixed.
+ III. Record isize into size tree according to data/truncation/inode
+ nodes.
+ IV. Apply nodes to TNC & LPT, update property for bud LEBs.
+ a) Corrupted/Invalid node searched from TNC, skip node and set
+ %FR_LPT_INCORRECT in lpt status for danger mode and normal mode
+ with 'yes' answer, other modes will exit. The space statistics
+ depend on a valid TNC, so this problem must be fixed.
+ b) Corrupted/Invalid index node read from TNC, danger mode with
+ rebuild_fs and normal mode with 'yes' answer will turn to
+ rebuild filesystem, other modes will exit. The space statistics
+ depend on a valid TNC, so this problem must be fixed.
+ c) Corrupted/Invalid lpt node, Set %FR_LPT_CORRUPTED for lpt status
+ and ignore the error.
+ d) Incorrect LEB property: Set %FR_LPT_INCORRECT for lpt status and
+ ignore the error.
+ e) If lpt status is not empty, skip updating lpt, because incorrect
+ LEB property could trigger assertion failure in ubifs_change_lp.
+ Step 3. Handle orphan nodes.
+ I. Scan orphan LEB to get all orphan nodes.
+ a) Corrupted/Invalid orphan node: danger mode and normal mode with
+ 'yes' answer will drop orphan LEB, other modes will exit.
+ Corrupted orphan area could lead to mounting/committing failure,
+ so this problem must be fixed.
+ II. Parse orphan node, find the original inode for each inum.
+ a) Corrupted/Invalid node searched from TNC, skip node for danger
+ mode and normal mode with 'yes' answer, other modes will exit.
+ b) Corrupted/Invalid index node read from TNC, danger mode with
+ rebuild_fs and normal mode with 'yes' answer will turn to
+ rebuild filesystem, other modes will exit. The space statistics
+ depend on a valid TNC, so this problem must be fixed.
+ III. Remove inode for each inum, update TNC & LPT.
+ a) Corrupted/Invalid node searched from TNC, skip node for danger
+ mode and normal mode with 'yes' answer, other modes will exit.
+ b) Corrupted/Invalid index node read from TNC, danger mode with
+ rebuild_fs and normal mode with 'yes' answer will turn to
+ rebuild filesystem, other modes will exit. The space statistics
+ depend on a valid TNC, so this problem must be fixed.
+ c) Corrupted/Invalid lpt node, Set %FR_LPT_CORRUPTED for lpt
+ status and ignore the error.
+ d) Incorrect LEB property: Set %FR_LPT_INCORRECT for lpt status
+ and ignore the error.
+ e) If lpt status is not empty, skip updating lpt, because
+ incorrect LEB property could trigger assertion failure in
+ ubifs_change_lp.
+ Step 4. Consolidate log area.
+ a) Corrupted data in log LEBs, danger mode with rebuild_fs and normal
+ mode with 'yes' answer will turn to rebuild filesystem, other modes
+ will exit. It could make commit failed by out of space in log area,
+ so this problem must be fixed.
+ Step 5. Recover isize.
+ I. Traverse size tree, lookup corresponding inode from TNC.
+ a) Corrupted/Invalid node searched from TNC, skip node for danger
+ mode and normal mode with 'yes' answer, other modes will exit.
+ b) Corrupted/Invalid index node read from TNC, danger mode with
+ rebuild_fs and normal mode with 'yes' answer will turn to
+ rebuild filesystem, other modes will exit. The space statistics
+ depend on a valid TNC, so this problem must be fixed.
+ II. Update isize for inode. Keep <inum, isize> in size tree for check
+ mode, remove <inum, isize> from the size tree and update inode
+ node in place for other modes.
+ Step 6. Traverse TNC and construct files.
+ I. Traverse TNC, check whether the leaf node is valid, remove invalid
+ nodes, construct file for valid node and insert the file into the
+ file tree.
+ a) Corrupted/Invalid node searched from TNC, remove corresponding
+ TNC branch for danger mode and normal mode with 'yes' answer,
+ other modes will exit. The space statistics depend on a valid
+ TNC, so this problem must be fixed.
+ b) Corrupted/Invalid index node read from TNC, danger mode with
+ rebuild_fs and normal mode with 'yes' answer will turn to
+ rebuild filesystem, other modes will exit. The space statistics
+ depend on a valid TNC, so this problem must be fixed.
+ II. Scan all LEBs(contain TNC) for non check mode(unclean LEBs cannot
+ be fixed in read-only mode, so scanning may fail in check mode,
+ then space statistics won't be checked in check mode), remove TNC
+ branch which points to corrupted LEB.
+ a) Corrupted data is found by scanning. If the current node is
+ index node, danger mode with rebuild_fs and normal mode with
+ 'yes' answer will turn to rebuild filesystem, other modes will
+ exit; If the current node is non-index node, danger mode and
+ normal mode with 'yes' answer will remove all TNC branches which
+ point to the corrupted LEB, other modes will exit. The space
+ statistics depend on valid LEB scanning, so this problem must
+ be fixed.
+ b) LEB contains both index and non-index nodes, danger mode with
+ rebuild_fs and normal mode with 'yes' answer will turn to
+ rebuild filesystem, other modes will exit. Invalid LEB will make
+ gc failed, so this problem must be fixed.
+ Step 7. Update files' size for check mode. Update files' size according to the
+ size tree for check mode.
+ Step 8. Check and handle invalid files. Similar to rebuild mode, but the
+ methods of handling are different:
+ a) Move unattached(file has no dentries) regular file into disconnected
+ list for safe mode, danger mode and normal mode with 'yes' answer,
+ let subsequent steps to handle them with lost+found. Other modes
+ will exit. Disconnected file affects the result of calculated
+ information(which will be used in subsequent steps) for its' parent
+ file(eg. nlink, size), so this problem must be fixed.
+ b) Make file type be consistent between inode, detries and data nodes
+ by deleting dentries or data nodes, for danger mode and normal mode
+ with 'yes' answer, other modes will exit.
+ c) Delete file for other invalid cases(eg. file has no inode) in
+ danger mode and normal mode with 'yes' answer, other modes will
+ exit.
+ Step 9. Extract reachable directory entries tree. Similar to rebuild mode, but
+ the methods of handling are different:
+ a) Remove unreachable dentry for danger mode and normal mode with 'yes'
+ answer, other modes will exit. Unreachable dentry affects the
+ calculated information(which will be used in subsequent steps) for
+ its' file(eg. nlink), so this problem must be fixed.
+ b) Delete unreachable non-regular file for danger mode and normal mode
+ with 'yes' answer, other modes will exit. Unreachable file affects
+ the calculated information(which will be used in subsequent steps)
+ for its' parent file(eg. nlink, size), so this problem must be
+ fixed.
+ c) Move unreachable regular file into disconnected list for safe mode,
+ danger mode and normal mode with 'yes' answer, let subsequent steps
+ to handle them with lost+found. Other modes will exit. Disconnected
+ file affects the calculated information(which will be used in
+ subsequent steps) for its' parent file(eg. nlink, size), so this
+ problem must be fixed.
+ Step 10.Correct the file information. Similar to rebuild mode, but the methods
+ of handling are different:
+ a) Correct the file information for safe mode, danger mode and normal
+ mode with 'yes' answer, other modes will exit. Incorrect file
+ information affects the new creations(which will be used in handling
+ lost+found), so this problem must be fixed.
+ Step 11.Check whether the TNC is empty. Empty TNC is equal to corrupted TNC,
+ which means that zero child count for root znode. If TNC is empty(All
+ nodes are invalid and are deleted from TNC), turn to rebuild mode for
+ danger mode with rebuild_fs and normal mode with 'yes' answer, other
+ modes will exit.
+ Step 12.Check and correct the space statistics.
+ I. Exit for check mode, if %FR_LPT_CORRUPTED or %FR_LPT_INCORRECT is
+ set in lpt status, the exit code should have %FSCK_UNCORRECTED.
+ II. Check lpt status, if %FR_LPT_CORRUPTED is set in lpt status, normal
+ mode with 'no' answer will exit, other modes will rebuild lpt. New
+ creations could be done in subsequent steps, which depends on
+ correct space statistics, so this problem must be fixed.
+ III. Traverse LPT nodes, check the correctness of nnode and pnode,
+ compare LEB scanning result with LEB properties.
+ a) LPT node is corrupted, normal mode with 'no' answer will exit,
+ rebuild lpt for other modes. New creations could be done in
+ subsequent steps, which depends on the correct space
+ statistics, so this problem must be fixed.
+ b) Incorrect nnode/pnode, normal mode with 'no' answer will exit,
+ other modes will correct the nnode/pnode. New creations could
+ be done in subsequent steps, which depends on correct space
+ statistics, so this problem must be fixed.
+ c) Inconsistent comparing result, normal mode with 'no' answer
+ will exit, other modes will correct the space statistics. New
+ creations could be done in subsequent steps, which depends on
+ correct space statistics, so this problem must be fixed.
+ IV. Compare LPT area scanning result with lprops table information.
+ a) LPT area is corrupted, normal mode with 'no' answer will exit,
+ rebuild lpt for other modes. Commit could fail in doing LPT gc
+ caused by scanning corrupted data, so this problem must be
+ fixed.
+ b) Inconsistent comparing result, normal mode with 'no' answer
+ will exit, other modes will correct the lprops table
+ information. Commit could fail in writing LPT with %ENOSPC
+ return code caused by incorrect space statistics in the LPT
+ area, so this problem must be fixed.
+ Step 13.Do commit, commit problem fixing modifications to disk. The index size
+ checking depends on this step.
+ Step 14.Check and correct the index size. Check and correct the index size by
+ traversing TNC just like dbg_check_idx_size does. This step should be
+ executed after first committing, because 'c->calc_idx_sz' can be
+ changed in 'ubifs_tnc_start_commit' and the initial value of
+ 'c->calc_idx_sz' read from the disk is untrusted. Correct the index
+ size for safe mode, danger mode and normal mode with 'yes' answer,
+ other modes will exit. New creations could be done in subsequent steps,
+ which depends on the correct index size, so this problem must be fixed.
+ Step 15.Check and create the root dir. Check whether the root dir exists,
+ create a new one if it is not found, for safe mode, danger mode and
+ normal mode with 'yes' answer, other modes will exit. Mounting depends
+ on the root dir, so this problem must be fixed.
+ Step 16.Check and create the lost+found.
+ I. If the root dir is encrypted, set lost+found as invalid. Because it
+ is impossible to check whether the lost+found exists in an encrypted
+ directory.
+ II. Search the lost+found under root dir.
+ a) Found a lost+found, lost+found is a non-encrypted directory, set
+ lost+found as valid, otherwise set lost+found as invalid.
+ b) Not found the lost+found, create a new one. If creation is
+ failed by %ENOSPC, set lost+found as invalid.
+ Step 17.Handle each file from the disconnected list.
+ I. If lost+found is invalid, delete file for danger mode and normal
+ mode with 'yes' answer, other modes will skip and set the exit code
+ with %FSCK_UNCORRECTED.
+ II. If lost+found is valid, link disconnected file under lost+found
+ directory with the name of the corresponding inode number
+ (INO_<inum>_<index>, index(starts from 0) is used to handle the
+ conflicted names).
+ a) Fails in handling conflicted file names, delete file for danger
+ mode and normal mode with 'yes' answer, other modes will skip
+ and set the exit code with %FSCK_UNCORRECTED.
+ b) Fails in linking caused by %ENOSPC, delete file for danger mode
+ and normal mode with 'yes' answer, other modes will skip and set
+ the exit code with %FSCK_UNCORRECTED.
+ Step 18.Do final commit, commit problem fixing modifications to disk and clear
+ %UBIFS_MST_DIRTY flag for master node.
+
+
+ Advantage
+ ---------
+ 1. Can be used for any UBIFS image, fsck has nothing to do with kernel version.
+ 2. Fsck is tolerant with power-cut, fsck will always succeed in a certain mode
+ without changing mode even power-cut happens in checking and repairing. In
+ other words, fsck won't let UBIFS image become worse in abnormal situations.
+ 3. It is compatible with FSCK(8), the exit code returned by fsck.ubifs is same
+ as FSCK, the command options used by fsck are supported in fsck.ubifs too.
+ 4. The UBIFS image can be fixed as long as the super block is not corrupted.
+ 5. Encrypted UBIFS image is supported, because dentry name and data content of
+ file are not necessary for fsck.
+
+
+ Limitations
+ -----------
+ 1. UBIFS image file is not supported(Not like ext4). The UBIFS image file is
+ not equal to UBI volume, empty LEBs are not included in image file, so UBIFS
+ cannot allocate empty space when file recovering is needed. Another reason
+ is that atomic LEB changing is not supported by image file.
+ 2. Authenticated UBIFS image is not supported, UBIFS metadata(TNC/LPT) parsing
+ depends on the authentication key which is not supported in fsck options.
+
+
+ Authors
+ -------
+ Zhihao Cheng <chengzhihao1 at huawei.com>
+ Zhang Yi <yi.zhang at huawei.com>
+ Xiang Yang <xiangyang3 at huawei.com>
+ Huang Xiaojia <huangxiaojia2 at huawei.com>
--
2.13.6
More information about the linux-mtd
mailing list