ubifs: Mean crash when dumping orphan node
Mats Kärrman
Mats.Karrman at tritech.se
Tue Aug 20 10:19:57 EDT 2013
Problem:
Found when running integck with power-cut emulation.
An orphan node ended up with a corrupted (huge) "len" field. The error is
detected by ubifs_check_node() that calls ubifs_dump_node(). This in turn
does not check the len and tries to dump from outside the leb buffer.
--> Kernel oops.
--> note: flush-ubifs_1_0[2143] exited with preempt_count 1
--> ubifs_bgt1_0 stuck in uninterruptible sleep.
Solution:
ubifs_dump_node() doesn't have access to leb number or offset, also it is
called from many places so adding those as arguments is not pleasing.
The following ugly patch limits the buffer len field in ubifs_check_node() in
case the len is out of range. Perhaps someone see a better alternative?
diff --git a/fs/ubifs/io.c b/fs/ubifs/io.c
index e18b988..0e6b1df 100644
--- a/fs/ubifs/io.c
+++ b/fs/ubifs/io.c
@@ -261,8 +261,11 @@ int ubifs_check_node(const struct ubifs_info *c, const void *buf, int lnum,
}
node_len = le32_to_cpu(ch->len);
- if (node_len + offs > c->leb_size)
+ if (node_len + offs > c->leb_size) {
+ /* Prevent crash when dumping node */
+ ch->len = cpu_to_le32(c->leb_size - offs);
goto out_len;
+ }
if (c->ranges[type].max_len == 0) {
if (node_len != c->ranges[type].len)
----------------------- Console log ---------------------------
UBI: attaching mtd1 to ubi1
UBI: physical eraseblock size: 131072 bytes (128 KiB)
UBI: logical eraseblock size: 130944 bytes
UBI: smallest flash I/O unit: 1
UBI: VID header offset: 64 (aligned 64)
UBI: data offset: 128
UBI: max. sequence number: 511497
UBI: attached mtd1 to ubi1
UBI: MTD device name: "integ_chk"
UBI: MTD device size: 11 MiB
UBI: number of good PEBs: 95
UBI: number of bad PEBs: 0
UBI: number of corrupted PEBs: 0
UBI: max. allowed volumes: 128
UBI: wear-leveling threshold: 4096
UBI: number of internal volumes: 1
UBI: number of user volumes: 1
UBI: available PEBs: 0
UBI: total number of reserved PEBs: 95
UBI: number of PEBs reserved for bad PEB handling: 0
UBI: max/mean erase counter: 30449/29798
UBI: image sequence number: 537117096
UBI: background thread "ubi_bgt1d" started, PID 2141
UBIFS: background thread "ubifs_bgt1_0" started, PID 2142
UBIFS: recovery needed
UBIFS: recovery completed
UBIFS: mounted UBI device 1, volume 0, name "integ_chk"(null)
UBIFS: LEB size: 130944 bytes (127 KiB), min./max. I/O unit sizes: 8 bytes/512 bytes
UBIFS: FS size: 10606464 bytes (10 MiB, 81 LEBs), journal size 1047553 bytes (0 MiB, 6 LEBs)
UBIFS: reserved for root: 500969 bytes (489 KiB)
UBIFS: media format: w4/r0 (latest is w4/r0), UUID 23ADF8B1-A415-4349-9ED5-E8348AFA64BF, small LPT model
UBIFS: background thread "ubifs_bgt1_0" stops
UBIFS: background thread "ubifs_bgt1_0" started, PID 2148
UBIFS warning (pid 21052): power_cut_emulated: failing after 6062 calls
UBIFS warning (pid 2143): power_cut_emulated: failing in GC head LEB 49
UBIFS warning (pid 2143): power_cut_emulated: ========== Power cut emulated ==========
Call Trace:
[ce54da50] [c000884c] show_stack+0x48/0x168 (unreliable)
[ce54da90] [c011927c] power_cut_emulated+0x2d8/0x668
[ce54dab0] [c011980c] dbg_leb_write+0x64/0x1b4
[ce54dae0] [c00f1df8] ubifs_leb_write+0x64/0x124
[ce54db00] [c00f2568] ubifs_wbuf_write_nolock+0x4a0/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
UBIFS warning (pid 2143): corrupt_data: filled bytes 346-511 with random data
UBIFS warning (pid 2143): dbg_leb_write: actually write 512 bytes to LEB 49:110592 (the buffer was corrupted)
UBIFS error (pid 2143): ubifs_leb_write: writing 512 bytes to LEB 49:110592 failed, error -30
UBIFS warning (pid 2143): ubifs_ro_mode: switched to read-only mode, error -30
Call Trace:
[ce54da90] [c000884c] show_stack+0x48/0x168 (unreliable)
[ce54dad0] [c00f0ef4] ubifs_ro_mode+0x70/0x74
[ce54dae0] [c00f1e84] ubifs_leb_write+0xf0/0x124
[ce54db00] [c00f2568] ubifs_wbuf_write_nolock+0x4a0/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
Call Trace:
[ce54daa0] [c000884c] show_stack+0x48/0x168 (unreliable)
[ce54dae0] [c00f1e88] ubifs_leb_write+0xf4/0x124
[ce54db00] [c00f2568] ubifs_wbuf_write_nolock+0x4a0/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
UBIFS error (pid 2143): ubifs_wbuf_write_nolock: cannot write 63 bytes to LEB 49:110592, error -30
magic 0x6101831
crc 0x5e5c8e6c
node_type 2 (direntry node)
group_type 1 (in node group)
sqnum 27165524
len 63
key (1019352, direntry, 0x258e374)
inum 1019371
type 1
nlen 6
name 522879
Call Trace:
[ce54dac0] [c000884c] show_stack+0x48/0x168 (unreliable)
[ce54db00] [c00f24b0] ubifs_wbuf_write_nolock+0x3e8/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
(pid 2143) start dumping LEB 49
UBIFS error (pid 2143): ubifs_check_node: bad node length 673382463
UBIFS error (pid 2143): ubifs_check_node: bad node at LEB 49:110920
magic 0x6101831
crc 0xa416d062
node_type 11 (orphan node)
group_type 64 (unknown)
sqnum 27165548
len 673382463
commit number 65620158181089514
last node flag 0
84172803 orphan inode numbers:
ino 3562129533828147616
ino 8117303588418149870
ino 4640262820834542106
ino 6726456995349585035
ino 14489074808184778633
ino 13946985241486444853
ino 10749874589530534200
ino 6464459056807361480
ino 6626419564033690030
ino 11788878205927043913
ino 6555818361502561268
ino 17704711455940586508
ino 14549808009372943550
ino 8481426162481423107
ino 10339464532359928517
ino 3645254393624201195
ino 14333515036885505844
ino 13565517142754118993
ino 16678727471166591753
ino 18446744073709551615
ino 18446744073709551615
//snip//
ino 18446744073709551615
ino 18446744073709551615
ino 18446744073709551615
ino 9821846592259568493
ino 11102932228019499002
ino 12626009155550599837
ino 2895740556906901342
ino 3426259527302190334
ino 4358966078677474170
ino 10542940169215034251
ino 13900150150557332777
ino 2325828422616975107
ino 6561521274669858056
ino 16499483703353557986
ino 8672796104748479605
ino 11865067532203562597
ino 4607242104969698445
ino 17031714597478032265
ino 11075308794197177645
Unable to handle kernel paging request for data at address 0xd6d92000
Faulting instruction address: 0xc011be34
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT MPC5125
last sysfs file: /sys/devices/virtual/ubi/ubi1/mtd_num
Modules linked in: xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntr
ack iptable_filter ip_tables x_tables rfcomm l2cap bluetooth ext2 vfat fat sg sd
_mod usb_storage scsi_mod usb_libusual ehci_hcd usbcore
NIP: c011be34 LR: c011be6c CTR: c01829f4
REGS: ce54d910 TRAP: 0300 Tainted: G W (2.6.35.14-debug-3)
MSR: 00009032 <EE,ME,IR,DR> CR: 44002024 XER: 20000000
DAR: d6d92000, DSISR: 20000000
TASK = cf30e7f0[2143] 'flush-ubifs_1_0' THREAD: ce54c000
GPR00: c011be6c ce54d9c0 cf30e7f0 00000022 120f6f8a ffffffff c03265e2 00010000
GPR08: c03269c8 00000001 00000004 120f6f8a 44002022 00000000 c029ec2c c02a6350
GPR16: c0262094 c029ec2c 00100100 00200200 c02a499c c0318c80 c03222b0 c03222a8
GPR24: cf325ca0 00000031 00000031 c02ada68 000009d3 d6d91fe0 ce54c000 05046003
NIP [c011be34] ubifs_dump_node+0x310/0x1700
LR [c011be6c] ubifs_dump_node+0x348/0x1700
Call Trace:
[ce54d9c0] [c011be6c] ubifs_dump_node+0x348/0x1700 (unreliable)
[ce54da40] [c00f171c] ubifs_check_node+0x1a8/0x2b8
[ce54da60] [c00fad7c] ubifs_scan_a_node+0x1b0/0x388
[ce54da90] [c00faff0] ubifs_scan+0x9c/0x394
[ce54dad0] [c011ea9c] ubifs_dump_leb+0x5c/0x158
[ce54db00] [c00f24bc] ubifs_wbuf_write_nolock+0x3f4/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
Instruction dump:
5009421e 5009c00e 3929ffe0 553fe8fe 7fe4fb78 48134761 2f9f0000 419efe94
3d20c02b 7f9de378 3b69da68 3b800000 <813d0020> 3b9c0001 7f63db78 801d0024
---[ end trace 47de3350cfddc1f3 ]---
note: flush-ubifs_1_0[2143] exited with preempt_count 1
^Z[1]+ Stopped time ./integck.2 -eP /sys/kernel/debug/ubifs/ubi1_0/tst_recovery -m 1 /home/root/ubifs
root at debug:~# ps
PID USER VSZ STAT COMMAND
1 root 1884 S init [5]
2 root 0 SW [kthreadd]
3 root 0 SW [ksoftirqd/0]
4 root 0 SW [events/0]
5 root 0 SW [khelper]
6 root 0 SW [async/mgr]
7 root 0 SW [sync_supers]
8 root 0 SW [bdi-default]
9 root 0 SW [kblockd/0]
10 root 0 SW [kswapd0]
11 root 0 SW [aio/0]
18 root 0 SW [mtdblock0]
19 root 0 SW [mtdblock1]
20 root 0 SW [mtdblock2]
21 root 0 SW [mtdblock3]
22 root 0 SW [mtdblock4]
23 root 0 SW [mtdblock5]
24 root 0 SW [mtdblock6]
25 root 0 SW [mtdblock7]
26 root 0 SW [mtdblock8]
27 root 0 SW [80011200.spi]
28 root 0 DW [80011900.spi]
29 root 0 SW [ubi_bgt0d]
30 root 0 SW [ubifs_bgt0_0]
67 root 0 DW [spi_receive_thr]
81 root 0 SW [khubd]
93 root 2468 S < /sbin/udevd -d
671 root 0 SW [l2cap]
672 root 0 SW< [krfcommd]
772 messageb 2912 S /usr/bin/dbus-daemon --system
857 root 3764 S /bin/login --
959 root 3488 S -sh
1076 root 3292 S /sbin/syslogd -n -C64 -m 20
2141 root 0 SW [ubi_bgt1d]
2148 root 0 DW [ubifs_bgt1_0]
10938 root 2464 S < /sbin/udevd -d
19488 root 2464 S < /sbin/udevd -d
21051 root 3096 T time ./integck.2 -eP /sys/kernel/debug/ubifs/ubi1_0/
21052 root 2060 D ./integck.2 -eP /sys/kernel/debug/ubifs/ubi1_0/tst_r
29046 root 3488 R ps
More information about the linux-mtd
mailing list