ubifs: Mean crash when dumping orphan node

Mats Kärrman Mats.Karrman at tritech.se
Tue Aug 20 10:19:57 EDT 2013


Problem:
Found when running integck with power-cut emulation.
An orphan node ended up with a corrupted (huge) "len" field. The error is
detected by ubifs_check_node() that calls ubifs_dump_node(). This in turn
does not check the len and tries to dump from outside the leb buffer.
--> Kernel oops.
--> note: flush-ubifs_1_0[2143] exited with preempt_count 1
--> ubifs_bgt1_0 stuck in uninterruptible sleep.

Solution:
ubifs_dump_node() doesn't have access to leb number or offset, also it is
called from many places so adding those as arguments is not pleasing.
The following ugly patch limits the buffer len field in ubifs_check_node() in
case the len is out of range. Perhaps someone see a better alternative?


diff --git a/fs/ubifs/io.c b/fs/ubifs/io.c
index e18b988..0e6b1df 100644
--- a/fs/ubifs/io.c
+++ b/fs/ubifs/io.c
@@ -261,8 +261,11 @@ int ubifs_check_node(const struct ubifs_info *c, const void *buf, int lnum,
        }
 
        node_len = le32_to_cpu(ch->len);
-       if (node_len + offs > c->leb_size)
+       if (node_len + offs > c->leb_size) {
+               /* Prevent crash when dumping node */
+               ch->len = cpu_to_le32(c->leb_size - offs);
                goto out_len;
+       }
 
        if (c->ranges[type].max_len == 0) {
                if (node_len != c->ranges[type].len)


----------------------- Console log ---------------------------

UBI: attaching mtd1 to ubi1
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    130944 bytes
UBI: smallest flash I/O unit:    1
UBI: VID header offset:          64 (aligned 64)
UBI: data offset:                128
UBI: max. sequence number:       511497
UBI: attached mtd1 to ubi1
UBI: MTD device name:            "integ_chk"
UBI: MTD device size:            11 MiB
UBI: number of good PEBs:        95
UBI: number of bad PEBs:         0
UBI: number of corrupted PEBs:   0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     1
UBI: available PEBs:             0
UBI: total number of reserved PEBs: 95
UBI: number of PEBs reserved for bad PEB handling: 0
UBI: max/mean erase counter: 30449/29798
UBI: image sequence number:  537117096
UBI: background thread "ubi_bgt1d" started, PID 2141
UBIFS: background thread "ubifs_bgt1_0" started, PID 2142
UBIFS: recovery needed
UBIFS: recovery completed
UBIFS: mounted UBI device 1, volume 0, name "integ_chk"(null)
UBIFS: LEB size: 130944 bytes (127 KiB), min./max. I/O unit sizes: 8 bytes/512 bytes
UBIFS: FS size: 10606464 bytes (10 MiB, 81 LEBs), journal size 1047553 bytes (0 MiB, 6 LEBs)
UBIFS: reserved for root: 500969 bytes (489 KiB)
UBIFS: media format: w4/r0 (latest is w4/r0), UUID 23ADF8B1-A415-4349-9ED5-E8348AFA64BF, small LPT model
UBIFS: background thread "ubifs_bgt1_0" stops
UBIFS: background thread "ubifs_bgt1_0" started, PID 2148
UBIFS warning (pid 21052): power_cut_emulated: failing after 6062 calls
UBIFS warning (pid 2143): power_cut_emulated: failing in GC head LEB 49
UBIFS warning (pid 2143): power_cut_emulated: ========== Power cut emulated ==========
Call Trace:
[ce54da50] [c000884c] show_stack+0x48/0x168 (unreliable)
[ce54da90] [c011927c] power_cut_emulated+0x2d8/0x668
[ce54dab0] [c011980c] dbg_leb_write+0x64/0x1b4
[ce54dae0] [c00f1df8] ubifs_leb_write+0x64/0x124
[ce54db00] [c00f2568] ubifs_wbuf_write_nolock+0x4a0/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
UBIFS warning (pid 2143): corrupt_data: filled bytes 346-511 with random data
UBIFS warning (pid 2143): dbg_leb_write: actually write 512 bytes to LEB 49:110592 (the buffer was corrupted)
UBIFS error (pid 2143): ubifs_leb_write: writing 512 bytes to LEB 49:110592 failed, error -30
UBIFS warning (pid 2143): ubifs_ro_mode: switched to read-only mode, error -30
Call Trace:
[ce54da90] [c000884c] show_stack+0x48/0x168 (unreliable)
[ce54dad0] [c00f0ef4] ubifs_ro_mode+0x70/0x74
[ce54dae0] [c00f1e84] ubifs_leb_write+0xf0/0x124
[ce54db00] [c00f2568] ubifs_wbuf_write_nolock+0x4a0/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
Call Trace:
[ce54daa0] [c000884c] show_stack+0x48/0x168 (unreliable)
[ce54dae0] [c00f1e88] ubifs_leb_write+0xf4/0x124
[ce54db00] [c00f2568] ubifs_wbuf_write_nolock+0x4a0/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
UBIFS error (pid 2143): ubifs_wbuf_write_nolock: cannot write 63 bytes to LEB 49:110592, error -30
        magic          0x6101831
        crc            0x5e5c8e6c
        node_type      2 (direntry node)
        group_type     1 (in node group)
        sqnum          27165524
        len            63
        key            (1019352, direntry, 0x258e374)
        inum           1019371
        type           1
        nlen           6
        name           522879
Call Trace:
[ce54dac0] [c000884c] show_stack+0x48/0x168 (unreliable)
[ce54db00] [c00f24b0] ubifs_wbuf_write_nolock+0x3e8/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
(pid 2143) start dumping LEB 49
UBIFS error (pid 2143): ubifs_check_node: bad node length 673382463
UBIFS error (pid 2143): ubifs_check_node: bad node at LEB 49:110920
        magic          0x6101831
        crc            0xa416d062
        node_type      11 (orphan node)
        group_type     64 (unknown)
        sqnum          27165548
        len            673382463
        commit number  65620158181089514
        last node flag 0
        84172803 orphan inode numbers:
          ino 3562129533828147616
          ino 8117303588418149870
          ino 4640262820834542106
          ino 6726456995349585035
          ino 14489074808184778633
          ino 13946985241486444853
          ino 10749874589530534200
          ino 6464459056807361480
          ino 6626419564033690030
          ino 11788878205927043913
          ino 6555818361502561268
          ino 17704711455940586508
          ino 14549808009372943550
          ino 8481426162481423107
          ino 10339464532359928517
          ino 3645254393624201195
          ino 14333515036885505844
          ino 13565517142754118993
          ino 16678727471166591753
          ino 18446744073709551615
          ino 18446744073709551615
//snip//
          ino 18446744073709551615
          ino 18446744073709551615
          ino 18446744073709551615
          ino 9821846592259568493
          ino 11102932228019499002
          ino 12626009155550599837
          ino 2895740556906901342
          ino 3426259527302190334
          ino 4358966078677474170
          ino 10542940169215034251
          ino 13900150150557332777
          ino 2325828422616975107
          ino 6561521274669858056
          ino 16499483703353557986
          ino 8672796104748479605
          ino 11865067532203562597
          ino 4607242104969698445
          ino 17031714597478032265
          ino 11075308794197177645
Unable to handle kernel paging request for data at address 0xd6d92000
Faulting instruction address: 0xc011be34
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT MPC5125
last sysfs file: /sys/devices/virtual/ubi/ubi1/mtd_num
Modules linked in: xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntr
ack iptable_filter ip_tables x_tables rfcomm l2cap bluetooth ext2 vfat fat sg sd
_mod usb_storage scsi_mod usb_libusual ehci_hcd usbcore
NIP: c011be34 LR: c011be6c CTR: c01829f4
REGS: ce54d910 TRAP: 0300   Tainted: G        W    (2.6.35.14-debug-3)
MSR: 00009032 <EE,ME,IR,DR>  CR: 44002024  XER: 20000000
DAR: d6d92000, DSISR: 20000000
TASK = cf30e7f0[2143] 'flush-ubifs_1_0' THREAD: ce54c000
GPR00: c011be6c ce54d9c0 cf30e7f0 00000022 120f6f8a ffffffff c03265e2 00010000
GPR08: c03269c8 00000001 00000004 120f6f8a 44002022 00000000 c029ec2c c02a6350
GPR16: c0262094 c029ec2c 00100100 00200200 c02a499c c0318c80 c03222b0 c03222a8
GPR24: cf325ca0 00000031 00000031 c02ada68 000009d3 d6d91fe0 ce54c000 05046003
NIP [c011be34] ubifs_dump_node+0x310/0x1700
LR [c011be6c] ubifs_dump_node+0x348/0x1700
Call Trace:
[ce54d9c0] [c011be6c] ubifs_dump_node+0x348/0x1700 (unreliable)
[ce54da40] [c00f171c] ubifs_check_node+0x1a8/0x2b8
[ce54da60] [c00fad7c] ubifs_scan_a_node+0x1b0/0x388
[ce54da90] [c00faff0] ubifs_scan+0x9c/0x394
[ce54dad0] [c011ea9c] ubifs_dump_leb+0x5c/0x158
[ce54db00] [c00f24bc] ubifs_wbuf_write_nolock+0x3f4/0xa00
[ce54db50] [c010096c] move_node+0x40/0xb4
[ce54db80] [c0101118] ubifs_garbage_collect_leb+0x608/0x874
[ce54dbe0] [c010150c] ubifs_garbage_collect+0x188/0x704
[ce54dc60] [c00e2c04] make_reservation+0x110/0x64c
[ce54dcd0] [c00e59e4] ubifs_jnl_write_data+0x124/0x354
[ce54dd40] [c00e7ae4] do_writepage+0x88/0x1e0
[ce54dd80] [c006226c] __writepage+0x24/0x8c
[ce54dd90] [c0062ce4] write_cache_pages+0x200/0x38c
[ce54de30] [c00aa380] writeback_single_inode+0xfc/0x364
[ce54de60] [c00aabc0] writeback_sb_inodes+0x160/0x270
[ce54dea0] [c00ab4ec] wb_writeback+0x224/0x2dc
[ce54df20] [c00ab648] wb_do_writeback+0xa4/0x1a8
[ce54df70] [c00ab79c] bdi_writeback_task+0x50/0x124
[ce54dfa0] [c0070074] bdi_start_fn+0x8c/0x114
[ce54dfc0] [c003ca3c] kthread+0x78/0x7c
[ce54dff0] [c00119a8] kernel_thread+0x4c/0x68
Instruction dump:
5009421e 5009c00e 3929ffe0 553fe8fe 7fe4fb78 48134761 2f9f0000 419efe94
3d20c02b 7f9de378 3b69da68 3b800000 <813d0020> 3b9c0001 7f63db78 801d0024
---[ end trace 47de3350cfddc1f3 ]---
note: flush-ubifs_1_0[2143] exited with preempt_count 1
^Z[1]+  Stopped                    time ./integck.2 -eP /sys/kernel/debug/ubifs/ubi1_0/tst_recovery -m 1 /home/root/ubifs
root at debug:~# ps
  PID USER       VSZ STAT COMMAND
    1 root      1884 S    init [5]
    2 root         0 SW   [kthreadd]
    3 root         0 SW   [ksoftirqd/0]
    4 root         0 SW   [events/0]
    5 root         0 SW   [khelper]
    6 root         0 SW   [async/mgr]
    7 root         0 SW   [sync_supers]
    8 root         0 SW   [bdi-default]
    9 root         0 SW   [kblockd/0]
   10 root         0 SW   [kswapd0]
   11 root         0 SW   [aio/0]
   18 root         0 SW   [mtdblock0]
   19 root         0 SW   [mtdblock1]
   20 root         0 SW   [mtdblock2]
   21 root         0 SW   [mtdblock3]
   22 root         0 SW   [mtdblock4]
   23 root         0 SW   [mtdblock5]
   24 root         0 SW   [mtdblock6]
   25 root         0 SW   [mtdblock7]
   26 root         0 SW   [mtdblock8]
   27 root         0 SW   [80011200.spi]
   28 root         0 DW   [80011900.spi]
   29 root         0 SW   [ubi_bgt0d]
   30 root         0 SW   [ubifs_bgt0_0]
   67 root         0 DW   [spi_receive_thr]
   81 root         0 SW   [khubd]
   93 root      2468 S <  /sbin/udevd -d
  671 root         0 SW   [l2cap]
  672 root         0 SW<  [krfcommd]
  772 messageb  2912 S    /usr/bin/dbus-daemon --system
  857 root      3764 S    /bin/login --
  959 root      3488 S    -sh
 1076 root      3292 S    /sbin/syslogd -n -C64 -m 20
 2141 root         0 SW   [ubi_bgt1d]
 2148 root         0 DW   [ubifs_bgt1_0]
10938 root      2464 S <  /sbin/udevd -d
19488 root      2464 S <  /sbin/udevd -d
21051 root      3096 T    time ./integck.2 -eP /sys/kernel/debug/ubifs/ubi1_0/
21052 root      2060 D    ./integck.2 -eP /sys/kernel/debug/ubifs/ubi1_0/tst_r
29046 root      3488 R    ps



More information about the linux-mtd mailing list