UBIFS volume is filled up by power cuts

[Walter Stoll]

Hi all

During stress tests we observe NAND flashes being filled up by power cuts
without effectively holding according data. We were able to reproduce the
effect with a simplified test procedure as follows:

1) Switch on the device
2) Run the following script (with no parameters)

    #!/bin/bash
    size=${1:-4000000}
    loops=${2:-20}
    df | grep ubi0
    for i in $(seq 1 ${loops}); do
        head -c ${size} /dev/urandom > rnd.bin
        rm rnd.bin
    done

3) Switch off the device


Please note that the number of loops and the file size in the script above are
crucial. We had to play around with these parameters until the effect was
clearly visible.


Effect description
------------------

When we execute the test 10 times, we see the following script output:

ubi0:rootfs             439340     27760    411580   6% /
ubi0:rootfs             439340     31508    407832   7% /
ubi0:rootfs             439340     35248    404092   8% /
ubi0:rootfs             439340     39228    400112   9% /
ubi0:rootfs             439340     42968    396372  10% /
ubi0:rootfs             439340     46700    392640  11% /
ubi0:rootfs             439340     50680    388660  12% /
ubi0:rootfs             439340     54420    384920  12% /
ubi0:rootfs             439340     58160    381180  13% /
ubi0:rootfs             439340     62144    377196  14% /

=> We loose about 3800 1k-blocks per power-cut. As a result, the flash is left
    unusable after about 100 test executions.


Environment description
-----------------------

Our devices run with a vanialla v5.4.69-rt39 kernel on a TI AM335x (Sitara)
based SoC (ARM aarch32 architecture). We run a very basic Linux system with the
following user space processes:  

  127 root      2784 S    /sbin/udevd -d
  331 root      2480 S    /usr/sbin/dropbear -r /etc/dropbear/dropbear_rsa_hos
  341 ntp       7568 S    /usr/sbin/ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
  348 root      2304 S    /sbin/syslogd -n -O /var/log/messages -s 1000 -b 3
  351 root      2304 S    /sbin/klogd -n
  356 root      5232 S    /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
  362 root      2904 S    {start_getty} /bin/sh /bin/start_getty 115200 ttyO0
  363 root      2304 S    /sbin/getty 38400 tty1
  377 root      3128 S    -sh

We use mtd-utils version 2.1.2 commit 7b986779342021bda87c04da3bf729718736d8ab
from git://git.infradead.org/mtd-utils.git.

We use a Micron MT29F8G08ADAFAWP NAND flash (256k block size).
mtdinfo:
    Name:                           rootfs1
    Type:                           nand
    Eraseblock size:                262144 bytes, 256.0 KiB
    Amount of eraseblocks:          1984 (520093696 bytes, 496.0 MiB)
    Minimum input/output unit size: 4096 bytes
    Sub-page size:                  1024 bytes
    OOB size:                       256 bytes
    Character device major/minor:   90:12
    Bad blocks are allowed:         true
    Device is writable:             true

UBI/UBIFS relate kernel configuration:

    CONFIG_UBIFS_FS=y
    # CONFIG_UBIFS_FS_ADVANCED_COMPR is not set
    CONFIG_UBIFS_FS_LZO=y
    CONFIG_UBIFS_FS_ZLIB=y
    CONFIG_UBIFS_FS_ZSTD=y
    # CONFIG_UBIFS_ATIME_SUPPORT is not set
    CONFIG_UBIFS_FS_XATTR=y
    CONFIG_UBIFS_FS_SECURITY=y
    # CONFIG_UBIFS_FS_AUTHENTICATION is not set

    CONFIG_MTD_UBI=y
    CONFIG_MTD_UBI_WL_THRESHOLD=4096
    CONFIG_MTD_UBI_BEB_LIMIT=20
    # CONFIG_MTD_UBI_FASTMAP is not set
    # CONFIG_MTD_UBI_GLUEBI is not set
    # CONFIG_MTD_UBI_BLOCK is not set


Further information
-------------------

We executed the following further tests:

* Run mtd tests:
    - nandbiterrs
    - flash_stress
    - flash_readtest
    - nandpagetest
    - nandsubpagetest
    
    All tests pass except flash_readtest. This test returns us one error per
    block:

        libmtd: error!: Cannot write 256 OOB bytes to address 1024 (OOB \
        offset 1024) - mtd6 OOB size is only 256 bytes
        Error reading OOB in block 0, page 1
        ...
        libmtd: error!: Cannot write 256 OOB bytes to address 519832576 (OOB \
        offset 1024) - mtd6 OOB size is only 256 bytes
        Error reading OOB in block 1983, page 1

    We have no clue how to interprete this error. OOB starts at address 4096
    not 1024.

* Run test with a non realtime v5.4.69 vanialla kernel.
    => same results
    
* Run test with a Micron MT29F8G08ABABAWP NAND flash (512k block size)
    mtdinfo:
        Name:                           rootfs1
        Type:                           nand
        Eraseblock size:                524288 bytes, 512.0 KiB
        Amount of eraseblocks:          992 (520093696 bytes, 496.0 MiB)
        Minimum input/output unit size: 4096 bytes
        Sub-page size:                  1024 bytes
        OOB size:                       224 bytes
        Character device major/minor:   90:12
        Bad blocks are allowed:         true
        Device is writable:             true
    => same results

* Run modified script with sync instruction in the loop:

        #!/bin/bash
        size=${1:-4000000}
        loops=${2:-20}
        df | grep ubi0
        for i in $(seq 1 ${loops}); do
            head -c ${size} /dev/urandom > rnd.bin
            rm rnd.bin
            sync
        done

    => The effect disappears. The same is true if we mount the partition
        synchronously.