Bad bits and blocks in JFFS2

[Ricard Wanderlof]

Hi all,

We are trying to determine how JFFS2 behaves in the case of a file system 
in NAND flash exhibiting bit flips and/or bad blocks.

To start with, we ran a series of tests to try and determine reasonable 
failure modes for flash chips. The data sheets are notoriously vague, just 
specifying the maximum number of bad blocks over the lifetime of the chip, 
and that ECC should be used to handle bit errors. Some application notes 
give further hints.

So what we did was erase/write torture a couple of eraseblocks in an 
ST NAND256W3 flash chip, program them with a test pattern, and then 
repeatedly read back the data to see if and when bit errors occurred.

The following table is the result of the test. 'First ecc correction' 
means that we did the indicated number of reads before the mtd ecc 
corrected counter started to increase. 'Ecc correction every read' means 
that from this read cycle, each read attempt was met with a corresponding 
increase in the ecc correction counter. And, similarly for the ecc failure 
counter.

(The test focused on number of read cycles, but it would seem that another 
factor affecting bit decay is the time between writing and reading. We did 
some tests with this too, but it seems that the most significant factor is 
the number of read cycles.)

---
(Table shows number of read cycles until the condition in the left hand
column becomes true).

Erase/write cycles		~20	 10 000	 100 000 1.7e+6	 3.7e+6
------------------------------------------------------------------------
First ecc correction		>4.5e+10 140e+6	 15e+6	 195000	 550
Ecc correction every read	>4.5e+10 163e+6	 19e+6	 231000	 n/a
First ecc failure		>4.5e+10 >991e+6 43e+6	 1.5e+6	 9600
Ecc failure every read		>4.5e+10 >991e+6 76e+6	 2.2e+6	 n/a
Notes				*1	 *2			 *3,*4
*1: Test ran over a three-month period, and the ecc stats remained at 0
     all the time.
*2: The test didn't run long enough to experience ecc failures.
*3: Values reported are after complete rewrite of device, not after
     write torture terminated with I/O errors, as this might have left
     the chip in an indeterminate state.
*4: Torture was run until it terminated with an I/O error.
(Also note that the maximum number of erase/write cycles are specified as
100 000 for this chip, so the last two columns are way over spec in terms
of erase/write cycle counts).
---

Of course, this is just a test on one single individual device of one type 
from one manufacturer. Still I believe it gives some insight into
the chip behavior. For larger memories, I'd expect the same behavior but 
perhaps scaled up a magnitude due to the smaller geometries involved.

The conclusions we drew from this are basically that:
a) Reading the device slowly causes the bits to decay, i.e. not the
    bit cells themselves, but the data (charge) therein.
b) The more times a cell has been erased/written, the faster this decay
    occurs. The cell wears out.
c) Cells which have been written few times exhibit no practical bit decay.
d) At least when torturing the blocks in this way, there's no point
    when a block becomes 'bad' in the sense that it becomes terminally
    unusable, however, the number of permitted read cycles before the
    data decays to unreadability becomes impractically small. (i.e. this
    limit is bound by the application using the flash.)

(An interesting thing not shown in the table above was that when doing the 
3.7e+6 cycle torture test, the torture module terminated with an I/O 
error. However, upon restarting the torture, it ran for a further 6000 
cycles without terminating (again with an I/O error), and the eraseblock 
that caused the error was not the same as the first time.)

The results above would seem to indicate that for program storage or other 
usage where the data is very seldom read, at least this NAND flash type 
would seem to exhibit a very low bit failure rate. ECC should cope with 
any spurious bit flips without problems.

However, let's assume that part of the flash is configured as a JFFS2 file 
system. Furthermore, assume that during the lifetime of a product, a lot 
of writes are performed, so that the maximum specified erase/write cycle 
count is reached and even exceeded. How does JFFS2 cope with the resulting 
errors?

I've looked in the code to try and see how JFFS2 handles this. Since my 
knowledge of JFFS2 internals is very dim, perhaps someone could shed some 
light on this...

In wbuf.c, from what I can tell, when mtd returns EBADMSG (= uncorrectable 
ecc occurred when reading) or EUCLEAN (= corrected ecc occurred), the 
resulting data is treated as if it was ok, and the failure assumed to be 
detected using the JFFS2 node CRC. There doesn't seem to be any attempt at 
rewriting the block or bumping a node read error counter so that the block 
could later be rewritten or marked bad. To me, it would therefore seem 
that ultimately, as bits decay and errors become uncorrectable, that one 
would simply get a read error on the file, instead of the data being moved 
to a good block while there is time. Or am I missing something completely 
here (very likely, but if so, what)?

Regarding write or erase failures, in jffs2_write_nand_badblock, a block 
is marked bad (this seems to be the only place it is done) if it fails to 
erase after two attempts (the function is called from 
erase.c:jffs2_erase_failed only). While this seems all good and well, the 
torture tests show that when the chip returns an erase or write failure, 
the number of erase/write cycles has far exceeded the specifications, and 
reads at this point cause significant bit decay. So while it's definitely 
a good idea to mark the block bad at this point, it seems to me that this 
should have done earlier, perhaps in conjunction with a significant number 
of read errors above.

Are these conclusions correct? Or what am I missing here?

/Ricard
--
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30