[PATCH 1/2] mtd: nand: add erased-page bitflip correction

[Bill Pringlemeir]

On 11 Mar 2014, Brian Norris wrote:

> Upper layers (e.g., UBI/UBIFS) expect that pages that have been erased
> will return all 1's (0xff). However, modern NAND (MLC, and even some
> SLC) experience bitflips in unprogrammed pages, and so they may not read
> back all 1's. This is problematic for drivers whose ECC cannot correct
> bitflips in an all-0xff page, as they will report an ECC error
> (-EBADMSG) when they come across such a page. This appears to UBIFS as
> "corrupt empty space".

> Several others [1][2] are attempting to solve this problem, but none is
> generically useful, and most (all?) have subtle bugs in their reasoning. Let's
> implement a simple, generic, and correct solution instead.

> To handle such situations, we should implement the following software
> workaround for drivers that need it: when the hardware driver reports an
> ECC error, nand_base should "verify" this error by

> * Re-reading the page without ECC
> * counting the number of 0 bits in each ECC sector (N[i], for i = 0, 1,
> ..., chip->ecc.steps)
> * If any N[i] exceeds the ECC strength (and thus, the maximum allowable
> bitflips) then we consider this to be an uncorrectable sector.
> * If all N[i] are less than the ECC strength, then we "correct" the
> output to all-0xff and report max-bitflips appropriately

One issue is that a raw read will never see 'stuck at one' errors.  I
believe that Elie had a good diagnosis of the issue,

> 3. I read something but failed to correct it.
> The third case can have two causes:
> 3.a you read valid data with bitflips exceeding what the BCH could
>   correct
> 3.b you read an erased page with bitflips.

For 3.b, the permitted value of bitflips should probably be based on the
flash device and not the ECC controller.  If the chip is giving bit
flips on an SLC NAND device, do we wish to continue on 3.b?  I believe
that maybe only some MLC NAND devices might want to permit this.  If the
conclusion is that this is an erased page, then someone is going to write
to it and possibly then see 'stuck at one' issues.  At first glance,
using the ECC strength seems correct, but I don't think that this is
simple data correction in this case.

Another issue is that the management of flash is not at the MTD layer.
The other layers general know when a sector is erased.  There is no hint
ever given to the MTD driver.  For instance, many drivers implement
sub-pages by doing a full page read followed by a sub-page write, where
just the sub-page data is updated in the originally read page.  If this
is happening multiple times (read page w ECC, read page w/o ECC, write
page), the performance to write a sub-page in a known erased sector
could be pretty horrid.  This maybe a fairly common case.

So, I think this statement,

> Obviously, this sequence can be compute intensive if applied heavily.
> Ideally, MTD users should not need to read un-programmed pages very
> often and would require this software check, for instance, only during
> attach/mount checks or certain recovery situations.

... is not quite correct.  It seems common for some upper layers to ask
to read erased data during normal operation?  Or the only MTD drivers I
have looked at have sub-page handling broken and need to fixing.

If this happens only at boot, then I don't think people would be as
concerned about performance.

See: 
 http://git.infradead.org/linux-mtd.git/blob/HEAD:/drivers/mtd/nand/fsmc_nand.c#l800

for another driver which is doing this check.  I think you are right to
ask for an generic API solution to this issue.  However, I believe we
only need to determine whether it is an all xff page or an erased page.
If an upper layer gave a hint that this page is 'known to be written',
then this could be avoided.  I don't think we have such hints?

Fwiw,
Bill Pringlemeir.