Preventing JFFS2 partial page writes?

[Artem Bityutskiy]

On Tue, 2011-06-28 at 14:56 -0400, Peter Barada wrote:
> On 06/28/2011 05:34 AM, Artem Bityutskiy wrote:
> > On Mon, 2011-06-27 at 10:31 -0400, Peter Barada wrote:
> >> On 06/24/2011 03:26 PM, Artem Bityutskiy wrote:
> >>> On Wed, 2011-06-22 at 11:28 -0400, Peter Barada wrote:
> >>>> Thoughts?
> >>> Sorry, could you please define the problem you are trying to solve?
> >>> Sorry if you did define it in your long post, but I could not easily
> >>> find it.
> >> The problem I'm trying to solve is that the Micron NAND  I'm using has
> >> an internal 4-bit ECC engine and uses four 8-byte ECCs that provide
> >> 4-bit protection per 512 data bytes + four OOB bytes.  The ecclayout I'm
> >> using is:
> >>
> >> ecclayout = {
> >>           eccbytes = 32,
> >>           eccpos = { 8, 9, 10, 11, 12, 13, 14, 15, /* ECC data bytes
> >> 0-511 + OOB bytes 4-7 */
> >>                   24, 25, 26, 27, 28, 19, 30, 31, /* ECC data bytes
> >> 512-1023 + OOB bytes 20-23 */
> >>                   40, 41, 42, 43, 44, 45, 46, 47,  /* ECC data bytes
> >> 1024-1535 + OOB bytes 36-39 */
> >>                   56, 57, 58, 59, 60, 61, 62, 63},  /* ECC data bytes
> >> 1536-2047 + OOB bytes 52-55 */
> >>           .oobfree = {
> >>                   { .offset = 4,
> >>                     .length = 4},
> >>                   { .offset = 20,
> >>                     .length = 4},
> >>                   { .offset = 36,
> >>                     .length = 4},
> >>                   { .offset = 52,
> >>                     .length = 4},
> >>           },
> >> };
> >>
> >> After the JFFS2 cleanmarker is written into bytes 4-7 and 16-23 of the
> >> OOB, nanddump shows:
> >>
> >>     OOB Data: ff ff ff ff 85 19 03 20 5a e3 da 69 01 40 f1 36
> >>     OOB Data: ff ff ff ff 08 00 00 00 91 99 3c 05 01 d0 5d b3
> >>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>     OOB Data: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >>
> >> Note that the ECC bytes 8-15 and 24-31 are no longer FF due to bytes 4-7
> >> and bytes 20-23 being written with non-zero data.
> >>
> >> When data is later written to this same page (w/o an intervening erase
> >> of its block) reading the page causes an uncorrectable ECC error.
> >>
> >> There are eight additional bytes of OOB space available for writing, but
> >> they are not ECC'd.
> >>
> >> The issue I'm trying to solve is how to communicate from MTD to JFFS2
> >> that some bytes of the oobfree array perturb the data ECC and can not be
> >> used to write the cleanmarker.
> > OK, thanks for explanation. I am not very good in this area as I do not
> > have much experience dealing with OOB, but here is what I thing.
> >
> > 1. Linux MTD code was _not_ designed for "ECC'ed OOB".
> > 2. I do not really know what MTD_OOB_RAW is, and the comment in mtd.h
> >     is not very verbose.
> > 3. But in my opinion MTD_OOB_AUTO makes most sense and should be used
> >     everywhere except for some tricky cases when you want to test things
> >     by writing incorrect ECC, or you have an image with ECC and you want
> >     to flash it as is.
> > 4. In general, OOB should be considered as belonging to the driver, and
> >     modern software should not rely on OOB at all.
> > 5. So MTD_OOB_AUTO make free bytes in OOB look like a contiguous buffer
> >     which the _user_ can freely and _independently_ use.
> > 6. In your case only this assumption does not work and your ecclayout is
> >     incorrect because the OOB areas you expose are not independent.
> > 7. So in your case your ecclayout should be changed and you should
> >     expose only independent ECC bytes.
> The independent ECC bytes available only total to eight, which makes it 
> impossible to use YAFFS (which needs at least 16 bytes to stor e its 
> metadata in). 

Are you sure YAFFS can use the dependent bytes? I am not sure, but I
thought YAFFS wants OOB to be independently writable, no?

>  I can add the independent bytes at the end of the layout 
> (and tweak YAFFS to ignore them since it needs only 16 bytes if the 
> metadata is ECC'ed), but I still need to create a fix in JFFS2 (and 
> u-boot and utilities) to skip the first 16 bytes of oobfree area. 

As I said, I think you need to take a good look at MTD and think how you
can teach it to distinguish with dependent and independent OOB bytes.

> That's why I think another ioctl call (GETOOBLAYOUT?) would be useful to 
> describe the oobfree list, as well as lists(or bitfields) that indicates 
> which bytes in oobfree are either ECC'ed as part of the data ECC, or 
> ECC'ed independently.

Why new ioctl? Why not add another OOB access type (like MTD_OOB_AUTO)?
Or may be one of the existing can be re-use? This should be carefully
analyzed.

> Another issue this exposes is that JFFS2 reads/compares the cleanmarker 
> w/o any ECC in the marker data to verify its validity - if a bitflip in 
> an unECC'd cleanmarker is read back, then I think JFFS2 will fail to use 
> that block.

No, I think what JFFS2 should do is just assume the eraseblock needs
erasure and just erase it. The whole purpose of clean markers is to
erase less. If it is corrupted - we just do an extra erase - not big
deal.

> Also, from what I can find, MTD does not provide a method of programming 
> OOB NAND data using MTD_OOB_AUTO as mtd_do_writeoob (called in mtdchar.c 
> from the MTDWRITEOOB ioctl) uses only MTD_OOB_PLACE.

Probably MTD_OOB_AUTO is just a layer on top of MTD_OOB_PLACE? Anyway,
this whole area of OOB is complex and needs some analysis, and even
better documentation, and then you can choose the best way to solve your
issue.

-- 
Best Regards,
Artem Bityutskiy