NAND timeout issues with blank chip and Marvell NFC

Wed Apr 25 14:16:44 PDT 2018

On 25/04/18 03:50, Steve deRosier wrote:
> Hi Chris,
> 
> On Mon, Apr 23, 2018 at 10:31 PM, Chris Packham
> <Chris.Packham at alliedtelesis.co.nz> wrote:
>> Hi,
>>
>> We're in the process of qualifying new NAND chips (Macronix
>> MX30LF2G18AC) for one of our Armada-385 based devices and we're
>> experiencing some long startup times on units with factory fresh NAND
>> chips. Anecdotally I think I've also seen this behaviour on the old
>> chips as well (Micron MT29F2G08ABAEAWP-ITX:E).
>>
>> On 4.17.0-rc2 with the newly re-written NAND infrastructure we see
>>
>> nand: device found, Manufacturer ID: 0xc2, Chip ID: 0xda
>> nand: Macronix MX30LF2G18AC
>> nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
>> marvell-nfc f10d0000.flash: Timeout on CMDD (NDSR: 0x00000080)
>> marvell-nfc f10d0000.flash: Timeout on CMDD (NDSR: 0x00000280)
>> Bad block table not found for chip 0
>> Bad block table not found for chip 0
>> Scanning device for bad blocks
>>
>> (nothing for some time)

I should correct this. I left it overnight and it's still at this point 
after >24hrs. My original statement was based on the fact that the old 
driver would eventually complete.

I can't be 100% sure that this is the same result as a factory fresh chip.

>>
>> On an older kernel we see
>>
>> pxa3xx-nand f10d0000.flash: This platform can't do DMA on this device
>> nand: device found, Manufacturer ID: 0xc2, Chip ID: 0xda
>> nand: Macronix MX30LF2G18AC
>> nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
>> pxa3xx-nand f10d0000.flash: ECC strength 16, ECC step size 2048
>> Bad block table not found for chip 0
>> Bad block table not found for chip 0
>> Scanning device for bad blocks
>> pxa3xx-nand f10d0000.flash: Wait time out!!!
>> pxa3xx-nand f10d0000.flash: Wait time out!!!
>> pxa3xx-nand f10d0000.flash: Wait time out!!!
>> pxa3xx-nand f10d0000.flash: Wait time out!!!
>> pxa3xx-nand f10d0000.flash: Wait time out!!!
>> ...
>> (time outs continue for some time)
>>
>> Presumably the new driver in 4.17.0-rc2 is experiencing the same wait
>> time out but just not complaining about it.
>>
>> If we leave the system running long enough (in the order of 30 minutes)
>> things seem to sort themselves out and bootup continues, the subsequent
>> boots are fine. If we run 'nand erase.chip' from u-boot on a fresh unit
>> and then boot into the kernel then things are also fine.
>>
>> If we run 'nand scrub.chip -y' from u-boot we are able to re-create the
>> problem.
>>
>> Our suspicion is that erased state of the chip is probably not agreeable
>> with either the ecc data or the bad block table location (or both). By
>> erasing it from u-boot this must fill in valid data in the expected
>> places and the kernel is happy.
>>
> 
> During your very first boot, Linux can't find the bad-block table and
> thus does a full scan of the chip, each and every block, to find the
> manufacturer bad block marks and then constructs the table. 

That's what I assumed was going on.

> I imagine
> you've got a parameter incorrect somewhere that's causing it to wait
> for timeouts at read points, instead of quickly able to read through
> the 2k or 4k blocks on that flash.  On subsequent boots, you don't see
> this issue because the BBT is found and Linux just uses that. Same
> deal if you do a `nand erase.chip`, because the BBT is itself marked
> with a bad-block marker and gets skipped during a normal erase.

Any suggestion as to which setting I may have missed. I haven't adjusted 
my board dts to use any of the new capabilities from the updated 
framework but it does look pretty much the same as every other user of 
this driver. It is just inheriting the setup from armada-38x.dtsi and 
setting up the CS, BBT and ECC params. The final version looks something 
like this

   flash at d0000 {
           compatible = "marvell,armada370-nand";
           reg = <0xd0000 0x54>;
           #address-cells = <0x1>;
           #size-cells = <0x1>;
           interrupts = <0x0 0x54 0x4>;
           clocks = <0xe 0x0>;
           status = "okay";
           num-cs = <0x1>;
           nand-ecc-strength = <0x4>;
           nand-ecc-step-size = <0x200>;
           marvell,nand-enable-arbiter;
           nand-on-flash-bbt;
   };

> Now, I don't know if you're aware of this, but by doing the `nand
> scub.chip -y`, you've ruined the flash chip.  That device can not be
> relied upon anymore. A scrub will ignore the factory bad-block-marks
> and erase them. Unless you stored this information off-chip and
> rewrite the markers, you've now lost the bad-block information from
> the manufacturer's tests.  In any case, this erases the BBT, so your
> next boot triggers Linux to rebuild the BBT.

I was aware and dumped out the BBT before scrubbing. These are sample 
chips anyway so I'm fine with burning them.

> 
>> We could update our manufacturing procedures to run 'nand erase.chip'
>> before the first boot but this feels wrong. Some of our devices boot
>> over the network so the nand is not normally touched by the bootloader.
>> It seems that there is some unhandled error condition that is stopping
>> the kernel from seeing that the chip is completely blank and making
>> forward progress.
>>
> 
> erase chip won't fix your issue. The BBT scan is going to happen
> anyway. There is however clearly some parameter that is setup
> incorrectly that's causing it to wait for the timeout instead of being
> able to quickly read pages. I don't see why that'd be unique to the
> BBT scan however, I'd expect you to see the problem on all reads, thus
> slowing down the system noticeably in general.
> 
> Your hint is likely these lines:
>      " marvell-nfc f10d0000.flash: Timeout on CMDD (NDSR: 0x00000080)
>        marvell-nfc f10d0000.flash: Timeout on CMDD (NDSR: 0x00000280)"
> 
> You can go look at that in the driver and compare with the relevant
> behavior in the datasheets. Sorry, but I can't help more specifically,
> I'd have to know your particular hardware and datasheets and spend
> some time looking at the code.

Those messages seem to come out in both the "good" and "bad" cases. I've 
been ignoring them up to now. I'll go take a closer look.