net: stmmac: dwmac-meson8b: interface sometimes does not come up at boot

[Erico Nunes]

Hello,

I've been tracking down an issue with network interfaces from
meson8b-dwmac sometimes not coming up properly at boot.
The target systems are AML-S805X-CC boards (Amlogic S805X SoC), I have
a group of them as part of a CI test farm that uses nfsroot.

After hopefully ruling out potential platform/firmware and network
issues I managed to bisect this commit in the kernel to make a big
difference:

  46f69ded988d2311e3be2e4c3898fc0edd7e6c5a net: stmmac: Use resolved
link config in mac_link_up()

With a kernel before that commit, I am able to submit hundreds of test
jobs and the boards always start the network interface properly.

After that commit, around 30% of the jobs start hitting this:

  [    2.178078] meson8b-dwmac c9410000.ethernet eth0: PHY
[0.e40908ff:08] driver [Meson GXL Internal PHY] (irq=48)
  [    2.183505] meson8b-dwmac c9410000.ethernet eth0: Register
MEM_TYPE_PAGE_POOL RxQ-0
  [    2.200784] meson8b-dwmac c9410000.ethernet eth0: No Safety
Features support found
  [    2.202713] meson8b-dwmac c9410000.ethernet eth0: PTP not supported by HW
  [    2.209825] meson8b-dwmac c9410000.ethernet eth0: configuring for
phy/rmii link mode
  [    3.762108] meson8b-dwmac c9410000.ethernet eth0: Link is Up -
100Mbps/Full - flow control off
  [    3.783162] Sending DHCP requests ...... timed out!
  [   93.680402] meson8b-dwmac c9410000.ethernet eth0: Link is Down
  [   93.685712] IP-Config: Retrying forever (NFS root)...
  [   93.756540] meson8b-dwmac c9410000.ethernet eth0: PHY
[0.e40908ff:08] driver [Meson GXL Internal PHY] (irq=48)
  [   93.763266] meson8b-dwmac c9410000.ethernet eth0: Register
MEM_TYPE_PAGE_POOL RxQ-0
  [   93.779340] meson8b-dwmac c9410000.ethernet eth0: No Safety
Features support found
  [   93.781336] meson8b-dwmac c9410000.ethernet eth0: PTP not supported by HW
  [   93.788088] meson8b-dwmac c9410000.ethernet eth0: configuring for
phy/rmii link mode
  [   93.807459] random: fast init done
  [   95.353076] meson8b-dwmac c9410000.ethernet eth0: Link is Up -
100Mbps/Full - flow control off

This still happens with a kernel from master, currently 5.17-rc2 (less
frequently but still often hit by CI test jobs).
The jobs still usually get to work after restarting the interface a
couple of times, but sometimes it takes 3-4 attempts.

Here is one example and full dmesg:
https://gitlab.freedesktop.org/enunes/mesa/-/jobs/16452399/raw

Note that DHCP does not seem to be an issue here, besides the fact
that the problem only happens since the mentioned commit under the
same setup, I did try to set up the boards to use a static ip but then
the interfaces just don't communicate at all from boot.

For test purposes I attempted to revert
46f69ded988d2311e3be2e4c3898fc0edd7e6c5a on top of master but that
does not apply trivially anymore, and by trying to revert it manually
I haven't been able to get a working interface.

Any advice on how to further debug or fix this?

Thanks

Erico