Fwd: Question abount VSOL/CarlitoxxPro SFP Patch on Marvell Armada (MCBIN DS)

[Russell King (Oracle)]

On Mon, Mar 07, 2022 at 09:12:45PM +0100, Giammarco lynx wrote:
> Il giorno lun 7 mar 2022 alle ore 21:09 Giammarco lynx
> <stich86 at gmail.com> ha scritto:
> > Then after "ifconfig eth3 up" i've removed and reinserted the SFP,
> > this is the log:
> >
> > [   56.127227] sfp sfp-eth3: module removed
> > [   56.131175] sfp sfp-eth3: tx disable 0 -> 1

TX_DISABLE asserted.

> > [   56.135432] sfp sfp-eth3: SM: exit empty:up:down
> > [   56.140067] sfp sfp-eth3: SM: enter empty:up:down event tx_fault
> > [   56.146103] sfp sfp-eth3: SM: exit empty:up:down
> > [   59.350731] sfp sfp-eth3: mod-def0 0 -> 1

Module inserted, and starts booting...

> > [   59.354757] sfp sfp-eth3: SM: enter empty:up:down event insert
> > [   59.360619] sfp sfp-eth3: SM: exit probe:up:down
> > [   59.680123] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   59.686185] sfp sfp-eth3: SM: exit probe:up:down
> > [   59.800122] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   59.806179] sfp sfp-eth3: SM: exit probe:up:down
> > [   59.920122] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   59.926179] sfp sfp-eth3: SM: exit probe:up:down
> > [   60.040122] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   60.046178] sfp sfp-eth3: SM: exit probe:up:down
> > [   60.160123] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   60.166178] sfp sfp-eth3: SM: exit probe:up:down
> > [   60.270122] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   60.276177] sfp sfp-eth3: SM: exit probe:up:down
> > [   60.380122] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   60.386178] sfp sfp-eth3: SM: exit probe:up:down
> > [   60.500122] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   60.506177] sfp sfp-eth3: SM: exit probe:up:down
> > [   60.620121] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   60.626176] sfp sfp-eth3: SM: exit probe:up:down
> > [   60.740122] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   60.746177] sfp sfp-eth3: please wait, module slow to respond
> > [   60.751982] sfp sfp-eth3: SM: exit probe:up:down
> > [   65.840198] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   65.846262] sfp sfp-eth3: SM: exit probe:up:down
> > [   70.870128] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   70.876192] sfp sfp-eth3: SM: exit probe:up:down
> > [   75.920133] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   75.926196] sfp sfp-eth3: SM: exit probe:up:down
> > [   80.896801] mvpp2 f4000000.ethernet eth3: mac link up

Link established with the host MAC at 1G, probably as part of the
module detecting what speed the host is operating at.

> > [   80.960124] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   80.966183] sfp sfp-eth3: SM: exit probe:up:down
> > [   81.330385] sfp sfp-eth3: los 1 -> 0
> > [   81.333978] sfp sfp-eth3: tx-fault 1 -> 0

Some of the modules IOs change state.

> > [   81.338004] sfp sfp-eth3: SM: enter probe:up:down event tx_clear
> > [   81.344044] sfp sfp-eth3: SM: exit probe:up:down
> > [   81.348680] sfp sfp-eth3: SM: enter probe:up:down event los_low
> > [   81.354629] sfp sfp-eth3: SM: exit probe:up:down
> > [   86.000154] sfp sfp-eth3: SM: enter probe:up:down event timeout
> > [   86.013661] sfp sfp-eth3: Detected broken RTL8672/RTL9601C emulated EEPROM

Module emulated EEPROM becomes readable about 27 seconds after
insertion Firmware probably almost finished booting.

> > [   86.020569] sfp sfp-eth3: Switching to reading EEPROM to one byte at a time
> > [   86.067758] sfp sfp-eth3: module OEM              V2801F
> > rev 1.0  sn 202101195032     dc 210119
> > [   86.077555] mvpp2 f4000000.ethernet eth3: requesting link mode
> > inband/1000base-x with support 0000000,00000200,00000440
> > [   86.088390] sfp sfp-eth3: tx disable 1 -> 0

We've now detected what the module is, and we configure for it, which
the EEPROM indicates it supports 1200Mbaud, which is 1G speed.

> > [   86.092636] sfp sfp-eth3: SM: exit present:up:wait
> > [   86.097448] sfp sfp-eth3: skipping hwmon device registration due to
> > broken EEPROM
> > [   86.104965] sfp sfp-eth3: diagnostic EEPROM area cannot be read
> > atomically to guarantee data coherency
> > [   86.160128] sfp sfp-eth3: SM: enter present:up:wait event timeout
> > [   86.166252] sfp sfp-eth3: SM: exit present:up:link_up

sfp.c determines link up...

> > [   86.171404] mvpp2 f4000000.ethernet eth3: Link is Up - 1Gbps/Full -
> > flow control off
> > [   86.179193] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready

Network device now indicates that link has been established at 1G
speed.

> > In that way it goes up. So how can we avoid this behaviour?

Very difficult to avoid it. My guess is that when you power up the
system with the module inserted, it sees the interface operating at
2.5G speed and locks itself to 2.5G speed.

Eventually, our kernel boots, and sfp reads the EEPROM. The EEPROM
indicates that it supports 1G speed, so we switch the host interface
to 1000base-X, and that causes link to be lost (although I don't see
any sign of that happening in your debug.)

That means that the module has decided to use 2.5G speed, but its
reporting it supports 1G speed in the EEPROM... so its doing something
different. No surprises that the link fails to come up.

We could add an entry to the sfp_quirks[] table in sfp-bus.c for this
module using the sfp_quirk_2500basex function to also indicate it
supports 2.5G speed:

	}, {
		// VSOL/CarlitoxxPro SFP can also work at 2.5G speed
		.part = "V2801F",
		.modes = sfp_quirk_2500basex,

However, we are always going to be stuck in the situation that this is
going to be unreliable. Whatever speed the network interface is in when
this module is inserted, it's going to lock to that speed. If we then
change it later, we'll never link.

A switch will likely only ever use one speed on its SFP port, meaning
it will always present 1000base-X, or it will always present 2500base-X,
so it will always work with a switch, but not a more capable MAC
interface that can switch between these at will.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!