ClearFog GT-8K and GPON SFP modules

[Russell King - ARM Linux admin]

On Sun, Nov 22, 2020 at 09:56:55PM +0100, Thomas Schreiber wrote:
> Hi,
> I have a ClearFog GT-8K that I use as a home router, I would like to
> report one patch that helped me to support GPON SFP module SERCOMM
> FGS202 (based on Intel (Lantiq) PEB98036 chip) and kindly ask for some
> advice to support CarlitoxxPro CPGOS03-0490 v2.0 module, rebranded
> from VSOL V2801F (Realtek RTL9601C chip).

Thank you for getting in touch, I'm aware of your issues, since you
reported them to SolidRun who then came to me. I've been waiting for
some feedback since.

> The patch for SERCOMM/FGS202 forces the phy to 1000BASE-X when such a
> module is detected, adding a GPON quirk in sfp-bus.c

Note that many GPON modules support 2500BASE-X as well, and attempt
to auto-detect what the host is using. This can be very tempermental
depending on how it's implemented. Some modules need to see valid
2500BASE-X on the serdes as soon as they're plugged in for example,
which would be impossible for the GT-8K to do (since it's a chicken
and egg problem, and the port supports 10G.)

> diff --git a/drivers/net/phy/sfp-bus.c b/drivers/net/phy/sfp-bus.c
> index 58014feedf6c..57e334739445 100644
> --- a/drivers/net/phy/sfp-bus.c
> +++ b/drivers/net/phy/sfp-bus.c
> @@ -44,6 +44,12 @@ static void sfp_quirk_2500basex(const struct
> sfp_eeprom_id *id,
> phylink_set(modes, 2500baseX_Full);
> }
> +static void sfp_quirk_1000basex(const struct sfp_eeprom_id *id,
> + unsigned long *modes)
> +{
> + phylink_set(modes, 1000baseX_Full);
> +}
> +
> static const struct sfp_quirk sfp_quirks[] = {
> {
> // Alcatel Lucent G-010S-P can operate at 2500base-X, but
> @@ -63,6 +69,10 @@ static const struct sfp_quirk sfp_quirks[] = {
> .vendor = "HUAWEI",
> .part = "MA5671A",
> .modes = sfp_quirk_2500basex,
> + }, {
> + .vendor = "SERCOMM",
> + .part = "FGS202",
> + .modes = sfp_quirk_1000basex,
> },
> };
> 
> [root at clearfog-gt-8k ~]# ethtool eth0
> Settings for eth0:
> Supported ports: [ FIBRE ]
> Supported link modes: 1000baseX/Full
> Supported pause frame use: Symmetric Receive-only
> Supports auto-negotiation: Yes
> Supported FEC modes: Not reported
> Advertised link modes: 1000baseX/Full
> Advertised pause frame use: Symmetric Receive-only
> Advertised auto-negotiation: Yes
> Advertised FEC modes: Not reported
> Speed: 1000Mb/s
> Duplex: Full
> Auto-negotiation: on
> Port: FIBRE
> PHYAD: 0
> Transceiver: internal
> Link detected: yes
> 
> [root at clearfog-gt-8k ~]# dmesg | grep eth0
> [ 3.168319] sfp sfp-cp0-eth0: Host maximum power 2.0W
> [ 3.197420] mvpp2 f2000000.ethernet eth0: Using random mac address
> xx:xx:xx:xx:xx:xx
> [ 3.520165] sfp sfp-cp0-eth0: module SERCOMM FGS202 rev 0001 sn
> SCOM2118C321 dc 190725
> [ 3.530303] sfp sfp-cp0-eth0: module address swap to access page 0xA2
> is not supported.

That's annoying that it's one of these modules that requires a special
sequence to switch to the other EEPROM address.

> [ 3.538350] mvpp2 f2000000.ethernet eth0: switched to
> inband/1000base-x link mode
> [ 7.919348] mvpp2 f2000000.ethernet eth0: configuring for
> inband/1000base-x link mode
> [ 7.985612] mvpp2 f2000000.ethernet eth0: Link is Up - 1Gbps/Full -
> flow control rx/tx
> [ 8.007872] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> 
> [root at clearfog-gt-8k ~]# ethtool -m eth0
> Identifier : 0x03 (SFP)
> Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
> Connector : 0x01 (SC)
> Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Encoding : 0x03 (NRZ)
> BR, Nominal : 1200MBd
> Rate identifier : 0x00 (unspecified)
> Length (SMF,km) : 20km
> Length (SMF) : 20000m
> Length (50um) : 0m
> Length (62.5um) : 0m
> Length (Copper) : 0m
> Length (OM3) : 0m
> Laser wavelength : 1310nm
> Vendor name : SERCOMM
> Vendor OUI : 00:00:00
> Vendor PN : FGS202
> Vendor rev : 0001
> Option values : 0x00 0x1a
> Option : RX_LOS implemented
> Option : TX_FAULT implemented
> Option : TX_DISABLE implemented
> BR margin, max : 0%
> BR margin, min : 0%
> Vendor SN : SCOM2118C321
> Date code : 190725
> 
> Now about CPGOS03-0490 v2.0, the EEPROM must be read in single-byte
> mode and my router manufacturer provided me with a patch written by
> Russell King to do that and indeed the EEPROM can subsequently be
> read:

I wrote the patch in response to the query coming through SolidRun,
but there was insufficient information to be able to write a full
patch; the "WHATEVER" string was supposed to be updated so that we
always use single byte reads to this module.

Without that change, we will initially read the first few bytes of
the EEPROM using single byte reads, and then switch back to 16-byte
reads for everything else, including your "ethtool -m" command below.
That makes the assertion you had from the CPGOS03-0490 vendor rather
"interesting" as it seems it becomes false!

> root at clearfog-gt-8k ~]# ethtool -m eth0
> Identifier : 0x03 (SFP)
> Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
> Connector : 0x01 (SC)
> Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Encoding : 0x03 (NRZ)
> BR, Nominal : 1200MBd
> Rate identifier : 0x00 (unspecified)
> Length (SMF,km) : 20km
> Length (SMF) : 20000m
> Length (50um) : 0m
> Length (62.5um) : 0m
> Length (Copper) : 0m
> Length (OM3) : 0m
> Laser wavelength : 1310nm
> Vendor name : VSOL
> Vendor OUI : 00:00:00
> Vendor PN : V2801F
> Vendor rev : 1.0
> Option values : 0x00 0x1c
> Option : RX_LOS implemented, inverted
> Option : TX_FAULT implemented
> Option : TX_DISABLE implemented

This looks possible, and it passes the checksum verification, so we
have to assume this is correct, and RX_LOS is really inverted on
this module.

> BR margin, max : 0%
> BR margin, min : 0%
> Vendor SN : CP202003180377
> Date code : 200408
> Optical diagnostics support : Yes
> Laser bias current : 0.208 mA
> Laser output power : 0.0000 mW / -inf dBm
> Receiver signal average optical power : 0.0000 mW / -inf dBm

Either it doesn't implement reporting this, or the module is not
getting any signal.

> Module temperature : 49.31 degrees C / 120.76 degrees F
> Module voltage : 3.2514 V
> Alarm/warning flags implemented : Yes
> Laser bias current high alarm : Off
> Laser bias current low alarm : On
> Laser bias current high warning : Off
> Laser bias current low warning : On
> Laser output power high alarm : Off
> Laser output power low alarm : On
> Laser output power high warning : Off
> Laser output power low warning : On
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser rx power high alarm : Off
> Laser rx power low alarm : On
> Laser rx power high warning : Off
> Laser rx power low warning : On

which is reflected in these two alarm and warning flags.

> Laser bias current high alarm threshold : 75.000 mA
> Laser bias current low alarm threshold : 1.000 mA
> Laser bias current high warning threshold : 70.000 mA
> Laser bias current low warning threshold : 3.000 mA
> Laser output power high alarm threshold : 3.1622 mW / 5.00 dBm
> Laser output power low alarm threshold : 1.1220 mW / 0.50 dBm
> Laser output power high warning threshold : 2.8183 mW / 4.50 dBm
> Laser output power low warning threshold : 1.2589 mW / 1.00 dBm
> Module temperature high alarm threshold : 90.00 degrees C / 194.00 degrees F
> Module temperature low alarm threshold : -10.00 degrees C / 14.00 degrees F
> Module temperature high warning threshold : 85.00 degrees C / 185.00 degrees F
> Module temperature low warning threshold : -5.00 degrees C / 23.00 degrees F
> Module voltage high alarm threshold : 3.6000 V
> Module voltage low alarm threshold : 3.0000 V
> Module voltage high warning threshold : 3.4700 V
> Module voltage low warning threshold : 3.1299 V
> Laser rx power high alarm threshold : 0.1584 mW / -8.00 dBm
> Laser rx power low alarm threshold : 0.0015 mW / -28.24 dBm
> Laser rx power high warning threshold : 0.1258 mW / -9.00 dBm
> Laser rx power low warning threshold : 0.0019 mW / -27.21 dBm
> 
> [root at clearfog-gt-8k ~]# ethtool eth0
> Settings for eth0:
> Supported ports: [ FIBRE ]
> Supported link modes: 1000baseX/Full
> Supported pause frame use: Symmetric Receive-only
> Supports auto-negotiation: Yes
> Supported FEC modes: Not reported
> Advertised link modes: 1000baseX/Full
> Advertised pause frame use: Symmetric Receive-only
> Advertised auto-negotiation: Yes
> Advertised FEC modes: Not reported
> Speed: 1000Mb/s
> Duplex: Full
> Auto-negotiation: on
> Port: FIBRE
> PHYAD: 0
> Transceiver: internal
> Link detected: no
> 
> [root at clearfog-gt-8k ~]# dmesg | grep eth0
> [ 3.170263] sfp sfp-cp0-eth0: Host maximum power 2.0W
> [ 3.199313] mvpp2 f2000000.ethernet eth0: Using random mac address
> be:28:7d:48:12:91
> [ 3.528841] sfp sfp-cp0-eth0: module VSOL V2801F rev 1.0 sn
> CP202003180377 dc 200408
> [ 3.538985] mvpp2 f2000000.ethernet eth0: switched to
> inband/1000base-x link mode
> [ 7.889360] mvpp2 f2000000.ethernet eth0: configuring for
> inband/1000base-x link mode
> 
> However, link is down for the module even after forcing 1000BASE-X
> using GPON quirk. The module is documented as working in 1000BASE-X or
> SGMII mode and it is working fine when used in a MC220L converter.

It's possible that the MC220L ignores the RX_LOS signal. GPON modules
seem to be a law to themselves about their behaviours and so far I've
found that their compliance with the SFF MSAs is pretty abismal,
requiring some form of special casing for almost every one.

> Could you help me troubleshoot why the link is not up in mode
> 1000BASE-X ? Or can I force the link with SGMII maybe ? This is beyond
> my knowledge and kindly ask for some directions.
> 
> Patch for single byte read mode:
> 
>  drivers/net/phy/sfp.c | 35 +++++++++++++++++++++++++++++++----
>  1 file changed, 31 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/phy/sfp.c b/drivers/net/phy/sfp.c
> index a4fd1d2e7ba8..b1f10b801790 100644
> --- a/drivers/net/phy/sfp.c
> +++ b/drivers/net/phy/sfp.c
> @@ -224,6 +224,7 @@ struct sfp {
>   struct sfp_bus *sfp_bus;
>   struct phy_device *mod_phy;
>   const struct sff_data *type;
> + size_t i2c_block_size;
>   u32 max_power_mW;
> 
>   unsigned int (*get_state)(struct sfp *);
> @@ -359,8 +360,8 @@ static int sfp_i2c_read(struct sfp *sfp, bool a2,
> u8 dev_addr, void *buf,
> 
>   while (len) {
>   this_len = len;
> - if (this_len > 16)
> - this_len = 16;
> + if (this_len > sfp->i2c_block_size)
> + this_len = sfp->i2c_block_size;
> 
>   msgs[1].len = this_len;
> 
> @@ -1975,14 +1976,20 @@ static int sfp_sm_mod_probe(struct sfp *sfp,
> bool report)
>   u8 check;
>   int ret;
> 
> - ret = sfp_read(sfp, false, 0, &id, sizeof(id));
> + /* Some modules (CarlitoxxPro CPGOS03-0490) do not support multibyte
> + * reads from the EEPROM, so start by reading the base identifying
> + * information one byte at a time.
> + */
> + sfp->i2c_block_size = 1;
> +
> + ret = sfp_read(sfp, false, 0, &id.base, sizeof(id.base));
>   if (ret < 0) {
>   if (report)
>   dev_err(sfp->dev, "failed to read EEPROM: %d\n", ret);
>   return -EAGAIN;
>   }
> 
> - if (ret != sizeof(id)) {
> + if (ret != sizeof(id.base)) {
>   dev_err(sfp->dev, "EEPROM short read: %d\n", ret);
>   return -EAGAIN;
>   }
> @@ -2021,6 +2028,26 @@ static int sfp_sm_mod_probe(struct sfp *sfp, bool report)
>   }
>   }
> 
> + /* Some modules (Nokia 3FE46541AA) lock up if byte 0x51 is read as a
> + * single read. Switch back to reading 16 byte blocks unless we have
> + * a CarlitoxxPro module. The nice thing about manufacturers is they
> + * know how to make life difficult.
> + */
> + if (memcmp(id.base.vendor_name, "WHATEVER        ", 16))
> + sfp->i2c_block_size = 16;
> +
> + ret = sfp_read(sfp, false, SFP_CC_BASE + 1, &id.ext, sizeof(id.ext));
> + if (ret < 0) {
> + if (report)
> + dev_err(sfp->dev, "failed to read EEPROM: %d\n", ret);
> + return -EAGAIN;
> + }
> +
> + if (ret != sizeof(id.ext)) {
> + dev_err(sfp->dev, "EEPROM short read: %d\n", ret);
> + return -EAGAIN;
> + }
> +
>   check = sfp_check(&id.ext, sizeof(id.ext) - 1);
>   if (check != id.ext.cc_ext) {
>   if (cotsworks) {

I would also like a binary dump of both these modules EEPROM for my
archive of EEPROM contents (which allows me to test the EEPROM parsing
without needing the module itself) and also to complete my patch above.

$ ethtool -m eth0 raw on > module.bin

Thanks.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!