MacchiatoBin SingleShot 2500BASE-X port (eth3) with GPON SFP on mvpp2x

Russell King - ARM Linux admin linux at armlinux.org.uk
Thu Mar 14 17:22:21 PDT 2019


On Thu, Mar 14, 2019 at 03:08:18PM -0400, Marc Micalizzi wrote:
> Mar 14 14:42:21 empire kernel: [   81.073356] bnx2x:
> [bnx2x_attn_int_deasserted:5146(eth0)]attn: 00000010 00000000 02180000
> 00000000 00000000
> Mar 14 14:42:21 empire kernel: [   81.073359] bnx2x:
> [bnx2x_attn_int_deasserted:5156(eth0)]group[0]: 00008010 fff55fff
> 0000ffff f00003e0 000000fc

This is the start of processing the module insertion

> Mar 14 14:42:21 empire kernel: [   81.073364] bnx2x:
> [bnx2x_sfp_set_transmitter:7855(eth0)]Setting SFP+ transmitter to 1
> Mar 14 14:42:21 empire kernel: [   81.073367] bnx2x:
> [bnx2x_sfp_e3_set_transmitter:4492(eth0)]Setting WC TX to 1
> Mar 14 14:42:21 empire kernel: [   81.073369] bnx2x:
> [bnx2x_set_epio:395(eth0)]Setting EPIO pin 21 to 0
> Mar 14 14:42:21 empire kernel: [   81.073374] bnx2x:
> [bnx2x_set_sfp_module_fault_led:8593(eth0)]Setting SFP+ module fault
> LED to 1
> Mar 14 14:42:21 empire kernel: [   81.073377] bnx2x:
> [bnx2x_set_e3_module_fault_led:8585(eth0)]Setting Fault LED to 1 using
> pin cfg 5
> Mar 14 14:42:21 empire kernel: [   81.073382] bnx2x:
> [bnx2x_set_gpio:2132(eth0)]Set GPIO 0 (shift 4) -> output high
> Mar 14 14:42:21 empire kernel: [   81.073479] bnx2x:
> [bnx2x_power_sfp_module:8622(eth0)]Setting SFP+ power to 1
> Mar 14 14:42:21 empire kernel: [   81.073490] bnx2x:
> [bnx2x_warpcore_power_module:7944(eth0)]Setting SFP+ module power to 1
> using pin cfg 16
> Mar 14 14:42:21 empire kernel: [   81.073499] bnx2x:
> [bnx2x_set_epio:395(eth0)]Setting EPIO pin 7 to 0

Here, we power up the module - no idea what state the modules
TX_DISABLE signal is.

> Mar 14 14:42:21 empire kernel: [   81.073512] bnx2x:
> [bnx2x_set_gpio_int:2226(eth0)]Clear GPIO INT 2 (shift 2) -> output
> low
> Mar 14 14:42:21 empire kernel: [   81.073516] bnx2x:
> [bnx2x_bsc_module_sel:3080(eth0)]Setting BSC switch
> Mar 14 14:42:21 empire kernel: [   81.073521] bnx2x:
> [bnx2x_set_epio:395(eth0)]Setting EPIO pin 29 to 1
> Mar 14 14:42:21 empire kernel: [   81.084303] bnx2x:
> [bnx2x_bsc_read:3129(eth0)]wr 0 byte timed out after 1002 try

We're trying to read the module EEPROM, which fails each time we see
the above message.

> Mar 14 14:42:21 empire kernel: [   81.084306] bnx2x:
> [bnx2x_bsc_module_sel:3080(eth0)]Setting BSC switch
> Mar 14 14:42:21 empire kernel: [   81.084307] bnx2x:
> [bnx2x_set_epio:395(eth0)]Setting EPIO pin 29 to 1
> Mar 14 14:42:21 empire kernel: [   81.094984] bnx2x:
> [bnx2x_bsc_read:3129(eth0)]wr 0 byte timed out after 1002 try
> Mar 14 14:42:21 empire kernel: [   81.094986] bnx2x:
> [bnx2x_bsc_module_sel:3080(eth0)]Setting BSC switch
> ++++++++++++repeats 50-100 times++++++++++++

It'll only try twice, then power down the module and power it back up,
then try one more time, and repeat that whole process 60 times.

...
> [bnx2x_bsc_read:3129(eth0)]wr 0 byte timed out after 1002 try
> Mar 14 14:42:23 empire kernel: [   83.614992] bnx2x:
> [bnx2x_warpcore_power_module:7944(eth0)]Setting SFP+ module power to 0
> using pin cfg 16
> Mar 14 14:42:23 empire kernel: [   83.614993] bnx2x:
> [bnx2x_set_epio:395(eth0)]Setting EPIO pin 7 to 1
> Mar 14 14:42:23 empire kernel: [   83.616568] bnx2x:
> [bnx2x_warpcore_power_module:7944(eth0)]Setting SFP+ module power to 1
> using pin cfg 16
> Mar 14 14:42:23 empire kernel: [   83.616571] bnx2x:
> [bnx2x_set_epio:395(eth0)]Setting EPIO pin 7 to 0

That's one of the module power cycles.

> Mar 14 14:42:23 empire kernel: [   83.616575] bnx2x:
> [bnx2x_bsc_module_sel:3080(eth0)]Setting BSC switch
> Mar 14 14:42:23 empire kernel: [   83.616576] bnx2x:
> [bnx2x_set_epio:395(eth0)]Setting EPIO pin 29 to 1
> Mar 14 14:42:23 empire kernel: [   83.627247] bnx2x:
> [bnx2x_bsc_read:3129(eth0)]wr 0 byte timed out after 1002 try
> Mar 14 14:42:23 empire kernel: [   83.627248] bnx2x:
> [bnx2x_handle_module_detect_int:8805(eth0)]SFP+ module is not
> initialized

and here, we give up trying in bnx2x_handle_module_detect_int().
So, at this point, we have not read the EEPROM, and I don't see
any further attempts to try and read the EEPROM.

> Mar 14 14:42:23 empire kernel: [   83.627251] bnx2x:
> [bnx2x_attn_int_deasserted:5181(eth0)]about to mask 0xfffffffe at IGU
> addr 0x442d10
> Mar 14 14:42:23 empire kernel: [   83.627255] bnx2x:
> [bnx2x_attn_int_deasserted:5194(eth0)]aeu_mask 16  newly deasserted 1
> Mar 14 14:42:23 empire kernel: [   83.627255] bnx2x:
> [bnx2x_attn_int_deasserted:5196(eth0)]new mask 17
> Mar 14 14:42:23 empire kernel: [   83.627257] bnx2x:
> [bnx2x_attn_int_deasserted:5201(eth0)]attn_state 801
> Mar 14 14:42:23 empire kernel: [   83.627258] bnx2x:
> [bnx2x_attn_int_deasserted:5203(eth0)]new state 800
> Mar 14 14:42:23 empire kernel: [   83.627259] bnx2x:
> [bnx2x_igu_ack_sb_gen:652(eth0)]write 0x01400002 to IGU addr 0x442000
> Mar 14 14:42:23 empire kernel: [   83.627287] bnx2x:
> [bnx2x_igu_ack_sb_gen:652(eth0)]write 0x02200000 to IGU addr 0x442000
> Mar 14 14:42:23 empire kernel: [   83.627422] bnx2x:
> [bnx2x_analyze_link_error:13741(eth0)]Analyze TX Fault

This appears to be a periodic task executed once per second.

> Mar 14 14:42:23 empire kernel: [   83.627423] bnx2x:
> [bnx2x_analyze_link_error:13747(eth0)]Link changed:[0 0]->1
> Mar 14 14:42:23 empire kernel: [   83.627424] bnx2x:
> [bnx2x_sfp_tx_fault_detection:13895(eth0)]Change TX_Fault LED: ->0
> Mar 14 14:42:23 empire kernel: [   83.627426] bnx2x:
> [bnx2x_set_e3_module_fault_led:8585(eth0)]Setting Fault LED to 0 using
> pin cfg 5
> Mar 14 14:42:23 empire kernel: [   83.627429] bnx2x:
> [bnx2x_set_gpio:2123(eth0)]Set GPIO 0 (shift 4) -> output low
> Mar 14 14:42:23 empire kernel: [   83.627432] bnx2x:
> [bnx2x_sp_task:5681(eth0)]sp task invoked
> Mar 14 14:42:23 empire kernel: [   83.627434] bnx2x:
> [bnx2x_sp_task:5690(eth0)]status 1
> Mar 14 14:42:23 empire kernel: [   83.627434] bnx2x:
> [bnx2x_sp_task:5691(eth0)]setting interrupt_occurred to 0
> Mar 14 14:42:23 empire kernel: [   83.627436] bnx2x:
> [bnx2x_attn_int:5221(eth0)]attn_bits 0  attn_ack 801  asserted 0
> deasserted 800
> Mar 14 14:42:23 empire kernel: [   83.627444] bnx2x:
> [bnx2x_attn_int_deasserted:5146(eth0)]attn: 00000000 00000000 02180000
> 00000000 00000000
> Mar 14 14:42:23 empire kernel: [   83.627446] bnx2x:
> [bnx2x_attn_int_deasserted:5181(eth0)]about to mask 0xfffff7ff at IGU
> addr 0x442d10
> Mar 14 14:42:23 empire kernel: [   83.627449] bnx2x:
> [bnx2x_attn_int_deasserted:5194(eth0)]aeu_mask 17  newly deasserted
> 800
> Mar 14 14:42:23 empire kernel: [   83.627449] bnx2x:
> [bnx2x_attn_int_deasserted:5196(eth0)]new mask 17
> Mar 14 14:42:23 empire kernel: [   83.627451] bnx2x:
> [bnx2x_attn_int_deasserted:5201(eth0)]attn_state 800
> Mar 14 14:42:23 empire kernel: [   83.627452] bnx2x:
> [bnx2x_attn_int_deasserted:5203(eth0)]new state 0
> Mar 14 14:42:23 empire kernel: [   83.627453] bnx2x:
> [bnx2x_igu_ack_sb_gen:652(eth0)]write 0x01400003 to IGU addr 0x442000
> Mar 14 14:42:24 empire kernel: [   84.193629] bnx2x:
> [bnx2x_stats_handle:1397(eth0)]state 0 -> event 2 -> state 0
> Mar 14 14:42:24 empire kernel: [   84.641795] bnx2x:
> [bnx2x_analyze_link_error:13741(eth0)]Analyze TX Fault
> Mar 14 14:42:24 empire kernel: [   84.641798] bnx2x:
> [bnx2x_analyze_link_error:13747(eth0)]Link changed:[0 0]->1
> Mar 14 14:42:24 empire kernel: [   84.641801] bnx2x:
> [bnx2x_sfp_tx_fault_detection:13895(eth0)]Change TX_Fault LED: ->0
> Mar 14 14:42:24 empire kernel: [   84.641804] bnx2x:
> [bnx2x_set_e3_module_fault_led:8585(eth0)]Setting Fault LED to 0 using
> pin cfg 5
> Mar 14 14:42:24 empire kernel: [   84.641809] bnx2x:
> [bnx2x_set_gpio:2123(eth0)]Set GPIO 0 (shift 4) -> output low
> Mar 14 14:42:25 empire kernel: [   85.217675] bnx2x:
> [bnx2x_stats_handle:1397(eth0)]state 0 -> event 2 -> state 0
> Mar 14 14:42:25 empire kernel: [   85.665873] bnx2x:
> [bnx2x_analyze_link_error:13741(eth0)]Analyze TX Fault
> Mar 14 14:42:25 empire kernel: [   85.665876] bnx2x:
> [bnx2x_analyze_link_error:13747(eth0)]Link changed:[0 0]->1
> Mar 14 14:42:25 empire kernel: [   85.665878] bnx2x:
> [bnx2x_sfp_tx_fault_detection:13895(eth0)]Change TX_Fault LED: ->0
> Mar 14 14:42:25 empire kernel: [   85.665882] bnx2x:
> [bnx2x_set_e3_module_fault_led:8585(eth0)]Setting Fault LED to 0 using
> pin cfg 5
> ++++++++++++fault repeats for a while++++++++++++
> Mar 14 14:43:09 empire kernel: [  129.251715] bnx2x:
> [bnx2x_stats_handle:1397(eth0)]state 0 -> event 2 -> state 0
> Mar 14 14:43:09 empire kernel: [  129.699906] bnx2x:
> [bnx2x_analyze_link_error:13741(eth0)]Analyze TX Fault
> Mar 14 14:43:09 empire kernel: [  129.699910] bnx2x:
> [bnx2x_analyze_link_error:13747(eth0)]Link changed:[0 0]->1
> Mar 14 14:43:09 empire kernel: [  129.699912] bnx2x:
> [bnx2x_sfp_tx_fault_detection:13895(eth0)]Change TX_Fault LED: ->0
> Mar 14 14:43:09 empire kernel: [  129.699915] bnx2x:
> [bnx2x_set_e3_module_fault_led:8585(eth0)]Setting Fault LED to 0 using
> pin cfg 5
> Mar 14 14:43:09 empire kernel: [  129.699920] bnx2x:
> [bnx2x_set_gpio:2123(eth0)]Set GPIO 0 (shift 4) -> output low
> Mar 14 14:43:10 empire kernel: [  130.275764] bnx2x:
> [bnx2x_stats_handle:1397(eth0)]state 0 -> event 2 -> state 0

This being the last of them...

> Mar 14 14:43:10 empire kernel: [  130.439097] bnx2x:
> [bnx2x_igu_ack_sb_gen:652(eth0)]write 0x02200000 to IGU addr 0x442000
> Mar 14 14:43:10 empire kernel: [  130.439148] bnx2x:
> [bnx2x_sp_task:5681(eth0)]sp task invoked
> Mar 14 14:43:10 empire kernel: [  130.439152] bnx2x:
> [bnx2x_sp_task:5690(eth0)]status 1
> Mar 14 14:43:10 empire kernel: [  130.439153] bnx2x:
> [bnx2x_sp_task:5691(eth0)]setting interrupt_occurred to 0
> Mar 14 14:43:10 empire kernel: [  130.439157] bnx2x:
> [bnx2x_attn_int:5221(eth0)]attn_bits 100  attn_ack 0  asserted 100
> deasserted 0
> Mar 14 14:43:10 empire kernel: [  130.439162] bnx2x:
> [bnx2x_attn_int_asserted:4026(eth0)]aeu_mask 17  newly asserted 100
> Mar 14 14:43:10 empire kernel: [  130.439164] bnx2x:
> [bnx2x_attn_int_asserted:4028(eth0)]new mask 17
> Mar 14 14:43:10 empire kernel: [  130.439166] bnx2x:
> [bnx2x_attn_int_asserted:4033(eth0)]attn_state 0
> Mar 14 14:43:10 empire kernel: [  130.439168] bnx2x:
> [bnx2x_attn_int_asserted:4035(eth0)]new state 100
> Mar 14 14:43:10 empire kernel: [  130.439174] bnx2x:
> [bnx2x_stats_handle:1397(eth0)]state 0 -> event 3 -> state 0
> Mar 14 14:43:10 empire kernel: [  130.439255] bnx2x:
> [bnx2x_link_update:6841(eth0)]port 0, XGXS?1, int_status 0x0

This looks like we got an XGXS interrupt, indicating that the link
came up at 2500Mbps - it looks like bnx2x has the ability to
detect the link speed from the module, which we don't have on the
mcbin.

> Mar 14 14:43:10 empire kernel: [  130.439259] bnx2x:
> [bnx2x_link_update:6848(eth0)]int_mask 0x0 MI_INT 0, SERDES_LINK 0
> Mar 14 14:43:10 empire kernel: [  130.439262] bnx2x:
> [bnx2x_link_update:6852(eth0)] 10G 0, XGXS_LINK f
> Mar 14 14:43:10 empire kernel: [  130.439337] bnx2x:
> [bnx2x_warpcore_read_status:5735(eth0)]0x81d1 = 0x400
> Mar 14 14:43:10 empire kernel: [  130.439411] bnx2x:
> [bnx2x_warpcore_read_status:5808(eth0)]lane 2 gp_speed 0x203
> Mar 14 14:43:10 empire kernel: [  130.439413] bnx2x:
> [bnx2x_get_link_speed_duplex:5548(eth0)]phy link up
> Mar 14 14:43:10 empire kernel: [  130.439416] bnx2x:
> [bnx2x_get_link_speed_duplex:5624(eth0)] phy_link_up 1 line_speed 2500

and here it decodes the speed by reading from the PHY.

So, it seems bnx2x also tries to read the module EEPROM, but fails
to, and while trying it keeps power cycling the module, eventually
giving up.  It doesn't power down the module, but leaves it powered.
After a while, the PHY on the board detects that one of its SERDES
lanes (which is connected to the SFP port) has link at 2.5Gbps, and
configures appropriately for that.

So, bnx2x is not doing this through being able to read the EEPROM
and configure based on what it found, but by having the capability
to detect the SERDES signal coming from the SFP, work out what
speed and (presumably) negotiation protocol (802.3z vs SGMII) is
in use and use that to bring up the link.

The Armada 8040 does not have that capability: it is unable to
automatically detect the SERDES protocol, needing software to
program several different hardware blocks according to the
desired speed.  The only thing I can think of doing in this
circumstance is to allow complete manual configuration. Hmm.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up



More information about the linux-arm-kernel mailing list