SM8250 USB-C DP alt mode: DP AUX times out in reversed (CC2) cable orientation

[BurningHoryd]

Hi,

I'm chasing a DisplayPort-over-USB-C alt mode bug on an SM8250 (Retroid
Pocket 5) board and would appreciate a pointer, as it looks like it may
be in the nb7vpq904m redriver or the qmp-combo DP AUX path.

Hardware / software
-------------------
- SoC: SM8250, qcom,sm8250-qmp-usb3-dp-phy combo PHY (88e8000.phy)
- USB-C redriver: onnn,nb7vpq904m (i2c, typec mux/switch)
- External sink: VITURE Beast XR glasses, 4-lane DP-only (pin assignment
  C, TYPEC_DP_STATE_C), HBR2, 1920x1200 at 120
- Kernel: v7.2 base, plus three local qmp-combo patches that are the
  subject of a separate downstream review (keep usb_init_count across the
  DP-only mux switch / skip USB3 power on/off while already in DP-only /
  the residual pipe_clk-in-common timeout hunk). The AUX failure below
  reproduces with those applied; I have not been able to bisect against
  pristine v7.2 because plain v7.2 has an unrelated regression on this
  board (qcom_pmic_typec now requires connector/vbus-supply, which the
  board DT lacks, so DP alt mode never powers up at all without a
  separate DT fix).

Symptom
-------
Normal cable orientation: DP alt mode works perfectly - link trains,
1920x1200 at 120, stable.

Reversed cable orientation (Type-C orientation = reverse / CC2): DP never
comes up. The DP alt mode HPD notification reaches msm_dp
(msm_dp_bridge_hpd_notify status=1), msm_dp starts bring-up
(msm_dp_display_host_phy_init, phy_init OK), but the very first DPCD read
- SINK_COUNT at 0x200 in msm_dp_hpd_plug_handle - polls for ~4 seconds
and never gets a reply:

  [drm:msm_dp_hpd_plug_handle] Before, sink_count=0
  [drm:msm_dp_display_host_phy_init] core_init=1 phy_init=1
  ... ~4 s ...
  [drm:msm_dp_hpd_plug_handle] After, sink_count=0
  [drm:msm_dp_bridge_detect] aux link status: 0
  [drm:msm_dp_bridge_detect] failed to read caps

msm_dp then gives up and the connector stays disconnected. Link training
is never reached. Stock Android on the exact same hardware works in both
orientations, so this is a driver-side issue, not a board/silicon limit.

What I've verified (instrumented kernel, dev_info in the phy + nb7 set
paths)
------------------------------------------------------------------------
- It is NOT a state / refcount / mux ordering artifact: a clean
  reverse-first plug (fresh boot, reversed cable plugged before any
  normal plug) fails identically.
- On the reversed plug, in order:
  * qmp_combo_typec_switch_set(REVERSE) runs, qmp_combo_com_init(force)
    writes QPHY_V3_DP_COM_TYPEC_CTRL = SW_PORTSELECT_MUX|SW_PORTSELECT_VAL
    (0x3).
  * nb7vpq904m_set() runs for TYPEC_DP_STATE_C with reverse=1, writes
    AUX_CC_REG (0x09) = 0x1.
  * qmp_combo_typec_mux_set() does the DP-only transition, com_init(force)
    again (mode=DP_ONLY, TYPEC_CTRL=0x3) - this happens AFTER the nb7 DP
    config.
  * qmp_combo_dp_init() -> qmp_v4_dp_aux_init() runs.
  So the PHY and the redriver are both configured for reverse, in a sane
  order, before msm_dp attempts AUX. And AUX still times out.
- qmp-combo has no DP-AUX orientation register that I can find - AUX
  orientation on this design is entirely nb7vpq904m AUX_CC_REG.
- Inverting SW_PORTSELECT_VAL and/or nb7 AUX_CC (making the registers
  bit-identical to the working normal-orientation case) does not help -
  reverse still fails.
- Forcing repeated full qmp_combo_com_exit/com_init cycles during the
  DPCD poll window does not help.

The one thing that changes anything
-----------------------------------
I added a knob to override GEN_DEV_SETTINGS OP_MODE in the nb7
TYPEC_DP_STATE_C branch (it normally hard-codes
GEN_DEV_SET_OP_MODE_DP_4LANE):

  OP_MODE = DP_4LANE (2)      -> DPCD read: 0 successes (many tries)
  OP_MODE = DP_CC1   (1)      -> DPCD read: 0 successes
  OP_MODE = DP_CC2   (0)      -> DPCD read succeeds intermittently
                                 (~1 in 4 plugs, sink_count 0->1,
                                 connector goes connected briefly).
                                 Link training then fails, which is
                                 expected since DP_CC2 is a 2-lane
                                 op-mode and the source is driving 4.
  OP_MODE = 3, 4             -> 0 successes

DP_CC2 is the "flipped orientation" op-mode. It is the only setting that
ever gets an AUX transaction through in reversed orientation. That
strongly suggests the nb7vpq904m SBU/AUX switch is not being put into the
flipped routing for a 4-lane DP session - AUX_CC_REG alone does not seem
to do it in DP_4LANE op-mode - but I don't have the full NB7VPQ904M
register map to confirm (the public datasheets are image-only).

Questions
---------
1. Is nb7vpq904m's DP_4LANE path missing SBU/AUX orientation handling
   that the DP_CC1/DP_CC2 paths get implicitly from the op-mode? Should
   AUX_CC_REG be sufficient in DP_4LANE mode, or is another
   GEN_DEV_SETTINGS / AUX register bit needed for CC2?
2. Is there an SM8250 qmp-combo DP-AUX orientation step that mainline is
   missing (something the downstream PHY driver does for a reversed
   4-lane DP session)?
3. Any known-good reference for reversed-orientation 4-lane DP on an
   SM8250 + nb7vpq904m board?

I can share the full instrumented traces (normal vs reverse) and test the
register map if someone can point at the right bits. Happy to turn a fix
into a proper patch.

Thanks,
BurningHoryd