[PATCH 2/3] iommu/rockchip: Disable the device link during resume

[Jagan Teki]

Hi Heiko/Kever/Simon,

On Tue, Apr 4, 2023 at 1:21 PM Jagan Teki <jagan at amarulasolutions.com> wrote:
>
> On Thu, Mar 30, 2023 at 7:13 PM Robin Murphy <robin.murphy at arm.com> wrote:
> >
> > On 2023-03-30 14:17, Jagan Teki wrote:
> > > Rockchip iommu is trying to enable the associated device at runtime
> > > resume however some devices might enable the iommu during their pm
> > > runtime resume operation which indeed leads iommu to use the wrong
> > > domain and this leads to device iommu page fault.
> > >
> > > An example of this behavior has been observed in Rockchip RK3328, where
> > > iommu stalls request timeout dring VOP device enablement.
> > >
> > > Here is the dmesg log for the same:
> > >
> > > rockchip-drm display-subsystem: bound ff370000.vop (ops vop_component_ops)
> > > dwhdmi-rockchip ff3c0000.hdmi: supply avdd-0v9 not found, using dummy regulator
> > > rk_iommu ff373f00.iommu: Enable stall request timed out, status: 0x00004b
> > > dwhdmi-rockchip ff3c0000.hdmi: supply avdd-1v8 not found, using dummy regulator
> > > rk_iommu ff373f00.iommu: Disable paging request timed out, status: 0x00004b
> > > dwhdmi-rockchip ff3c0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (inno_dw_hdmi_phy2)
> > > dwhdmi-rockchip ff3c0000.hdmi: registered DesignWare HDMI I2C bus driver
> > > rockchip-drm display-subsystem: bound ff3c0000.hdmi (ops dw_hdmi_rockchip_ops)
> > > [drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 0
> > >
> > > This issue is reproduced if we enable the display in U-Boot however
> > > U-Boot is not even touched any iommu register as the U-Boot display
> > > uses the simple frame buffer like other Rockchip platforms RK3399,
> > > and RK3328 do.
> > >
> > > When VOP is trying to enable the iommu using runtime resume call
> > > pm_runtime_resume_and_get from @vop_enable then the iommu runtime
> > > resume call @rk_iommu_resume will try to attach the VOP in the wrong
> > > domain via @rk_iommu_enable will lead to the vop iommu page fault.
> >
> > That sounds like a driver bug. The whole point of the device link is
>
> Do you mean the bug in rockchip-iommu.c or vop?
>
> > supposed to be that the IOMMU gets suspended after the VOP, and resumed
> > before it, so it can make sure that whatever translations the VOP was
> > using are restored *before* the VOP starts trying to access them again.
> > If the IOMMU driver is failing to restore the correct state on resume,
> > no amount of DT abuse is the right answer.
>
> Then how can we handle the co-relation b/w them as VOP already
> attaching the iommu and at the same time IOMMU trying to enable VOP
> device but referring to the wrong domain? Any suggestions?
>
> >
> > I can understand if the IOMMU itself expects to be idle for the initial
> > configuration at probe time, and gets unhappy if we try to reset it
> > while (bypass) VOP traffic for the bootloader framebuffer is still going
> > through, but that's an entirely different issue, and again hacking
>
> Does it mean accessing VOP traffic at the bootloader stage effecting
> iommu even though the VOP drivers in the bootloader are not using
> iommu at all?

Any suggestions on this issue? we found similar issues even with
upcoming RK SoCs - RV1126, RK3566, RK3588.

Thanks,
Jagan.