[BUG] rkvdec-vdpu383-h264: wrong pixels at horizontal de-blocking edges y=4 and y=12

[Simon Wright]

Hi Piotr,

Thanks - glad it's useful. If your RK3576 H.264 decode shows wrong pixels at the
same horizontal edges (luma rows 4 and 12 within each macroblock row, propagating
through the following P-frames), it's very likely the same bug.

On a PoC for 7.1, good news first: I checked the in-kernel driver, and the whole
drivers/media/platform/rockchip/rkvdec/ directory is byte-identical between 7.0 and
7.1-rc7 (as is the V4L2 stateless control uAPI). So there's nothing 7.1-specific to
do - the same fix covers both.

The less convenient part is that the fix is not a small patch to
rkvdec-vdpu383-h264.c. The root cause is that the BSP runs a one-shot "warmup"
priming decode at every decoder power-up (rk3576_workaround_run) that mainline
omits, and that warmup is driven through the VDPU383's link/CCU register bank.
Mainline's single-shot path doesn't carry that link infrastructure, so the warmup
lives in an out-of-tree file (rkvdec-link.c) that is part of a broader downstream
tree I've been building on top of mainline - link-mode support plus V4L2 stateless
VP9 and AV1 decoders for the VDPU383 (neither is upstream yet). The warmup isn't
self-contained in the H.264 file; it needs that link plumbing around it. So rather
than hand you a fragile micro-diff, the practical way to verify is to build and load
that downstream module, where the warmup is on by default:

    git clone https://github.com/SympleNZ/rkvdec-vdpu383-vp9
    cd rkvdec-vdpu383-vp9
    # against your running kernel's headers/source:
    KBUILD_MODPOST_WARN=1 make -C /lib/modules/$(uname -r)/build M=$PWD/src modules

    sudo rmmod rockchip_vdec 2>/dev/null    # unload the in-tree driver
    sudo insmod src/rockchip-vdec.ko        # warmup is default-on, no params needed

(If rockchip-vdec is built into your kernel rather than a module, you'll need it as
a module - or blacklisted - first. The module name may differ slightly on your
build.)

Then decode your H.264 as usual (v4l2slh264dec) and compare against avdec_h264 - the
rows-4/12 corruption should be gone. That repo is the VP9 work, but its src/ builds
the full VDPU381/383 H.264/H.265/VP9/AV1 driver, so H.264 is included.

One honest caveat for the media-player use case: the warmup fixes correctness (the
wrong pixels), not throughput - and the throughput limit isn't specific to H.264.
It's the mainline single-shot submit model (no link-mode pipelining yet), so it
applies equally to HEVC. (HEVC, incidentally, already decodes correctly on stock
mainline - the deblock bug is H.264-only - but it sits under the same throughput
ceiling.) Mainline decode on RK3576 is still well short of the BSP's rate, so
real-time 1080p/4K playback can still drop frames regardless of codec. This will
tell you whether your corruption is this bug; smooth playback across codecs is a
separate piece of work.

If it builds and runs, I'd be very interested in the result on your three boards -
independent confirmation across more RK3576 hardware would strengthen the report.
Happy to help if the build trips up.

Full write-up (root cause, the register kick sequence, validation):
https://github.com/SympleNZ/rkvdec-vdpu383-h264-bug/blob/master/FIX.md

Regards,
Simon