[RFCv2 0/3] vp9 v4l2 stateless uapi

Wed May 5 05:38:33 PDT 2021

Dear All,

This is a follow-up work for https://patchwork.linuxtv.org/project/linux-media/list/?series=5268
I addressed Hans's comments.

Changes:

v1..v2:
- improve documentation
- imrpove coding style
- factor out of common vp9 code into v4l2-vp9.c
- rename V4L2_CID_STATELESS_VP9_FRAME_DECODE_PARAMS into V4L2_CID_STATELESS_VP9_FRAME

This is still sent as an RFC because the works for adding the second driver (g2 at imx8)
are ongoing.

The v1 was an RFC on stateless uapi for vp9 decoding with v4l2, which was based on
https://lkml.org/lkml/2020/11/2/1043, but had been substantially reworked. The important
change was that the v4l2 control used to pass boolean decoder probabilities had been made
unidirectional, and was renamed V4L2_CID_STATELESS_VP9_COMPRESSED_HDR_PROBS.

In the original proposal from Boris, to queue a frame the userspace must fully dequeue
the previous one, which effectively results in a forced lockstep behavior and defeats
vb2's capability to enqueue multiple buffers. Such a design was a consequence of backward
probability updates being performed by the kernel driver (which has direct access to
appropriate counter values) but forward probability updates being coupled with compressed
header parsing performed by the userspace.

In vp9 the boolean decoder used to decode the bitstream needs certain parameters to work.
Those are probabilities, which change with each frame. After each frame is decoded it is
known how many times a given symbol occured in the frame, so the probabilities can be adapted.
This process is known as backward probabilities update. A next frame header can also contain
information which modifies probabilities resulting from backward update. The said modification
is called forward probabilities update. The data for backward update is generated by the decoder
hardware, while the data for forward update is prepared by reading the compressed frame header.
The natural place to parse something is userspace, while the natural place to access
hardware-provided counters is the kernel. Such responsibilties assignment was used in the
original work.

To overcome the lockstep, we moved forward probability updates to the kernel, while leaving
parsing them in userspace. This way the v4l2 control which is used to pass the probs becomes
unidirectional (user->kernel) and the userspace can keep parsing and enqueueing succeeding
frames.

If a particular driver parses the compressed header and does backward probability updates
on its own then V4L2_CID_STATELESS_VP9_COMPRESSED_HDR_PROBS does not need to be used.

This series adds vp9 uapi in proper locations, which means it is a proper, "official" uapi,
as opposed to staging uapi which was proposed in the above mentioned lkml thread.

The series adds vp9 support to rkvdec driver.

Rebased onto media_tree, requires this patch:

https://patchwork.linuxtv.org/project/linux-media/patch/20210505122347.7576-2-andrzej.p@collabora.com/

You can test rkvdec implementation with gstreamer, please clone gstreamer and then
use these branches for -base and -bad:

https://gitlab.freedesktop.org/dwlsalmeida/gst-plugins-base/-/tree/vp9-upstream-padding
https://gitlab.freedesktop.org/dwlsalmeida/gst-plugins-bad/-/tree/vp9-upstream

Example invocation:

without format conversion:

gst-launch-1.0 filesrc location=Big_Buck_Bunny_1080_10s_1MB.webm ! parsebin ! v4l2slvp9dec ! filesink location=out.yuv

with format conversion to match vpxdec output:

gst-launch-1.0 filesrc location=Big_Buck_Bunny_1080_10s_1MB.webm ! parsebin ! v4l2slvp9dec ! videoconvert ! video/x-raw,format=I420 ! filesink location=out.yuv

I kindly ask for your comments.

Andrzej Pietrasiewicz (2):
  media: uapi: Add VP9 stateless decoder controls
  media: uapi: Add VP9 v4l2 library

Boris Brezillon (1):
  media: rkvdec: Add the VP9 backend

 .../userspace-api/media/v4l/biblio.rst        |   10 +
 .../media/v4l/ext-ctrls-codec-stateless.rst   |  547 +++++
 .../media/v4l/pixfmt-compressed.rst           |   15 +
 .../media/v4l/vidioc-g-ext-ctrls.rst          |    8 +
 .../media/v4l/vidioc-queryctrl.rst            |   12 +
 .../media/videodev2.h.rst.exceptions          |    2 +
 drivers/media/v4l2-core/Kconfig               |    4 +
 drivers/media/v4l2-core/Makefile              |    1 +
 drivers/media/v4l2-core/v4l2-ctrls.c          |  229 +++
 drivers/media/v4l2-core/v4l2-ioctl.c          |    1 +
 drivers/media/v4l2-core/v4l2-vp9.c            | 1831 +++++++++++++++++
 drivers/staging/media/rkvdec/Kconfig          |    1 +
 drivers/staging/media/rkvdec/Makefile         |    2 +-
 drivers/staging/media/rkvdec/rkvdec-vp9.c     | 1084 ++++++++++
 drivers/staging/media/rkvdec/rkvdec.c         |   52 +-
 drivers/staging/media/rkvdec/rkvdec.h         |    6 +
 include/media/v4l2-ctrls.h                    |    4 +
 include/media/v4l2-vp9.h                      |  168 ++
 include/uapi/linux/v4l2-controls.h            |  425 ++++
 include/uapi/linux/videodev2.h                |    6 +
 20 files changed, 4403 insertions(+), 5 deletions(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-vp9.c
 create mode 100644 drivers/staging/media/rkvdec/rkvdec-vp9.c
 create mode 100644 include/media/v4l2-vp9.h

base-commit: 0b276e470a4d43e1365d3eb53c608a3d208cabd4
prerequisite-patch-id: d148a5f17bbb03e88c3744a166d1dd2c6088ce7d
prerequisite-patch-id: 1817161060ef577cff00a8f7892eb9cbbfa2f327
prerequisite-patch-id: 5d4236886083146c81f4234d5f97acf6fd6dd4e4
-- 
2.17.1