[RFC] V4L2 unified low-level decoder API

Fri Nov 4 06:55:10 PDT 2016

Hi Randy,

thanks for reply, some comments below:

On 10/27/2016 03:08 AM, Randy Li wrote:
>
>
> On 10/26/2016 11:09 PM, Hugues FRUCHET wrote:
>> Hi,
>>
>>
>>
>> This RFC aims to start discussions in order to define the codec specific
>> controls structures to fulfill the low-level decoder API needed by non
>> “Stream API” based decoders (“stateless” or “Frame API” based decoders).
>>
>> Several implementation exists now which runs on several SoC and various
>> software frameworks.
>>
>> The idea is to find the communalities between all those implementations
>> and SoC to define a single unified interface in V4L2 includes.
>>
>> Even if “Request API” is needed to pass those codec specific controls
>> from userspace down to kernel on a per-buffer basis, we can start
>> discussions and define the controls in parallel of its development.
> Yes, I have sent a one for H.264 decoder and JPEG encoder.
>>
>> We can even propose some implementations based on existing V4L2 control
>> framework (which doesn’t support “per-frame” basis) by ensuring
>> atomicity of sequence S_EXT_CTRL(header[i])/QBUF(stream[i]). Constraint
>> can then be relaxed when “Request API” is merged.
>>
>>
>>
>> I would like to propose to work on a “per-codec” basis, having at least
>> 2 different SoC and 2 different frameworks to test and validate controls.
>>
>> To do so, I have tried to identify some people that have worked on this
>> subject and have proposed some implementations, feel free to correct me
>> and enhance the list if needed:
>>
>> * MPEG2/MPEG4
>>
>>    - Florent Revest for Allwinner A13 CedarX support [1] tested with VLC
>> -> libVA + sunxi-cedrus-drv-video -> V4L2
>>
>>    - Myself for STMicroelectronics Delta support [2] tested with
>> GStreamer V4L2 -> libv4l2 + libv4l-delta plugin -> V4L2
>>
>>
>>
>> * VP8
>>
>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [3] tested with
>> Chromium -> V4L2
>>
>> - Jung Zhao for Rockchip RK3288 VPU support [4] <cannot find the
>> framework used>
> There is rockchip VDPAU driver supporting it, but it is .

Could you point out the code that is used ? Which application is used on 
top of VDPAU ?

>>
>>
>>
>> * H264
>>
>> - Pawel Osciak for Rockchip RK3288, RK3399? VPU Support [5] tested with
>> Chromium -> V4L2
>>
>> - Randy Li for Rockchip RK3288  VPU support [6] tested with VLC? ->
>> libVA + rockchip-va-driver -> V4L2
> I only tested it with Gstreamer -> VA-API element -> Rockchip VA-API
> driver -> V4L2

OK got it, thks !

>>
>>                                                                                                                          VLC?
>> -> libVDPAU + rockchip-va-driver -> V4L2
>>
>> I can work to define MPEG2/MPEG4 controls and propose functional
>> implementations for those codecs, and will be glad to co-work with you
>> Florent.
> But it may not work with Rockchip's SoC, you may check the following branch
> https://github.com/hizukiayaka/rockchip-video-driver/tree/rk_v4l2_mix

I have checked code and I have only found H264 support, do I miss 
something ?

>>
>> I can help on H264 on a code review basis based on the functional H264
>> setup I have in-house and codec knowledge, but I cannot provide
>> implementation in a reasonable timeframe, same for VP8.
>>
>>
>>
>> Apart of very details of each codec, we have also to state about generic
>> concerns such as:
>>
>> -          new pixel format introduction (VP8 => VP8F, H264 => S264,
>> MPG2 => MG2F, MPG4 => MG4F)
> I don't think it is necessary.

But currently it is done that way in all patches proposals I have seen 
so far, including rockchip:
rockchip_decoder_v4l2.c:{VAProfileH264Baseline,V4L2_PIX_FMT_H264_SLICE},

We have to state about it all together. Seems natural to me to do this 
way instead of device caps.
Doing so user knows that the driver is based on "Frame API" -so 
additional headers are required to decode input stream- and not
on "Stream API" -H264 stream can be decoded directly-.

>>
>> -          new device caps to indicate that driver requires extra
>> headers ? maybe not needed because redundant with new pixel format
> I prefer this one.
>>
>> -          continue to modify v4l2-controls.h ? or do we add some new
>> specific header files (H264 is huge!) ?
> Not huge. You could check rockchip's kernel.

Checking 
https://github.com/hizukiayaka/rockchip-video-driver/blob/rk_v4l2_mix/librkdec/include/linux/v4l2-controls.h
it's already 400 lines of types definition for H264, VP8 and VP9.

>>
>> -          how to manage sequence header & picture header,
>> optional/extended controls (MPEG2 sequence/picture extensions, H264 SEI,
>> …). Personally I have added flags inside a single control structure,
>> H264 is done in a different way using several controls
>> (SPS/PPS/SLICE/DECODE/…)
> the last one is dpb, except the dpb, it would have the same numbers of
> controls to those structures defined in VA-API H264 decoder.

I'm a bit lost because of differences I see between implementations [5] 
and [6].
Digging into rk_v4l2_mix, I see that implementation is done using [5] 
interface:
https://github.com/hizukiayaka/rockchip-video-driver/blob/rk_v4l2_mix/librkdec/include/linux/v4l2-controls.h
and not the one pushed on V4L2 mailing list ([6] 
http://www.spinics.net/lists/linux-media/msg105095.html).

>>
>>
>> Thanks you to all of you for your attention and feel free to react on
>> this topic if you are interested to work on this subject.
> Currently, I have to pause the process of VA-API drive, and moving to
> the other idea I have said before, creating a new API in userspace

Could you point out the thread which gives details on this ?
Does this break the H264 V4L2 decoder settings defined in [5] ?

> (but won't archive the goal I set before in this step). There are some
> shortages in VA-API I have said in last email making the performance in
> 4K video and extending the Gstreamer VA-API is a little difficult job
> and need more time.
> And the development for the new VPU driver for rockchip would pause a
> while as well.
>
> It would not be a long time(a few weeks) and I am still available in my
> free time(at home). It is good to know the wheel begin to roll. And do
> feel free to assign job to me.

Thanks Randy, let's continue discussion and try to involve more people 
on subject in order that we converge.

>>
>>
>>
>> Best regards,
>>
>> Hugues.
>>
>>
>>
>> [0] [ANN] Codec & Request API Brainstorm meeting Oct 10 & 11
>> https://www.spinics.net/lists/linux-media/msg106699.html
>>
>> [1] MPEG2 A13 CedarX http://www.spinics.net/lists/linux-media/msg104823.html
>>
>> [1] MPEG4 A13 CedarX http://www.spinics.net/lists/linux-media/msg104817.html
>>
>> [2] MPEG2 STi4xx Delta
>> http://www.spinics.net/lists/linux-media/msg106240.html
>>
>> [2] MPEG4 STi4xx Delta is also supported but not yet pushed
>>
>> [3] VP8 Rockchip RK3288, RK3399? VPU
>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0002-CHROMIUM-v4l-Add-VP8-low-level-decoder-API-controls.patch
>>
>>
>> [4] VP8 Rockchip RK3288 VPU
>> http://www.spinics.net/lists/linux-media/msg97997.html
>>
>> [5] H264 Rockchip RK3288, RK3399? VPU
>> https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch
>>
>> [6] H264 Rockchip RK3288 VPU
>> http://www.spinics.net/lists/linux-media/msg105095.html
>>
>>
>>
>