[PATCH v2 00/27] media: Add support for multi-context operations
John Cox
jc at kynesim.co.uk
Thu Aug 20 09:09:41 PDT 2026
>
> Modern ISPs are designed to handle multiple "streams" of data, not necessarily
> related to the same image or video stream.
>
> The hardware resources are generally time-multiplexed between different
> execution contexts at the hardware or firmware level and in order to operate the
> ISP with multiple video sources it is necessary for drivers to keep track of
> per-context data and resources.
>
> In V4L2 the M2M framework supports multiple contexts through multiple opens of
> the same video device. This doesn't however support drivers exposing multiple
> video devices and sub-devices. Several out-of-tree drivers implement
> multi-context support by registering multiple 'logical' instances of the same
> media graph, one for each context. This effectively multiplies the number of
> video device nodes and subdevice nodes. Userspace applications open one media
> graph instance and operate on the corresponding video devices, under the
> impression of dealing with a dedicated instance of the sole underlying hardware
> resource.
>
> This solution is however a short term hack, it doesn't scale well when the
> number of contexts grow. ISPs such as the Mali C55 have been designed to process
> 8 cameras concurrently, and other ISPs may do more.
>
> For this reason, a solution to expose and manage multiple execution contexts
> without duplicating the number of media, video and sub-devices registered to
> userspace is needed to improve support for multi-context devices in V4L2.
>
> The series enables userspace to multiplex the usage of a media device and of
> video devices without duplicating the number of devnodes in userspace, by
> introducing the following concept in the framework:
>
> - Media Device Context: a context created at media-device open time and stored
> in the media-fh file handle. To a media device context is associated a list
> of media entity contexts which are 'bound' to it.
>
> - Video Device Context: represents an isolated execution context of a
> video device. By storing the data and the configuration of a video
> device userspace is allowed to effectively multiplex the usage of a
> device node.
>
> - Video Subdevice Context: represents an isolated execution context of
> v4l2 subdevice. It stores that subdev state which is now available in
> three locations:
> - in the file handle: for TRY formats
> - in the subdevice: for ACTIVE formats to support operations of subdevices
> drivers that do not implement context support
> - in the subdevice default context: for ACTIVE formats to support
> operations of subdevice drivers that implement context support and
> are operated by non-context-aware userspace
> - in the subdevice file-handle context: for ACTIVE formats to support
> operations of subdevices drivers that implement context support and
> are bound to a media device context
>
> Both the Video Device Context and the V4L2 Subdevice
> Context extend the Media Entity Context base type so that the MC and
> V4L2 layers are kept independent one from each other.
>
> - A Video Device Context is created by a new ioctl VIDIOC_BIND_CONTEXT and is
> stored in the v4l2-fh file handle.
>
> - A V4L2 Subdevice context is created by a new ioctl
> VIDIOC_SUBDEV_BIND_CONTEXT and is stored in the v4l2_subdev_fh file
> handle.
>
> The VIDIOC_BIND_CONTEXT and VIDIOC_SUBDEV_BIND_CONTEXT ioctls associates a
> Video/Subdevice Context to a Media Device Context. By binding a set of video
> devices and subdevices to a media device context userspace can create several
> isolated 'execution contexts' which can be operated independently one from
> each other.
>
> - A V4L2 Video Device and V4L2 Subdevice default context is made
> created to allow drivers that implement multi-context support but are
> operated by userspace that is not context aware (IOW doesn't call
> VIDIOC_BIND_CONTEXT).
>
> The first 20 patches implement the above described changes in the
> framework:
>
> media: v4l2-subdev: Validate media links with context
> media: mc-entity: Add link_validate_context
> media: media-entity: Support context in pipeline_start
> media: v4l2-subdev: Get state from context
> media: v4l2-subdev: Add subdev state accessor helpers
> media: v4l2_subdev: Introduce default context
> media: Documentation: Add VIDIOC_SUBDEV_BIND_CONTEXT
> media: v4l2-subdv: Introduce VIDIOC_SUBDEV_BIND_CONTEXT
> media: v4l2-subdev: Introduce v4l2 subdev context
> media: videobuf2-v4l2: Support vb2_queue embedded in a context
> media: v4l2-dev: Add video_device_context_from_queue()
> media: v4l2-dev: Add video_device_context_from_file()
> media: Introduce default contexts
> media: v4l2-dev: Documentation: Add VIDIOC_BIND_CONTEXT
> media: v4l2-ioctl: Introduce VIDIOC_BIND_CONTEXT
> media: v4l2-dev: Introduce video device context
> media: media-device: Introduce media device context
> media: media-entity: Introduce media_entity_context
> media: mc: Maintain a list of open file handles in a media device
> media: mc: Add per-file-handle data support
>
> For testing the implementation I used a Raspberry Pi5, implementing
> support for multiple contexts in the ISP driver. The PiSP BE driver
> however only implements video devices, as the single ISP subdev is not
> exposed to userspace and doesn't require any configuration. To test the
> V4L2 subdev context operations I had to first expose the ISP subdev to
> userspace and implement link validation and pipeline allocation for it.
> This is implemented in the following 6 patches NOT FOR INCLUSION in
> mainline but just here for reference
>
> [DNI] media: pisp_be: Register devnode to userspace
> [DNI] media: pisp_be: Implement link validation
> [DNI] media: pisp_be: Implement set/get_pad_fmt
> [DNI] media: pisp_be: Add support for subdev state
> [DNI] media: pisp_be: Start and stop the media pipeline
>
> Finally, the last patch (again NOT FOR INCLUSION) ports the ISP driver
> to support multi-context operations and provides a reference for the
> multi-context API usage by drivers:
>
> [DNI] media: pisp_be: Add support for multi-context
>
> The multi-context support has been tested with a version of libcamera
> that binds all video devices and the ISP subdevice in an execution
> context and allows to stream two cameras at the same time using the same
> media graph.
>
> In order to validate the fact that introducing context support in
> drivers doesn't break existing userspace which is not context-aware,
> capturing from a single camera with [mainline libcamera + ISP subdev
> configuration] has been tested as well.
>
> CI pipeline:
> - https://gitlab.freedesktop.org/linux-media/users/jmondi/-/pipelines/1472617
> - checkpatch errors are due to "DNI" prefix in last patches
> - static check errors seems unrelated to this set of patches but to
> existing code
> - checkpatch error are due to the usage of "Generic" which checkpatch
> doesn't like
>
> Branch for testing:
> - kernel patches:
> https://gitlab.freedesktop.org/linux-media/users/jmondi/-/commits/multicontext/mainline/2025
> - libcamera multi-context:
> https://git.libcamera.org/libcamera/jmondi/libcamera.git/log/?h=multicontext/rpi/mc
> - libcamera single context:
> https://git.libcamera.org/libcamera/jmondi/libcamera.git/log/?h=multicontext/rpi/subdev-no-multi-context
>
> Signed-off-by: Jacopo Mondi <jacopo.mondi at ideasonboard.com>
> ---
> Changes in v2:
> - Add "media: uapi: Add 'flags' to media_device_info"
> - Fix robots complaints by protecting users of symbols only defined if
> CONFIG_MEDIA_CONTROLLER with pre-processor guards
> - Link to v1: https://lore.kernel.org/r/20250717-multicontext-mainline-2025-v1-0-81ac18979c03@ideasonboard.com
>
> rfc->v1 Compared to the RFC version sent in September 2024:
> - Implement support for V4L2 Subdevice context
> - Implement context-aware pipeline start and link validation
> - Break out from Sakari's media lifetime series the only 2 required
> patches that introduce media_device_fh
> - Test the BE ISP subdevice by implementing support for the ISP
> subdevice in the BE driver
>
> ---
> Jacopo Mondi (25):
> media: media-entity: Introduce media_entity_context
> media: media-device: Introduce media device context
> media: v4l2-dev: Introduce video device context
> media: v4l2-ioctl: Introduce VIDIOC_BIND_CONTEXT
> media: Documentation: Add VIDIOC_BIND_CONTEXT
> media: v4l2-dev: Introduce default contexts
> media: v4l2-dev: Add video_device_context_from_file()
> media: v4l2-dev: Add video_device_context_from_queue()
> media: videobuf2-v4l2: Support vb2_queue embedded in a context
> media: v4l2-subdev: Introduce v4l2 subdev context
> media: v4l2-subdev: Introduce VIDIOC_SUBDEV_BIND_CONTEXT
> media: Documentation: Add VIDIOC_SUBDEV_BIND_CONTEXT
> media: v4l2_subdev: Introduce default context
> media: v4l2-subdev: Add subdev state accessor helpers
> media: v4l2-subdev: Get state from context
> media: media-entity: Support context in pipeline_start
> media: mc-entity: Add link_validate_context
> media: v4l2-subdev: Validate media links with context
> media: uapi: Add 'flags' to media_device_info
> [DNI] media: pisp_be: Start and stop the media pipeline
> [DNI] media: pisp_be: Add support for subdev state
> [DNI] media: pisp_be: Implement set/get_pad_fmt
> [DNI] media: pisp_be: Implement link validation
> [DNI] media: pisp_be: Register devnode to userspace
> [DNI] media: pisp_be: Add support for multi-context
>
> Laurent Pinchart (1):
> media: mc: Add per-file-handle data support
>
> Sakari Ailus (1):
> media: mc: Maintain a list of open file handles in a media device
>
> .../media/mediactl/media-ioc-device-info.rst | 6 +-
> .../userspace-api/media/mediactl/media-types.rst | 13 +
> .../userspace-api/media/v4l/user-func.rst | 2 +
> .../media/v4l/vidioc-bind-context.rst | 80 +++
> .../media/v4l/vidioc-subdev-bind-context.rst | 81 +++
> drivers/media/common/videobuf2/videobuf2-v4l2.c | 139 ++--
> drivers/media/mc/mc-device.c | 213 +++++-
> drivers/media/mc/mc-devnode.c | 20 +-
> drivers/media/mc/mc-entity.c | 220 +++++-
> .../media/platform/raspberrypi/pisp_be/pisp_be.c | 775 ++++++++++++++++-----
> drivers/media/v4l2-core/v4l2-dev.c | 158 ++++-
> drivers/media/v4l2-core/v4l2-device.c | 11 +-
> drivers/media/v4l2-core/v4l2-fh.c | 1 +
> drivers/media/v4l2-core/v4l2-ioctl.c | 72 ++
> drivers/media/v4l2-core/v4l2-subdev.c | 414 ++++++++++-
> include/media/media-device.h | 230 ++++++
> include/media/media-devnode.h | 21 +-
> include/media/media-entity.h | 163 ++++-
> include/media/media-fh.h | 37 +
> include/media/v4l2-dev.h | 252 +++++++
> include/media/v4l2-fh.h | 3 +
> include/media/v4l2-ioctl.h | 7 +
> include/media/v4l2-subdev.h | 293 ++++++--
> include/uapi/linux/media.h | 6 +-
> include/uapi/linux/v4l2-subdev.h | 11 +
> include/uapi/linux/videodev2.h | 11 +
> 26 files changed, 2890 insertions(+), 349 deletions(-)
> ---
> base-commit: d968e50b5c26642754492dea23cbd3592bde62d8
> change-id: 20250716-multicontext-mainline-2025-3479c7c24b7a
>
> Best regards,
> --
> Jacopo Mondi <jacopo.mondi at ideasonboard.com>
Thanks for this patchset - I've been writing a driver (not ready for upstreaming
for several reasons), and it works well to enable processing several streams
at once. I would be interested in seeing it finished. I have however spotted
a number of issues which I will address against the relevant patches.
Regards
John Cox
More information about the linux-arm-kernel
mailing list