[PATCH 00/26] media: Add support for multi-context operations
Jacopo Mondi
jacopo.mondi at ideasonboard.com
Thu Jul 17 03:45:26 PDT 2025
Modern ISPs are designed to handle multiple "streams" of data, not necessarily
related to the same image or video stream.
The hardware resources are generally time-multiplexed between different
execution contexts at the hardware or firmware level and in order to operate the
ISP with multiple video sources it is necessary for drivers to keep track of
per-context data and resources.
In V4L2 the M2M framework supports multiple contexts through multiple opens of
the same video device. This doesn't however support drivers exposing multiple
video devices and sub-devices. Several out-of-tree drivers implement
multi-context support by registering multiple 'logical' instances of the same
media graph, one for each context. This effectively multiplies the number of
video device nodes and subdevice nodes. Userspace applications open one media
graph instance and operate on the corresponding video devices, under the
impression of dealing with a dedicated instance of the sole underlying hardware
resource.
This solution is however a short term hack, it doesn't scale well when the
number of contexts grow. ISPs such as the Mali C55 have been designed to process
8 cameras concurrently, and other ISPs may do more.
For this reason, a solution to expose and manage multiple execution contexts
without duplicating the number of media, video and sub-devices registered to
userspace is needed to improve support for multi-context devices in V4L2.
The series enables userspace to multiplex the usage of a media device and of
video devices without duplicating the number of devnodes in userspace, by
introducing the following concept in the framework:
- Media Device Context: a context created at media-device open time and stored
in the media-fh file handle. To a media device context is associated a list
of media entity contexts which are 'bound' to it.
- Video Device Context: represents an isolated execution context of a
video device. By storing the data and the configuration of a video
device userspace is allowed to effectively multiplex the usage of a
device node.
- Video Subdevice Context: represents an isolated execution context of
v4l2 subdevice. It stores that subdev state which is now available in
three locations:
- in the file handle: for TRY formats
- in the subdevice: for ACTIVE formats to support operations of subdevices
drivers that do not implement context support
- in the subdevice default context: for ACTIVE formats to support
operations of subdevice drivers that implement context support and
are operated by non-context-aware userspace
- in the subdevice file-handle context: for ACTIVE formats to support
operations of subdevices drivers that implement context support and
are bound to a media device context
Both the Video Device Context and the V4L2 Subdevice
Context extend the Media Entity Context base type so that the MC and
V4L2 layers are kept independent one from each other.
- A Video Device Context is created by a new ioctl VIDIOC_BIND_CONTEXT and is
stored in the v4l2-fh file handle.
- A V4L2 Subdevice context is created by a new ioctl
VIDIOC_SUBDEV_BIND_CONTEXT and is stored in the v4l2_subdev_fh file
handle.
The VIDIOC_BIND_CONTEXT and VIDIOC_SUBDEV_BIND_CONTEXT ioctls associates a
Video/Subdevice Context to a Media Device Context. By binding a set of video
devices and subdevices to a media device context userspace can create several
isolated 'execution contexts' which can be operated independently one from
each other.
- A V4L2 Video Device and V4L2 Subdevice default context is made
created to allow drivers that implement multi-context support but are
operated by userspace that is not context aware (IOW doesn't call
VIDIOC_BIND_CONTEXT).
The first 20 patches implement the above described changes in the
framework:
media: v4l2-subdev: Validate media links with context
media: mc-entity: Add link_validate_context
media: media-entity: Support context in pipeline_start
media: v4l2-subdev: Get state from context
media: v4l2-subdev: Add subdev state accessor helpers
media: v4l2_subdev: Introduce default context
media: Documentation: Add VIDIOC_SUBDEV_BIND_CONTEXT
media: v4l2-subdv: Introduce VIDIOC_SUBDEV_BIND_CONTEXT
media: v4l2-subdev: Introduce v4l2 subdev context
media: videobuf2-v4l2: Support vb2_queue embedded in a context
media: v4l2-dev: Add video_device_context_from_queue()
media: v4l2-dev: Add video_device_context_from_file()
media: Introduce default contexts
media: v4l2-dev: Documentation: Add VIDIOC_BIND_CONTEXT
media: v4l2-ioctl: Introduce VIDIOC_BIND_CONTEXT
media: v4l2-dev: Introduce video device context
media: media-device: Introduce media device context
media: media-entity: Introduce media_entity_context
media: mc: Maintain a list of open file handles in a media device
media: mc: Add per-file-handle data support
For testing the implementation I used a Raspberry Pi5, implementing
support for multiple contexts in the ISP driver. The PiSP BE driver
however only implements video devices, as the single ISP subdev is not
exposed to userspace and doesn't require any configuration. To test the
V4L2 subdev context operations I had to first expose the ISP subdev to
userspace and implement link validation and pipeline allocation for it.
This is implemented in the following 6 patches NOT FOR INCLUSION in
mainline but just here for reference
[DNI] media: pisp_be: Register devnode to userspace
[DNI] media: pisp_be: Implement link validation
[DNI] media: pisp_be: Implement set/get_pad_fmt
[DNI] media: pisp_be: Add support for subdev state
[DNI] media: pisp_be: Start and stop the media pipeline
Finally, the last patch (again NOT FOR INCLUSION) ports the ISP driver
to support multi-context operations and provides a reference for the
multi-context API usage by drivers:
[DNI] media: pisp_be: Add support for multi-context
The multi-context support has been tested with a version of libcamera
that binds all video devices and the ISP subdevice in an execution
context and allows to stream two cameras at the same time using the same
media graph.
In order to validate the fact that introducing context support in
drivers doesn't break existing userspace which is not context-aware,
capturing from a single camera with [mainline libcamera + ISP subdev
configuration] has been tested as well.
CI pipeline:
- https://gitlab.freedesktop.org/linux-media/users/jmondi/-/pipelines/1472617
- checkpatch errors are due to "DNI" prefix in last patches
- static check errors seems unrelated to this set of patches but to
existing code
- checkpatch error are due to the usage of "Generic" which checkpatch
doesn't like
Branch for testing:
- kernel patches:
https://gitlab.freedesktop.org/linux-media/users/jmondi/-/commits/multicontext/mainline/2025
- libcamera multi-context:
https://git.libcamera.org/libcamera/jmondi/libcamera.git/log/?h=multicontext/rpi/mc
- libcamera single context:
https://git.libcamera.org/libcamera/jmondi/libcamera.git/log/?h=multicontext/rpi/subdev-no-multi-context
rfc->v1 Compared to the RFC version sent in September 2024:
- Implement support for V4L2 Subdevice context
- Implement context-aware pipeline start and link validation
- Break out from Sakari's media lifetime series the only 2 required
patches that introduce media_device_fh
- Test the BE ISP subdevice by implementing support for the ISP
subdevice in the BE driver
Signed-off-by: Jacopo Mondi <jacopo.mondi at ideasonboard.com>
---
Jacopo Mondi (24):
media: media-entity: Introduce media_entity_context
media: media-device: Introduce media device context
media: v4l2-dev: Introduce video device context
media: v4l2-ioctl: Introduce VIDIOC_BIND_CONTEXT
media: Documentation: Add VIDIOC_BIND_CONTEXT
media: v4l2-dev: Introduce default contexts
media: v4l2-dev: Add video_device_context_from_file()
media: v4l2-dev: Add video_device_context_from_queue()
media: videobuf2-v4l2: Support vb2_queue embedded in a context
media: v4l2-subdev: Introduce v4l2 subdev context
media: v4l2-subdev: Introduce VIDIOC_SUBDEV_BIND_CONTEXT
media: Documentation: Add VIDIOC_SUBDEV_BIND_CONTEXT
media: v4l2_subdev: Introduce default context
media: v4l2-subdev: Add subdev state accessor helpers
media: v4l2-subdev: Get state from context
media: media-entity: Support context in pipeline_start
media: mc-entity: Add link_validate_context
media: v4l2-subdev: Validate media links with context
[DNI] media: pisp_be: Start and stop the media pipeline
[DNI] media: pisp_be: Add support for subdev state
[DNI] media: pisp_be: Implement set/get_pad_fmt
[DNI] media: pisp_be: Implement link validation
[DNI] media: pisp_be: Register devnode to userspace
[DNI] media: pisp_be: Add support for multi-context
Laurent Pinchart (1):
media: mc: Add per-file-handle data support
Sakari Ailus (1):
media: mc: Maintain a list of open file handles in a media device
.../userspace-api/media/v4l/user-func.rst | 2 +
.../media/v4l/vidioc-bind-context.rst | 80 +++
.../media/v4l/vidioc-subdev-bind-context.rst | 81 +++
drivers/media/common/videobuf2/videobuf2-v4l2.c | 135 ++--
drivers/media/mc/mc-device.c | 212 +++++-
drivers/media/mc/mc-devnode.c | 20 +-
drivers/media/mc/mc-entity.c | 220 +++++-
.../media/platform/raspberrypi/pisp_be/pisp_be.c | 773 ++++++++++++++++-----
drivers/media/v4l2-core/v4l2-dev.c | 158 ++++-
drivers/media/v4l2-core/v4l2-device.c | 11 +-
drivers/media/v4l2-core/v4l2-fh.c | 1 +
drivers/media/v4l2-core/v4l2-ioctl.c | 64 ++
drivers/media/v4l2-core/v4l2-subdev.c | 414 ++++++++++-
include/media/media-device.h | 223 ++++++
include/media/media-devnode.h | 21 +-
include/media/media-entity.h | 163 ++++-
include/media/media-fh.h | 37 +
include/media/v4l2-dev.h | 252 +++++++
include/media/v4l2-fh.h | 3 +
include/media/v4l2-ioctl.h | 7 +
include/media/v4l2-subdev.h | 293 ++++++--
include/uapi/linux/v4l2-subdev.h | 11 +
include/uapi/linux/videodev2.h | 11 +
23 files changed, 2845 insertions(+), 347 deletions(-)
---
base-commit: d968e50b5c26642754492dea23cbd3592bde62d8
change-id: 20250716-multicontext-mainline-2025-3479c7c24b7a
Best regards,
--
Jacopo Mondi <jacopo.mondi at ideasonboard.com>
More information about the linux-arm-kernel
mailing list