[PATCH v5 06/16] media: Documentation: Add Mali-C55 ISP Documentation
Dan Scally
dan.scally at ideasonboard.com
Wed May 29 14:11:32 PDT 2024
On 29/05/2024 21:51, Laurent Pinchart wrote:
> On Wed, May 29, 2024 at 09:35:08PM +0100, Daniel Scally wrote:
>> On 29/05/2024 21:22, Laurent Pinchart wrote:
>>> On Wed, May 29, 2024 at 04:28:48PM +0100, Daniel Scally wrote:
>>>> Add a documentation page for the mali-c55 driver, which gives a brief
>>>> overview of the hardware and explains how to use the driver's capture
>>>> devices and the crop/scaler functions.
>>>>
>>>> Acked-by: Nayden Kanchev <nayden.kanchev at arm.com>
>>>> Co-developed-by: Jacopo Mondi <jacopo.mondi at ideasonboard.com>
>>>> Signed-off-by: Jacopo Mondi <jacopo.mondi at ideasonboard.com>
>>>> Signed-off-by: Daniel Scally <dan.scally at ideasonboard.com>
>>>> ---
>>>> Changes in v5:
>>>>
>>>> - None
>>>>
>>>> Changes in v4:
>>>> - None
>>>>
>>>> Changes in v3:
>>>> - Documented the synchronised buffer sequence numbers (Sakari)
>>>> - Clarified that the downscale pipe cannot output raw data, the ISP'S
>>>> resolution limits and choice of media bus format code (Kieran)
>>>>
>>>> Changes in v2:
>>>>
>>>> - none
>>>>
>>>> .../admin-guide/media/mali-c55-graph.dot | 19 +
>>>> Documentation/admin-guide/media/mali-c55.rst | 333 ++++++++++++++++++
>>>> .../admin-guide/media/v4l-drivers.rst | 1 +
>>>> 3 files changed, 353 insertions(+)
>>>> create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot
>>>> create mode 100644 Documentation/admin-guide/media/mali-c55.rst
>>>>
>>>> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot
>>>> new file mode 100644
>>>> index 000000000000..0775ba42bf4c
>>>> --- /dev/null
>>>> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot
>>>> @@ -0,0 +1,19 @@
>>>> +digraph board {
>>>> + rankdir=TB
>>>> + n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> + n00000001:port0 -> n00000003:port0 [style=dashed]
>>>> + n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> + n00000003:port1 -> n00000007:port0 [style=bold]
>>>> + n00000003:port2 -> n00000007:port2 [style=bold]
>>>> + n00000003:port1 -> n0000000b:port0 [style=bold]
>>>> + n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> + n00000007:port1 -> n0000000e [style=bold]
>>>> + n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> + n0000000b:port1 -> n00000012 [style=bold]
>>>> + n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
>>>> + n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
>>>> + n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> + n00000022:port1 -> n00000003:port0
>>>> + n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
>>>> + n00000027:port0 -> n00000022:port0 [style=bold]
>>>> +}
>>>> \ No newline at end of file
>>>> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst
>>>> new file mode 100644
>>>> index 000000000000..cf4176cb1e44
>>>> --- /dev/null
>>>> +++ b/Documentation/admin-guide/media/mali-c55.rst
>>>> @@ -0,0 +1,333 @@
>>>> +.. SPDX-License-Identifier: GPL-2.0
>>>> +
>>>> +==========================================
>>>> +ARM Mali-C55 Image Signal Processor driver
>>>> +==========================================
>>>> +
>>>> +Introduction
>>>> +============
>>>> +
>>>> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
>>>> +driver is located under drivers/media/platform/arm/mali-c55.
>>>> +
>>>> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
>>>> +sensors through either a parallel interface or a memory bus before processing it
>>>> +and outputting it through an internal DMA engine. Two output pipelines are
>>>> +possible (though one may not be fitted, depending on the implementation). These
>>>> +are referred to as "Full resolution" and "Downscale", but the naming is historic
>>>> +and both pipes are capable of cropping/scaling operations. The full resolution
>>>> +pipe is also capable of outputting RAW data, bypassing much of the ISP's
>>>> +processing. The downscale pipe cannot output RAW data. An integrated test
>>>> +pattern generator can be used to drive the ISP and produce image data in the
>>>> +absence of a connected camera sensor. The driver module is named mali_c55, and
>>>> +is enabled through the CONFIG_VIDEO_MALI_C55 config option.
>>>> +
>>>> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
>>>> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
>>>> +
>>>> +Mali-C55 ISP hardware
>>>> +=====================
>>>> +
>>>> +A high level functional view of the Mali-C55 ISP is presented below. The ISP
>>>> +takes input from either a live source or through a DMA engine for memory input,
>>>> +depending on the SoC integration.::
>>>> +
>>>> + +---------+ +----------+ +--------+
>>>> + | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA |
>>>> + +---------+ +----------+ |\ Output +--->| Writer |
>>>> + | | \ | +--------+
>>>> + | | \ +----------+ +------+---> Streaming I/O
>>>> + +------------+ +------->| | | | |
>>>> + | | | |-->| Mali-C55 |--+
>>>> + | DMA Reader |--------------->| | | ISP | |
>>>> + | | | / | | | +---> Streaming I/O
>>>> + +------------+ | / +----------+ | |
>>>> + |/ +------+
>>>> + | +--------+
>>>> + +--->| DMA |
>>>> + "Downscaled" | Writer |
>>>> + Output +--------+
>>> You have a mix of tabs and spaces here.
>>>
>>>> +
>>>> +Media Controller Topology
>>>> +=========================
>>>> +
>>>> +An example of the ISP's topology (as implemented in a system with an IMX415
>>>> +camera sensor and generic CSI-2 receiver) is below:
>>>> +
>>>> +
>>>> +.. kernel-figure:: mali-c55-graph.dot
>>>> + :alt: mali-c55-graph.dot
>>>> + :align: center
>>>> +
>>>> +The driver has 4 V4L2 subdevices:
>>>> +
>>>> +- `mali_c55 isp`: Responsible for configuring input crop and color space
>>>> + conversion
>>>> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
>>>> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer
>>>> +- `mali_c55 resizer ds`: The Downscale pipe resizer
>>>> +
>>>> +The driver has 2 V4L2 video devices:
>>>> +
>>>> +- `mali-c55 fr`: The full-resolution pipe's capture device
>>>> +- `mali-c55 ds`: The downscale pipe's capture device
>>>> +
>>>> +Frame sequences are synchronised across to two capture devices, meaning if one
>>>> +pipe is started later than the other the sequence numbers returned in its
>>>> +buffers will match those of the other pipe rather than starting from zero.
>>>> +
>>>> +Frame sequences are synchronised across to two capture devices, meaning if one
>>>> +pipe is started later than the other the sequence numbers returned in its
>>>> +buffers will match those of the other pipe rather than starting from zero.
>>> I think you can explain this once only.
>> Oops...
>>
>>>> +
>>>> +Idiosyncrasies
>>>> +--------------
>>>> +
>>>> +**mali-c55 isp**
>>>> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
>>>> +should be connected. The active source is selected by enabling the appropriate
>>>> +media link and disabling all others.
>>> Modelling this with links prevents switching between sources at runtime.
>>> It also makes it possible to misconfigure the pipeline by disconnecting
>>> the two sources. This would be caught at pipeline validation time, but
>>> it still adds complexity.
>>>
>>> I was considering using the subdev routing API instead, which would
>>> allow runtime reconfiguration, and prevent invalid configuration in the
>>> first place. The downside is that we would need a mux subdev in front of
>>> the ISP. In terms of additional complexity, that's clearly not great.
>>>
>>> Given that switching between the sensor and TPG at runtime is likely not
>>> an important use case, and that the harware may not even support it at
>>> all, we can probably keep the existing graph and driver implementation.
>> I suppose in the long term we need to think about how this should be
>> modeled in a multi-context system...when we have a media graph with 8
>> cameras connected (somehow) to the ISP's single sink pad how should we
>> select the right input device for a context? Whatever the answer is
>> there, probably we should do it here...if we can't do it at runtime
>> with links then probably it has to be a mux...or some novel thing.
> Good point. Although... In M2M operation, will we need the TPG at all ?
> Userspace could easily supply test patterns in the input buffers.
True, we wouldn't need it.
> I don't think we can time-multiplex the ISP in a mode where some of the
> inputs would come from memory, and some from a live source (be it the
> TPG or a sensor).
Yeah I'm not sure how we could get the timing of that right.
> Sticking to the technical discussion, I envision that the contexts will
> be handled by opening the media device multiple times. Each file handle
> will be associated with one context. The link state would then be stored
> in the context. The links will still not be configurable at runtime
> within a context, but could differ between contexts.
OK I see. In that case keeping the link here gets my vote
>
>>>> The ISP has two source pads, reflecting the
>>>> +different paths through which it can internally route data. Tap points within
>>>> +the ISP allow users to divert data to avoid processing by some or all of the
>>>> +hardware's processing steps. The diagram below is intended only to highlight how
>>>> +the bypassing works and is not a true reflection of those processing steps; for
>>>> +a high-level functional block diagram see ARM's developer page for the
>>>> +ISP [3]_::
>>>> +
>>>> + +--------------------------------------------------------------+
>>>> + | Possible Internal ISP Data Routes |
>>>> + | +------------+ +----------+ +------------+ |
>>>> + +---+ | | | | | Colour | +---+
>>>> + | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 |
>>>> + +---+ | | | | | | Conversion | +---+
>>>> + | | +------------+ +----------+ +------------+ |
>>>> + | | +---+
>>>> + | +---------------------------------------------------| 2 |
>>>> + | +---+
>>>> + | |
>>>> + +--------------------------------------------------------------+
>>>> +
>>>> +
>>>> +.. flat-table::
>>>> + :header-rows: 1
>>>> +
>>>> + * - Pad
>>>> + - Direction
>>>> + - Purpose
>>>> +
>>>> + * - 0
>>>> + - sink
>>>> + - Data input, connected to the TPG and camera sensors
>>>> +
>>>> + * - 1
>>>> + - source
>>>> + - RGB/YUV data, connected to the FR and DS V4L2 subdevices
>>>> +
>>>> + * - 2
>>>> + - source
>>>> + - RAW bayer data, connected to the FR V4L2 subdevices
>>>> +
>>>> +The ISP is limited to both input and output resolutions between 640x480 and
>>>> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
>>>> +operations.
>>>> +
>>>> +**mali-c55 resizer fr**
>>>> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
>>>> +insertion points in the hardware (either RAW or demosaiced data):
>>>> +
>>>> +.. flat-table::
>>>> + :header-rows: 1
>>>> +
>>>> + * - Pad
>>>> + - Direction
>>>> + - Purpose
>>>> +
>>>> + * - 0
>>>> + - sink
>>>> + - Data input connected to the ISP's demosaiced stream.
>>>> +
>>>> + * - 1
>>>> + - source
>>>> + - Data output connected to the capture video device
>>>> +
>>>> + * - 2
>>>> + - sink
>>>> + - Data input connected to the ISP's raw data stream
>>>> +
>>>> +The data source in use is selected through the routing API; two routes each of a
>>>> +single stream are available:
>>>> +
>>>> +.. flat-table::
>>>> + :header-rows: 1
>>>> +
>>>> + * - Sink Pad
>>>> + - Source Pad
>>>> + - Purpose
>>>> +
>>>> + * - 0
>>>> + - 1
>>>> + - Demosaiced data route
>>>> +
>>>> + * - 2
>>>> + - 1
>>>> + - Raw data route
>>>> +
>>>> +
>>>> +If the demosaiced route is active then the FR pipe is only capable of output
>>>> +in RGB/YUV formats. If the raw route is active then the output reflects the
>>>> +input (which may be either Bayer or RGB/YUV data).
>>>> +
>>>> +Using the driver to capture video
>>>> +=================================
>>>> +
>>>> +Using the media controller APIs we can configure the input source and ISP to
>>>> +capture images in a variety of formats. In the examples below, configuring the
>>>> +media graph is done with the v4l-utils [1]_ package's media-ctl utility.
>>>> +Capturing the images is done with yavta [2]_.
>>>> +
>>>> +Configuring the input source
>>>> +----------------------------
>>>> +
>>>> +The first step is to set the input source that we wish by enabling the correct
>>>> +media link. Using the example topology above, we can select the TPG as follows:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
>>>> + media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
>>>> +
>>>> +Capturing bayer data from the source and processing to RGB/YUV
>>>> +--------------------------------------------------------------
>>> Missing blank line.
>>>
>>>> +To capture 1920x1080 bayer data from the source and push it through the ISP's
>>>> +full processing pipeline, we configure the data formats appropriately on the
>>>> +source, ISP and resizer subdevices and set the FR resizer's routing to select
>>>> +processed data. The media bus format on the resizer's source pad will be either
>>>> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
>>>> +YUV. The ISP's debayering block outputs RGB data natively, setting the source
>>>> +pad format to YUV10_1X30 enables the colour space conversion block.
>>>> +
>>>> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
>>>> +source pad's format:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + # Set formats on the TPG and ISP
>>>> + media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]"
>>>> + media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]"
>>>> + media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]"
>>>> +
>>>> + # Set routing on the FR resizer
>>>> + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
>>>> +
>>>> + # Set format on the resizer, must be done AFTER the routing.
>>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
>>>> +
>>>> +The downscale output can also be used to stream data at the same time. In this
>>>> +case since only processed data can be captured through the downscale output no
>>>> +routing need be set:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + # Set format on the resizer
>>>> + media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
>>>> +
>>>> +Following which images can be captured from both the FR and DS output's video
>>>> +devices (simultaneously, if desired):
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
>>>> +
>>>> +Cropping the image
>>>> +~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +Both the full resolution and downscale pipes can crop to a minimum resolution of
>>>> +640x480. To crop the image simply configure the resizer's sink pad's crop and
>>>> +compose rectangles and set the format on the video device:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]"
>>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
>>>> + yavta -f RGB565 -s 640x480 -c10 /dev/video0
>>>> +
>>>> +Downscaling the image
>>>> +~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +Both the full resolution and downscale pipes can downscale the image by up to 8x
>>>> +provided the minimum 640x480 resolution is adhered to. For the best image result
>>> Maybe "minimum 640x480 output resolution".
>>>
>>>> +the scaling ratio for each dimension should be the same. To configure scaling we
>>> s/dimension/direction/
>>>
>>>> +use the compose rectangle on the resizer's sink pad:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]"
>>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
>>>> + yavta -f RGB565 -s 640x480 -c10 /dev/video0
>>>> +
>>>> +Capturing images in YUV formats
>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +If we need to output YUV data rather than RGB the color space conversion block
>>>> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
>>>> +resizer's source pad. We can then configure a capture format like NV12 (here in
>>>> +its multi-planar variant)
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
>>>> + yavta -f NV12M -s 1920x1080 -c10 /dev/video0
>>>> +
>>>> +Capturing RGB data from the source and processing it with the resizers
>>>> +----------------------------------------------------------------------
>>>> +
>>>> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
>>>> +case although none of the image quality blocks would be used it can still
>>>> +crop/scale the data in the usual way.
>>>> +
>>>> +To achieve this, the ISP's sink pad's format is set to
>>>> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
>>>> +work with the ISP. Converting the camera sensor's output to that format is the
>>>> +responsibility of external hardware.
>>>> +
>>>> +In this example we ask the test pattern generator to give us RGB data instead of
>>>> +bayer.
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
>>>> + media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
>>>> +
>>>> +Cropping or scaling the data can be done in exactly the same way as outlined
>>>> +earlier.
>>> Do we use the ISP's output on pad 1 or pad 2 in this case ? The text
>>> seems to imply that the ISP is bypassed, but the example doesn't mention
>>> any routing change. You may want to clarify this.
>> Pad 1 for bypassed RGB data. I'll make it more clear.
> Explaining that this doesn't follow the bypass path, but goes through
> the processing path and disables the processing blocks that are not
> relevant, would help.
Well; kinda. It takes a _different_ bypass path, but in this case the decision as to whether or not
that path should be taken is just based on the ISP sink pad's format.
>
>>>> +
>>>> +Capturing raw data from the source and outputting it unmodified
>>>> +-----------------------------------------------------------------
>>>> +
>>>> +The ISP can additionally capture raw data from the source and output it on the
>>>> +full resolution pipe only, completely unmodified. In this case the downscale
>>>> +pipe can still process the data normally and be used at the same time.
>>>> +
>>>> +To configure raw bypass the FR resizer's subdevice's routing table needs to be
>>>> +configured, followed by formats in the appropriate places:
>>>> +
>>>> +.. code-block:: none
>>>> +
>>>> + # We need to configure the routing table for the resizer to use the bypass
>>>> + # path along with set formats on the resizer's bypass sink pad. Doing this
>>>> + # necessitates a single media-ctl command, as multiple calls to the program
>>>> + # reset the routing table.
>>> Really ?
>> Yeah
>>
>>> Does -V reset the routing table ? That surprises me.
>> It's not -V, it's the fact of opening a subdev which calls
>> .init_state(), which resets the routing table...
> Only for the TRY state, not the ACTIVE state. If the TRY state leaks to
> the ACTIVE state, you may have a driver bugs :-)
Hmm, of course that's right...maybe I'm misremembering what the problem was, or maybe there's no
problem at all. I will check and see.
>
>> in a single process the fds are held open throughout so all is well, but if you run the
>> program twice they're opened each time and the routing is reset.
>>
>>>> + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\
>>>> + -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\
>>>> + "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\
>>>> + "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
>>>> +
>>>> + # Set format on the video device and stream
>>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
>>>> +
>>>> +References
>>>> +==========
>>>> +.. [1] https://git.linuxtv.org/v4l-utils.git/
>>>> +.. [2] https://git.ideasonboard.org/yavta.git
>>>> +.. [3] https://developer.arm.com/Processors/Mali-C55
>>>> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
>>>> index 4120eded9a13..1d9485860d93 100644
>>>> --- a/Documentation/admin-guide/media/v4l-drivers.rst
>>>> +++ b/Documentation/admin-guide/media/v4l-drivers.rst
>>>> @@ -18,6 +18,7 @@ Video4Linux (V4L) driver-specific documentation
>>>> ipu3
>>>> ipu6-isys
>>>> ivtv
>>>> + mali-c55
>>>> mgb4
>>>> omap3isp
>>>> omap4_camera
More information about the linux-arm-kernel
mailing list