[PATCH v2] media: imx-jpeg: Align upwards buffer size

mirela.rabulea at oss.nxp.com mirela.rabulea at oss.nxp.com
Thu Jun 16 00:30:35 PDT 2022


Hi Ming,

On 13.06.2022 08:25, Ming Qian wrote:
>> From: Mirela Rabulea (OSS) <mirela.rabulea at oss.nxp.com>
>> Sent: 2022年6月13日 6:56
>> To: Ming Qian <ming.qian at nxp.com>; mchehab at kernel.org;
>> hverkuil-cisco at xs4all.nl
>> Cc: shawnguo at kernel.org; s.hauer at pengutronix.de; kernel at pengutronix.de;
>> festevam at gmail.com; dl-linux-imx <linux-imx at nxp.com>;
>> linux-media at vger.kernel.org; linux-kernel at vger.kernel.org;
>> devicetree at vger.kernel.org; linux-arm-kernel at lists.infradead.org
>> Subject: Re: [PATCH v2] media: imx-jpeg: Align upwards buffer size
>>
>> Hi Ming,
>>
>> On 30.05.2022 10:49, Ming Qian wrote:
>>> The hardware can support any image size WxH, with arbitrary W (image
>>> width) and H (image height) dimensions.
>>>
>>> Align upwards buffer size for both encoder and decoder.
>>> and leave the picture resolution unchanged.
>>>
>>> For decoder, the risk of memory out of bounds can be avoided.
>>> For both encoder and decoder, the driver will lift the limitation of
>>> resolution alignment.
>>>
>>> For example, the decoder can support jpeg whose resolution is 227x149
>>
>> I doubt 227x149 is working. I have tried 127x127, with your patch applied,
>> both on encoder and decoder, the image does not look ok. The
>> 126x127 seems to work. Having an odd resolution seems strange, I see not
>> even gstreamer supports it (tried videotestsrc & filesink with BGR,
>> 127x128 produces a 128x128 size).
>>
>> We need to gain more clarity on this one.
>> And when we do, if we really can support any arbitrary resolution, from both
>> the jpeg core and wrapper point of view, we have stuff to clean up.
>> The assumption that I started with was, as stated in the comments:
>>    * The alignment requirements for the resolution depend on the format,
>>    * multiple of 16 resolutions should work for all formats.
>>    * Special workarounds are made in the driver to support NV12 1080p.
>> With h_align/v_align defined in mxc_formats[].
>>
>> Regards,
>> Mirela
> 
> Hi Mirela,
>      I think you are confusing picture size and buffer size.
>      In this patch, driver will enlarge the buffer size to align 16x16. But let the picture size unchanged.
>      And if you display the decoded 227x149 picture directly on imx8q, you may meet some drm error, and the display looks abnormal,
> The error message like below:
> [   36.381015] [drm] [CRTC:38:crtc-0] dpu_crtc_atomic_flush: wait for content shdld done timeout
> [   36.389630] [drm] [CRTC:38:crtc-0] dpu_crtc_atomic_flush: FrameGen requests to read empty FIFO
> [   49.469022] [drm] [CRTC:38:crtc-0] dpu_crtc_atomic_flush: wait for content shdld done timeout
> 
> But if you save the decoded picture data in to a file, then check the data, you will find it's correct.
> And if you treat the decoded buffer as a picture with resolution 240x160, and with some padding content, it's also correct.
> 
> So in my first patch, I let the driver report the aligned resolution in g_fmt,
> and implement a g_selection to report the actual resolution through the crop info.
> but this solution will fail your labgrid jpeg test.
> 
> So I choose the current solution that keep the actual picture size, and align upwards the buffer size.
> 
> The display of 227x149 is abnormal, I think it's not the jpeg codec's limitation, but the imx8q drm's limitation.

I did not try to display with gstreamer. I just tried to generate some 
test files. For example, this one:
gst-launch-1.0 videotestsrc pattern=smpte75 num-buffers=1 ! 
video/x-raw,width=227,height=149,format=BGR ! filesink 
location=bgr_227x149.raw
generates a 228x149 file, not 227x149, with the 228 column black 
(padding?). For visualizing, I used vooya, on host machine.

I then tried the pattern generator: https://github.com/NXPmicro/nxp-patgen
./patgen.exe -pix_fmt yuv444 -p colorbar -vsize 227x149

This generates a raw yuv444 227x149, without padding.
I used the unit test application to encode it.
The resulting jpeg is reported by JPEGSnoop to have Image Size = 
227x149, and it looks bad, every line is shifted, as if it would have 
228 width.
I did not check yet to see if any adjustments are needed in the unit test.

I'll send you my test files.

Regards,
Mirela

> 
> And in my opinion, I prefer the first solution that implementing a g_selection to report the actual picture size.
> If you can accept that changing your labgrid jpeg testcase, I can prepare a v3 patch to switch to this solution.
> 
> Ming
> 
>>
>>> the encoder can support nv12 1080P, won't change it to 1920x1072.
>>>
>>> Fixes: 2db16c6ed72ce ("media: imx-jpeg: Add V4L2 driver for i.MX8 JPEG
>>> Encoder/Decoder")
>>> Signed-off-by: Ming Qian <ming.qian at nxp.com>
>>> ---
>>> v2
>>> - add Fixes tag
>>>    .../media/platform/nxp/imx-jpeg/mxc-jpeg.c    | 88 ++++++++-----------
>>>    1 file changed, 37 insertions(+), 51 deletions(-)
>>>
>>> diff --git a/drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c
>>> b/drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c
>>> index c0fd030d0f19..9a1c8df522ed 100644
>>> --- a/drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c
>>> +++ b/drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c
>>> @@ -894,8 +894,8 @@ static void mxc_jpeg_config_enc_desc(struct
>> vb2_buffer *out_buf,
>>>    	jpeg->slot_data[slot].cfg_stream_size =
>>>    			mxc_jpeg_setup_cfg_stream(cfg_stream_vaddr,
>>>    						  q_data->fmt->fourcc,
>>> -						  q_data->w_adjusted,
>>> -						  q_data->h_adjusted);
>>> +						  q_data->w,
>>> +						  q_data->h);
>>>
>>>    	/* chain the config descriptor with the encoding descriptor */
>>>    	cfg_desc->next_descpt_ptr = desc_handle | MXC_NXT_DESCPT_EN; @@
>>> -977,7 +977,7 @@ static bool mxc_jpeg_source_change(struct mxc_jpeg_ctx
>> *ctx,
>>>    				      &q_data_cap->h_adjusted,
>>>    				      q_data_cap->h_adjusted, /* adjust up */
>>>    				      MXC_JPEG_MAX_HEIGHT,
>>> -				      q_data_cap->fmt->v_align,
>>> +				      0,
>>>    				      0);
>>>
>>>    		/* setup bytesperline/sizeimage for capture queue */ @@ -1161,18
>>> +1161,30 @@ static int mxc_jpeg_queue_setup(struct vb2_queue *q,
>>>    {
>>>    	struct mxc_jpeg_ctx *ctx = vb2_get_drv_priv(q);
>>>    	struct mxc_jpeg_q_data *q_data = NULL;
>>> +	struct mxc_jpeg_q_data tmp_q;
>>>    	int i;
>>>
>>>    	q_data = mxc_jpeg_get_q_data(ctx, q->type);
>>>    	if (!q_data)
>>>    		return -EINVAL;
>>>
>>> +	tmp_q.fmt = q_data->fmt;
>>> +	tmp_q.w = q_data->w_adjusted;
>>> +	tmp_q.h = q_data->h_adjusted;
>>> +	for (i = 0; i < MXC_JPEG_MAX_PLANES; i++) {
>>> +		tmp_q.bytesperline[i] = q_data->bytesperline[i];
>>> +		tmp_q.sizeimage[i] = q_data->sizeimage[i];
>>> +	}
>>> +	mxc_jpeg_sizeimage(&tmp_q);
>>> +	for (i = 0; i < MXC_JPEG_MAX_PLANES; i++)
>>> +		tmp_q.sizeimage[i] = max(tmp_q.sizeimage[i], q_data->sizeimage[i]);
>>> +
>>>    	/* Handle CREATE_BUFS situation - *nplanes != 0 */
>>>    	if (*nplanes) {
>>>    		if (*nplanes != q_data->fmt->colplanes)
>>>    			return -EINVAL;
>>>    		for (i = 0; i < *nplanes; i++) {
>>> -			if (sizes[i] < q_data->sizeimage[i])
>>> +			if (sizes[i] < tmp_q.sizeimage[i])
>>>    				return -EINVAL;
>>>    		}
>>>    		return 0;
>>> @@ -1181,7 +1193,7 @@ static int mxc_jpeg_queue_setup(struct
>> vb2_queue *q,
>>>    	/* Handle REQBUFS situation */
>>>    	*nplanes = q_data->fmt->colplanes;
>>>    	for (i = 0; i < *nplanes; i++)
>>> -		sizes[i] = q_data->sizeimage[i];
>>> +		sizes[i] = tmp_q.sizeimage[i];
>>>
>>>    	return 0;
>>>    }
>>> @@ -1381,11 +1393,6 @@ static int mxc_jpeg_parse(struct mxc_jpeg_ctx
>> *ctx, struct vb2_buffer *vb)
>>>    	}
>>>    	q_data_out->w = header.frame.width;
>>>    	q_data_out->h = header.frame.height;
>>> -	if (header.frame.width % 8 != 0 || header.frame.height % 8 != 0) {
>>> -		dev_err(dev, "JPEG width or height not multiple of 8: %dx%d\n",
>>> -			header.frame.width, header.frame.height);
>>> -		return -EINVAL;
>>> -	}
>>>    	if (header.frame.width > MXC_JPEG_MAX_WIDTH ||
>>>    	    header.frame.height > MXC_JPEG_MAX_HEIGHT) {
>>>    		dev_err(dev, "JPEG width or height should be <= 8192: %dx%d\n",
>> @@
>>> -1748,22 +1755,17 @@ static int mxc_jpeg_try_fmt(struct v4l2_format *f,
>> const struct mxc_jpeg_fmt *fm
>>>    	pix_mp->num_planes = fmt->colplanes;
>>>    	pix_mp->pixelformat = fmt->fourcc;
>>>
>>> -	/*
>>> -	 * use MXC_JPEG_H_ALIGN instead of fmt->v_align, for vertical
>>> -	 * alignment, to loosen up the alignment to multiple of 8,
>>> -	 * otherwise NV12-1080p fails as 1080 is not a multiple of 16
>>> -	 */
>>> +	pix_mp->width = w;
>>> +	pix_mp->height = h;
>>>    	v4l_bound_align_image(&w,
>>> -			      MXC_JPEG_MIN_WIDTH,
>>> -			      w, /* adjust downwards*/
>>> +			      w, /* adjust upwards*/
>>> +			      MXC_JPEG_MAX_WIDTH,
>>>    			      fmt->h_align,
>>>    			      &h,
>>> -			      MXC_JPEG_MIN_HEIGHT,
>>> -			      h, /* adjust downwards*/
>>> -			      MXC_JPEG_H_ALIGN,
>>> +			      h, /* adjust upwards*/
>>> +			      MXC_JPEG_MAX_HEIGHT,
>>> +			      0,
>>>    			      0);
>>> -	pix_mp->width = w; /* negotiate the width */
>>> -	pix_mp->height = h; /* negotiate the height */
>>>
>>>    	/* get user input into the tmp_q */
>>>    	tmp_q.w = w;
>>> @@ -1889,35 +1891,19 @@ static int mxc_jpeg_s_fmt(struct mxc_jpeg_ctx
>>> *ctx,
>>>
>>>    	q_data->w_adjusted = q_data->w;
>>>    	q_data->h_adjusted = q_data->h;
>>> -	if (jpeg->mode == MXC_JPEG_DECODE) {
>>> -		/*
>>> -		 * align up the resolution for CAST IP,
>>> -		 * but leave the buffer resolution unchanged
>>> -		 */
>>> -		v4l_bound_align_image(&q_data->w_adjusted,
>>> -				      q_data->w_adjusted,  /* adjust upwards */
>>> -				      MXC_JPEG_MAX_WIDTH,
>>> -				      q_data->fmt->h_align,
>>> -				      &q_data->h_adjusted,
>>> -				      q_data->h_adjusted, /* adjust upwards */
>>> -				      MXC_JPEG_MAX_HEIGHT,
>>> -				      q_data->fmt->v_align,
>>> -				      0);
>>> -	} else {
>>> -		/*
>>> -		 * align down the resolution for CAST IP,
>>> -		 * but leave the buffer resolution unchanged
>>> -		 */
>>> -		v4l_bound_align_image(&q_data->w_adjusted,
>>> -				      MXC_JPEG_MIN_WIDTH,
>>> -				      q_data->w_adjusted, /* adjust downwards*/
>>> -				      q_data->fmt->h_align,
>>> -				      &q_data->h_adjusted,
>>> -				      MXC_JPEG_MIN_HEIGHT,
>>> -				      q_data->h_adjusted, /* adjust downwards*/
>>> -				      q_data->fmt->v_align,
>>> -				      0);
>>> -	}
>>> +	/*
>>> +	 * align up the resolution for CAST IP,
>>> +	 * but leave the buffer resolution unchanged
>>> +	 */
>>> +	v4l_bound_align_image(&q_data->w_adjusted,
>>> +			      q_data->w_adjusted,  /* adjust upwards */
>>> +			      MXC_JPEG_MAX_WIDTH,
>>> +			      q_data->fmt->h_align,
>>> +			      &q_data->h_adjusted,
>>> +			      q_data->h_adjusted, /* adjust upwards */
>>> +			      MXC_JPEG_MAX_HEIGHT,
>>> +			      q_data->fmt->v_align,
>>> +			      0);
>>>
>>>    	for (i = 0; i < pix_mp->num_planes; i++) {
>>>    		q_data->bytesperline[i] = pix_mp->plane_fmt[i].bytesperline;



More information about the linux-arm-kernel mailing list