Working with the OV13850 camera sensor on the NanoPC-T4
Sebastian Fricke
sebastian.fricke.linux at gmail.com
Wed Nov 18 10:09:09 EST 2020
[RESEND with all CC Email addresses]
On 16.11.2020 08:31, Helen Koike wrote:
>Hi Sebastian,
Hey Helen,
just wanted to leave a quick thank you for your advice!
I have left a few remarks below.
>
>On 11/15/20 3:41 PM, Ezequiel Garcia wrote:
>> On Sun, 15 Nov 2020 at 08:11, Sebastian Fricke
>> <sebastian.fricke.linux at gmail.com> wrote:
>>>
>>> Hello,
>>>
>>
>> Hello Sebastian,
>>
>> Let me first add my colleagues Helen and Dafna, who maintain this driver,
>> and who will surely yell if I stop making sense here.
>>
>>> I am currently trying to get the OV13850 camera sensor
>>> (https://www.friendlyarm.com/index.php?route=product/product&product_id=228) to work on my friendlyElec NanoPC-T4.
>>>
>>> I have problems with connecting the RkISP1 ISP to the OV13850 sensor and I am not sure,
>>> where the problem could be. The device tree seems to load correctly and
>>> I can detect the sensor as a device on the i2c bus:
>>>
>>> root at nanopct4:~# cat /sys/bus/i2c/devices/1-0010/name
>>> ov13850
>>>
>>
>> OK, good start :-)
>>
>>> And the driver module is loaded as well:
>>>
>>> root at nanopct4:~# lsmod | grep ov13850
>>> ov13850 28672 0
>>> v4l2_fwnode 28672 2 rockchip_isp1,ov13850
>>> videodev 266240 9 rockchip_vdec,v4l2_fwnode,rockchip_isp1,videobuf2_v4l2,hantro_vpu,rockchip_rga,videobuf2_common,v4l2_mem2mem,ov13850
>>> mc 61440 8 rockchip_vdec,videodev,rockchip_isp1,videobuf2_v4l2,hantro_vpu,videobuf2_common,v4l2_mem2mem,ov13850
>>>
>>> The driver reports using dummy regulators instead of the requested ones,
>>> I am not sure yet if this is part of the problem, as the driver doesn't
>>> bail out after requesting the regulators. But from what I currently
>>> understand, these warnings mean that for some reason my system didn't
>>> map these regulators but acts as if they were there.
>>>
>>> More info below, I hope that someone can help to find the error I made,
>>> thanks in advance!
>>>
>>> -----------------------------
>>>
>>> I attached the two patches I created:
>>> 1. For the device tree I combined a patch from Helen Koike (which is not merged yet),
>>> where she adds the isp0 to the rk3399.dtsi file, with my addition which
>>> activates the mipi_dphy_rx0, adds the camera sensor to i2c1 and
>>> connects the pads of the ISP with the sensor. I followed the
>>> documentation for the ISP part and got most of the camera sensor
>>> parts from the BSP Kernel:
>>> (https://github.com/friendlyarm/kernel-rockchip/blob/nanopi4-linux-v4.4.y/arch/arm64/boot/dts/rockchip/rk3399-nanopi4-rkisp1.dtsi#L52).
>
>If you take a look at this patch:
>
> https://patchwork.linuxtv.org/project/linux-media/patch/20201020193850.1460644-10-helen.koike@collabora.com/
>
>You'll see that the dts of the isp has
>
>ports {
> port at 0 {
> ...
> }
>}
>
>which differ from the patch you attached, and I guess this is why your
>sensor is not being detected by the rkisp1 driver.
>
>Please check:
>
> https://git.linuxtv.org/media_tree.git/tree/drivers/staging/media/rkisp1/Documentation/devicetree/bindings/media/rockchip-isp1.yaml
>
This was exactly the problem. Thank you very much!
>>> 2. I ported the driver from the BSP kernel of friendlyElec:
>>> (https://github.com/friendlyarm/kernel-rockchip/blob/nanopi4-linux-v4.4.y/drivers/media/i2c/ov13850.c)
>>> I changed a few lines in order to have the module compile correctly.
>>> ```
>>> +#include <linux/compat.h>
>>> - sd->entity.type = MEDIA_ENT_T_V4L2_SUBDEV_SENSOR;
>>> - ret = media_entity_init(&sd->entity, 1, &ov13850->pad, 0);
>>> + sd->entity.function = MEDIA_ENT_F_CAM_SENSOR;
>>> + ret = media_entity_pads_init(&sd->entity, 1, &ov13850->pad);
>>> ```
>>>
>>
>> Keep in mind that, with some exceptions, the upstream community
>> doesn't provide much help with downstream/vendor kernels.
>>
>>> ----------------------------------------
>>>
>>> I was able to create an armbian image for the media_tree (https://git.linuxtv.org/media_tree.git/):
>>
>> Ah, upstream is better :-)
>>
>>> root at nanopct4:~# uname -a
>>> Linux nanopct4 5.10.0-rc1-rockchip64 #trunk SMP PREEMPT Fri Nov 13 15:08:05 CET 2020 aarch64 GNU/Linux
>>>
>>> When I boot up the board I can spot the following messages in the kernel
>>> log:
>>> [ 7.216307] ov13850 1-0010: driver version: 00.01.01
>>> [ 7.216322] ov13850 1-0010: could not get module information!
>>> [ 7.216565] ov13850 1-0010: supply avdd not found, using dummy regulator
>>> [ 7.216761] ov13850 1-0010: supply dovdd not found, using dummy regulator
>>> [ 7.216846] ov13850 1-0010: supply dvdd not found, using dummy regulator
>>> [ 7.219535] ov13850 1-0010: Detected OV00d850 sensor, REVISION 0xb1
>>
>> OK, good.
>>
>> I can be wrong (since I haven't looked at your driver) but this
>> usually indicates
>> your sensor is powered and properly configured to at least read
>> some CHIP_ID register.
>>
>> The regulators are likely always-on in your sensor module,
>> so maybe probably that's why it works.
>>
>> See for instance arch/arm64/boot/dts/renesas/aistarvision-mipi-adapter-2.1.dtsi
>> for an example of how regulators can be declared. There are other ways,
>> it's just an example.
>>
>>> ...
>>> [ 7.352292] rockchip_isp1: module is from the staging directory, the quality is unknown, you have been warned.
>>> ...
>>> [ 7.356178] rkisp1 ff910000.isp0: Adding to iommu group 4
>>> ...
>>> [ 7.357637] rkisp1: registered rkisp1_mainpath as /dev/video0
>>> [ 7.357816] rkisp1: registered rkisp1_selfpath as /dev/video1
>>>
>>> ----------------------------------------
>>>
>>> And this command (try to stream 50 frames from video1 which the mainpath
>>> on the RkISP1):
>>> root at nanopct4:~# v4l2-ctl --stream-to /home/basti/test.raw --stream-mmap 50 -d /dev/video0 --verbose
>>>
>>> I get this output:
>>> VIDIOC_STREAMON returned -1 (No such device)
>>>
>>> And this kernel log message:
>>> [16939.667867] rkisp1 ff910000.isp0: No link between isp and sensor
>>>
>>
>> This error seems useful. It would indicate your sensor is not
>> connected (software-connected) to the ISP.
>>
>> See below.
>>
>>> -----------------------------------------
>>>
>>> Here is the output for media-ctl -p:
>>>
>>> Media controller API version 5.10.0
>>>
>>> Media device information
>>> ------------------------
>>> driver rkisp1
>>> model rkisp1
>>> serial
>>> bus info platform:rkisp1
>>> hw revision 0x0
>>> driver version 5.10.0
>>>
>>> Device topology
>>> - entity 1: rkisp1_isp (4 pads, 4 links)
>>> type V4L2 subdev subtype Unknown flags 0
>>> device node name /dev/v4l-subdev0
>>> pad0: Sink
>>> [fmt:SRGGB10_1X10/800x600 field:none
>>> crop.bounds:(0,0)/800x600
>>> crop:(0,0)/800x600]
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>
>> Note that here the rkisp1_isp entity sink pad 0
>> should be connected to the bayer sensor, but it
>> seems not connected to anything.
>>
>> I can be wrong, but I don't see your sensor
>> appearing anywhere in the topology.
>>
>> See for instance, the example in the driver
>> documentation:
>>
>> https://www.kernel.org/doc/html/latest/admin-guide/media/rkisp1.html
>
>Just to complement what Ezequiel mentioned, to better understand, you can
>check the topology with
>
> media-ctl --print-dot
>
>and copy the results to https://dreampuf.github.io/GraphvizOnline/
>
>You'll see there is no sensor connected in pad 0 of the rkisp1_isp node.
>
Cool website didn't know it exists
>>
>> And note the section where the topology is set, connecting
>> the imx219 sensor to rkisp1_isp sink pad0:
>>
>> "media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 ->
>> 'rkisp1_isp':0 [1]"
>>
>> So I would say you are very much on the right track,
>> but you still need a bit more work to construct the capture pipeline.
>>
>> Not sure if this helps, or makes things more complicated, but instead
>> of v4l2-ctl, I would personally start with libcamera, and work from there.
>>
>> Cheers,
>> Ezequiel
>>
>After correcting the topology, you should configure the stream as indicated
>in the docs:
>
> https://www.kernel.org/doc/html/latest/admin-guide/media/rkisp1.html#capturing-video-frames-example
>
It is difficult to spot anything on the video, as it is super dark but
this works. My current efforts are focused on getting to work on the
RkISP1 IPAs so I hope to change this to the better soon. ;)
>Or use libamera with, for instance:
>
> cam -c 1 -C
>
This sadly didn't work directly, the libcamera pipeline handler has to
be fixed as it has a maximum resolution of 4032x3024, while not
configuring the output resolution of the sensor correctly:
(This was pointed out by Laurent Pinchart on the libcamera IRC, I will try to fix this.)
basti at nanopct4:~$ LIBCAMERA_LOG_LEVELS=0 cam -c 1 -C
[0:10:02.777886361] [2102] DEBUG IPAProxy ipa_proxy.cpp:290 Registered proxy "IPAProxyLinux"
[0:10:02.778607938] [2102] DEBUG IPAProxy ipa_proxy.cpp:290 Registered proxy "IPAProxyThread"
[0:10:02.782258408] [2102] DEBUG IPAModule ipa_module.cpp:330 ipa_rkisp1.so: IPA module /usr/local/lib/aarch64-linux-gnu/libcamera/ipa_rkisp1.so is signed
[0:10:02.783011777] [2102] DEBUG IPAManager ipa_manager.cpp:238 Loaded IPA module '/usr/local/lib/aarch64-linux-gnu/libcamera/ipa_rkisp1.so'
[0:10:02.788297608] [2102] DEBUG IPAModule ipa_module.cpp:330 ipa_rpi.so: IPA module /usr/local/lib/aarch64-linux-gnu/libcamera/ipa_rpi.so is signed
[0:10:02.789237642] [2102] DEBUG IPAManager ipa_manager.cpp:238 Loaded IPA module '/usr/local/lib/aarch64-linux-gnu/libcamera/ipa_rpi.so'
[0:10:02.790346258] [2102] DEBUG IPAModule ipa_module.cpp:330 ipa_vimc.so: IPA module /usr/local/lib/aarch64-linux-gnu/libcamera/ipa_vimc.so is signed
[0:10:02.790817587] [2102] DEBUG IPAManager ipa_manager.cpp:238 Loaded IPA module '/usr/local/lib/aarch64-linux-gnu/libcamera/ipa_vimc.so'
[0:10:02.791825287] [2102] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+2030-d14a3f8a
[0:10:02.792881987] [2103] DEBUG Camera camera_manager.cpp:105 Starting camera manager
[0:10:02.817883446] [2103] DEBUG DeviceEnumerator device_enumerator.cpp:223 New media device "hantro-vpu" created from /dev/media2
[0:10:02.818795772] [2103] DEBUG DeviceEnumerator device_enumerator_udev.cpp:94 Defer media device /dev/media2 due to 2 missing dependencies
[0:10:02.823322693] [2103] DEBUG DeviceEnumerator device_enumerator_udev.cpp:319 All dependencies for media device /dev/media2 found
[0:10:02.823521900] [2103] DEBUG DeviceEnumerator device_enumerator.cpp:251 Added device /dev/media2: hantro-vpu
[0:10:02.825636174] [2103] DEBUG DeviceEnumerator device_enumerator.cpp:223 New media device "rkvdec" created from /dev/media1
[0:10:02.825903630] [2103] DEBUG DeviceEnumerator device_enumerator_udev.cpp:94 Defer media device /dev/media1 due to 1 missing dependencies
[0:10:02.827886364] [2103] DEBUG DeviceEnumerator device_enumerator_udev.cpp:319 All dependencies for media device /dev/media1 found
[0:10:02.828109487] [2103] DEBUG DeviceEnumerator device_enumerator.cpp:251 Added device /dev/media1: rkvdec
[0:10:02.830667966] [2103] DEBUG DeviceEnumerator device_enumerator.cpp:223 New media device "rkisp1" created from /dev/media0
[0:10:02.832361952] [2103] DEBUG DeviceEnumerator device_enumerator_udev.cpp:94 Defer media device /dev/media0 due to 7 missing dependencies
[0:10:02.844721516] [2103] DEBUG DeviceEnumerator device_enumerator_udev.cpp:319 All dependencies for media device /dev/media0 found
[0:10:02.844916639] [2103] DEBUG DeviceEnumerator device_enumerator.cpp:251 Added device /dev/media0: rkisp1
[0:10:02.846288920] [2103] DEBUG Camera camera_manager.cpp:148 Found registered pipeline handler 'PipelineHandlerIPU3'
[0:10:02.847350869] [2103] DEBUG Camera camera_manager.cpp:148 Found registered pipeline handler 'PipelineHandlerRPi'
[0:10:02.847648075] [2103] DEBUG Camera camera_manager.cpp:148 Found registered pipeline handler 'PipelineHandlerRkISP1'
[0:10:02.848204862] [2103] DEBUG DeviceEnumerator device_enumerator.cpp:311 Successful match for media device "rkisp1"
[0:10:02.849379686] [2103] DEBUG V4L2 v4l2_videodevice.cpp:577 /dev/video3[cap]: Opened device platform:rkisp1: rkisp1: rkisp1_stats
[0:10:02.849756516] [2103] DEBUG V4L2 v4l2_videodevice.cpp:577 /dev/video4[out]: Opened device platform:rkisp1: rkisp1: rkisp1_params
[0:10:02.850229304] [2103] DEBUG V4L2 v4l2_videodevice.cpp:577 /dev/video1[cap]: Opened device platform:rkisp1: rkisp1: rkisp1
[0:10:02.850775591] [2103] DEBUG V4L2 v4l2_videodevice.cpp:577 /dev/video2[cap]: Opened device platform:rkisp1: rkisp1: rkisp1
[0:10:02.883320946] [2103] DEBUG IPAManager ipa_manager.cpp:316 IPA module /usr/local/lib/aarch64-linux-gnu/libcamera/ipa_rkisp1.so signature is valid
[0:10:02.888229655] [2103] DEBUG Camera camera_manager.cpp:160 Pipeline handler "PipelineHandlerRkISP1" matched
[0:10:02.888665694] [2103] DEBUG Camera camera_manager.cpp:148 Found registered pipeline handler 'SimplePipelineHandler'
[0:10:02.888949483] [2103] DEBUG Camera camera_manager.cpp:148 Found registered pipeline handler 'PipelineHandlerUVC'
[0:10:02.889104648] [2103] DEBUG Camera camera_manager.cpp:148 Found registered pipeline handler 'PipelineHandlerVimc'
Using camera /base/i2c at ff110000/ov13850 at 10
[0:10:02.891430671] [2102] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
[0:10:02.892222831] [2102] INFO Camera camera.cpp:830 configuring streams: (0) 1920x1920-NV12
[0:10:02.893203989] [2103] DEBUG MediaDevice media_device.cpp:808 /dev/media0[rkisp1]: rkisp1_isp[2] -> rkisp1_resizer_mainpath[0]: 0
[0:10:02.893346613] [2103] DEBUG MediaDevice media_device.cpp:808 /dev/media0[rkisp1]: rkisp1_isp[2] -> rkisp1_resizer_selfpath[0]: 0
[0:10:02.893470862] [2103] DEBUG MediaDevice media_device.cpp:808 /dev/media0[rkisp1]: ov13850 1-0010[0] -> rkisp1_isp[0]: 0
[0:10:02.893602111] [2103] DEBUG RkISP1 rkisp1.cpp:997 Enabling link from sensor 'ov13850 1-0010' to ISP
[0:10:02.893698360] [2103] DEBUG MediaDevice media_device.cpp:808 /dev/media0[rkisp1]: ov13850 1-0010[0] -> rkisp1_isp[0]: 1
[0:10:02.893819401] [2103] DEBUG MediaDevice media_device.cpp:808 /dev/media0[rkisp1]: rkisp1_isp[2] -> rkisp1_resizer_mainpath[0]: 1
[0:10:02.893921775] [2103] DEBUG RkISP1 rkisp1.cpp:684 Configuring sensor with 4224x3136-SBGGR10_1X10
[0:10:02.894077815] [2103] DEBUG RkISP1 rkisp1.cpp:690 Sensor configured with 4224x3136-SBGGR10_1X10
[0:10:02.894252231] [2103] DEBUG RkISP1 rkisp1.cpp:701 ISP input pad configured with 4032x3024-SBGGR10_1X10
[0:10:02.894355771] [2103] DEBUG RkISP1 rkisp1.cpp:705 Configuring ISP output pad with 4032x3024-YUYV8_2X8
[0:10:02.894458729] [2103] DEBUG RkISP1 rkisp1.cpp:711 ISP output pad configured with 800x600-YUYV8_2X8
[0:10:02.894572770] [2103] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
[0:10:02.894674560] [2103] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 1920x1920-YUYV8_2X8
[0:10:02.894781893] [2103] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 1920x1920-YUYV8_1_5X8
[0:10:02.915852010] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1137 /dev/video1[cap]: 4 buffers requested.
[0:10:02.917560871] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1137 /dev/video1[cap]: 0 buffers requested.
[0:10:02.918685528] [2102] DEBUG Camera camera.cpp:967 Starting capture
[0:10:02.919501021] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1137 /dev/video4[out]: 4 buffers requested.
[0:10:02.921160591] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1137 /dev/video3[cap]: 4 buffers requested.
[0:10:02.923722570] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1137 /dev/video1[cap]: 4 buffers requested.
[0:10:02.923808902] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1350 /dev/video1[cap]: Prepared to import 4 buffers
[0:10:02.923921193] [2103] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13850 1-0010': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device
[0:10:02.924034067] [2103] ERROR CameraSensor camera_sensor.cpp:513 'ov13850 1-0010': Failed to construct camera sensor info: the camera sensor does not report the active area
[0:10:02.924062650] [2103] WARN RkISP1 rkisp1.cpp:906 Camera sensor information not available
[0:10:02.924368606] [2103] INFO IPARkISP1 rkisp1.cpp:115 Exposure: 4-3324 Gain: 16-248
[0:10:02.924925977] [2103] DEBUG Timer timer.cpp:105 Starting timer 0xffffb0015ea0: deadline 0:10:02.929769061
[0:10:02.925217932] [2103] DEBUG Event event_dispatcher_poll.cpp:217 timeout 0.004577378
Capture until user interrupts by SIGINT
[0:10:02.925964885] [2103] DEBUG Event event_dispatcher_poll.cpp:217 timeout 0.003814385
[0:10:02.926044509] [2103] DEBUG Event event_dispatcher_poll.cpp:217 timeout 0.003727469
[0:10:02.926105758] [2103] DEBUG Event event_dispatcher_poll.cpp:217 timeout 0.003665928
[0:10:02.926152716] [2103] DEBUG Event event_dispatcher_poll.cpp:217 timeout 0.003618970
[0:10:02.930000060] [2103] DEBUG Timer timer.cpp:105 Starting timer 0xffffb0015ea0: deadline 0:10:02.935715803
[0:10:02.930041184] [2103] DEBUG Event event_dispatcher_poll.cpp:217 timeout 0.005677244
[0:10:02.935862511] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video4[out]: Queueing buffer 0
[0:10:02.935981802] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video3[cap]: Queueing buffer 0
[0:10:02.936049759] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video1[cap]: Queueing buffer 0
[0:10:02.936136967] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video4[out]: Queueing buffer 1
[0:10:02.936177508] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video3[cap]: Queueing buffer 1
[0:10:02.936212508] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video1[cap]: Queueing buffer 1
[0:10:02.936276091] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video4[out]: Queueing buffer 2
[0:10:02.936310799] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video3[cap]: Queueing buffer 2
[0:10:02.936344049] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video1[cap]: Queueing buffer 2
[0:10:02.936418131] [2103] ERROR V4L2 v4l2_videodevice.cpp:1443 /dev/video1[cap]: Failed to queue buffer 2: Broken pipe
[0:10:02.936458673] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video4[out]: Queueing buffer 3
[0:10:02.936493089] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video3[cap]: Queueing buffer 3
[0:10:02.936526922] [2103] DEBUG V4L2 v4l2_videodevice.cpp:1439 /dev/video1[cap]: Queueing buffer 3
[0:10:02.936584963] [2103] ERROR V4L2 v4l2_videodevice.cpp:1443 /dev/video1[cap]: Failed to queue buffer 3: Broken pipe
^CExiting
>
>I hope this helps.
>Regards,
>Helen
Greetings,
Sebastian
More information about the Linux-rockchip
mailing list