RPi Camera v2: Cropped image at low resolution

[larsll]

I am trying to use camera_ros to get a full FoV picture at a low resolution (640x480 or even 160x120) on Raspberry Pi 4, using the Camera Module v2, on Ubuntu 22.04/24.04.

Using the old legacy camera support then 640x480 was the default, but with libcamera (and hence camera_ros) it picks a sensor mode that leads to cropping, rather than scaling down.

Is there a way to get the sensor input for the lowest full FoV resolution (1640x1232), but then use the GPU to scale it to the desired lower resolution to avoid 'overloading' the ROS nodes having to serialize/deserialize the larger image?

Reference:
https://picamera.readthedocs.io/en/release-1.13/fov.html#sensor-modes

[christianrauch]

I don't fully understand what you want to do. If you just want to get a lower resolution from the camera, why aren't you setting that resolution via the parameters? The default selected pixel format and resolution depends on the selected stream mode. But you can change these things via ROS parameters.

[larsll]

The problem is visualized in the PiCamera page:
https://picamera.readthedocs.io/en/release-1.13/_images/sensor_area_2.png

If I ask for 640x480 using the normal parameters I get a cropped image (the smallest box in the middle); I obviously want the entire picture, just scaled down.

[christianrauch]

I see, you mean cropping within the same aspect ratio.

Indeed, some stream formats will use the same sensor configuration, while other formats with the same aspect ratio will use a different sensor configuration. E.g. 160x120-XRGB8888 and 640x480-XRGB8888 both map to 1536x864-SBGGR10_1X10:

$ ros2 run camera_ros camera_node --ros-args -p height:=120 -p width:=160
[...]
INFO Camera camera.cpp:1202 configuring streams: (0) 160x120-XRGB8888
INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 1536x864-SBGGR10_1X10 - Selected CFE format: 1536x864-PC1B

$ ros2 run camera_ros camera_node --ros-args -p height:=480 -p width:=640
[...]
INFO Camera camera.cpp:1202 configuring streams: (0) 640x480-XRGB8888
INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 1536x864-SBGGR10_1X10 - Selected CFE format: 1536x864-PC1B

while 1600x1200-XRGB8888 (i.e. 10x scale of 160x120-XRGB8888) will map to 2304x1296-SBGGR10_1X10:

$ ros2 run camera_ros camera_node --ros-args -p height:=1200 -p width:=1600
[...]
INFO Camera camera.cpp:1202 configuring streams: (0) 1600x1200-XRGB8888
INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected CFE format: 2304x1296-PC1B

But this is managed by the IPAs in libcamera and the ROS node has no control over this. You will get the same for the libcamera examples:

$ cam -c1 -C --stream width=160,height=120 
[...]
INFO Camera camera.cpp:1202 configuring streams: (0) 160x120-XRGB8888
INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 1536x864-SBGGR10_1X10 - Selected CFE format: 1536x864-PC1B

$ cam -c1 -C --stream width=640,height=480
[...]
INFO Camera camera.cpp:1202 configuring streams: (0) 640x480-XRGB8888
INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 1536x864-SBGGR10_1X10 - Selected CFE format: 1536x864-PC1B

$ cam -c1 -C --stream width=1600,height=1200
[...]
INFO Camera camera.cpp:1202 configuring streams: (0) 1600x1200-XRGB8888
INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected CFE format: 2304x1296-PC1B

You can inspect the "raw" sensor configurations by setting the role to "raw":

$ cam -c1 -I --stream role=raw
[...]
Using camera /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a as cam0
0: 4608x2592-BGGR_PISP_COMP1
 * Pixelformat: SRGGB10_CSI2P (1536x864)-(4608x2592)/(+0,+0)
  - 1536x864
  - 2304x1296
  - 4608x2592

So here you see that any of the stream roles formats map to one of these three (for the Camera Module 3) raw formats.

I don't know how much control a user has over this. You better ask for support on this with the libcamera developers. If it is possible to further control this manually, e.g. how many pixels should be binned together, I would add this to the documentation of the repo.

What you can always do, is, as you suggested, to manually downscale the larger image with the larger FoV. But I don't think this is something the camera node should do. You can write your own composable "downscaling" node or just reuse image_proc::CropDecimateNode or image_proc::ResizeNode from the image_proc package.

[larsll]

Thank you for the input; I will investigate a bit more; looking at the RPi software there seems to be a mode switch. (https://www.raspberrypi.com/documentation/computers/camera_software.html).

I need to install Raspberry Pi OS to see how it works; the ros-humble-libcamera is old.

[Woojin-Crive]

you can get a hint from my repo's commits

https://github.com/Woojin-Crive/camera_ros

[larsll]

@Woojin-Crive great! Thank you.

I expect this one to solve the problem:
Woojin-Crive@91ac144

It does not seem like cam has these options, but rpicam-still/vid has:
rpicam-still --mode 1640:1232:10 --width 160 --height 120 -o image.jpg

rpicam-still --width 160 --height 120 -o image_2.jpg

[larsll]

@christianrauch I sent the PR for review, more as an initial draft review, as documentation etc. would also have to be discussed.