Enable selection of sensor format to avoid a cropped FoV

[larsll]

Work to resolve #99

It enables the raw role, and adjusts the size of it to match the desired input, this avoids cropping where libcamera by default selects a sensor mode that is not having the full FoV. This is a problem with RPi Camera V2 (see https://picamera.readthedocs.io/en/release-1.13/fov.html#sensor-modes).

Example:

ros2 run camera_ros camera_node --ros-args -p format:=BGR888 -p height:=120 -p width:=160 -p mode_width:=1640 -p mode_height:=1232

Output:

[INFO] [1740686517.425357897] [camera]: 1640x1232
[INFO] [1740686517.425436891] [camera]: Mode configuration:1640x1232-SBGGR12_CSI2P
[WARN] [1740686517.425989703] [camera]: stream configuration adjusted from "160x120-BGR888" to "160x120-BGR888"
[0:46:50.002516247] [3249]  INFO Camera camera.cpp:1033 configuring streams: (0) 160x120-BGR888 (1) 1640x1232-SBGGR10_CSI2P
[0:46:50.003349197] [3259]  INFO RPI vc4.cpp:512 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1640x1232-SBGGR10_1X10 - Selected unicam format: 1640x1232-pBAA
[INFO] [1740686517.429272329] [camera]: camera "/base/soc/i2c0mux/i2c@1/imx219@10" configured with 160x120-BGR888 stream

[christianrauch]

Thanks for the PR.

Can you add a before/after comparison to this PR? This will make it more clear what the new parameter do.

The new parameters and their behaviour and interaction with the other sizes and stream roles needs documentation, if this should make it into the upstream code. I think having two sets of sizes is confusing as it is not clear from the parameter names what the effect is.

[christianrauch]

What is the meaning of mode in the parameter prefix? I think these new parameters also need to be documented, as it is not clear what they do without looking at the code.

[larsll]

The term mode seems to be the term for this -- and is a short for sensor mode;

• https://www.raspberrypi.com/documentation/computers/camera_software.html#common-options
• How to change the default sensor mode that gets selected by the video configuration? raspberrypi/picamera2#756

We could opt for a style similar to the rpicam tools; where mode is width:height:bit:pack, and maybe calling it raw_mode or sensor_mode would make abit more sense.

[christianrauch]

According to the first source, "sensor mode" refers to what the node uses as format, e.g. SRGGB10_CSI2P:

Sensor mode identifiers have the following form: S<Bayer order><Bit-depth>_<Optional packing> : <Resolution list>

How are the rpicam-apps terming these parameters?

We could opt for a style similar to the rpicam tools; where mode is width:height:bit:pack, and maybe calling it raw_mode or sensor_mode would make abit more sense.

This would also work. In contrast to the other stream roles, the raw roles are always discrete, i.e. the stream configuration for the raw stream will always be adjusted to one of the raw formats.

[christianrauch]

What happens if a user already selected raw for the role? Are we then adding the same role twice with potentially different conflicting configurations?

[larsll]

I agree, a check should be made, that if role is raw then this is not having an effect (and then ignore it).

[christianrauch]

Alternatively, we could remove raw from the options for role. The device specific raw formats aren't supported by the ROS message anyway, and I am not sure if there is a benefit of publishing in Bayer format and then doing the debayering somewhere else.

[larsll]

For now I have edited the code to not use my changes if the first role is a Raw.

[christianrauch]

Have you had a look at the std::optional<SensorConfiguration> sensorConfig; inside the CameraConfiguration? This seems to store the sensor configuration, such as the binning, size and crop region. Can you achieve the same effect by setting those values?

[christianrauch]

When you update the PR, can you rebase and squash your commits? I merged a change that avoids referring to the read-only parameters multiple times. Have a look at this and try following the same with the new read-only parameters that you want to add.

[christianrauch]

Have you had a look at the std::optional<SensorConfiguration> sensorConfig; inside the CameraConfiguration? This seems to store the sensor configuration, such as the binning, size and crop region. Can you achieve the same effect by setting those values?

Setting the sensorConfig (SensorConfiguration)

  libcamera::SensorConfiguration sensor;
  sensor.outputSize = libcamera::Size(4608, 2592);
  sensor.bitDepth = 10;

  if (sensor.isValid())
    cfg->sensorConfig = sensor;
  else
    throw std::runtime_error("invalid sensor configuration");

before the CameraConfiguration is validated, indeed has the same effect:

INFO Camera camera.cpp:1202 configuring streams: (0) 160x120-XRGB8888
INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 4608x2592-SBGGR10_1X10 - Selected CFE format: 4608x2592-PC1B

without the need to manage an additional stream.

But the size and bit depth have to match exactly for the validation to pass.