Enable Raspberry Pi CSI2 cameras (#50)

* initial testing changes for ribbon cam * fix syntax * add specific options and convert to csi2 * finalize one csi2 camera and no autodiscover * Automatically reformatting code with black and isort * update readme and resolution error * Automatically reformatting code with black and isort * update version number --------- Co-authored-by: Auto-format Bot <[email protected]>
groundlight · Jul 25, 2024 · 7680413 · 7680413
1 parent 447f0aa
commit 7680413
Show file tree

Hide file tree

Showing 3 changed files with 80 additions and 28 deletions.
diff --git a/README.md b/README.md
@@ -21,11 +21,12 @@ pip install framegrab
 ```
 
 ## Optional Dependencies
-To use a Basler USB or GigE camera, you must separately install the `pypylon` package.
+Certain camera types have additional dependencies that must be installed separately. If you don't intend to use these camera types, you don't need to install these extra packages. 
 
-Similarly, to use Intel RealSense cameras, you must install `pyrealsense2`. 
+- To use a Basler USB or GigE camera, you must separately install the `pypylon` package.
+- To use Intel RealSense cameras, you must install `pyrealsense2`.
+- To use a Raspberry Pi "CSI2" camera (connected with a ribbon cable), you must install the `picamera2` library. See install instructions at the [picamera2 github repository](https://github.com/raspberrypi/picamera2).
 
-If you don't intend to use these camera types, you don't need to install these extra packages. 
 
 ## Usage
 
@@ -44,7 +45,7 @@ lists the sub-commands, including `autodiscover` and `preview`.
 Frame Grabbers are defined by a configuration dict which is usually stored as YAML.  The configuration combines the camera type, the camera ID, and the camera options.  The configuration is passed to the `FrameGrabber.create_grabber` method to create a grabber object.  The grabber object can then be used to grab frames from the camera.
 
 
-`config` can contain many details and settings about your camera, but only `input_type` is required. Available `input_type` options are: `generic_usb`, `rtsp`, `realsense`, and `basler`.
+`config` can contain many details and settings about your camera, but only `input_type` is required. Available `input_type` options are: `generic_usb`, `rtsp`, `realsense`, `basler`, and `rpi_csi2`.
 
 Here's an example of a single USB camera configured with several options:
 ```python
@@ -93,7 +94,7 @@ When you are done with the camera, release the resource by running:
 grabber.release()
 ```
 
-You might have several cameras that you want to use in the same application. In this case, you can load the configurations from a yaml file and use `FrameGrabber.create_grabbers`.
+You might have several cameras that you want to use in the same application. In this case, you can load the configurations from a yaml file and use `FrameGrabber.create_grabbers`. Note that currently only a single Raspberry Pi CSI2 camera is supported, but these cameras can be used in conjunction with other types of cameras. 
 
 If you have multiple cameras of the same type plugged in, it's recommended that you include serial numbers in the configurations; this ensures that each configuration is paired with the correct camera. If you don't provide serial numbers in your configurations, configurations will be paired with cameras in a sequential manner.
 
@@ -146,35 +147,35 @@ for grabber in grabbers.values():
 ```
 ### Configurations
 The table below shows all available configurations and the cameras to which they apply.
-| Configuration Name         | Example         | Generic USB     | RTSP      | Basler    | Realsense |
-|----------------------------|-----------------|------------|-----------|-----------|-----------|
-| name                       | On Robot Arm    | optional   | optional  | optional  | optional  |
-| input_type                 | generic_usb    | required   | required  | required  | required  |
-| id.serial_number           | 23458234       | optional   | -         | optional  | optional  |
-| id.rtsp_url                | rtsp://…        | -          | required  | -         | -         |
-| options.resolution.height  | 480            | optional   | -         | -         | optional  |
-| options.resolution.width   | 640            | optional   | -         | -         | optional  |
-| options.zoom.digital       | 1.3            | optional   | optional  | optional  | optional  |
-| options.crop.pixels.top    | 100            | optional   | optional  | optional  | optional  |
-| options.crop.pixels.bottom | 400            | optional   | optional  | optional  | optional  |
-| options.crop.pixels.left   | 100            | optional   | optional  | optional  | optional  |
-| options.crop.pixels.right  | 400            | optional   | optional  | optional  | optional  |
-| options.crop.relative.top  | 0.1            | optional   | optional  | optional  | optional  |
-| options.crop.relative.bottom | 0.9          | optional   | optional  | optional  | optional  |
-| options.crop.relative.left | 0.1            | optional   | optional  | optional  | optional  |
-| options.crop.relative.right | 0.9            | optional   | optional  | optional  | optional  |
-| options.depth.side_by_side | 1              | -          | -         | -         | optional  |
-| options.num_90_deg_rotations | 2              | optional          | optional         | optional         | optional  |
-| options.keep_connection_open | True              | -          | optional         | -         | -  |
-| options.max_fps | 30              | -          | optional         | -         | -  |
+| Configuration Name         | Example         | Generic USB     | RTSP      | Basler    | Realsense | Raspberry Pi CSI2 |
+|----------------------------|-----------------|------------|-----------|-----------|-----------|-----------|
+| name                       | On Robot Arm    | optional   | optional  | optional  | optional  | optional  |
+| input_type                 | generic_usb    | required   | required  | required  | required  | required  |
+| id.serial_number           | 23458234       | optional   | -         | optional  | optional  | -  |
+| id.rtsp_url                | rtsp://…        | -          | required  | -         | -         | -         |
+| options.resolution.height  | 480            | optional   | -         | -         | optional  | -  |
+| options.resolution.width   | 640            | optional   | -         | -         | optional  | -  |
+| options.zoom.digital       | 1.3            | optional   | optional  | optional  | optional  | optional  |
+| options.crop.pixels.top    | 100            | optional   | optional  | optional  | optional  | optional  |
+| options.crop.pixels.bottom | 400            | optional   | optional  | optional  | optional  | optional  |
+| options.crop.pixels.left   | 100            | optional   | optional  | optional  | optional  | optional  |
+| options.crop.pixels.right  | 400            | optional   | optional  | optional  | optional  | optional  |
+| options.crop.relative.top  | 0.1            | optional   | optional  | optional  | optional  | optional  |
+| options.crop.relative.bottom | 0.9          | optional   | optional  | optional  | optional  | optional  |
+| options.crop.relative.left | 0.1            | optional   | optional  | optional  | optional  | optional  |
+| options.crop.relative.right | 0.9            | optional   | optional  | optional  | optional  | optional  |
+| options.depth.side_by_side | 1              | -          | -         | -         | optional  | optional  |
+| options.num_90_deg_rotations | 2              | optional          | optional         | optional         | optional  | optional  |
+| options.keep_connection_open | True              | -          | optional         | -         | -  | - |
+| options.max_fps | 30              | -          | optional         | -         | -  | - |
 
 
 
 
 In addition to the configurations in the table above, you can set any Basler camera property by including `options.basler.<BASLER PROPERTY NAME>`. For example, it's common to set `options.basler.PixelFormat` to `RGB8`.
 
 ### Autodiscovery
-Autodiscovery automatically connects to all cameras that are plugged into your machine or discoverable on the network, including `generic_usb`, `realsense`, `basler`, and ONVIF supported `rtsp` cameras. Default configurations will be loaded for each camera. Note that discovery of RTSP cameras will be disabled by default but can be enabled by setting `rtsp_discover_mode`. Refer to [RTSP Discovery](#rtsp-discovery) section for different options.
+Autodiscovery automatically connects to cameras that are plugged into your machine or discoverable on the network, including `generic_usb`, `realsense`, `basler`, and ONVIF supported `rtsp` cameras. Note that `rpi_csi2` cameras are not yet supported by autodiscover. Default configurations will be loaded for each camera. Note that discovery of RTSP cameras will be disabled by default but can be enabled by setting `rtsp_discover_mode`. Refer to [RTSP Discovery](#rtsp-discovery) section for different options.
 
 Autodiscovery is great for simple applications where you don't need to set any special options on your cameras. It's also a convenient method for finding the serial numbers of your cameras (if the serial number isn't printed on the camera).
 ```python

diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "framegrab"
-version = "0.6.1"
+version = "0.6.2"
 description = "Easily grab frames from cameras or streams"
 authors = ["Groundlight <[email protected]>"]
 license = "MIT"

diff --git a/src/framegrab/grabber.py b/src/framegrab/grabber.py
@@ -30,6 +30,12 @@
 except ImportError as e:
     rs = UnavailableModule(e)
 
+# Only used for CSI2 cameras with Raspberry Pi, not required otherwise
+try:
+    from picamera2 import Picamera2
+except ImportError as e:
+    Picamera2 = UnavailableModule(e)
+
 OPERATING_SYSTEM = platform.system()
 DIGITAL_ZOOM_MAX = 4
 NOISE = np.random.randint(0, 256, (480, 640, 3), dtype=np.uint8)  # in case a camera can't get a frame
@@ -42,6 +48,7 @@ class InputTypes:
     RTSP = "rtsp"
     REALSENSE = "realsense"
     BASLER = "basler"
+    RPI_CSI2 = "rpi_csi2"
     MOCK = "mock"
 
     def get_options() -> list:
@@ -248,6 +255,8 @@ def create_grabber(config: dict, autogenerate_name: bool = True, warmup_delay: f
             grabber = BaslerFrameGrabber(config)
         elif input_type == InputTypes.REALSENSE:
             grabber = RealSenseFrameGrabber(config)
+        elif input_type == InputTypes.RPI_CSI2:
+            grabber = RaspberryPiCSI2FrameGrabber(config)
         elif input_type == InputTypes.MOCK:
             grabber = MockFrameGrabber(config)
         else:
@@ -296,6 +305,7 @@ def autodiscover(warmup_delay: float = 1.0, rtsp_discover_mode: AutodiscoverMode
             InputTypes.GENERIC_USB,
             InputTypes.BASLER,
             InputTypes.RTSP,
+            InputTypes.RPI_CSI2,
         )
 
         # Autodiscover the grabbers
@@ -1035,6 +1045,47 @@ def _apply_camera_specific_options(self, options: dict) -> None:
             pass
 
 
+class RaspberryPiCSI2FrameGrabber(FrameGrabber):
+    "For CSI2 cameras connected to Raspberry Pis through their dedicated camera port"
+
+    def __init__(self, config: dict):
+        self.config = config
+
+        # This will also detect USB cameras, but according to the documentation CSI2
+        # cameras attached to the dedicated camera port will always come before USB
+        # cameras in the resulting list of camera dictionaries
+        # https://datasheets.raspberrypi.com/camera/picamera2-manual.pdf#page=66.21
+        cameras = Picamera2.global_camera_info()
+
+        if not cameras:
+            raise ValueError("No CSI2 cameras were found. Is your camera connected?")
+
+        # Since global_camera_info() also finds USB cameras, we will only use the first
+        # entry. USB cameras must be found through their specific FrameGrabber. Note
+        # that only a single CSI2 camera is supported at this time.
+        picam2 = Picamera2()
+        picam2.configure(picam2.create_still_configuration())
+        picam2.start()
+        logger.info(f"Connected to Raspberry Pi CSI2 camera with id {(cameras[0])['Id']}")
+
+        self.camera = picam2
+
+    def _grab_implementation(self) -> np.ndarray:
+        frame = self.camera.capture_array()
+
+        # Convert to BGR for opencv
+        frame = cv2.cvtColor(np.asanyarray(frame), cv2.COLOR_BGR2RGB)
+
+        return frame
+
+    def _apply_camera_specific_options(self, options: dict) -> None:
+        if options.get("resolution"):
+            raise ValueError("FrameGrab does not support setting resolution on Raspberry Pi CSI2 cameras.")
+
+    def release(self) -> None:
+        self.camera.close()
+
+
 class MockFrameGrabber(FrameGrabber):
     """A mock camera class for testing purposes"""