Batched inference, tensorRT

[Kishore-Yogaraj]

Need to use TensorRT, something like https://github.com/noahmr/yolov5-tensorrt for yolov8
➝ https://github.com/triple-Mu/YOLOv8-TensorRT/blob/main/infer-det.py

will be in C++

also look at https://github.com/NVIDIA-AI-IOT/YOLOv5-with-Isaac-ROS

[justin] read 2.1. Batching section here https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-700/pdf/TensorRT-Best-Practices.pdf
Approach: There are three subtasks for this:

• Create a node to synchronize multiple camera messages to publish a batched_camera_message
• Research how to convert yolov8 to tensorrt and modify the model output to support batched inference (see https://medium.com/@DeeperAndCheaper/yolov8-batch-inference-implementation-using-tensorrt-2-converting-to-batch-model-engine-e02dc203fc8b )
• Create a node that subscribes to the batched_camera_message and runs inference on the batch of images

[Arfan12630]

Buffer for getting messages from all 6 camera angles
https://wiki.ros.org/message_filters#registerCallback.28.29

[Kishore-Yogaraj]

• Preprocessing batching 6 images from 6 different camera angles (using nuscenes) dataset has been completed
• The current YOLOV8 pytorch model was converted to ONNX format
• TensorRT engine has been built
• Bounding box logic is currently being worked on
• Error with allocating GPU for use with TensorRT engine
• Memory needs to be allocated to buffers for TensorRT inference

[Kishore-Yogaraj]

• Allocated GPU for use with TensorRT engine
• Had an error with inferencing due to a change in Cuda version
• Fixed error by adjusting to new cuda version and rebuilding enginer
• Working on bounding box visualization

[Arfan12630]

• Bounding boxes, class IDs, and confidence scores have been stored in a structured results dictionary.
• Detection results have been organized into a detection2D array for further processing.
• Currently working on visualizing the detections in Foxglove

[Arfan12630]

• Getting detections on foxglove
• Working on refactoring and ensuring there are no incorrect bounding boxes and overlap
• Working on visualizing all batched images

[Arfan12630]

• Getting detections for all camera angles
• Nuscenes batch inferencing complete
• Waiting on eve rosbags for testing
• Waiting on approval for PR