Implement RTMDet to Perception Pipeline #7235
Comments
StepTurtle
added
the
component:perception
|
Here is the results of pre-trained models shared in this link from mmdetection. Results:
For now, I tested the pre-trained models shared by mmdetection using mmdetection tools. Also they provide a couple of tools to convert Right now I am trying to handle how can use TensorRT models in |
|
I deploy TensorRT engine to Python and I get some consistent results.
Warning In some parts of the video, you may see incorrect class names. For example, you might see both truck and car class names assigned to a vehicle. This is because I didn't run NMS when I deployed it in Python. I plan to fix this when I deploy it in C++. So, my next plan is doing same things in C++. Also, the detection times looks a little bit more. I did not understand the reason right now but I am working on it. |
xmfcx
changed the title
|
I am sharing the results from TensorRT deployment to C++, but currently, there are some issues with the results. I cannot see exactly the same results as with Python deployment. When I check the bounding box and score results, everything appears to be the same as in Python. However, when I check the labels, the results are very different. I am currently trying to resolve this issue.
You can find the scripts I used for deployment at these links: |
|
Fatih suggest to:
|
|
I just started to deploy RTMDet to ROS 2. Here is the repository: https://github.com/leo-drive/tensorrt_rtmdet/
I plan to complete the porting to ROS2 by performing the following steps:
|
|
✅ CUDA preprocessing is OK Preprocess Time Comparison per Image:
✅ NMS is added When I checked the model, I saw that the existing NMS algorithm within the model was already working. Despite NMS working, the reason we saw overlapping boxes was because the detected objects belonged to different classes. To prevent this, I added a simple NMS algorithm that selects the one with the higher score for overlapping detections.
NMS Time per Image
✅ Batch sizes which greater than
✅ Hard Coded Parts, Launch and Config files
🟨 Plugin Loading Autoware has the same logic that I used to load the TensorRT plugin. for (const auto & plugin_path : plugin_paths) {
int32_t flags{RTLD_LAZY};
void * handle = dlopen(plugin_path.c_str(), flags); // Plugin path is '/path/to/plugin.so'
if (!handle) {
logger_.log(nvinfer1::ILogger::Severity::kERROR, "Could not load plugin library");
}
}After compilation, a file with the extension '.so' is created. This file stored in Is there any information about can we handle this in Cmake. If we cannot, how can I provide the path to the file located inside the 'build' folder? I was able to load the plugin using the file paths below:
🟨 There are three precision option (
🟨 Post process and Visualization In current implementation visualization and postprocess parts works on CPU. I can't figure out how I can do these on the GPU.
Following table shows the total time for
❓Message Type for Outputs YOLOX semantic segmentation uses following message types:
There is no message defination for instance segmentation, so my plan is creating a new message type which combines the current detection message and new instance segmentation information. Should I create under autoware_msgs or tier4_autoware_msgs 📝 An RTX 3090 GPU was used for the time calculations and benchmarking |
|
When I ran 8 separate RTMDet nodes, I obtained the results in the table below, numbers represents the process times per image:
Computer Specifications
|
|
Following tables shows the process times for Computer Specifications
RTMDetSingle Camera
Multiple Camera
YOLOXSingle Camera
Multiple Camera
|
|
Discussion on new message type for instance segmentation results: https://github.com/orgs/autowarefoundation/discussions/5047 |
|
Last updates: https://www.youtube.com/watch?v=N8qrGAxzSJM |
|
Some options to use RTMDet results on camera-lidar pipeline: 1) Only run RTMDet and use roi outputs with current pipeline. Since the bounding box results from RTMDet is same with YOLOX, you can directly use RTMDet with current camera-lidar pipeline. 2) Fuse instance segmentation mask with clusters from euclidean clustering and assign clusters to label. The outputs of euclidean clustering are only point clouds, and it is It may perform better with objects that are close to each other and overlapping. I haven't tested it yet. code: https://github.com/StepTurtle/autoware.universe/tree/feat/mask_cluster_fusion 3) Fuse the point cloud with instance segmentation masks and take the points within the mask as object. code: https://github.com/StepTurtle/autoware.universe/tree/feat/mask_pointcloud_fusion 4) Fuse the point cloud with the instance segmentation mask. Filter out the points that do not correspond to objects within the mask and create a filtered point cloud. Couldn't find a useful place where we can use the filtered point cloud. PR: #8167 |
|
@StepTurtle Hi, first of all, thank you for your contribution to Autoware 🙏 However, I am not sure whether it would be better to merge this into Autoware at this moment (sorry for bringing this up after all of the reviews 🙏 ) I have several questions to ask:
Let me know your thoughts. |
It also has competitive speed. 
Why project boxes when you can project instance segmentation masks? ❌ Poor bounding box to point cloud performance
final-yolox.mp4ROI cluster fusion code has very complicated rules: ❌ Reduced performance with
|
I think this is a very backwards way of thinking. The universe is supposed to be for community contributions. A lot of effort is already been put into this integration. And the author was very cooperative during the process. The benefits are very clear and I don't see the reason for refusing to accept a feature like this. It doesn't even affect the existing repositories. I am very confused about your proposal. |
This is too late to suggest. All work has been done and it has to be merged to the main now. I am sorry for not giving any other options but these comments had to be made earlier. |
|
@xmfcx @armaganarsln @StepTurtle First, I would like to apologize if I have been disrespectful in any way. My primary concern was to understand "specifically which use cases posed challenges." However, now it make sense to me with the Fatih's comment here. Let me provide this feedback to our team too 🙏 |
|
@kminoda san, thank you and I am sorry for misunderstanding you on your approach. I thought you don't want it to be used or be in the main. Anyway there is still a lot work to be done for it to be useful so let's focus on that right now and see if there will be an improvement in the end in the overall false detections. |
|
Hey @kminoda , could you update me on your latest decision? Are the reviews ongoing, or are there other concerns you're currently considering? It would be helpful to know the latest status so I can continue working. |
|
Closing the discussion and PRs as |
|
@xmfcx @mitsudome-r could you please check why there are no reviews on this? |
|
@StepTurtle (cc: @mitsudome-r @xmfcx ) I am sorry I overlooked the notification and also failed to assign reviewers properly. We have been obsessed with some internal development items recently. Please kindly open the PR again if you are willing to merge. If so, let us assign reviewers again. Thank you for your understanding and patience. |
|
@armaganarsln @StepTurtle We've discussed this in Sensing & Perception WG today, and as @kminoda -san mentioned, the reviewers were busy for the past few months with some projects and wasn't able to spend enough time on it. That being said, I will raise this in TSC and SPC because we definitely need improvements in our review pipeline to speed up the process. It doesn't really make sense to have TIER IV "blocking" the PRs for packages that is not owned by TIER IV. We have set the rule that if a PR doesn't have reaction for over a week, either @xmfcx and @mitsudome-r will pick them up, but we probably need more reviewers to support us. |
|
Thank you both for your replies, @kminoda and @mitsudome-r. I believe this is an important topic that we should discuss further at both TSC and SPC. I have a straightforward and practical solution to address this issue and similar problems, but it will require being open and making changes. My proposed solution is simple: tasks should be created, assigned, and organized by reviewers/decision-makers, and no one should be able to create large tasks for AV Software (besides bugfix etc.), such as this one, and then ask for a review. From my observation, similar issues created by others in the past have often ended in failure (including AWSIM Labs). If managers are willing to focus on creating and organizing tasks rather than directly developing them or saying no to other proposals, we could eliminate this inefficiency. Specifically, my expectation would be for @kminoda-san not to focus on developing but rather on creating tasks, issues, and reviewing or assigning people to review them. If this approach is not adopted, I fear that meaningful and impactful contributions to AV software will remain limited to TIER IV. In the past, I had hoped that @xmfcx and @mitsudome-r could take on this role, and this was the idea behind Autoware Labs—fixing inefficiencies through direct task management—but unfortunately, it didn’t work out because in the end the reviewer is someone else. To summarize, my proposal is to encourage decision-makers to take a more active role in managing and determining what they want to be developed in Autoware not their own repository. If individuals like @kminoda, @TakaHoribe, and other managers in perception, planning, etc., are willing to create and review issues, I believe we can achieve success. |
|
Hey @kminoda -san, thanks for your reply. These are the PR's I closed;
Also, we have a branch for Also we created a separate repository for a custom TensorRT plugin, and we plan to add it to autoware.repos; If we agree to continue on this task, I will reopen the PRs and create a new PR for 'mask_cluster_fusion'. However, if we still decide that this is not beneficial for users, I will not reopen them. I think Leo Drive team and Fatih want to add instance segmentation. |
|
@StepTurtle Hello, thank you for the summary, I understood the situation. Let us know once you reopen the PR 🙏 |
|
@kminoda OK. Old ones reopened, i will create for PR for |
|
@StepTurtle |
|
I had a discussion with @StepTurtle on this to clarify who would be working on each task explained here. There are currently two one PRs:
We need the following action items to be done:
|
|
@mitsudome-r, I update the items from the TODO list in your message and let me know if there's anything else. |
|
@mitsudome-r Reviews seem to be nearly complete. Did you upload the artifacts? |
|
@StepTurtle Thanks for the reminder. I have uploaded the artifacts here: https://autoware-files.s3.us-west-2.amazonaws.com/models/tensorrt_rtmdet_onnx_models.tar.gz |
|
@mitsudome-r I have following error when I tried to download the data: > wget -P /tmp/ https://autoware-files.s3.us-west-2.amazonaws.com/models/tensorrt_rtmdet_onnx_models.tar.gz
--2024-12-13 11:56:35-- https://autoware-files.s3.us-west-2.amazonaws.com/models/tensorrt_rtmdet_onnx_models.tar.gz
Resolving autoware-files.s3.us-west-2.amazonaws.com (autoware-files.s3.us-west-2.amazonaws.com)... 52.92.194.34, 52.92.229.170, 52.218.216.57, ...
Connecting to autoware-files.s3.us-west-2.amazonaws.com (autoware-files.s3.us-west-2.amazonaws.com)|52.92.194.34|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-12-13 11:56:36 ERROR 403: Forbidden. |
|
@mitsudome-r ping |
|
It was missing a public read permission, fixed it @mitsudome-r @StepTurtle the link should work for everyone now. |
|
@xmfcx Thanks! |
Updatescc. @mitsudome-r @StepTurtle @amadeuszsz @YoshiRi @kminoda Merged
Remaining
|
Checklist
Description
We plan to add the RTMDet model in addition to the existing YOLOX model in Autoware Universe. While the YOLOX model is successful in the bounding box task, its instance segmentation layer is weak. We aim to improve instance segmentation results by adding RTMDet model.
Purpose
Our goal is to enhance the lidar image fusion pipeline by adding the RTMDet model to Autoware for image segmentation.
Possible approaches
We can convert pre-trained PyTorch models to ONNX and TensorRT formats, and we can create a ROS 2 package to handle the TensorRT models in Autoware Universe perception to implement them in Autoware.
Definition of done
The text was updated successfully, but these errors were encountered: