GUI App for Automatic Extrinsic Calibration of Range and Visual Sensors (Karnik Ram R) #2
Comments
|
Student: @karnikram |
EduFdez
changed the title
|
Hi @EduFdez, @jlblancoc, and @jolting! During this bonding period I want to work on the GUI skeleton and get it ready. I'll also be spending time to get more familiar with the mrpt libraries and the coding style guides. Is there anything else that you guys want me to get started on? I also wanted to know about when and how the existing calibration codebases will be made available, so that I can start reading them. Thank you again! |
|
Hi @karnikram , I have been looking at my previous code that I wrote for the experiments that are presented in some of the papers related to this project. This code is divided in different projects, some in Matlab and some in C++, and in some projects it's not very well structured. So I will save you some pain by putting the main resources that you will need into the project. I forked the project from your git, I will add some structure and files and I'll do my pull request to your fork, giving you some indications on where to start, and where to download first datasets to work with. I will come to you soon :-) Best regards, |
|
Thanks @EduFdez, looking forward to it! |
|
Hi @karnikram ! You can see that I added some dependencies:
You can download and compile these libraries to start using them in this project. Then, you can have a look at my unfinished MRPT application for calibrating 2D LiDARs, if look down to the main function you can have an idea of the algorithm. I'll try to split this long file into the classes of autocalib-sensor-extrinsics so that we get soon a basic example to build on. |
|
Looks neat! I shall get started with it. |
|
Hi @EduFdez, I've built and installed the latest mrpt libraries (v-1.9.9) from source but I'm not able to build our project since it includes mrpt-base which I believe is not included in mrpt 1.9.9. Shall I go back to the maintenance branch (v-1.5) or shall I just not include mrpt-base? I've also been looking into your MRPT application for calibrating 2D LiDARs, it looks interesting, but calibrating 2D LiDARs wasn't actually included in my proposal! We'd only included calibrating 3D LiDARs, RGB-D cameras, RGB cameras, and any combination between them. And out of them I'd wanted to implement Extrinsic calibration of a set of range cameras in 5 seconds without pattern first. So what do you suggest we do here? :) And also just wanted to update that I haven't been able to access my proposal or any of the other students' proposals on the issue tickets. I guess the links are broken now. |
|
Hi @karnikram !
It's better to move forward to the new mrpt master branch (1.9.9). Please, remove the reference to mrpt-base to the list of libraries that the code really needs. Typically you can find it out by checking the namespaces used by your code. Make sure of reading the guide on porting old code to mrpt-2: since you'll have to apply it. Let us know if you find further incidences. |
|
Hi @jlblancoc! I shall move forward with the latest branch. Until now there has been only one reference to mrpt-base, I shall read the documents you linked for any future conflicts. One more minor issue, earlier when I ran cmake while building mrpt I got the following warning even though I have vtk6, libvtk6-dev, and python-vtk6 properly installed. Is this expected? |
|
Yes, that vtk error is "well known"... it can be fixed installing
additional packages, but it's harmless anyway so you can safely ignore it.
|
|
Hi @karnikram !
I forgot that 2D LiDAR calibration was finally not included in the proposal. Maybe it could be addressed at the end of this GSoC if there is time for it. I agree about starting with the calibration of 3D LiDAR / range images. You can have a look at the method PbMapMaker::detectPlanesCloud which contains a wrapper of PCL to extract planes by region growing. Have a look also to the class CFeatureLines that I was using to segment lines from images. I have just done a pull request to include this class in MRPT. I just set you through "WeTransfer" a rawlog of an RGB-D video from 2 sensors to start working on 3D range calibration (Extrinsic calibration of a set of range cameras in 5 seconds without pattern) and RGB to Depth calibration. The first thing that you could try to do is to extract the 3D planes in this dataset, and then perform data association and display it.
The links to the student proposals work for me, can you check again and tell me if there is any broken link? Please. |
|
Hi @EduFdez! Thanks for the code files and the dataset, they are interesting.
Sounds good!
I'm still not able to access any of the proposals linked here. I think only the mentors have access to them! |
You were right. I have updated the link of your proposal, and we'll update those of other students too. |
|
@EduFdez, @jlblancoc But I'm having difficulty with one part. After loading the input .rawlog file, I believe the user should be able to inspect the observations and visualize the images, point clouds within the app just like how RawLogViewer does it. Could you give me some pointers about how to go about bringing RawLogViewer's functionality (tree-view of observations, visual preview) into our app? Or is this even the right thing to do? |
|
What I'm doing in all my custom apps (outside of MRPT, using MRPT
libs) is to create a CDisplayWindow3D with a COpenGLScene in which all
objects (observations, lines, etc.) are rendered using mrpt::opengl
objects.
Using overlay text messages (see the methods in CDisplayWindow3D) to
show key-based menus, the current state of the app, results, etc. it's
a very convenient way of designing a "GUI" pretty fast and without
having to work with wxWidgets or Qt.
Alternatively, depending on how much time you want to spend on the
GUI, you can create a Qt app (see robot-maps-gui
https://github.com/MRPT/mrpt/tree/master/apps/robot-map-gui as a
pretty good example), but it might eat your time that should probably
better spent in the algorithms themselves...
Cheers,
|
|
Hi @jlblancoc, |
|
Hi @jlblancoc,
I've been reading the robot-maps-gui, it's very well designed! Since I already have a start with Qt, I'll continue with it with this as an example. I'm looking to complete the observation tree and the viewer before the coding period starts.
Great! Recall to commit often, explaining what's your progress.
Relatively small commits are normally preferred to huge, bulk
one-app-out-of-the-blue commits ;-)
In which repo are you working right now?
|
|
Sure :) |
|
Hi, I'm having difficulty in de-serializing the observation objects from the rawlog file. When I run, CFileGZInputStream rawlog(file_name);
CSerializable::Ptr obj;
archiveFrom(rawlog) >> obj;I get the following exception Aren't all the mrpt classes already registered automatically? mrpt::rtti::getAllRegisteredClasses()I get no output. Am I missing something? |
|
Hi, |
|
Hi @EduFdez! In the example that you've mentioned the rawlog is being loaded in one step. But for our use case I'm actually trying to load the observations one by one and insert them into a tree structure, so that they can be presented easily as a tree view. I was following this example for loading but it doesn't seem to work.. |
|
Your code to deserialize seems correct. Make sure of adding to cmake's FIND_PACKAGE(MRPT ...), the list of all libraries whose objects you are deserializing, e.g. slam, obs, maps,... Also: are you compiling mrpt from sources, right? Make sure of not building as static libraries |
|
Hi @jlblancoc! I've included most of the modules in my cmake script I have also built mrpt from source. I have pushed the code to my fork |
|
Hi, I've built your code. First thing first: it builds cleanly, which is a really good thing to start! The list of mrpt libs is correctly set via cmake and it makes its way to the linker: But the executable does NOT actually depend (so it neither load) any of the "important" mrpt libs, the ones in which the deserialized objects live: |
|
I can't find an obvious reason for this, apart of the usual optimizations of the linker. For now, just adding one variable like this: mrpt::obs::CObservation3DRangeScan o; Anywhere (e.g. at the global scope of the GUI app) just fixes the problem. Give it a try, run ldd again (it worked for me and should for you as well) and try the serialization again. I think it will work now. |
|
Hi @jlblancoc, Thank you so much, it works now! |
|
Update:
Hope it's okay. The changes are in my fork. I'm now re-reading your paper on fast extrinsic calibration of rgbd sensors using planes, and PbMap to implement plane extraction and association. |
|
Hi, I'm glad that you solved the compilation problems and the GUI is in a good track. I am taking some holidays now and I cannot get internet all the time, but I am online at least once every 2 days. |
|
Hi @EduFdez @jlblancoc Shall I do this conversion by myself or is there any other existing method that I can use? |
|
@karnikram |
|
Not yet @EduFdez, compiling the latest version of PCL is taking forever and it is also exceeding my 4GB physical memory. Even while compiling with just |
|
OK, I will test it and I will tell you if that solves the problem, otherwise I think we should better go back to PCL 1.7 since this kind of error is very difficult to track. |
|
I compiled PCL 1.8 from sources, but I am not able to use it in "autocalib-sensor-extrinsics" since CMake complains about Eigen, and I do not know how to solve this properly. The CMake msg is:
@jlblancoc do you have any idea on how to solve this? |
|
@EduFdez, are you sure Eigen is installed properly? And where is it placed? |
|
@karnikram yes, it is installed but it is not found correctly by PCLConfig.cmake and I'm not sure on how to modify this file. Probably the best thing to do is to come back to the version 1.7 to avoid losing time on this. |
|
Hi @karnikram , I fixed some small errors in order to compile the PCL version 1.7.2. (I sent you an email with the patch to fix these errors). I compiled your project against this version of PCL, but unluckily the Segmentation Fault is not solved. In the meanwhile, you may comment out the code for the plane segmentation and start working on line segmentation, and also the adaptation of the solver from the code I gave you. |
|
Hi @EduFdez, I've already raised an issue on the PCL issues page. But now since it's not working for you on 1.7.2 either (on 18.04) I think the problem is with the eigen libraries (version 3.3.4-4) in 18.04. Because it worked for me (no segmentation faults) with PCL 1.7.2 on 16.04 that has Eigen version 3.3~beta. Some people had a similar issue like here and they solved it by using an older version of Eigen. I wanted to try this but I do not know how to make PCL 1.8 binaries to use an older version of Eigen without compiling pcl by myself (which is not possible on my system due to memory constraints). Let me know what you think. |
|
You both can try just downloading a specific version of Eigen in some
directory (NO need to "make install" them, they're just headers), then
setting EIGEN3_DIR to the corresponding directory in CMake of the
project you want to use a specific version of Eigen.
I've done it in the past without problems.
I think there **might not** be problems in using the system-provided
PCL with a custom version of Eigen.
|
|
Thanks @jlblancoc, |
|
Is there any way to check which version of Eigen is being used by the pcl binaries?
Yes: going to the ubuntu package website and see its dependencies for
your distro.
The actual original info is under the file "debian/control" of the pcl
package, which you can also download via: "dget
http://xxxx/pcl-xxxx.dsc"
Also, the Segmentation Fault could be caused by using package binaries that have been compiled with different c++ standards, as explained here or here.
That also makes sense... If you try building your test case (the one
reported to PCL's repo) without the set(CMAKE_CXX_STANDARD 17), with
different versions of C++, let's see if the crash happens.
Then it may be caused by a different ABI in the gnu c++ lib. If that's
the problem, we'll find a solution, but please first confirm if that's
the problem...
|
|
I tested it by compiling PCL1.8 (master branch) with the -std=c++11 flag and it worked out. I just added the following line to PCL's CMakelists.txt: @karnikram you will need then to compile it like this, because the option of coming back to ubuntu16 seems a much worse choice. What memory constraints do you have? |
|
I installed Eigen 3.2.10 headers in /usr/local/include and my example segmentation program listed here works now (with PCL 1.8 binaries). The program is able to use the 3.2.10 headers. But our project is still crashing. gdb backtrace suggests it's still using the headers in /usr/include/eigen3 (version 3.3.4-4) I am not pointing to any Eigen3 directory in either of the CMake scripts. |
|
@karnikram the calibration project does not crash anymore if you compile PCL with -std=c++11 as I said in my previous post. |
|
But unfortunately I'm not able to compile PCL 1.8 on my system :/ I got another solution, that was posted here just now. But removing Contents of |
|
I think that the main problem that causes the SegFault is that the binaries of PCL (and its dependencies) are compiled with a c++ standard previous to c++11. Compiling PCL with c++11 seems to solve the problem. But another option may be to remove SET(CMAKE_CXX_STANDARD 17) from your project. |
|
However, the new mrpt2 requires c++17... :-(
There are options, anyway (*always* there is a solution ;-):
* Move all the invokations to pcl to one (or a small set) or .cpp files.
* Manually instruct CMake to add the c++11 gcc flag to those files
Et voilà!
|
|
Are you using 16.04 or 18.04? I think 18.04 should work. I backported 1.8 to 16.04. Maybe that will work. https://launchpad.net/~jolting/+archive/ubuntu/backport-mrpt FYI, the Ubuntu build farm is free for open source. I've never seen PCL fail to build on their farm. https://opensourcehacker.com/2013/03/20/how-to-backport-packages-on-ubuntu-linux/ |
|
@EduFdez, @jlblancoc, @jolting I rebuilt mrpt in Debug mode, with debugging symbols turned on, and now somehow everything's working fine. Earlier, removing Now after rebuilding mrpt in Debug mode, this SegFault too doesn't occur anymore, and I can run everything properly. (I'm using PCL 1.8 binaries on 18.04) |
|
That's great! Now you can continue working on the project, and we may try later to solve the problem to build in Release mode. |
|
Hi @karnikram |
|
I believe mrpt::system::createThreadFromObjectMethod and threads.h have been removed from master. |
|
Yes, please use std::thread, that's much better since it's standard ;-)
|
|
Hi @karnikram
You almost have it @karnikram , just a final push is needed! |
|
Hi @EduFdez, Calibration by plane-matching will be implemented fully and tested on the datasets by this week. |
|
Calibration from ego-motion is one of the most important requirements of the project. You should try to speed up in the calibration from planes/lines and try to finish it this week in order to address the ego-motion method next week. |
|
Update: Plane extraction and matching for the first algorithm, line extraction and matching for the second algorithm are done along with their visualizations. I've now started testing them and am integrating the rotation and translation solvers for both. I hope to finish it in a couple of days, with @EduFdez 's help! After that I will start to gather the documentation and write unit tests for the algorithms and fix some gui issues. |
|
Cool! I will start testing them this evening. And I will include the translation solver and global solver functions for plane-based calibration, then you can adapt it to line-based calibration. Regarding the creation of unit-tests you need to follow the guidelines of MRPT. What I normally do is to create unit tests whenever I find I recurrent bug or failure that is likely to happen again upon some code modifications. So, my advice is that you take the 2 datasets that I gave you and you make a good downsampling of them with rawlog-edit to extract 6-10 observations (separated away along the original dataset) into a new dataset. You will use these 2 new datasets for the tests, so that you can load them, do the different calibrations, and tests that the results are around some given values (for different initializations). |
|
Final Work Product Link - MRPT/autocalib-sensor-extrinsics#1 Will continue to fix other issues now. |
Initial description
Range and visual sensors are being increasingly used alongside one another in mobile robots. With their increasing use, calibration techniques that can accurately estimate the 6-DoF transformation between them are becoming increasingly important. In this regard, an end-to-end application with an easy-to-use graphical user interface for the extrinsic calibration of different types of sensors is proposed. The app will be able to calibrate the extrinsics of 3D LiDARs, RGB-D cameras, RGB cameras, and any combination between them. Automatic and target-less calibration algorithms based on line matching, plane matching, and trajectory matching will be implemented and integrated into the app. The user will be able to directly visualize the calibration results inside the app and also compare different algorithms wherever possible, and significantly reduce calibration efforts.
See the GSoC project proposal
Progress: All comments below reflect the interactions between the GSoC student and the mentors.
The text was updated successfully, but these errors were encountered: