nan return value after calling dual_conic.bounds() in realsense.py example script

[M4D-SC1ENTIST]

Hi. I am currently trying to test the realsense example script However, I noticed that when calling dual_conic.bounds() for data association in the script, the return value is always nan.

This causes the linear_sum_assignment() function from scipy unable to solve the optimization problem.

I tried to debug the issue, and I noticed that the problem might related to the input pose when calling gtsam_quadrics.QuadricCamera.project(quadric, pose, self.calibration). When the camera is not moving, the pose should be an identity matrix. When I replaced the pose with a manually defined value pasted from the simple_example script, the program work but cannot do any data association (of course).

I was wondering if the unchanging pose between frame is the cause of the problem? If so, how can we use the realsense.py script since when holding the camera, the movement should be continuous, and how can we achieve the result as you showed in the video? I was wondering if you can also share the source code from the paper so that we can reproduce the result?

Thanks! I`m looking forward to your reply.

[lachlan-nicholson]

At a quick glance, that dual conic doesn't seem to be well defined. The mathematics for projecting dual quadrics should be well defined even at edge cases, but perhaps there is something we have missed. Could you please let us know what the parameters of your quadric, camera pose and calibration were during this test?

What might be happening, is that the naïve data association is attempting to look at a quadric outside the cameras current view. Typically we expect this to result in an open conic that stretches across the screen and a large bounding box that exists mostly outside the screen, but the math should hold up.