Rectified images and annotation

[Shashvatb]

Hi @SeaOtocinclus ,

Is there a way to easily obtain rectified images along with the annotations (bbox, camera calib, 2D keypoints) which is inbuilt?

thanks for all the help!

[SeaOtocinclus]

Hi @Shashvatb ,

Context

• As GT information has been computed for the FishEye camera model, projecting them directly on Linear rectilinear camera would not directly make sense (i.e a amodal Bounding Box for a FishEye image does not map to a Bounding Box on a Linear image.)

Recommendation

• We would advise you to directly project 3D models on the Linear Image/Camera and compute their amodal bounding box after projection.

Here some tips on how to retrieve image/camera and manage world to camera projection

Image rectification & Linear Camera model

• Rectification of the images are easy with our provided API thanks to using get_undistorted_image in AriaDataProvider or QuestDataProvider classes.
  https://github.com/facebookresearch/hot3d/blob/main/hot3d/data_loaders/AriaDataProvider.py#L118C9-L118C30
• Obtaining the corresponding camera calibration for it can be done using the following:

Retrieving Camera Calibration	Code
Aria	[T_device_camera, pinhole_camera_online_calibration] = (self.get_online_camera_calibration(stream_id, timestamp_ns=timestamp_ns, camera_model=LINEAR))
Quest	[T_device_camera, pinhole_camera_calibration] = self.get_camera_calibration(stream_id, camera_model=LINEAR)

World to Camera projection

You can then use the camera calibration to project a world 3D point as following (see #14):

• pinhole_camera_calibration.project( (headset_pose3d.T_world_device @ T_device_camera).inverse() @ X)

Loading 3D models

You can use trimesh to load the GT 3D models:

# Load the scene.
scene = trimesh.load_mesh(
    path,
    process=True,
    merge_primitives=True,
    skip_materials=True,
)

# Represent the scene by a single mesh.
mesh = scene.dump(concatenate=True)

• You can then retrieve the vertices, move them to the right location given the pose of the 3D object at the given timestamp and project it to the image of your choice.

Comment about visibility

• Visibility score should remain unchanged (we don't change how much of the object is seen) but do notice that the linear projection is cropping some image content and so limiting what is visible.

Comment on speed

• You could subsample the mesh points to perform the world to camera projection

Final comments

• We are planning soon to release a code example showing how to perform 3D model reprojection for Linear and FishEye images with pyrender and trimesh. It would enable you to perform a similar task, but using hardware rasterization rather than software projection.

[Shashvatb]

Thanks a lot for this information! Will use it accordingly!