RoboKeyGen

This is an official code for ICRA 2024 paper "RoboKeyGen: Robot Pose and Joint Angles Estimation via Diffusion-based 3D Keypoint Generation".

[PDF] [Video] [Webpage]

Code Release Schedule

• Installation
• Dataset Release and Pretrained Model Release
• Training Code Release
• Inference Code Release

Installation

Dataset and Pretrained Model Checkpoints Release

Synthetic Training Dataset SimRGBD-Franka

We have created a huge synthetic training dataset SimRGBD-Franka (~hundreds of GB). For downloading conveniently, we divide the whole dataset into six parts: color.zip, mask.zip, ir.zip, depth_60.zip, simDepthImage.zip, and meta.zip. We could download the dataset from the webpage website You can download any kind of data as you want.

Diffusion Model Training Dataset

Our diffusion model predicts 3D keypoints conditioned on 2D keypoints, without using any RGB, Depth information. Therefore, we have created a hugh training dataset only containing meta information to train the diffusion model. We can download the dataset from the link with password irGq.

Real-world Testing Dataset RealSense-Franka and AzureKinect-Franka

To download the real-world testing dataset, RealSense-Franka and AzureKinect-Franka. We can download the file Real_Test_0613.tar.gz from the webpage website. After extracting this file, we can find 12 folders containing 12 different scenes. We have selected 9 out of 12 scenes as the final testing scenes according to the quality of annotation. The folders 1_D415_front_0, 2_D415_front_1, 6_D415_left_1, 12_D415_right_1 constitutes RealSense-Franka, and the folders 3_kinect_front_0, 4_kinect_front_1, 8_kinect_left_1, 9_kinect_left_0, 10_kinect_right_1 constitutes AzureKinect-Franka.

Pretrained Model Checkpoints

We can download the pretrained checkpoints from the link with password b2MV.