Real-time 3D human pose and motion reconstruction from monocular RGB videos

With this repository you have access to all the Unity tools presented in the demo.

📄 Publication DOI »

🎬 Video Demo · 📘 Thesis · 🥇 Graphics Lab · 🤺 OpenPose · 🕹️ OpenPose Unity Plugin

Abstract

Real-time three-dimensional (3D) pose estimation is of high interest in interactive applications, virtual reality, activity recognition, and most importantly, in the growing gaming industry. In this work, we present a method that captures and reconstructs the 3D skeletal pose and motion articulation of multiple characters using a monocular RGB camera. Our method deals with this challenging, but useful, task by taking advantage of the recent development in deep learning that allows two-dimensional (2D) pose estimation of multiple characters and the increasing availability of motion capture data. We fit 2D estimated poses, extracted from a single camera via OpenPose, with a 2D multiview joint projections database that is associated with their 3D motion representations. We then retrieve the 3D body pose of the tracked character, ensuring throughout that the reconstructed movements are natural, satisfy the model constraints, are within a feasible set, and are temporally smooth without jitters. We demonstrate the performance of our method in several examples, including human locomotion, simultaneously capturing of multiple characters, and motion reconstruction from different camera views.

Getting Started

This project runs on Unity Engine 2020.3.24f1 LTS, on Windows Operating System.

To run the Real-Time feature, you will need:

• to have a webcam connected to your pc.
• to follow steps 2,3 from Installation below, to complete the OpenPose setup.

Otherwise, you can explore the offline features.

Installation

1. Clone the repo

   git clone https://github.com/tasyiann/2Dto3DMotion

2. (For Real-Time usage) Install trained models for OpenPose, by running the script:

   Assets/StreamingAssets/getModels.bat
   

3. (For Real-Time usage) Install OpenPose plugins, by running the script:

   Assets/OpenPose/getPlugins.bat
   

Usage

Always, start with the 0-MAIN_SCENE_LOAD DATABASE scene, to automatically load the dataset.

Then, proceed either with Offline, or Real-Time.

In the Scenes, try interacting using w, a, s & d keys.

Offline

• Run the 0-MAIN_SCENE_LOAD DATABASE scene.
• Click on Input and select the example scenario from Scenarios\example.
• Click on GO OFFLINE.
• Navigate through the different views, using the UI.

Real-Time

• Run the 0-MAIN_SCENE_LOAD DATABASE scene.
• Click on GO REAL-TIME
• Make sure you have completed the OpenPose setup from the Installation steps.
• A webcam is required.

License

Distributed under the GNU Affero General Public License v3.0 License. See LICENSE.txt for more information.

Acknowledgments

• OpenPose
• OpenPose Unity Plugin
• Winterdust