ST-GCN with demo

Runnable Demo(Attention!!!)

After config the conda environment, you can use the following commands to run the demo without the installation of openpose but the code of openpose

Before run the demo, you need to:

1.Move the demo_offline.py(top level folder) into the folder(./processor) to replace the previous demo_offline.py

2.Download the openpose pytorch repo into top level folder and replace the openpose.py with the openpose.py(top level folder)

3.Remove the demo_offline.py and openpose.py(top level folder).

The structure of the repo like this:

|-- configs
|-- feeder
|-- models
|-- net
|-- processor
    |-- demo_offline.py (have been replaced)
    |-- ...
|-- openpose(Download)
    |-- openpose.py     (have been replaced)
    |-- ...
|-- ...

# only with offline pose estimation
python main.py demo_offline [--video ${PATH_TO_VIDEO}]

Introduction

This repository holds the codebase, dataset and models for the paper:

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition Sijie Yan, Yuanjun Xiong and Dahua Lin, AAAI 2018. [Arxiv Preprint]

Visulization of ST-GCN in Action

Our demo for skeleton-based action recognition:

ST-GCN is able to exploit local pattern and correlation from human skeletons. Below figures show the neural response magnitude of each node in the last layer of our ST-GCN.

Touch head	Sitting down	Take off a shoe	Eat meal/snack	Kick other person
Hammer throw	Clean and jerk	Pull ups	Tai chi	Juggling ball

The first row of above results is from NTU-RGB+D dataset, and the second row is from Kinetics-skeleton.

Prerequisites

• Python3 (>3.5)
• PyTorch
• Openpose with Python API. (Optional: for demo only)
• Other Python libraries can be installed by pip install -r requirements.txt

Installation

git clone https://github.com/yysijie/st-gcn.git; cd st-gcn
cd torchlight; python setup.py install; cd ..

Get pretrained models

We provided the pretrained model weithts of our ST-GCN. The model weights can be downloaded by running the script

bash tools/get_models.sh

You can also obtain models from GoogleDrive or BaiduYun, and manually put them into ./models.

Optional arguments:

• PATH_TO_VIDEO: Filename of the input video.

Data Preparation

We experimented on two skeleton-based action recognition datasts: Kinetics-skeleton and NTU RGB+D. Before training and testing, for convenience of fast data loading, the datasets should be converted to proper file structure. You can download the pre-processed data from GoogleDrive and extract files with

cd st-gcn
unzip <path to st-gcn-processed-data.zip>

Otherwise, for processing raw data by yourself, please refer to below guidances.

Kinetics-skeleton

Kinetics is a video-based dataset for action recognition which only provide raw video clips without skeleton data. Kinetics dataset include To obatin the joint locations, we first resized all videos to the resolution of 340x256 and converted the frame rate to 30 fps. Then, we extracted skeletons from each frame in Kinetics by Openpose. The extracted skeleton data we called Kinetics-skeleton(7.5GB) can be directly downloaded from GoogleDrive or BaiduYun.

After uncompressing, rebuild the database by this command:

python tools/kinetics_gendata.py --data_path <path to kinetics-skeleton>

NTU RGB+D

NTU RGB+D can be downloaded from their website. Only the 3D skeletons(5.8GB) modality is required in our experiments. After that, this command should be used to build the database for training or evaluation:

python tools/ntu_gendata.py --data_path <path to nturgbd+d_skeletons>

where the <path to nturgbd+d_skeletons> points to the 3D skeletons modality of NTU RGB+D dataset you download.

Testing Pretrained Models

To evaluate ST-GCN model pretrained on Kinetcis-skeleton, run

python main.py recognition -c config/st_gcn/kinetics-skeleton/test.yaml

For cross-view evaluation in NTU RGB+D, run

python main.py recognition -c config/st_gcn/ntu-xview/test.yaml

For cross-subject evaluation in NTU RGB+D, run

python main.py recognition -c config/st_gcn/ntu-xsub/test.yaml

To speed up evaluation by multi-gpu inference or modify batch size for reducing the memory cost, set --test_batch_size and --device like:

python main.py recognition -c <config file> --test_batch_size <batch size> --device <gpu0> <gpu1> ...

Results

The expected Top-1 accuracy of provided models are shown here:

Model	Kinetics- skeleton (%)	NTU RGB+D Cross View (%)	NTU RGB+D Cross Subject (%)
Baseline[1]	20.3	83.1	74.3
ST-GCN (Ours)	31.6	88.8	81.6

[1] Kim, T. S., and Reiter, A. 2017. Interpretable 3d human action analysis with temporal convolutional networks. In BNMW CVPRW.

Training

To train a new ST-GCN model, run

python main.py recognition -c config/st_gcn/<dataset>/train.yaml [--work_dir <work folder>]

where the <dataset> must be ntu-xsub, ntu-xview or kinetics-skeleton, depending on the dataset you want to use. The training results, including model weights, configurations and logging files, will be saved under the ./work_dir by default or <work folder> if you appoint it.

You can modify the training parameters such as work_dir, batch_size, step, base_lr and device in the command line or configuration files. The order of priority is: command line > config file > default parameter. For more information, use main.py -h.

Finally, custom model evaluation can be achieved by this command as we mentioned above:

python main.py recognition -c config/st_gcn/<dataset>/test.yaml --weights <path to model weights>

Acknowledge

1.Thanks for Sijie Yan and Yuanjun Xiong and Dahua Lin for their work about st-gcn

@inproceedings{stgcn2018aaai,
  title     = {Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition},
  author    = {Sijie Yan and Yuanjun Xiong and Dahua Lin},
  booktitle = {AAAI},
  year      = {2018},
}

2.Thanks for Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh for their work about openpose(pytorch) code

@InProceedings{cao2017realtime,
  title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
  author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2017}
  }

Contact

For any question, feel free to contact

Gaoqi Li     : [email protected]