Skip to content

Latest commit

 

History

History
257 lines (234 loc) · 9.17 KB

README.md

File metadata and controls

257 lines (234 loc) · 9.17 KB

Installation

Requirements

  • Linux or macOS with Python ≥ 3.8
  • CUDA>=11.7, lower CUDA versions may result in not successfully built on detectron2
  • pip install -r requirements.txt

Example virtualenv environment setup for kMaX-DeepLab

pip3 install virtualenv
python3 -m virtualenv kmax_deeplab --python=python3
source kmax_deeplab/bin/activate

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2
pip install -r requirements.txt
unzip detectron2.zip
cd detectron2
pip install -e .
pip install git+https://github.com/cocodataset/panopticapi.git

Example test model checkpoint of kMaX-DeepLab

  1. Download the checkpoint.

  2. Dataset preperation and structure for evaluation as below. You need to download 'relabeled_COCO_val' and rename it to 'panoptic_val2017', and rename the corresponding json file and download the images from COCO dataset.

datasets
└── coco
    ├── annotations 
    │   └── panoptic_val2017.json # relabeled_coco_val.json
    ├── panoptic_val2017  # relabeled_coco_val
    ├── val2017 # original COCO dataset val set images
  1. Use the script below to evaluate the model.
export DETECTRON2_DATASETS=YOUR_DATA_PATH
python3 train_net.py --num-gpus 8 --dist-url tcp://127.0.0.1:9999 \
--config-file configs/coco/panoptic-segmentation/kmax_convnext_large.yaml \
--eval-only MODEL.WEIGHTS YOUR_MODEL_PATH
  1. The provided checkpoint should give a PQ of 64.4 on relabeled COCO-val.

Example demo

Use the following script to demo images in a directory.

python3 demo.py --config-file ../configs/coco/panoptic-segmentation/kmax_convnext_large.yaml     \
--input YOUR_IMG_FOLDER_PATH \
--output  OUTPUT_VIS_FOLDER  \
--opts MODEL.WEIGHTS  YOUR_MODEL_PATH

Distributed training

Need to set up your environment variables to run the training script below.

export DETECTRON2_DATASETS=YOUR_DATASET_PATH
python3 train_net.py --num-gpus 8 --num-machines $WORKER_NUM \
--machine-rank $WORKER_ID --dist-url tcp://$WORKER_0_HOST:$port \
--config-file configs/coco/panoptic-segmentation/kmax_convnext_large.yaml

Explore COCONut-Large

COCONut-Large consists of three subsets from COCO train2017, COCO unlabeled set and subsets from Objects365. To use the COCONut-Large panoptic masks, you should follow the steps below:

  1. Download the panoptic masks and annotation json file from huggingface
  2. Download the images from Objects365. The images are organized using patches, please download the corresponding raw patches: patch25,patch32,patch35,patch40 from the official website.
  3. Follow the instruction to set up COCONut-B, which is used to build COCONut-L. The folder organization should be as follow:
datasets
└── coco
   ├── annotations 
   │   └── panoptic_train2017.json # coconut-b.json
   ├── panoptic_train2017  # coconut-b
   ├── train2017 # original COCO dataset train and unlabeled set images
  1. Link the Objects365 images and panoptic masks to the coco/train_2017 and coco/panoptic_train2017 respectively using the dataset path of COCONut-B.
objects365/images ----> coco/train2017
object365/panoptic_masks ----> coco/panoptic_train2017
  1. Merge the object365 json files to COCONut-B json files using the 'merged.py' script. Then it is ready to be used.

Explore COCONut-val

  1. Similar to COCONut-Large, images need to be downloaded from Ojbects365, we provide a link to download the selected val set images. Link the image to COCO val2017.
  2. Download the panoptic masks from huggingface. Link the panoptic masks to COCO panoptci_val2017.
  3. Then merge the downloaded COCONut-val and relabeled COCO-val jsons using merged.py.

Model zoo

coco-val relabeled coco-val coconut-val checkpoint
backbone training set PQ AP_mask mIoU PQ AP_mask mIoU PQ AP_mask mIoU gdrive huggingface
ResNet 50 COCO 53.3 39.6 61.7 55.1 40.6 63.9 53.1 37.1 62.5 download download
COCONut-S 51.7 37.5 59.4 58.9 44.4 64.4 56.7 41.2 63.6 download download
COCONut-B 53.4 39.3 62.6 60.2 45.2 65.7 58.1 42.9 64.7 download download
coco-val relabeled coco-val coconut-val checkpoint
backbone training set PQ AP_mask mIoU PQ AP_mask mIoU PQ AP_mask mIoU gdrive huggingface
ConvNeXt-Large COCO 57.9 45.0 66.9 60.4 46.4 69.9 58.3 44.1 66.4 download download
COCONut-S 55.9 41.9 66.1 64.4 50.8 71.4 59.4 45.7 67.8 download download
COCONut-B 57.8 44.8 66.6 64.9 51.2 71.8 61.3 46.5 69.5 download download
COCONut-L 57.9 45.1 67.1 65.0 51.1 71.9 62.4 47.5 69.9 download download