concat dataset train,val and test problem（连接数据集的训练验证和测试问题）

[Evil1m]

I got 2 dataset
Dataset_A have 15 classes with COCO format
Dataset_B have 20 classes with VOC format

I followed the format in https://mmdetection.readthedocs.io/en/latest/tutorials/customize_dataset.html#concatenate-dataset note part to create a concated dataset and the config like this:

img_norm_cfg = dict(
          mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
      train_pipeline = [
          dict(type='LoadImageFromFile'),
          dict(type='LoadAnnotations', with_bbox=True),
          dict(type='Resize', img_scale=(1024, 1024), keep_ratio=True),
          dict(type='RandomFlip', flip_ratio=0.5),
          dict(type='Normalize', **img_norm_cfg),
          dict(type='Pad', size_divisor=32),
          dict(type='DefaultFormatBundle'),
          dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
      ]
      test_pipeline = [
          dict(type='LoadImageFromFile'),
          dict(
              type='MultiScaleFlipAug',
              img_scale=(1024, 1024),
              flip=False,
              transforms=[
                  dict(type='Resize', keep_ratio=True),
                  dict(type='RandomFlip'),
                  dict(type='Normalize', **img_norm_cfg),
                  dict(type='Pad', size_divisor=32),
                  dict(type='ImageToTensor', keys=['img']),
                  dict(type='Collect', keys=['img']),
              ])
      ]
      dota_train_dict=dict(
          type='RepeatDataset',
          times=1,
          dataset=dict(
              type='CocoDataset',
              ann_file='ann_path/train_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=train_pipeline
          )
      )
      dota_val_dict=dict(
              type='CocoDataset',
              ann_file='ann_path/val_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=test_pipeline    
      )
      dota_test_dict=dict(
              type='CocoDataset',
              ann_file='ann_path/val_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=test_pipeline
           
      )
      mar20_train_dict=dict(
          type='RepeatDataset',
          times=3,
          dataset=dict(
              type='VOCDataset',
              ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
              img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
              pipeline=train_pipeline
          )
      )
      # mar20_val_dict=dict(
      #     dataset=dict(
      #         type='VOCDataset',
      #        ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
      #        img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
      #         pipeline=test_pipeline
      #     )    
      # )
      # mar20_test_dict=dict(
      #     dataset=dict(
      #         type='VOCDataset',
      #        ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
      #        img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
      #         pipeline=test_pipeline
      #     )    
      # )
      data = dict(
          samples_per_gpu=4,
          workers_per_gpu=2,
          train=[dota_train_dict,mar20_train_dict],
          val=dota_val_dict,
          test=dota_test_dict
          )
      evaluation = dict(interval=1, metric='bbox')

I try to train this concat in a normal faster rcnn r50 fpn network.
so i set the num_classes to a+b which is 35 here

then i change both /mmdet/datasets/coco.py and /mmdet/datasets/voc.py file CLASSES part to

which also num = 35

CASE1: when I started training, everything was normal ,but i will got issue when this 1st training process and the 1st val process finished.

2022-09-27 15:04:44,246 - mmdet - INFO - Epoch [1][1850/1898]   lr: 1.000e-04, eta: 3:05:32, time: 0.528, data_time: 0.022, memory: 8168, loss_rpn_cls: 0.1822, loss_rpn_bbox: 0.0940, loss_cls: 0.3737, acc: 91.3897, loss_bbox: 0.2509, loss: 0.9008
2022-09-27 15:05:16,011 - mmdet - INFO - Saving checkpoint at 1 epochs
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 10131/10130, 42.3 task/s, elapsed: 240s, ETA:     0s

Traceback (most recent call last):
  File "/home/xxxxx/xxxxx/mmdetection/./tools/train.py", line 185, in <module>
    main()
  File "/home/xxxxx/xxxxx/mmdetection/./tools/train.py", line 174, in main
    train_detector(
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/apis/train.py", line 203, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/epoch_based_runner.py", line 54, in train
    self.call_hook('after_train_epoch')
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/hooks/evaluation.py", line 267, in after_train_epoch
    self._do_evaluate(runner)
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/core/evaluation/eval_hooks.py", line 123, in _do_evaluate
    key_score = self.evaluate(runner, results)
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/hooks/evaluation.py", line 361, in evaluate
    eval_res = self.dataloader.dataset.evaluate(
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/datasets/coco.py", line 445, in evaluate
    result_files, tmp_dir = self.format_results(results, jsonfile_prefix)
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/datasets/coco.py", line 390, in format_results
    result_files = self.results2json(results, jsonfile_prefix)
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/datasets/coco.py", line 322, in results2json
    json_results = self._det2json(results)
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/datasets/coco.py", line 259, in _det2json
    data['category_id'] = self.cat_ids[label]
IndexError: list index out of range

list out of range here

CASE2: so I change the faster rcnn r50 fpn num_classes to 15 which equal to dataset_a class num
at the same time i change the /mmdet/dataset/coco.py CLASSES part to only include dataset_A classes and the /mmdet/dataset/voc.py CLASSES part to only include dataset_B classes

but i still got issue

Done (t=0.94s)
creating index...
Done (t=0.96s)
creating index...
index created!
index created!
2022-09-27 15:40:25,563 - mmcv - INFO - Reducer buckets have been rebuilt in this iteration.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:194: nll_loss_forward_no_reduce_cuda_kernel: block: [1,0,0], thread: [0,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:194: nll_loss_forward_no_reduce_cuda_kernel: block: [1,0,0], thread: [1,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:194: nll_loss_forward_no_reduce_cuda_kernel: block: [1,0,0], thread: [2,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/home/shuzhilian/anaconda3/envs/openmmlab/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 14 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

CASE3: SO I TRY LIKE THIS WAY
and this num_classes still 35

train_class=('plane', 'baseball-diamond',"bridge","ground-track-field", "small-vehicle", "large-vehicle","ship", "tennis-court", 
              "basketball-court","storage-tank","soccer-ball-field","roundabout","harbor","swimming-pool","helicopter",'A1', 'A2', 'A3', 'A4', 
               'A5', 'A6', 'A7',
               'A8', 'A9', 'A10', 'A11', 'A12', 'A13',
               'A14', 'A15', 'A16', 'A17', 'A18', 'A19',
               'A20')
dota_class=('plane', 'baseball-diamond',"bridge","ground-track-field", "small-vehicle", "large-vehicle","ship", "tennis-court", 
              "basketball-court","storage-tank","soccer-ball-field","roundabout","harbor","swimming-pool","helicopter")
mar20_class=('A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7',
               'A8', 'A9', 'A10', 'A11', 'A12', 'A13',
               'A14', 'A15', 'A16', 'A17', 'A18', 'A19',
               'A20')
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', img_scale=(1024, 1024), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1024, 1024),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
dota_train_dict=dict(
          type='RepeatDataset',
          times=1,
          dataset=dict(
              type='CocoDataset',
              classes=train_class,
              ann_file='ann_path/train_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=train_pipeline
          )
      )
      dota_val_dict=dict(
              type='CocoDataset',
               classes=dota_class,
              ann_file='ann_path/val_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=test_pipeline    
      )
      dota_test_dict=dict(
              type='CocoDataset',
               classes=dota_class,
              ann_file='ann_path/val_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=test_pipeline
           
      )
      mar20_train_dict=dict(
          type='RepeatDataset',
          times=3,
          dataset=dict(
              type='VOCDataset',
               classes=train_class,
              ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
              img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
              pipeline=train_pipeline
          )
      )
      # mar20_val_dict=dict(
      #     dataset=dict(
      #         type='VOCDataset',
      #        ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
      #        img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
      #         pipeline=test_pipeline
      #     )    
      # )
      # mar20_test_dict=dict(
      #     dataset=dict(
      #         type='VOCDataset',
      #        ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
      #        img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
      #         pipeline=test_pipeline
      #     )    
      # )
data = dict(
    samples_per_gpu=4,
    workers_per_gpu=2,
    train=[dota_train_dict,mar20_train_dict],
    val=dota_val_dict,
    test=dota_test_dict
    )
evaluation = dict(interval=1, metric='bbox')

IT SHOWS THE SAME PROBLEM AT CASE1

I really wonder how to correctly concat two different types of datasets for normal training, val and testing processes
and another question is : it seem that i can not val and test this two different types datasets together after each epoch finished use the config below：

       mar20_val_dict=dict(
           dataset=dict(
               type='VOCDataset',
              ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
             img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
              pipeline=test_pipeline
           )    
       )
       mar20_test_dict=dict(
           dataset=dict(
               type='VOCDataset',
              ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
              img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
               pipeline=test_pipeline
           )    
       )
      data = dict(
          samples_per_gpu=4,
          workers_per_gpu=2,
          train=[dota_train_dict,mar20_train_dict],
          val=[dota_val_dict,mar20_val_dict]
          test=[dota_test_dict,mar20_test_dict]
          )
      evaluation = dict(interval=1, metric='bbox')
      evaluation = dict(interval=1, metric='mAP')

[zachcoleman]

ConcatDataset is for concatenating datasets of the same type. There's no logic to map classes together like you're trying to do. See here:

mmdetection/mmdet/datasets/dataset_wrappers.py

Line 32 in 9d3e162

self.CLASSES = datasets[0].CLASSES