Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PQ calculation error #135

Open
choodly opened this issue Jun 20, 2020 · 2 comments
Open

PQ calculation error #135

choodly opened this issue Jun 20, 2020 · 2 comments

Comments

@choodly
Copy link

choodly commented Jun 20, 2020

Thank you for your excellent work. On my own dataset, PQ calculation of stuff class is wrong. Can you tell me what else I should pay attention to?

2020-06-19 19:17:10,088 | upsnet_end2end_test.py | line 158: test config:{'dataset': {'dataset': 'coco',
'dataset_path': './data/coco/',
'image_set': 'train2017',
'num_classes': 4,
'num_seg_classes': 6,
'root_path': './data/',
'test_image_set': 'val2017'},
'debug_mode': False,
'gpus': '1',
'model_prefix': 'upsnet_resnet_50_coco_',
'network': {'anchor_ratios': [0.5, 1, 2],
'anchor_scales': [8],
'backbone_fix_bn': True,
'backbone_freeze_at': 2,
'backbone_with_dconv': 100,
'backbone_with_dilation': False,
'backbone_with_dpyramid': False,
'bbox_reg_weights': [10.0, 10.0, 5.0, 5.0],
'binary_thresh': 0.5,
'cls_agnostic_bbox_reg': False,
'fcn_head': 'FCNHead',
'fcn_num_layers': 3,
'fcn_with_norm': 'none',
'fpn_feature_dim': 256,
'fpn_upsample_method': 'nearest',
'fpn_with_gap': True,
'fpn_with_norm': 'none',
'has_fcn_head': True,
'has_fpn': True,
'has_mask_head': True,
'has_mask_rcnn': True,
'has_panoptic_head': True,
'has_rcnn': True,
'has_rpn': True,
'mask_size': 28,
'num_anchors': 3,
'pixel_means': array([102.9801, 115.9465, 122.7717]),
'pretrained': './model/pretrained_model/resnet-50-caffe.pth',
'rcnn_feat_stride': 32,
'rcnn_with_norm': 'none',
'rpn_feat_stride': [4, 8, 16, 32, 64],
'rpn_with_norm': 'none',
'use_caffe_model': True,
'use_syncbn': False},
'output_path': './output/upsnet/coco',
'symbol': 'resnet_50_upsnet',
'test': {'batch_size': 1,
'max_det': 100,
'max_size': 2000,
'nms_thresh': 0.5,
'panoptic_score_thresh': 0.6,
'panoptic_stuff_area_limit': 500000,
'rpn_min_size': 0,
'rpn_nms_thresh': 0.7,
'rpn_post_nms_top_n': 1000,
'rpn_pre_nms_top_n': 1000,
'scales': [1200],
'score_thresh': 0.05,
'test_iteration': 90000,
'vis_mask': False},
'train': {'batch_rois': 512,
'batch_size': 1,
'bbox_loss_weight': 1.0,
'bbox_means': [0.0, 0.0, 0.0, 0.0],
'bbox_normalization_precomputed': True,
'bbox_regression_thresh': 0.5,
'bbox_stds': [0.1, 0.1, 0.2, 0.2],
'bbox_thresh': 0.5,
'bbox_weights': array([1., 1., 1., 1.]),
'begin_iteration': 0,
'bg_thresh_hi': 0.5,
'bg_thresh_lo': 0.0,
'crowd_filter_thresh': 0.7,
'decay_iteration': [60000, 80000],
'display_iter': 20,
'eval_data': True,
'fcn_focal_loss_gamma': 1.0,
'fcn_loss_weight': 0.2,
'fcn_use_focal_loss': False,
'fcn_with_negative_loss': False,
'fcn_with_roi_loss': True,
'fg_fraction': 0.25,
'fg_thresh': 0.5,
'flip': True,
'gt_min_area': -1,
'lr': 0.00125,
'lr_schedule': 'step',
'max_iteration': 90000,
'max_size': 2000,
'momentum': 0.9,
'panoptic_box_keep_fraction': 0.7,
'panoptic_focal_loss_gamma': 1.0,
'panoptic_loss_weight': 0.1,
'panoptic_use_focal_loss': False,
'panoptic_with_roi_loss': False,
'resume': False,
'rpn_batch_size': 256,
'rpn_bbox_weights': [1.0, 1.0, 1.0, 1.0],
'rpn_clobber_positive': False,
'rpn_fg_fraction': 0.5,
'rpn_individual_proposals': True,
'rpn_min_size': 0,
'rpn_negative_overlap': 0.3,
'rpn_nms_thresh': 0.7,
'rpn_positive_overlap': 0.7,
'rpn_positive_weight': -1.0,
'rpn_post_nms_top_n': 2000,
'rpn_pre_nms_top_n': 2000,
'rpn_straddle_thresh': 0,
'scales': [1200],
'shuffle': True,
'snapshot_step': 2000,
'use_horovod': False,
'warmup_iteration': 1500,
'wd': 0.0001}}

2020-06-19 19:17:17,126 | json_dataset.py | line 63 : Creating: coco_val2017
2020-06-19 19:17:17,789 | upsnet_end2end_test.py | line 274: Batch 1/85, data_time:0.182, net_time:0.000, post_time:0.021
2020-06-19 19:17:18,166 | upsnet_end2end_test.py | line 274: Batch 2/85, data_time:0.157, net_time:0.000, post_time:0.019
2020-06-19 19:17:18,539 | upsnet_end2end_test.py | line 274: Batch 3/85, data_time:0.151, net_time:0.000, post_time:0.018
2020-06-19 19:17:18,911 | upsnet_end2end_test.py | line 274: Batch 4/85, data_time:0.149, net_time:0.000, post_time:0.017
2020-06-19 19:17:19,321 | upsnet_end2end_test.py | line 274: Batch 5/85, data_time:0.147, net_time:0.000, post_time:0.025
2020-06-19 19:17:19,744 | upsnet_end2end_test.py | line 274: Batch 6/85, data_time:0.145, net_time:0.000, post_time:0.030
2020-06-19 19:17:20,141 | upsnet_end2end_test.py | line 274: Batch 7/85, data_time:0.143, net_time:0.000, post_time:0.032
2020-06-19 19:17:20,546 | upsnet_end2end_test.py | line 274: Batch 8/85, data_time:0.141, net_time:0.000, post_time:0.034
2020-06-19 19:17:20,958 | upsnet_end2end_test.py | line 274: Batch 9/85, data_time:0.142, net_time:0.000, post_time:0.034
2020-06-19 19:17:21,357 | upsnet_end2end_test.py | line 274: Batch 10/85, data_time:0.143, net_time:0.000, post_time:0.034
2020-06-19 19:17:21,749 | upsnet_end2end_test.py | line 274: Batch 11/85, data_time:0.141, net_time:0.214, post_time:0.035
2020-06-19 19:17:22,140 | upsnet_end2end_test.py | line 274: Batch 12/85, data_time:0.141, net_time:0.214, post_time:0.034
2020-06-19 19:17:22,561 | upsnet_end2end_test.py | line 274: Batch 13/85, data_time:0.141, net_time:0.213, post_time:0.036
2020-06-19 19:17:22,970 | upsnet_end2end_test.py | line 274: Batch 14/85, data_time:0.140, net_time:0.214, post_time:0.037
2020-06-19 19:17:23,388 | upsnet_end2end_test.py | line 274: Batch 15/85, data_time:0.140, net_time:0.214, post_time:0.038
2020-06-19 19:17:23,785 | upsnet_end2end_test.py | line 274: Batch 16/85, data_time:0.139, net_time:0.215, post_time:0.039
2020-06-19 19:17:24,204 | upsnet_end2end_test.py | line 274: Batch 17/85, data_time:0.139, net_time:0.215, post_time:0.039
2020-06-19 19:17:24,633 | upsnet_end2end_test.py | line 274: Batch 18/85, data_time:0.139, net_time:0.215, post_time:0.041
2020-06-19 19:17:25,016 | upsnet_end2end_test.py | line 274: Batch 19/85, data_time:0.138, net_time:0.215, post_time:0.040
2020-06-19 19:17:25,393 | upsnet_end2end_test.py | line 274: Batch 20/85, data_time:0.138, net_time:0.215, post_time:0.039
2020-06-19 19:17:25,780 | upsnet_end2end_test.py | line 274: Batch 21/85, data_time:0.137, net_time:0.215, post_time:0.039
2020-06-19 19:17:26,151 | upsnet_end2end_test.py | line 274: Batch 22/85, data_time:0.137, net_time:0.215, post_time:0.039
2020-06-19 19:17:26,536 | upsnet_end2end_test.py | line 274: Batch 23/85, data_time:0.136, net_time:0.215, post_time:0.039
2020-06-19 19:17:26,913 | upsnet_end2end_test.py | line 274: Batch 24/85, data_time:0.135, net_time:0.215, post_time:0.038
2020-06-19 19:17:27,286 | upsnet_end2end_test.py | line 274: Batch 25/85, data_time:0.135, net_time:0.215, post_time:0.038
2020-06-19 19:17:27,675 | upsnet_end2end_test.py | line 274: Batch 26/85, data_time:0.135, net_time:0.215, post_time:0.038
2020-06-19 19:17:28,077 | upsnet_end2end_test.py | line 274: Batch 27/85, data_time:0.135, net_time:0.215, post_time:0.038
2020-06-19 19:17:28,453 | upsnet_end2end_test.py | line 274: Batch 28/85, data_time:0.135, net_time:0.215, post_time:0.037
2020-06-19 19:17:28,839 | upsnet_end2end_test.py | line 274: Batch 29/85, data_time:0.135, net_time:0.215, post_time:0.037
2020-06-19 19:17:29,229 | upsnet_end2end_test.py | line 274: Batch 30/85, data_time:0.135, net_time:0.215, post_time:0.037
2020-06-19 19:17:29,615 | upsnet_end2end_test.py | line 274: Batch 31/85, data_time:0.134, net_time:0.215, post_time:0.037
2020-06-19 19:17:30,016 | upsnet_end2end_test.py | line 274: Batch 32/85, data_time:0.135, net_time:0.215, post_time:0.037
2020-06-19 19:17:30,406 | upsnet_end2end_test.py | line 274: Batch 33/85, data_time:0.134, net_time:0.215, post_time:0.037
2020-06-19 19:17:30,827 | upsnet_end2end_test.py | line 274: Batch 34/85, data_time:0.134, net_time:0.215, post_time:0.038
2020-06-19 19:17:31,256 | upsnet_end2end_test.py | line 274: Batch 35/85, data_time:0.134, net_time:0.215, post_time:0.038
2020-06-19 19:17:31,694 | upsnet_end2end_test.py | line 274: Batch 36/85, data_time:0.135, net_time:0.215, post_time:0.039
2020-06-19 19:17:32,134 | upsnet_end2end_test.py | line 274: Batch 37/85, data_time:0.135, net_time:0.215, post_time:0.040
2020-06-19 19:17:32,542 | upsnet_end2end_test.py | line 274: Batch 38/85, data_time:0.135, net_time:0.215, post_time:0.040
2020-06-19 19:17:33,003 | upsnet_end2end_test.py | line 274: Batch 39/85, data_time:0.135, net_time:0.215, post_time:0.041
2020-06-19 19:17:33,404 | upsnet_end2end_test.py | line 274: Batch 40/85, data_time:0.136, net_time:0.215, post_time:0.041
2020-06-19 19:17:33,858 | upsnet_end2end_test.py | line 274: Batch 41/85, data_time:0.136, net_time:0.215, post_time:0.041
2020-06-19 19:17:34,274 | upsnet_end2end_test.py | line 274: Batch 42/85, data_time:0.136, net_time:0.215, post_time:0.042
2020-06-19 19:17:34,667 | upsnet_end2end_test.py | line 274: Batch 43/85, data_time:0.136, net_time:0.215, post_time:0.042
2020-06-19 19:17:35,063 | upsnet_end2end_test.py | line 274: Batch 44/85, data_time:0.136, net_time:0.215, post_time:0.042
2020-06-19 19:17:35,457 | upsnet_end2end_test.py | line 274: Batch 45/85, data_time:0.135, net_time:0.215, post_time:0.042
2020-06-19 19:17:35,855 | upsnet_end2end_test.py | line 274: Batch 46/85, data_time:0.135, net_time:0.215, post_time:0.042
2020-06-19 19:17:36,277 | upsnet_end2end_test.py | line 274: Batch 47/85, data_time:0.135, net_time:0.215, post_time:0.042
2020-06-19 19:17:36,690 | upsnet_end2end_test.py | line 274: Batch 48/85, data_time:0.135, net_time:0.216, post_time:0.042
2020-06-19 19:17:37,092 | upsnet_end2end_test.py | line 274: Batch 49/85, data_time:0.135, net_time:0.215, post_time:0.043
2020-06-19 19:17:37,502 | upsnet_end2end_test.py | line 274: Batch 50/85, data_time:0.135, net_time:0.216, post_time:0.043
2020-06-19 19:17:37,882 | upsnet_end2end_test.py | line 274: Batch 51/85, data_time:0.135, net_time:0.216, post_time:0.043
2020-06-19 19:17:38,294 | upsnet_end2end_test.py | line 274: Batch 52/85, data_time:0.134, net_time:0.216, post_time:0.043
2020-06-19 19:17:38,692 | upsnet_end2end_test.py | line 274: Batch 53/85, data_time:0.134, net_time:0.216, post_time:0.043
2020-06-19 19:17:39,063 | upsnet_end2end_test.py | line 274: Batch 54/85, data_time:0.134, net_time:0.216, post_time:0.043
2020-06-19 19:17:39,447 | upsnet_end2end_test.py | line 274: Batch 55/85, data_time:0.134, net_time:0.216, post_time:0.043
2020-06-19 19:17:39,816 | upsnet_end2end_test.py | line 274: Batch 56/85, data_time:0.133, net_time:0.216, post_time:0.043
2020-06-19 19:17:40,184 | upsnet_end2end_test.py | line 274: Batch 57/85, data_time:0.133, net_time:0.216, post_time:0.042
2020-06-19 19:17:40,579 | upsnet_end2end_test.py | line 274: Batch 58/85, data_time:0.133, net_time:0.216, post_time:0.042
2020-06-19 19:17:40,978 | upsnet_end2end_test.py | line 274: Batch 59/85, data_time:0.133, net_time:0.216, post_time:0.042
2020-06-19 19:17:41,348 | upsnet_end2end_test.py | line 274: Batch 60/85, data_time:0.133, net_time:0.216, post_time:0.042
2020-06-19 19:17:41,718 | upsnet_end2end_test.py | line 274: Batch 61/85, data_time:0.133, net_time:0.216, post_time:0.042
2020-06-19 19:17:42,092 | upsnet_end2end_test.py | line 274: Batch 62/85, data_time:0.132, net_time:0.215, post_time:0.042
2020-06-19 19:17:42,460 | upsnet_end2end_test.py | line 274: Batch 63/85, data_time:0.132, net_time:0.215, post_time:0.041
2020-06-19 19:17:42,833 | upsnet_end2end_test.py | line 274: Batch 64/85, data_time:0.132, net_time:0.215, post_time:0.041
2020-06-19 19:17:43,247 | upsnet_end2end_test.py | line 274: Batch 65/85, data_time:0.132, net_time:0.216, post_time:0.041
2020-06-19 19:17:43,644 | upsnet_end2end_test.py | line 274: Batch 66/85, data_time:0.132, net_time:0.216, post_time:0.041
2020-06-19 19:17:44,008 | upsnet_end2end_test.py | line 274: Batch 67/85, data_time:0.132, net_time:0.215, post_time:0.041
2020-06-19 19:17:44,432 | upsnet_end2end_test.py | line 274: Batch 68/85, data_time:0.132, net_time:0.216, post_time:0.041
2020-06-19 19:17:44,845 | upsnet_end2end_test.py | line 274: Batch 69/85, data_time:0.132, net_time:0.216, post_time:0.041
2020-06-19 19:17:45,250 | upsnet_end2end_test.py | line 274: Batch 70/85, data_time:0.132, net_time:0.216, post_time:0.041
2020-06-19 19:17:45,670 | upsnet_end2end_test.py | line 274: Batch 71/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:46,080 | upsnet_end2end_test.py | line 274: Batch 72/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:46,478 | upsnet_end2end_test.py | line 274: Batch 73/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:46,908 | upsnet_end2end_test.py | line 274: Batch 74/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:47,338 | upsnet_end2end_test.py | line 274: Batch 75/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:47,732 | upsnet_end2end_test.py | line 274: Batch 76/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:48,111 | upsnet_end2end_test.py | line 274: Batch 77/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:48,496 | upsnet_end2end_test.py | line 274: Batch 78/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:48,874 | upsnet_end2end_test.py | line 274: Batch 79/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:49,281 | upsnet_end2end_test.py | line 274: Batch 80/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:49,697 | upsnet_end2end_test.py | line 274: Batch 81/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:50,072 | upsnet_end2end_test.py | line 274: Batch 82/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:50,446 | upsnet_end2end_test.py | line 274: Batch 83/85, data_time:0.132, net_time:0.216, post_time:0.042
2020-06-19 19:17:50,817 | upsnet_end2end_test.py | line 274: Batch 84/85, data_time:0.131, net_time:0.216, post_time:0.042
2020-06-19 19:17:51,193 | upsnet_end2end_test.py | line 274: Batch 85/85, data_time:0.131, net_time:0.216, post_time:0.042
2020-06-19 19:17:55,412 | base_dataset.py | line 194: Writing bbox results json to: /media/mubai/backupdata/UPSNet/output/upsnet/coco/upsnet_resnet50_coco_4gpu/val2017/results/bbox_coco_val2017_results.json
2020-06-19 19:17:55,687 | base_dataset.py | line 706: ~~~~ Mean and per-category AP @ IoU=[0.50,0.95] ~~~~
2020-06-19 19:17:55,687 | base_dataset.py | line 714: 0.815
2020-06-19 19:17:55,688 | base_dataset.py | line 714: 0.693
2020-06-19 19:17:55,688 | base_dataset.py | line 714: 0.252
2020-06-19 19:17:55,688 | base_dataset.py | line 716: 0.587
2020-06-19 19:17:55,688 | base_dataset.py | line 706: ~~~~ Mean and per-category AP @ IoU=[0.50,0.50] ~~~~
2020-06-19 19:17:55,688 | base_dataset.py | line 714: 0.986
2020-06-19 19:17:55,688 | base_dataset.py | line 714: 0.945
2020-06-19 19:17:55,689 | base_dataset.py | line 714: 0.505
2020-06-19 19:17:55,689 | base_dataset.py | line 716: 0.812
2020-06-19 19:17:55,689 | base_dataset.py | line 718: ~~~~ Summary metrics ~~~~
2020-06-19 19:17:55,692 | coco.py | line 214: Writing segmentation results json to: /media/mubai/backupdata/UPSNet/output/upsnet/coco/upsnet_resnet50_coco_4gpu/val2017/results/segmentations_coco_val2017_results.json
2020-06-19 19:17:56,087 | base_dataset.py | line 706: ~~~~ Mean and per-category AP @ IoU=[0.50,0.95] ~~~~
2020-06-19 19:17:56,088 | base_dataset.py | line 714: 0.768
2020-06-19 19:17:56,088 | base_dataset.py | line 714: 0.626
2020-06-19 19:17:56,088 | base_dataset.py | line 714: 0.000
2020-06-19 19:17:56,088 | base_dataset.py | line 716: 0.465
2020-06-19 19:17:56,089 | base_dataset.py | line 706: ~~~~ Mean and per-category AP @ IoU=[0.50,0.50] ~~~~
2020-06-19 19:17:56,089 | base_dataset.py | line 714: 0.986
2020-06-19 19:17:56,089 | base_dataset.py | line 714: 0.944
2020-06-19 19:17:56,089 | base_dataset.py | line 714: 0.000
2020-06-19 19:17:56,089 | base_dataset.py | line 716: 0.643
2020-06-19 19:17:56,089 | base_dataset.py | line 718: ~~~~ Summary metrics ~~~~
2020-06-19 19:17:56,092 | upsnet_end2end_test.py | line 307: unified pano result:
2020-06-19 19:18:25,855 | base_dataset.py | line 301: | PQ SQ RQ N
2020-06-19 19:18:25,855 | base_dataset.py | line 302: --------------------------------------
2020-06-19 19:18:25,855 | base_dataset.py | line 304: All | 2157.7 2206.5 63.1 6
2020-06-19 19:18:25,855 | base_dataset.py | line 304: Things | 52.1 56.1 61.8 3
2020-06-19 19:18:25,855 | base_dataset.py | line 304: Stuff | 4263.4 4356.8 64.4 3
2020-06-19 19:18:25,855 | base_dataset.py | line 306: IDX | PQ SQ RQ IoU TP FP FN
2020-06-19 19:18:25,856 | base_dataset.py | line 309: 0 | 10119.1 10241.0 98.8 8500.1 83 1 1
2020-06-19 19:18:25,856 | base_dataset.py | line 309: 1 | 0.0 0.0 0.0 0.0 0 0 30
2020-06-19 19:18:25,856 | base_dataset.py | line 309: 2 | 2671.2 2829.4 94.4 2150.3 76 1 8
2020-06-19 19:18:25,856 | base_dataset.py | line 309: 3 | 81.1 86.9 93.3 6.1 7 0 1
2020-06-19 19:18:25,856 | base_dataset.py | line 309: 4 | 75.0 81.5 92.1 502.6 617 43 63
2020-06-19 19:18:25,856 | base_dataset.py | line 309: 5 | 0.0 0.0 0.0 0.0 0 1 2
2020-06-19 19:18:30,706 | coco.py | line 260: evaluate segmentation:
2020-06-19 19:18:30,706 | coco.py | line 264: IU_array:
2020-06-19 19:18:30,706 | coco.py | line 266: 0.98239
2020-06-19 19:18:30,706 | coco.py | line 266: 0.87430
2020-06-19 19:18:30,707 | coco.py | line 266: 0.97881
2020-06-19 19:18:30,707 | coco.py | line 266: 0.75718
2020-06-19 19:18:30,707 | coco.py | line 266: 0.61269
2020-06-19 19:18:30,707 | coco.py | line 266: 0.08025
2020-06-19 19:18:30,707 | coco.py | line 267: meanIU:0.71427
2020-06-19 19:18:30,708 | coco.py | line 271: confusion_matrix:
2020-06-19 19:18:30,708 | coco.py | line 272: 0.996 0. 0.001 0. 0.002 0.
0.001 0.943 0.04 0. 0.015 0.
0. 0. 0.998 0. 0.001 0.
0.14 0. 0. 0.817 0.043 0.
0.164 0.011 0.184 0. 0.642 0.
0.276 0. 0. 0. 0.639 0.085
@rlangefe
Copy link

I also ran into this issue.

@wshilaji
Copy link

wshilaji commented Mar 3, 2021

I also ran into this issue.

I also ran into this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wshilaji @rlangefe @choodly