YOLOv8 Model in SAHI with “no_sliced_prediction=True” got 0.26 less mAP50 than before？

[WaderLaken]

First, thank you for your contributions to SAHI.
But I get an incomprehensible problem using SAHI with YOLOv8 model.

When I compute the metrics for val datesets using standard inference with a YOLOv8 Model, I could get 0.862 mAP50. However, I just get 0.607 mAP50 using sliced inference with the same model, simultaneously setting "no_sliced_prediction=True" or setting "no_sliced_prediction=False" but tuning the "slice_size" to an oversized value, like 8000x8000, definitively inferring only one slice on a picture. By the way, I ensure the other reasoning variables the same. What a surprising result!

What's more, I make another try, using sliced inference with am appropriate slice-size (such as 4032x4032), and the mAP50 rises to 0.931. Now I confirm that SAHI is effective to improve the recognition accuracy, especially for tiny objects. However, I really wonder why the values between the last two experiments are markedly different?

Looking forward to your reply. @sinanonur @fcakyon
Best wishes!

[sinanonur]

Hello @WaderLaken

I suspect mAP 's not calculated with the same library but It needs to be checked.
Using "standard inference" how do you calculate mAP50 and is it the same with SAHI prediction? (different libraries may yield different results)

Also can you make sure there is no post processing with your code?

You can also compare the results of no sliced prediction with standard inference. Also you can try to calculate the mAP in a They should be the same. It not it might require further investigation.

If prediction results are different Is it possible for you to share a notebook or code with a sample image demonstrating this situation?

Best

[WaderLaken]

Glad to hear from you @sinanonur

I will show you more effective details.

To be honest, I'm certain that I use the same library sahi/scripts/coco_evaluation.py when I calculate mAP for all inferences. And the predictions from standard YOLOv8 are converted from yolo to coco format before calculating. What's more, the inferences either using standard YOLOv8 or SAHI are based on the same setup, such as 0.001 conf, 0.5 iou, 1 batch-size, 640 image-size, 1000 max-detections and so on.

Also there has been no post processing added or modified with my code yet.

Well, to make it more intuitive, I list the results as follows:

1. standard YOLOv8: 0.862 mAP50 (converted from yolo to coco, then fed into sahi/scripts/coco_evaluation.py)
2. SAHI with no sliced prediction: 0.607 mAP50
3. SAHI with sliced prediction in an oversized slice-size (8000x8000): 0.607 mAP50
4. SAHI with sliced prediction in an appropriate slice-size (4032x4032): 0.931 mAP50

Specifically, I get the same result for 2. and 3., in terms of mAP, Precision, and Recall. And both of them do inference once for each image.

I pick a picture where the difference is obvious in four scenarios. (I've only picked the ones with significant differences. That doesn't mean all the images are different like this for 2. and 3.)

original image:

label image:

standard-YOLOv8 image:

SAHI image with no sliced prediction:

I'm not sure if the 4 high-resolution images will work on this page......

Here is my code using SAHI. (I do not change any original SAHI codes)

source_image_dir = "VOC2007/images/"
dataset_json_path='VOC2007/coco_labels/coco_val.json'
output_dir = 'output'
output_name = 'val' # output_dir/output_name/
yolov8_model_path = "runs/detect/train4/weights/best.pt"
model_type = "yolov8"
model_device = "cuda:0" # or 'cpu'
model_confidence_threshold = 0.001

slice_height = 8000
slice_width = 8000
overlap_height_ratio = 0.2
overlap_width_ratio = 0.2
model_confidence_threshold = 0.001

detection_model = AutoDetectionModel.from_pretrained(
    model_type=model_type,
    model_path=yolov8_model_path,
    confidence_threshold=model_confidence_threshold,
    image_size=640,
    device=model_device,
)
predict(
    model_type=model_type,
    model_path=yolov8_model_path,
    model_device=model_device,
    model_confidence_threshold=model_confidence_threshold,
    source=source_image_dir,
    image_size=640,
    slice_height=slice_height,
    slice_width=slice_width,
    overlap_height_ratio=overlap_height_ratio,
    overlap_width_ratio=overlap_width_ratio,
    postprocess_match_metric='IOU',
    postprocess_match_threshold=0.5,
    dataset_json_path=dataset_json_path,
    visual_bbox_thickness=2,
    project=output_dir,
    name=output_name,
    novisual= True, 
    no_sliced_prediction = True
)

Looking forward to your reply.

Best wishes!

[fcakyon]

@WaderLaken there was a bug in YOLOv8 prediction thresholds. This PR fixed the issue: #988