Strange drone image prediction

[greenhawktobias]

Hi,
I am trying to detect litter on RGB high resolution drone images (pixel size 1 cm).
I have about 70 annotated litter objects divided on 10 tiles (one tile is for validation). The litter is very sparsely divided.
I trained for 2 epochs and the statistics during the run in cmd shows a high average accuracy over 90 %.
However the prediction results on the validation image seems strange with a regular grided pattern of detected object.
That makes me think that I did something wrong. I also tried prediction on another image with the same type of results.

Validation image with annotations:

Prediction results on validation image

My code below:

import os
from os.path import join, basename

from rastervision.core.rv_pipeline import *
from rastervision.core.backend import *
from rastervision.core.data import *
from rastervision.core.analyzer import *
from rastervision.pytorch_backend import *
from rastervision.pytorch_learner import *
from rastervision.pytorch_backend.examples.utils import (get_scene_info,
                                                         save_image_crop)
from rastervision.pytorch_backend.examples.semantic_segmentation.utils import (
    example_multiband_transform, example_rgb_transform, imagenet_stats,
    Unnormalize)


def get_config(runner,
               multiband: bool = True,
               external_model: bool = False,
               augment: bool = False,
               nochip: bool = False,
               test: bool = False):
    root_uri = '/opt/data/output/'
    train_image_uris = ['/opt/data/data_input/images/1.tif','/opt/data/data_input/images/2.tif','/opt/data/data_input/images/3.tif','/opt/data/data_input/images/4.tif','/opt/data/data_input/images/5.tif','/opt/data/data_input/images/6.tif','/opt/data/data_input/images/7.tif','/opt/data/data_input/images/8.tif','/opt/data/data_input/images/9.tif']
    train_label_uris = ['/opt/data/data_input/labels/1.geojson','/opt/data/data_input/labels/2.geojson','/opt/data/data_input/labels/3.geojson','/opt/data/data_input/labels/4.geojson','/opt/data/data_input/labels/5.geojson','/opt/data/data_input/labels/6.geojson','/opt/data/data_input/labels/7.geojson','/opt/data/data_input/labels/8.geojson','/opt/data/data_input/labels/9.geojson']
    train_scene_ids = ['1','2','3','4','5','6','7','8','9']
    train_scene_list = list(zip(train_scene_ids, train_image_uris, train_label_uris))

    val_image_uri = '/opt/data/data_input/images/10.tif'
    val_label_uri = '/opt/data/data_input/labels/10.geojson'
    val_scene_id = '10'
  

    train_scenes_input = []
    
    channel_order = [0, 1, 2]
    channel_display_groups = None
    aug_transform = example_rgb_transform

    if augment:
        mu, std = imagenet_stats['mean'], imagenet_stats['std']
        mu, std = mu[channel_order], std[channel_order]

        base_transform = A.Normalize(mean=mu.tolist(), std=std.tolist())
        plot_transform = Unnormalize(mean=mu, std=std)

        aug_transform = A.to_dict(aug_transform)
        base_transform = A.to_dict(base_transform)
        plot_transform = A.to_dict(plot_transform)
    else:
        aug_transform = None
        base_transform = None
        plot_transform = None

    chip_sz = 300
    img_sz = chip_sz
    if nochip:
        chip_options = SemanticSegmentationChipOptions()
    else:
        chip_options = SemanticSegmentationChipOptions(
            window_method=SemanticSegmentationWindowMethod.sliding,
            stride=chip_sz)

    class_config = ClassConfig(
    names=['litter', 'background'], colors=['red', 'black'])

    def make_scene(scene_id, image_uri, label_uri):
     
        raster_source = RasterioSourceConfig(
            uris=[image_uri],
            channel_order=channel_order,
            transformers=[StatsTransformerConfig()])
        vector_source = GeoJSONVectorSourceConfig(
            uri=label_uri, default_class_id=0, ignore_crs_field=True)
        label_source = SemanticSegmentationLabelSourceConfig(
            raster_source=RasterizedSourceConfig(
                vector_source=vector_source,
                rasterizer_config=RasterizerConfig(background_class_id=1)))
                
        label_store = SemanticSegmentationLabelStoreConfig(
            rgb=True, vector_output=[PolygonVectorOutputConfig(class_id=0)])
        return SceneConfig(
            id=scene_id,
            raster_source=raster_source,
            label_source=label_source,
            label_store=label_store)

    for scene in train_scene_list:
        train_scenes_input.append(make_scene(*scene))
        
    dataset = DatasetConfig(
    class_config=class_config,
    train_scenes=
        train_scenes_input
 
    validation_scenes=[
        make_scene(val_scene_id, val_image_uri, val_label_uri)
    ])
    
    chip_sz = 300
    backend = PyTorchSemanticSegmentationConfig(
        data=SemanticSegmentationImageDataConfig(img_channels=len(channel_order), img_sz=img_sz),
        model=SemanticSegmentationModelConfig(backbone=Backbone.resnet50),
        solver=SolverConfig(lr=1e-4, num_epochs=2, batch_sz=4))
    chip_options = SemanticSegmentationChipOptions(window_method=SemanticSegmentationWindowMethod.sliding)

    return SemanticSegmentationConfig(
        root_uri=root_uri,
        dataset=dataset,
        backend=backend,
        train_chip_sz=chip_sz,
        predict_chip_sz=chip_sz)

Best, Tobias

[AdeelH]

shows a high average accuracy over 90 %

There seems to be a large class imbalance so I wouldn't recommend looking at the average metrics. How are the litter_precision, litter_recall, and litter_f1? If they are low, then the model is simply not trained enough.

Besides that I would also check if the images in <root_uri>/train/dataloaders look right. Are the images and labels correctly aligned in those images?

[greenhawktobias]

Hi!
litter_precision, litter_recall, and litter_f1 are all 0. Images are correctly labeled and in wgs84.

I can´t se any obvious error in <root_uri>/train/dataloaders:

I wonder if it will be a hard task generelly to train with such a few annotations on high res images? It seems when training that average accuracy did not increase much from 2 to 5 epochs when I did 5 epochs before. But maybe recall will increase with more epochs?

[greenhawktobias]

It should be annotions there right, not a black image?

images and labels in both in wgs84

[AdeelH]

• You are not passing chip_options to SemanticSegmentationConfig.
• Since you are using the chip stage, here is something you can do to address the class imbalance:

   chip_options = SemanticSegmentationChipOptions(
   	window_method=SemanticSegmentationWindowMethod.random_sample,
   	target_class_ids=[class_config.get_class_id('litter')],
   	# set this to whatever makes sense
   	chips_per_scene=100,
   	# minimum number of litter pixels for a chip to be considered positive
   	target_count_threshold=1,
   	# discard 80% of negative chips (ones with no litter pixels)
   	negative_survival_prob=0.2)

• You can also try weighting the loss for litter higher than background by setting class_loss_weights in SolverConfig to (for example) class_loss_weights=[10.0, 1.0].

[AdeelH]

It should be annotations there right, not a black image?

Yes, if the image actually contains litter.

[greenhawktobias]

I tried with class_loss_weights=[10.0, 1.0, 0] and to discard 90%. I guess I need more training data according to the statistics after 8 epochs (going for 15 epochs).