Add LoveDADataModule to the trainer tests

[calebrob6]

@adamjstewart the problem with LoveDA you likely ran into is that the test set doesn't have any masks, just imagery (which we accurately reflected when creating the test data)

[adamjstewart]

Looks great! I'll try to figure out the failing tests, enjoy your vacation!

[calebrob6]

Minimum tests have a pytorchlightning warning that gets treated as an error, I tried to ignore it but messed something up

[adamjstewart]

Just noticed that our data loader made their news: https://github.com/Junjue-Wang/LoveDA#news

[adamjstewart]

Okay, so looking into this more, I think it's strange to have a datamodule where test doesn't report an accuracy. I think we should instead split val (dataset) into val/test (datamodule) and move test (dataset) to predict (datamodule). We also use test during predict in the Inria datamodule, so there is precedence for this.

In terms of split sizes, we have:

• train: 2522
• val: 1669
• test: 1796

I think it makes the most sense to split val in half so that we have enough samples to train on and can still evaluate val/test accuracy.

@nilsleh @Junjue-Wang do you have any opinions on this?

[Junjue-Wang]

Okay, so looking into this more, I think it's strange to have a datamodule where test doesn't report an accuracy. I think we should instead split val (dataset) into val/test (datamodule) and move test (dataset) to predict (datamodule). We also use test during predict in the Inria datamodule, so there is precedence for this.

In terms of split sizes, we have:

• train: 2522
• val: 1669
• test: 1796

I think it makes the most sense to split val in half so that we have enough samples to train on and can still evaluate val/test accuracy.

@nilsleh @Junjue-Wang do you have any opinions on this?

Accurately, The test scores can be evaluated on the Codalab: https://codalab.lisn.upsaclay.fr/competitions/421#results
I don't think rearranging the data splits is a good solution.
We should align with the public benchmark.

[adamjstewart]

Will the test set masks ever be made public? If we can only make predictions on test and can't evaluate them locally then the datamodule becomes pretty limited.

[calebrob6]

I don't think so, even without test set masks you need the data module for training with torchgeo which is a massive plus. it will also make it easy to generate test predictions.

[adamjstewart]

Another option would be the following mapping:

Dataset	DataModule
train	train
val	val
val	test
test	predict

This feels a bit redundant, although it's what inria does when test_split_pct == 0.

[adamjstewart]

Another option would be:

Dataset	DataModule
train	train
val	val
test	predict

The datamodule wouldn't have a test_dataloader since there's no way to evaluate it anyway.

[adamjstewart]

Depends on #975 to get the tests to pass.