Fold "all" vs 5-folds in nnUnet

[ironb25]

Hello,

I was wondering what would be the difference in running 5 fold cross-validation vs running "all" as your fold(https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/common_questions.md#do-i-need-to-always-run-all-u-net-configurations) in nnUnet in terms of reporting scores.

Are the scores generated by running "all" basically just the training scores, since it takes all patients as your training data?
What about the dice scores generated in postprocessing.json by running 5 folds? Are those training scores or validation?

In general, I have 3 classes with 30 ish patients, but one of the class is present in 15 patients only.So when I run 5 fold cross validation models,the class in minority is not represented well in validation for each of the models. Would it be better to have a single "all fold" model in that case?

Thanks,
Rohan

[FabianIsensee]

when running the 5 fold CV you get scores for the respective validation sets.
If you set fold='all' then you train and validate on all training cases, so there are no scores you can report. In that case you need a holdout test or holdout validation set to report scores.

My recommendation is to always use the 5 fold cross-validation + have a heldout test set. Use the 5 models from the cross-validation as an ensemble to predict the test set

[ironb25]

Here are my 5 fold results and the allfold (in the end) from post processing.json:

fold 0:

"dc_per_class_raw": {
      "1": 0.6472641395718316,
      "2": 0.15167476174601918,
      "3": 0.675517139424981
  }

fold 1:

"dc_per_class_raw": {
        "1": 0.5736850155584694,
        "2": 0.14869908237920448,
        "3": 0.7546495891995467
    }

fold 2:

"dc_per_class_raw": {
        "1": 0.3510506645589176,
        "2": 0.17456428451561862,
        "3": 0.5163695772616183
    }

fold 3:

"dc_per_class_raw": {
        "1": 0.567096231259455,
        "2": 0.22166004459420946,
        "3": 0.23004841251919275
    }

fold 4:

"dc_per_class_raw": {
        "1": 0.4844174817969575,
        "2": 0.01642710472279261,
        "3": 0.6810303470058721
    }

allfold:

"dc_per_class_raw": {
        "1": 0.8688873874534866,
        "2": 0.8463373946576567,
        "3": 0.8741452051083302
    }

My second class has lower dice scores in all folds, and other classes also seem to work average. Any solutions I could try.?

[FabianIsensee]

all fold is not really a fold. all will do validation on the same data it di the training on. You cannot use these dice scores for anything