irace.assert(identical(sort(race.instances), seq_along(race.instances))) fails

[DE0CH]

Hi Manuel,
I ran into this bug

Error: == irace == identical(sort(race.instances), seq_along(race.instances)) is not TRUE
5: irace.assert(identical(sort(race.instances), seq_along(race.instances)))
4: elitist_race(scenario = scenario, configurations = raceConfigurations, 
       parameters = parameters, maxExp = currentBudget, minSurvival = minSurvival, 
       elite.data = elite.data, elitistNewInstances = if (firstRace) 0L else scenario$elitistNewInstances, 
       full_experiment_log = iraceResults$experimentLog)
3: irace_run(scenario = scenario, parameters = parameters)
2: irace_common(scenario = scenario, simple = FALSE)
1: irace.cmdline()
== irace == An unexpected condition occurred. Please report this bug to the authors of the irace package <https://github.com/MLopez-Ibanez/irace/issues>

Steps to reproduce

The relevant command line arguments, parameters.txt, and instances list can be found in my github repo. Getting the target runner to work is fairly involved so I just replaced it with a simple program that memories the output of the target runner for the relevant cases.

Thanks for your help!

[MLopez-Ibanez]

The problem is that we call elitist_race because you use max-time but the command-line disables elitism. I am not sure we ever supported using max-time and elitist=0. I am not even sure how that is supposed to work.

[MLopez-Ibanez]

My first thought is to simply give a sensible error to the user saying that using max-time requires elitist=1.

[ndangtt]

Hi Manuel,

Thank you very much for pointing that out. Sorry we weren't aware that the two options aren't compatible. Deyao is doing a number of experiments for tuning irace parameters and this is one of the configurations generated by meta-irace.

Here is the list of irace's parameters that we're looking at based on the previous notes. The first list (the simple list) is the one we're currently using.
DE0CH/irace-tuning3#4

Currently we're focusing on runtime scenarios only, and we use a fixed max-time per scenario to ensure that all irace runs during the meta-tuning are given the same tuning budget. I guess we should remove elitist from the list then. Since you're the expert here, could you please help to have a look at that list and see if there's anything we may miss?