After running test instances: assertion error in execute_evaluator()

[sbomsdorf]

Dear @MLopez-Ibanez,

Both a colleague and I found the same internal error in iracedump.rda files which have the following traceback:

> iracedump[["irace.internal.error(msg)"]][["bt"]]
[1] "7: irace.assert(isTRUE(all.equal(configurations_id, sapply(experiments, " 
[2] "       getElement, \"id_configuration\"))))"                              
[3] "6: execute_evaluator(race_state$target_evaluator, experiments, scenario, "
[4] "       target_output, configurations[[\".ID.\"]])"                        
[5] "5: testConfigurations(configurations, scenario)"                          
[6] "4: testing_common(configurations, scenario, iraceResults)"                
[7] "3: testing_fromlog(logFile = scenario$logFile)"                           
[8] "2: irace_common(scenario = scenario, simple = FALSE)"                     
[9] "1: irace_cmdline()"  

We execute irace with different code in different programming languages but on the same hardware using slurm and therefore both a target_runner and target_evaluator. The training works perfectly fine and also for the test instances, the output files are there. In the iracedump.rda the output also show, e.g., in iracedump[["testConfigurations(configurations, scenario)"]][["target_output"]][[1]][["outputRaw"]]. The above error did not show in the terminal output of irace but there the output ends with the table of elite configurations that are to be tested. Along the above lines, the irace.Rdata lacks the testing data.

Could you please point us to whether this can be a problem in some parameter definition in a scenario.txt file and/or how we can assess the actual data that causes the assertion to be thrown?

Please let us know if you need more information/data to further investigate the issue.

Many thanks in advance!

[MLopez-Ibanez]

Which version of irace is this? Could you try the current development version https://github.com/MLopez-Ibanez/irace?tab=readme-ov-file#github-development-version ? If it fails with the development version, could you share the iracedump.rda file? Thanks!

[sbomsdorf]

It happened with irace 4.0.886dd4c. We are currently running everything again using 4.2.0.c9d441b-dirty and will keep you posted on the results with the development version.

[MLopez-Ibanez]

It happened with irace 4.0.886dd4c. We are currently running everything again using 4.2.0.c9d441b-dirty and will keep you posted on the results with the development version.

Thanks. Please let me know if you detect anything wrong.

[MLopez-Ibanez]

Hi, @sbomsdorf any news about this?

[sbomsdorf]

Hi,

I personally encounter another issue (irace is stuck when evaluating the first batch of instances run using the target-evaluator; I suspect an issue with the file system and/or my code but not irace since my colleague's code runs) . My colleague currently performs the exact same runs as before, but has not reached the testing phase yet. We will keep you posted.

Regards,
Stefan

[MLopez-Ibanez]

Thanks.

Are you using target-evaluator just because of using slurm ? If you have some knowledge about R, it may be better to create a targetRunnerParallel function in your scenario.txt and the batchtools package to implement the parallelization:

https://mllg.github.io/batchtools/reference/makeClusterFunctionsSlurm
https://mllg.github.io/batchtools/reference/btlapply.html

Or the clustermq pacakge: https://mschubert.github.io/clustermq/

This may be more reliable than what target-evaluator is currently doing.