Error in get_confs_for_psrace()

[be-marc]

Hey, our test / examples sometimes fail in bbotk and mlr3tuning. It happens rarely but I can reproduce the error with a specific seed.

search_space = domain = ps(
  x1 = p_dbl(-5, 10),
  x2 = p_dbl(0, 15)
)

fun = function(xdt, instances) {
  data.table(y = branin(xdt[["x1"]], xdt[["x2"]], noise = as.numeric(instances)))
}

objective = ObjectiveRFunDt$new(fun = fun, domain = domain)

instance = OptimInstanceBatchSingleCrit$new(
  objective = objective,
  search_space = search_space,
  terminator = trm("evals", n_evals = 96))


optimizer = opt("irace", instances = rnorm(10, mean = 0, sd = 0.1))

x = withr::with_seed(68, optimizer$optimize(instance))

#> Error in if (any(left >= 0L)) { : missing value where TRUE/FALSE needed

The traceback says

21: get_confs_for_psrace(iraceResults, scenario$blockSize, iteration_elites, 
        max_experiments, conf_ids = conf_ids, rejected_ids = race_state$rejected_ids)
20: psRace(iraceResults, max_experiments = remainingBudget, iteration_elites = TRUE)
19: irace_finish(iraceResults, scenario, reason = "Not enough budget to race more than the minimum configurations")
18: irace_run(scenario = scenario)
17: irace_common(scenario, simple = TRUE)
16: irace::irace(scenario = scenario)
15: eval(expr, p)
14: eval(expr, p)
13: eval.parent(expr, n = 1L)
12: invoke(irace::irace, scenario = scenario, .opts = allow_partial_matching) at OptimizerBatchIrace.R#194
11: .__OptimizerBatchIrace__.optimize(self = self, private = private, 
        super = super, inst = inst)
10: get_private(optimizer)$.optimize(instance) at OptimizerBatch.R#61
9: doTryCatch(return(expr), name, parentenv, handler)
8: tryCatchOne(expr, names, parentenv, handlers[[1L]])
7: tryCatchList(expr, classes, parentenv, handlers)
6: tryCatch({
       get_private(optimizer)$.optimize(instance)
   }, terminated_error = function(cond) {
   }) at OptimizerBatch.R#60
5: optimize_batch_default(inst, self) at OptimizerBatch.R#25
4: .__OptimizerBatch__optimize(self = self, private = private, super = super, 
       inst = inst)
3: optimizer$optimize(instance)
2: with_preserve_seed({
       set_seed(list(seed = seed, rng_kind = rng_kind))
       code
   })
1: withr::with_seed(68, optimizer$optimize(instance))

Can you fix the error with this examples or do you need one without bbotk?

[MLopez-Ibanez]

Hi,

Could you try the version in git? If you plan to use r-universe to install the package, please wait until it rebuilds: https://mlopez-ibanez.r-universe.dev/builds

[be-marc]

Seems to work. Thanks!

[MLopez-Ibanez]

I will submit 4.1 to CRAN in the next few days. Fixed by ea2a71e

[MLopez-Ibanez]

Hi @be-marc

Could you test the latest git version? CRAN says that bbotk fails with it but I cannot see it:

Changes to worse in reverse depends:

Package: bbotk
Check: tests
New result: ERROR
    Running ‘testthat.R’ [72s/76s]
  Running the tests in ‘tests/testthat.R’ failed.
  Complete output:
    > if (requireNamespace("testthat", quietly = TRUE)) {
    +   library("testthat")
    +   library("checkmate")
    +   test_check("bbotk")
    + }
    Loading required package: bbotk
    Loading required package: paradox
    [ FAIL 1 | WARN 0 | SKIP 50 | PASS 730 ]
    
    ══ Skipped tests (50) ══════════════════════════════════════════════════════════
    • On CRAN (49): 'test_ArchiveAsync.R:2:3', 'test_ArchiveAsync.R:61:3',
      'test_ArchiveBatch.R:3:3', 'test_CallbackAsync.R:4:3',
      'test_CallbackAsync.R:29:3', 'test_CallbackAsync.R:55:3',
      'test_CallbackAsync.R:82:3', 'test_CallbackAsync.R:110:3',
      'test_CallbackAsync.R:144:3', 'test_CallbackAsync.R:171:3',
      'test_CallbackAsync.R:196:3', 'test_CallbackAsync.R:223:3',
      'test_CallbackAsync.R:251:3', 'test_CallbackAsync.R:277:3',
      'test_Objective.R:9:3', 'test_OptimInstanceAsyncSingleCrit.R:2:3',
      'test_OptimInstanceAsyncSingleCrit.R:25:3',
      'test_OptimInstanceAsyncSingleCrit.R:43:3',
      'test_OptimInstanceAsyncSingleCrit.R:61:3',
      'test_OptimInstanceAsyncSingleCrit.R:78:3',
      'test_OptimInstanceBatchMultiCrit.R:3:3',
      'test_OptimInstanceBatchSingleCrit.R:8:3',
      'test_OptimizerAsynDesignPoints.R:2:3', 'test_OptimizerAsynGridSearch.R:2:3',
      'test_OptimizerAsync.R:2:3', 'test_OptimizerAsync.R:24:3',
      'test_OptimizerAsync.R:61:3', 'test_OptimizerAsync.R:82:3',
      'test_OptimizerAsync.R:110:3', 'test_OptimizerAsyncRandomSearch.R:2:3',
      'test_OptimizerCmaes.R:28:3', 'test_OptimizerDesignPoints.R:5:3',
      'test_OptimizerFocusSearch.R:4:3', 'test_OptimizerGenSA.R:6:3',
      'test_OptimizerGridSearch.R:4:3', 'test_OptimizerLocalSearch.R:4:3',
      'test_OptimizerLocalSearch.R:26:3', 'test_OptimizerLocalSearch.R:49:3',
      'test_OptimizerNLoptr.R:9:3', 'test_OptimizerRandomSearch.R:4:3',
      'test_TerminatorClockTime.R:5:3', 'test_TerminatorCombo.R:6:3',
      'test_TerminatorEvals.R:5:3', 'test_TerminatorNone.R:5:3',
      'test_TerminatorPerfReached.R:4:3', 'test_TerminatorRunTime.R:2:5',
      'test_TerminatorStagnation.R:5:3', 'test_TerminatorStagnationBatch.R:5:3',
      'test_TerminatorStagnationBatch.R:19:3'
    • TRUE is TRUE (1): 'test_mlr_callbacks.R:19:3'
    
    ══ Failed tests ════════════════════════════════════════════════════════════════
    ── Error ('test_OptimizerIrace.R:23:3'): OptimizerBatchIrace minimize works ────
    Error in `.__OptimizerBatchIrace__.assign_result(self = self, private = private, 
        super = super, inst = inst)`: `irace::irace` did not return a result. The evaluated configurations are still accessible through the archive.
    Backtrace:
        ▆
     1. ├─utils::capture.output(optimizer$optimize(instance)) at test_OptimizerIrace.R:23:3
     2. │ └─base::withVisible(...elt(i))
     3. └─optimizer$optimize(instance)
     4.   └─bbotk:::.__OptimizerBatch__optimize(...)
     5.     └─bbotk::optimize_batch_default(inst, self)
     6.       └─get_private(optimizer)$.assign_result(instance)
     7.         └─bbotk:::.__OptimizerBatchIrace__.assign_result(...)
    
    [ FAIL 1 | WARN 0 | SKIP 50 | PASS 730 ]
    Error: Test failures
    Execution halted

[MLopez-Ibanez]

Fixed in bbotk.