running out of budget unexpectedly

[DE0CH]

What happened

irace thought it ran out of time budget when there was clearly a lot left. I set --max-time 172800 but after irace called target runner 10 times and used 17290 time, it said it only had 90 time left and stopped. An excerpt of the text log is provided below.

# 2024-01-27 21:14:01 GMT: Stopped because there is not enough budget to enforce the value of nbConfigurations.
# Iteration: 1
# nbIterations: 1
# experimentsUsedSoFar: 10
# timeUsed: 17290
# remainingBudget: 90

What did I expect to happen

I expected it to keep using the budget it has for more experiments.

Steps to reproduce

Run run.sh in my repo here https://github.com/DE0CH/irace-bug2.

[MLopez-Ibanez]

boundMax=10000 when you have a budget of 172800 does not make much sense. Nevertheless, I have made some changes to improve the behavior in this type of scenario.

But I noticed that your target-runner was reporting a large time even when given a time bound. irace will now check that the bound is respected and be more strict about it. Your target-runner should do:

#!/bin/sh -e
echo "1729.0 $5"

Let me know if it doesn't work! Thanks!

[DE0CH]

Thanks for the explanation!

my target runner actually returns min(result, boundMax + 1) because it’s running an surrogate module so it gets the answer regardless. +1 to make sure it’s recorded as a timed out run. I wasn’t sure how irace works with bound so there’s +1 to make sure it triggers the right behavior.

I’ll change my code to remove this +1.

[ndangtt]

hi @MLopez-Ibanez, we have a question related to your comment:

But I noticed that your target-runner was reporting a large time even when given a time bound. irace will now check that the bound is respected and be more strict about it. Your target-runner should do:

#!/bin/sh -e
echo "1729.0 $5"

I suppose $5 is the bound given to the target-runner. If we return the same value as the bound, how does irace know if this is a timeout run (and will penalise it with PAR) or if the algorithm finishes successfully but exactly at that cutoff time? Sorry if this is a dump question!

[MLopez-Ibanez]

That is a good question. Actually irace has always applied PAR if time >= bound. See:

irace/R/race.R

Lines 470 to 476 in 23d36b0

	applyPAR <- function(results, boundMax, boundPar)
	{
	# We do not want to change Inf or -Inf because those represent rejection.
	if (boundPar != 1)
	results[is.finite(results) & results >= boundMax] <- boundMax * boundPar
	return(results)
	}

Is there a reason to change this? What are other tuners doing? If the target-runner can return a status like 'SOLVED' or 'TIMEOUT', how do they handle a situation when the target-runner return 'SOLVED' but the runtime was 10 times the bound given?

Note that the condition for giving an error that I have just implemented is bound + scenario$minMeasurableTime < output$time, so one could return bound + 0.1 as the runtime and it will not give an error but it will be considered a timeout.

[ndangtt]

hi Manuel,

Ah I see, the condition bound + scenario$minMeasurableTime < output$time makes sense. Thank you for pointing that out, that was exactly what we were looking for! We'll make sure to return bound + 0.1 in the target-runner then :)

Regarding your first point:

"Is there a reason to change this?"

The current applyPAR function won't distinguish between cases where the solver successfully stops at exactly the bound, or if it fails and timeouts. But I suppose the first case is super rare (or never happens) so perhaps it doesn't really matter! (I was just curious how it would be handled in irace if that happens).

"What are other tuners doing? If the target-runner can return a status like 'SOLVED' or 'TIMEOUT', how do they handle a situation when the target-runner return 'SOLVED' but the runtime was 10 times the bound given?"

That is a very good point, and honestly I don't know the answer (I tried to look at SMAC3's source code but didn't see that case being handled. If the target-runner returns a runtime value, the tuner will just take it as it is).