Skip to content

Commit

Permalink
add sleep hack when deriving parallelism for solve to work around iss…
Browse files Browse the repository at this point in the history
…ue with determining number of executors
  • Loading branch information
trautmane committed Jan 8, 2024
1 parent 3159a15 commit 860c4fb
Showing 1 changed file with 9 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,15 @@ private static int deriveParallelismValues(final JavaSparkContext sparkContext,
setup.distributedSolve.threadsWorker = executorCores;
});

try {
final int sleepSeconds = 15;
LOG.info("deriveParallelismValues: sleeping {} seconds to give workers a chance to connect",
sleepSeconds);
Thread.sleep(sleepSeconds * 1000L);
} catch (final InterruptedException e) {
LOG.warn("deriveParallelismValues: interrupted while sleeping", e);
}

// set parallelism to number of worker executors
// see https://stackoverflow.com/questions/51342460/getexecutormemorystatus-size-not-outputting-correct-num-of-executors
final int numberOfExecutorsIncludingDriver = sparkContext.sc().getExecutorMemoryStatus().size();
Expand Down

0 comments on commit 860c4fb

Please sign in to comment.