Add support for build/test priority values so that users can, for example, have long tests start earlier in the flow

[dougthor42]

Description of the feature request:

Migrated from an Google Bazel Users chatroom topic on 2024-12-09.

I would like to be able to tell Bazel to start particular targets earlier on in the build/test process, because I know that these targets will take longer to build/test than others.

For example:

py_test(name = "A", ...)
py_test(
    name = "B",
    ...
    priority = 1,  # lowest, will be one of the last to start testing
)
py_test(name = "C", ...)
py_test(
    name = "D",
    ...
    priority = 10,  # highest, will be one of the first to start testing
)

The "priority" attribute would help Bazel schedule which builds/tests to run first. As a simple example, imagine two workers running 4 tests (defined above). There are many ways that the 4 tests get run on 2 workers, but here are two examples:

When the "long" test D is run first, the total time to run is 4 time units shorter.

Obviously this is not really relevant with 4 targets and 2 workers, but the overall time reduction can be significant when you have a bunch of targets, a bunch of workers, and multiple targets that take a long time to test.

Other notes:

• I originally thought that the scheduling priority could be tied to test size (larger size --> higher priority), but that wouldn't help with the build phase and some people might want larger tests to be lower priority.
• I'm not married to the "1..10" range.
  • If "1..10", then the default would be 5. Or I guess 5.5 haha.
• Unbounded upper is fine too, thus allowing people to set things like priority = 9000 to make sure it's done before others.
  • Then priority = -1 could mean "default" but it might be harder to place. Is the default 4500?

Which category does this issue belong to?

Configurability, Performance, Remote Execution

What underlying problem are you trying to solve with this feature?

As I understand it, builds and tests are run in a pseduo-randomized order. Obviously dependencies are built first, but assuming N targets have completely independent dep trees, there is nothing that say "prioritize building target A before target B".

We have O(1400) test targets and typically run with 40 workers. Some of our tests take 10 minutes to complete. If that test happens to get started at then end of the process, then we end up waiting for that one worker. If the test was started first, then it would complete while all the other small tests are running.

Which operating system are you running Bazel on?

gLinux

What is the output of bazel info release?

release 7.4.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

N/A

Have you found anything relevant by searching the web?

I had a small chat with @rickeylev about this in the Google Bazel Users chat. Here's the transcript:

@dthor says:
Would this be a reasonable Bazel / RBE feature request?

Start test targets with larger timeouts (sizes) earlier in the process.

That way, you don't get pseudo-randomly stuck with the 300s test starting at the very end of bazel test //...

Richard Levasseur, Dec 9, 3:56 PM
I'd rather they be started at the end, so that the small tests that identify a failure early don't wait for the slow tests to take 300s to identify the failure.

But I get your overall point. The ability to affecting test scheduling is rather limited

You, Dec 9, 4:12 PM
The way I see it, we're (basically always) running with some form of parallelization, so unless N(large_tests) > N(jobs), the smaller tests will still fail early.

But yeah having scheduling config would be nice. Maybe it doesn't get tied to test size/timeout and is just a priority attribute that defaults to "middle". After all, there may be some short/small tests that we want to know about early on.

Richard Levasseur, Dec 9, 4:22 PM
Yeah, that's the conundrum with my laptop. I have 8 cores, so Bazel wants to run max 8 tests. And there are definitely > more than 8 heavy integration tests
Meanwhile, CI has plenty of cores

Any other information, logs, or outputs that you want to share?

No response