Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple queues in BUILDKITE_QUEUE #205

Open
joemiller opened this issue Jan 25, 2025 · 0 comments
Open

Support multiple queues in BUILDKITE_QUEUE #205

joemiller opened this issue Jan 25, 2025 · 0 comments

Comments

@joemiller
Copy link

It would be a great addition if the autoscaler could be configured to listen on multiple queue names.

For instance, I have a queue named private that has been in place for a long time and a lot of repos target this queue. Now, I have introduced private-arm64 as we expand into multi-arch builds.

I have used the BuildkiteAgentTags input on the elastic-ci stack to add additional queue tags to private: queue: private-amd64 and queue: private-x86_64. This mostly works. If a job is created and asks for private-amd64 it will be assigned. But if there are no private queue workers running none will be created because the autoscaler is only watching for BUILDKITE_AGENT: private and not private-amd64.

Also, being able to target queue: private-{{matrix.arch}} is VERY useful to avoid duplication in pipelines.

This means I either need to:

  1. Switch over to queue names such as private-amd64 and private-arm64, which would require updating a lot of repos from private -> private-amd64
  2. Introduce an entirely new private-amd64 queue. Slightly inefficient on resource usage with extra agents hanging around.

I think the ideal would be for the autoscaler to support multiple queue names to trigger a scale event, eg: BUILDKITE_QUEUE: private,private-amd64,private-x86_64. This would also match the behavior of the buildkite-agent which is able to listen for and accept jobs on multiple queue tags.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant