Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Too many failed trials" block #1128

Open
poutine-dejeuner opened this issue Nov 25, 2024 · 0 comments
Open

"Too many failed trials" block #1128

poutine-dejeuner opened this issue Nov 25, 2024 · 0 comments

Comments

@poutine-dejeuner
Copy link

poutine-dejeuner commented Nov 25, 2024

When launching an array of jobs on Slurm, Orion will cancel all subsequent trials when too many of them have failed and then displays a message explaining this. However, it is not very clear from this message that the user needs to take action at this point by removing the failed trials from the experiment.
A short message like: "you can call orion db rm <exp name> --status broken to correct this. Orion will cancel all further trials in this experiment until this is done."
I now know my way around this problem but it took me a while to understand what was going on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant