Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make it easier to run tasks locally #536

Open
bhearsum opened this issue Jul 2, 2024 · 1 comment
Open

make it easier to run tasks locally #536

bhearsum opened this issue Jul 2, 2024 · 1 comment

Comments

@bhearsum
Copy link
Contributor

bhearsum commented Jul 2, 2024

@JohanLorenzo linked me to dagger.io recently. One of the things that it tries to solve is reproducability problems between local and CI executions of the same thing. It does by making it trivial to run exactly what CI does locally. We have a lot of issues with this at the moment (especially in Gecko), to the point where it is often avoided altogether, and I think taskgraph should be able to make this similarly easy for containerized tasks.

I'm envisioning something like a taskgraph run command that can be fed a label or some other unique task identifier. Taskgraph would then either build or download the image the task runs on, launch a container, and run the task.

The trickiest bit here is likely to be what to do with tasks that depend on things from other tasks. In many cases we'd probably want to fetch artifacts from existing tasks, but there may be cases where we'd want to rebuild those tasks locally. This is extra tricky in some Gecko scenarios where something like mozharness is built as part of the build task, but we probably don't want to do a full build to pick up changes to it. (Maybe we can ignore extra complicated scenarios like this to start with. The fix there could be not to build mozharness as part of builds...). Docker images are the most notable thing that nearly all containerized tasks depend on.

Other possible issues we may encounter:

  • run-task or other task runners currently only working on our deployed images
  • Tasks that use volume caches may need some adjustments to run locally
@bhearsum
Copy link
Contributor Author

bhearsum commented Jul 2, 2024

For a bit of additional context, one of the things that makes it so difficult to reproduce some things locally is all of the upstream artifact downloading and unpacking that taskgraph/run-task does. These are done opaquely in Python code, with no way to simply copy/paste some commands to have the equivalent done locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant