Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance #40

Open
Jolanrensen opened this issue Feb 6, 2024 · 3 comments
Open

Performance #40

Jolanrensen opened this issue Feb 6, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@Jolanrensen
Copy link
Owner

Running the processor with Gradle can take a long while in a project the size of DataFrame. This can be felt most using the @include and arg processors, mainly, probably, because they're used the most in the project.

Some performance steps need to be researched.

@Jolanrensen Jolanrensen mentioned this issue Feb 22, 2024
@Jolanrensen Jolanrensen added the enhancement New feature or request label Feb 22, 2024
@Jolanrensen
Copy link
Owner Author

#43 fixes a lot. For DataFrame it halves the build time!

Still, I think there's some potential speed to gain in the ArgDocProcessor; currently, all @set tags are processed before @get tags, but this happens in such a way that if a @get tag is found while @set tags are still present, it's returned without change, but still processed.

@Jolanrensen
Copy link
Owner Author

Jolanrensen commented Mar 12, 2024

Now there might be some more time saving in caching the builds with outputs.uptToDateWhen {}.
Currently, they are rerun every build, but they only need to be updated when:

  • the input has changed
  • the output has changed (if we make the output Read only, we only need to detect file deletions

Partial updates could also be done, but that would require more research.

@Jolanrensen
Copy link
Owner Author

Also, ArgDocProcessor could run in parallel :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant