[javasrc2cpg] Clear JavaParser caches after AST creation #4489
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xavierpinho
added
java
xavierpinho
requested review from
max-leuthaeuser,
johannescoetzee and
DavidBakerEffendi
DavidBakerEffendi
approved these changes
Apr 24, 2024
max-leuthaeuser
approved these changes
Apr 25, 2024
There was a problem hiding this comment.
Maybe we should make AstCreationPass here a Releasable and use Using (https://www.scala-lang.org/api/3.4.1/scala/util/Using$.html) to clean up stuff (cleanupDelombokOutput, clearJavaParserCaches)
Also this PR needs to be merged with master once #4490 is in.
xavierpinho
force-pushed
the
xavierp/clear-javaparser-instances
branch
from
30de7c4 to
b564995
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We reach out-of-memory exceptions when running consecutive scans (within the same process) and providing the so-called «inference jar paths» (
--inference-jar-paths).This can be observed if we run all javasrc2cpg unit-tests with some additional inference jar. In the following experiment, I've used the same 100MB .jar throughout unit-tests and the 8GB limit is quickly reached, precluding them from finishing.
Poking around, it seems clear that each time an inference jar is loaded its TypeSolver remains in memory.
By clearing JavaParser caches the situation improves considerably: