Indexing never completes on large Scala codebase #6552
Labels
Milestone
Comments
|
Thanks for reporting! We should not have any issues indexing a large workspace, but it might have never been tested on a one similar to yours. IT would be worth takin a look if it's actually not hanging due to GC issues. Might be worth bumping You can also add How large are specific files? We could try to generate something similar to take a look |
|
Hi @tgodzik, I started metals with Here's the metals.log: Click to expand logIt looks like that metals is pinging the build server (sbt in this case) and after about 10 minutes it stops. I have also tried running memory profiling in Visualvm and there's a lot of GC being performed in the beginning, but the heap usage never gets over 3 GB. After about 10 minutes, Visualvm reports that the application has stopped responding, which seems to coincide with the time when metals stops pinging the build server. 
I assume that this indicates that the problem might be with sbt rather than metals? Although, I can confirm that there are no issues compiling the project in sbt after starting it manually in a terminal.
The top 10 largest files: |
|
I think it does never stop indexing, there is no message that the indexing finished. Metals will stop pinging the server after a period of inactivity and nothing is being sent, since indexing never finished. Looks like we need more data 🤔 14804 line code is a lot, but the parser should handle it 🤔 I will try to add something. |
|
You could also try getting a stack trace in the meantime. I think if there is something hung we should see it there. |
|
@tgodzik - Hmm... I re-opened the project and metals again stopped pinging the build server after 10 minutes, but this time the process is not hanging. Plenty of activity even after almost 50 minutes:
Thread dump below. Looks like ScalametaParser is very busy parsing stuff. Could it be some edge case where it's stuck in some sort of never-ending loop? Click to expand log |
|
Looks like one of the files doesn't ever finish parsing, would probably be good to figure out which file it is and why it never finishes parsing, that would be the cause here. You could try loading those files directly into the parser and see which one fails. This Scala CLI script should help: //> using dep org.scalameta:scalameta_2.13:4.9.7
//> using toolkit default
import scala.meta._
@main
def main =
given Dialect = dialects.Scala3
println(os.read(os.pwd / "Main.scala").parse[Source].get.structure) |
|
Ok, it appears that one of the Scala files contains 40 nested if-statements, which seems to trip scalameta parser. :-) I did some timings and it looks like after 20 nested if-statements the parse time increases very sharply (exponentially?) after every additional if-statement:
Note that removing Let me know if you want me to create an issue in scalameta repository. Example piece of code with 40 nested if-statements.import scala.meta.*
@main def main() =
val program =
"""
object Test {
def apply(): Unit = {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (2 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
if (1 >= 10) {
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
"""
given Dialect = dialects.Scala3
println(program.parse[Source].get.structure)
|
|
Thanks for finding this! We can try to figure this one out |
This was referenced Jul 5, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 9, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 9, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 9, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 9, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 9, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 9, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 16, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 16, 2024
tgodzik
added a commit
to tgodzik/metals
that referenced
this issue
Jul 16, 2024
tgodzik
added a commit
that referenced
this issue
Jul 24, 2024
|
Issue is no longer encountered in Metals 1.3.5 |
|
Can this be re-opened? I believe it still occurs on 1.3.5. I will respond with some better evidence. |
|
It's probably best to open a new issue, this doesn't have to have anything to do with an old one |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Hi, we have a relatively large Scala codebase in terms of lines of code, but fairly average in terms of number of files. It's a bit of a special codebase, because the code was not hand written, but rather transpiled from a legacy domain specific language.
As you can see the project has over 1.3 million lines of Scala code across 5553 files.
When we open this project in Metals, we can see that the "indexing" phase never finishes. I have the project opened right now and the indexing has been running for nearly 2 hours:
We have witnessed the indexing phase running for over 3 hours before and never finishing.
We do not really see anything interesting in the
metals.logbesides warnings like this when trying to navigate to a definition:Are there any known problems/limitations in Metals when trying to index large Scala files?
Do you know if we could enable some additional logging to see what Metals is actually doing during the indexing phase?
Do you perhaps have any other suggestions that we could investigate or try out?
Thanks.
Expected behavior
Indexing phase completes in a reasonable amount of time.
Operating system
macOS
Editor/Extension
Nvim (nvim-metals)
Version of Metals
v1.3.2
Extra context or search terms
No response
The text was updated successfully, but these errors were encountered: