What happens to diagnostics sent by the LS

[mdbergmann]

I'm working with eglot on a Scala project using Metals.
This project, due to Scapegoat compiler plugin, produces tons of diagnostics.
It seems that Metals passes through all diagnostics of the complete project to the LSP client.
How are they handled by the eglot? Are they stored on disk, like a cache?
Assumingly they are passed over to flymake.
It is problematic when there are many diagnostics?
Is this actually the right use of the protocol, when diagnostics of the complete project are sent to the client even though only a tiny fraction are used in the open files?

[joaotavora]

How are they handled by the eglot? Are they stored on disk, like a cache?

No, there is no cache.

Is this actually the right use of the protocol, when diagnostics of the complete project are sent to the client even though only a tiny fraction are used in the open files?

It is a legal use of the protocol but Eglot, and mostly Flymake doesn't know what to do with that extra information right now. That could change, and I think it's a feature for Flymake. I have some ideas, but I've yet to find the manpower (which is mostly me) to implement it in a good architectural way. There is an issue about this: #504 and a pull request #640. I'm going to close the issue in favour of this discussion.

Before even touching the code, one needs to understand, under various clearly laid out scenarios, what should happen in terms of --

a) in-buffer annotation (i.e. the in-buffer underlines)
b) diagnostic summary under M-x flymake-diagnostics-buffer

in an hypothetical project with files foo.c, bar.c, baz.c, quuz.c when:

1. LSP sends diagnostics for all of them but only foo.c and bar.c are being visited by buffers.
2. LSP sends diagnostic for a subset of these files
3. Same as 1 but there were already some diagnostics previously recorded for "unvisited" files baz.c and quuz.c
4. Probably more that I'm not thinking of.

It's only when the combination of these scenarios is very clearly laid out and understood that we should begin coding. Most of this work should be done in Flymake. Flymake lives in the Emacs repo. Flymake has automatic tests. It would probably be a good beginning to begin speccing this in the automatic tests of Flymake, but it can maybe also be done in the automatic tests of Eglot.

[joaotavora]

I think this has to be broken down piece-wise. For now, I'd like to tackle the possibility of presentation of diagnostics for multiple files in Flymake.

Then we can look at how to make Eglot's processing of enormous amounts of diagnostic info coming from the LSP server more efficient.

These are two different problems.

[mdbergmann]

Sure. No prob.

For my immediate problem:

Then we can look at how to make Eglot's processing of enormous amounts of diagnostic info coming from the LSP server more efficient.

How does Eglot currently handle this large amount of data.
Is it just dropped? Or is it kept in memory?

[joaotavora]

How does Eglot currently handle this large amount of data.
Is it just dropped? Or is it kept in memory?

The parts relevant to files that are visited by buffers that Eglot happens to be managing are converted into Flymake diagnostics and serve to annotate the buffer and fill the diagnostic summary list. The other parts are discarded. But to process a big JSON object you have to read it to the end, so at one point it will be all in memory. Worse the big string will be in memory as a string and as Elisp objects.

One promising possibility of improvement is developing a lazy JSON parser that converts the string to Elisp objects "just in time".

[orge-dev]

Sorry If I'm being repetitive, or if this is no longer relevant, but I'm wondering if flymake-show-project-diagnostics could work with eglot to show diagnostics for all files (open or not) in the project. Any thoughts on the difficulty or usefulness of this @joaotavora ?

[joaotavora]

It's useful, but not 100% within the control of Eglot. Last I checked, some servers allow this, some servers don't. It depends on whether the server actually sends this information.