Improve preview performance during large document editing

[hackerwins]

What would you like to be added:

Currently, during recent tests with approximately 50 users editing a document containing around 40,000 lines in CodePair, we've observed significant delays caused by the rendering of the Markdown Preview while typing.

This experience can be improved through a couple of potential strategies:

A. Rendering Optimization

Introduce debouncing or optimize to update only the changed parts of the DOM tree.

B. Incremental Parsing

Rather than parsing the entire string into Markdown each time, employ incremental parsing methodologies. This can be explored in detail via libraries such as tree-sitter or CodeMirror Lezer.

Performance Tab	Code

Why is this needed:

Enhancing the performance of the Markdown Preview will drastically improve the user experience, especially in collaborative environments where multiple users are editing lengthy documents simultaneously.

[devleejb]

I believe it may be structurally challenging to apply incremental parsing to the current Markdown preview setup.

I couldn’t find any preview libraries that support this functionality either. However, it seems feasible to implement incremental parsing using tree-sitter-markdown and then write rendering method based on that.

While this approach would require a significant implementation effort, it could enable us to later modify the scroll sync from a ratio-based approach to one that aligns with the actual content (like HackMD’s method).

[blurfx]

As a proof of concept, I created a simple renderer (markdown to html) with tree-sitter-markdown and @lezer/markdown, but I decided that working with AST is not the right way to solve this problem.

Converting markdown to ast is already done well by CodeMirror with lezer, so we should focus on rendering optimizations rather than others.

[blurfx]

I'll handle this issue.

[devleejb]

we should focus on rendering optimizations rather than others.

It seems that Incremental DOM can be helpful for rendering optimizations.
(inspired by markdown-it-incremental-dom

[blurfx]

@devleejb actually, I was already working with it 😄

[devleejb]

In #397, incremental DOM was introduced to optimize rendering.

Afterward, we could consider introducing incremental parsing with tree-sitter or lezer. However, rendering directly from these parsers may be challenging, as they mainly focus on syntax highlighting.