Working with 10+MB strings

[sunny-chung]

Environment:

ktreesitter 0.23.0
Kotlin 1.9
JDK 17

Description:

ktreesitter takes 20 seconds to initialize a 13 MB string. All other stuffs, like incremental updating or querying the AST, work fine and quick. This is what I am using to initialize:

        ast = parser.parse { byte, point ->
            if (byte in 0u until text.length.toUInt()) {
                s.substring(byte.toInt() ..byte.toInt()).let {
                    val codePoints = it.codePoints().toArray()
                    if (codePoints.size > 1 || codePoints.first() > 255) {
                        "X" // replace multibyte char as single-byte char
                    } else {
                        it
                    }
                }
            } else {
                "" // the doc is wrong. null would result in crash
            }
        }

I tried to feed direct strings like below, and it doesn't speed up to an acceptable level, and got troubles with multi-byte characters.

ast = parser.parse(singleByteCharSequence)

My use case is to load and edit a 13 MB JSON with syntax highlighting. If this passes, I will feed in even larger JSON data.

Anything could be done or work around? Not sure if it helps, I am using a rope data structure on JVM side.