Does not work on darwin/arm64

[juli1]

I am trying to use the project for Darwin/ARM64. The problem I am facing is that the program crashes. I am trying to use tree-sitter-python. I am using the configuration directives from the README file.

The program I am using is the following

        try {
            Parser parser = new Parser();
            parser.setLanguage(Languages.python());
            Tree tree = parser.parseString("def foo(bar, baz):\n  print(bar)\n  print(baz)");
            Node root = tree.getRootNode();
            System.out.println(root.getType());

            Node function = root.getChild(0);
            System.out.println(function.getType());
        }
        catch (Exception e){
            System.out.println("exception");
            e.printStackTrace();
        }

At the instruction Node root = tree.getRootNode();, the program crashes. This is the stack trace

Command Line: -agentlib:jdwp=transport=dt_socket,server=n,suspend=y,address=127.0.0.1:51316 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant org.example.Main

Host: "Mac14,2" arm64, 8 cores, 16G, Darwin 22.3.0, macOS 13.2.1 (22D68)
Time: Fri Mar 17 18:59:24 2023 MDT elapsed time: 5.564390 seconds (0d 0h 0m 5s)

---------------  T H R E A D  ---------------

Current thread (0x0000000134808a00):  JavaThread "main" [_thread_in_native, id=5123, stack(0x000000016b594000,0x000000016b797000)]

Stack: [0x000000016b594000,0x000000016b797000],  sp=0x000000016b7969b0,  free space=2058k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libjava-tree-sitter.dylib+0x22884]  ts_tree_root_node+0x0
j  ai.serenade.treesitter.TreeSitter.treeRootNode(J)Lai/serenade/treesitter/Node;+0
j  ai.serenade.treesitter.Tree.getRootNode()Lai/serenade/treesitter/Node;+4
j  org.example.Main.main([Ljava/lang/String;)V+36
v  ~StubRoutines::call_stub
V  [libjvm.dylib+0x46e49c]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x390
V  [libjvm.dylib+0x4d2bf8]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*)+0x110
V  [libjvm.dylib+0x4d627c]  jni_CallStaticVoidMethod+0x130
C  [libjli.dylib+0x9ac8]  JavaMain+0xa80
C  [libjli.dylib+0xbdc0]  ThreadJavaMain+0xc
C  [libsystem_pthread.dylib+0x706c]  _pthread_start+0x94

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  ai.serenade.treesitter.TreeSitter.treeRootNode(J)Lai/serenade/treesitter/Node;+0
j  ai.serenade.treesitter.Tree.getRootNode()Lai/serenade/treesitter/Node;+4
j  org.example.Main.main([Ljava/lang/String;)V+36
v  ~StubRoutines::call_stub

FYI, I am using the JVM amazoncorretto17

[sogaiu]

I think it's possible that with what's in this PR you might have more luck.

[brodmo]

I have the same issue unfortunately, even with the PR.

Command Line: -ea -Didea.test.cyclic.buffer.size=1048576 -javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=61893:/Applications/IntelliJ IDEA CE.app/Contents/bin -Dfile.encoding=UTF-8 com.intellij.rt.junit.JUnitStarter -ideVersion5 -junit5 de.jplag.cpp3.MyParseTest

Host: "MacBookPro18,3" arm64, 10 cores, 16G, Darwin 22.5.0, macOS 13.4 (22F66)
Time: Thu Jun 22 18:11:48 2023 CEST elapsed time: 0.252633 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x0000000140009c00):  JavaThread "main" [_thread_in_native, id=9987, stack(0x000000016f6c4000,0x000000016f8c7000)]

Stack: [0x000000016f6c4000,0x000000016f8c7000],  sp=0x000000016f8c3890,  free space=2046k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libjava-tree-sitter.dylib+0x3a40c]  ts_tree_root_node+0x0
j  ai.serenade.treesitter.TreeSitter.treeRootNode(J)Lai/serenade/treesitter/Node;+0
j  ai.serenade.treesitter.Tree.getRootNode()Lai/serenade/treesitter/Node;+4
j  de.jplag.cpp3.MyParser.getRoot(Ljava/io/File;)Lai/serenade/treesitter/Node;+37
j  de.jplag.cpp3.MyParser.parse(Ljava/util/Set;)Ljava/util/List;+28
j  de.jplag.cpp3.MyParseTest.testMyParser()V+21
v  ~StubRoutines::call_stub

[brodmo]

I got it, the tree-sitter submodule just needs to be bumped to the latest version, the current one is two years out of date

[brodmo]

This doesn't get me much further unfortunately, getting the root works, getting its type doesn't:

Command Line: -ea -Didea.test.cyclic.buffer.size=1048576 -javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=62148:/Applications/IntelliJ IDEA CE.app/Contents/bin -Dfile.encoding=UTF-8 com.intellij.rt.junit.JUnitStarter -ideVersion5 -junit5 de.jplag.cpp3.MyParseTest

Host: "MacBookPro18,3" arm64, 10 cores, 16G, Darwin 22.5.0, macOS 13.4 (22F66)
Time: Thu Jun 22 18:39:32 2023 CEST elapsed time: 0.250809 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x000000011f81ca00):  JavaThread "main" [_thread_in_native, id=8451, stack(0x000000016aefc000,0x000000016b0ff000)]

Stack: [0x000000016aefc000,0x000000016b0ff000],  sp=0x000000016b0fb930,  free space=2046k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libjava-tree-sitter.dylib+0x2a924]  ts_node_type+0x28
j  ai.serenade.treesitter.TreeSitter.nodeType(Lai/serenade/treesitter/Node;)Ljava/lang/String;+0
j  ai.serenade.treesitter.Node.getType()Ljava/lang/String;+1
j  de.jplag.cpp3.MyParser.parse(Ljava/util/Set;)Ljava/util/List;+31
j  de.jplag.cpp3.MyParseTest.testMyParser()V+21
v  ~StubRoutines::call_stub

[sogaiu]

@mbrdl

Some months back I worked with someone to get java-tree-sitter working for them on macos. Looking through some of the logs, I think they were using arm64 which I guess implies Apple Silicon. I've got a fork of java-tree-sitter with some branches [1] that contain some of the related work from that occasion.

I don't know if it will help in your case, but FWIW, this branch (containing a shell script to avoid using gradle) was what I was informed worked for them. (The script mentions x86-64 and linux, but that was just for my own testing as I don't have access to macos/arm64 -- if JAVA_HOME is set appropriately, it should use whatever bits are appropriate.) [2]

The discussions took place at the tree-sitter discord server (the invite link should be at the top of GH discussions for the tree-sitter repository) starting around 2023-03-13 and extending to about the end of 2023-03-20, if you're interested in taking a look at the details -- this will require some combing through unrelated conversations though. (It's been a while and I don't regularly work with Java so I don't recall clearly ATM what was done (^^; )

Anyway, I don't know if any of that will help, but FWIW.

Good luck with whatever path you choose.

---

[1] The following branches might be worth looking at:

• sql
• tree-sitter-sql-bigquery
• build-and-run-from-shell-script
• clojure

[2] Note that the branch does not contain the second fix mentioned in #18:

I noticed that deleting tree cursors did not call the associated ts_tree_cursor_delete, so I added that call to the JNI shim.

[brodmo]

Thanks for your efforts, but their issue looks unrelated unfortunately. Building works fine for me.

[sogaiu]

Ah I see.

Out of curiosity, do all the tests pass for you?

[brodmo]

Oh, maybe I got something mixed up. Building the lib with ./build.py works for me. I haven't tried building java-tree-sitter itself and have instead been using the jitpack version from the README. With gradle build I get the following error:

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'java-tree-sitter'.
> Could not resolve all files for configuration ':classpath'.
   > Could not find org.codehaus.groovy.modules.http-builder:http-builder:0.7.2.
     Searched in the following locations:
       - https://plugins.gradle.org/m2/org/codehaus/groovy/modules/http-builder/http-builder/0.7.2/http-builder-0.7.2.pom
     If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
     Required by:
         project : > com.jfrog.bintray:com.jfrog.bintray.gradle.plugin:1.8.5 > com.jfrog.bintray.gradle:gradle-bintray-plugin:1.8.5

But the jitpack version should work, no?

[sogaiu]

I've not used the jitpack version and I don't know whether it works with macos + arm64. Based on other reports at this repository it wouldn't surprise me if it didn't, but this is just a guess.

I tried manually fetching the jar of the most recent version listed at jitpack.io here and looked inside. (For reference, this is what I downloaded.)

I didn't see any shared library, just .class files and a manifest:

$ tree
.
├── ai
│   └── serenade
│       └── treesitter
│           ├── Languages.class
│           ├── Node.class
│           ├── Parser.class
│           ├── Tree.class
│           ├── TreeCursor.class
│           ├── TreeCursorNode.class
│           └── TreeSitter.class
└── META-INF
    └── MANIFEST.MF

I wouldn't be surprised if one needs to arrange for the necessary "native" bits some other way. May be the instructions in the README:

Before you can start using java-tree-sitter, you need to build a shared library that Java can load using the build.py script.

imply that one must do some building to get things working.

FWIW, the branches I mentioned before have various changes in them including (but not limited to) things like modifying the gradle version and tweaking build.gradle.

[brodmo]

Right, I've built the shared library. It works in principle, as I'm able to parse a program and get its root node. I'm unable to get the root node's type, however.

[sogaiu]

Curious indeed.

If getting the root node's type is not working, I wonder if there is something wrong with the dynamic library.

The error output from here vaguely reminds me of some things that were tried to get things working here. I don't know if it's really relevant, but tweaking build.gradle and updating gradle itself were among the things done. If interested, the details are in this commit.

Dong the aforementioned was important in our case because as-is the java-tree-sitter repository content wasn't working on multiple operating system setups (it wasn't just macos). Once I got it working for Linux x86_64, there was a more solid basis from which to proceed.

However, ultimately I found that trying to get appropriate diagnostic information out of the gradle build process to be on the opaque and frustrating side -- that's what motivated the shell script approach. With a shell script, the details of the whole procecss were much more apparent and individual commands and their output could be more readily tweaked and examined.

So things we ended up doing at one point or another included:

• try on another OS / architecture
• try to get the tests from the official repository working
• try with another grammar

[juli1]

FYI, I forked and added functionalities for java-tree-sitter. It works on MacOS ARM64 and Linux ARM64 and AMD64

[sogaiu]

@juli1 Thanks for mentioning your results and fork.

I looked over some of the commits and see some of the additions you mentioned along with fixes.

[sogaiu]

On a side note, does anyone know if there have been attempts to hook up to tree-sitter via the relevant parts of Panama like may be this?

May be there will be enough in JDK 21 (2023-09) to allow movement away from use of JNI.

[1aidar1]

Easy workaround is to change variable cpp to True in build.py Seems to work.

[sogaiu]

On a side note, tree-sitter-ng might be a potential alternative depending on one's use case.