JavaScript API Discussion

[cwisniew]

If we are going to start supporting JavaScript as a scripting language properly going forwards then we are going to need to come up with a (at lest half decently) thought out API.

One of the issues that we have is the JavaScript that runs in HTML5/Dialog5/Overlay runs in the JavaFX thread which leads to two problems

1. JavaScript can not directly read many of the objects such as maps, tokens, token properties etc as they are not thread safe
2. We can block on this thread while we try to synchronise the above as will lead to unresponsive UI or even worse deadlocks.

Description	Benefits	Drawbacks
1) Use Promises for HTML5/No Promises for GraalVM	GraalVM code is not as complicated	Different API between HML5 + GraalVM (twice the upkeep, more confusing for users), users will find it more difficult to move code between two, depends on setup always remaining same in the future
2) Use Promises for Both	Same API (less upkeep) code can be moved between the two easily, less likely to break when there are changes in underlying MT setup in the future.	GraalVM JS is more difficult than it needs to be
3) Provide both promise and non promise versions of API to both	Where it works its easier to code	Just because it works when you test it doesn't mean it will work while you are playing...
4) JavaScript API for GraalVM, REST API for HTML5	Its still promises galore for HTML5 but would return a json object with the requested data in one go (which is what one would expect) so less promises overall, REST API could be useful for other things as well	It is still 2 APIs, REST API is more cumbersome than JavaScript API

While on the surface option 4 seems kinda crazy it does have a certain appeal as it resembles a web development model...

Once we have figured out how to approach it we can start detailing what is needed.

[dark-ether]

in my opinion we should make the code between HTML5 javascript and graalVM javascript as reusable as possible and maintaining 2 different APIs seems like a lot of development work that we could use in other areas ,so we should do either 2 or 3.

[perkinslr]

Because of the differences in internals on the two javascript engines, they won't be able to share much (if any) Java code. Fortunately, making functions accessible to GraalVM is fairly trivial: just annotate them as access allowed. So pursuing a unified API for the sake of ease of development is not a concern. That said, matching the HTML5 API should only require a small amount of extra work to wrap return values in already-fulfilled promises.

Being able to use the same javascript code with both is of value, but even that is not fully possible (there is no Document-Object-Model in GraalVM, HTML5 javascript cannot register UDFs). The major advantage of matching the APIs is to reduce the "why does this work here, but not there" questions. So minimising differences between them is a laudable goal.

[cwisniew]

in my opinion we should make the code between HTML5 javascript and graalVM javascript as reusable as possible and maintaining 2 different APIs seems like a lot of development work that we could use in other areas ,so we should do either 2 or 3.

Option 3 is really two APIs though... It's not quite as simple as wrapping the whole of the async API in a wrapper that resolves the promise.

[Azhrei]

Yep, number 2 is the way to go. While it's a bit more work creating Promise-based APIs initially, it'll shield user code from future changes better.

[cwisniew]

Yep, number 2 is the way to go. While it's a bit more work creating Promise-based APIs initially, it'll shield user code from future changes better.

I am not so much worried about the more work creating the API but more the more work to use it. For example with promises the below is going to be more complicated and possibly needlessly so for GraalVM

MapTool.tokens.getAllTokens().filter(t => t.isVisible())

[perkinslr]

Rather than a REST API, we could enable XMLHttpRequests (for both lib:// and macro:// URIs). That is the first "tool" any reasonably experience javascript frontend dev is going to reach for when they realize that Frame5's are embedded web pages. And building a layer on top of them for arbitrary access and function calling would be fairly simple. Currently, they don't work though.

---

As for promises in GraalVM... I'm unconvinced that it will help as much as it will hurt. The Java implementation side is easy enough: just wrap all returns in a Thennable. This causes two problems.

First, it masks the fundamental differences between GraalVM javascript and HTML5 javascript. In the first case, the promise always returns immediately fulfilled, so users can just skip waiting on the promise, and won't get bit by that. In the second case, promises won't be immediately fulfilled. So code may not be 100% portable.

Second, it complicates using the javascript API. Instead of

function foo(token) {
  token.setName("bar");
  token.setNotes("baz");
}

it's

function foo(token) {
  token.setName("bar").then(()=>{
    token.setNotes("baz");
    });
}

or, slightly better

async function foo(token) {
  await token.setName("bar")
  await token.setNotes("baz")
}

And then awaiting each call to foo. Which means wrapping all js UDFs in a non-async function just to async call the real function.

---

I think this becomes largely a non-problem if we consider the roles of GraalVM and HTML5 properly. HTML5 javascript is frontend, like the javascript running on a web page (because that is what it is). GraalVM javascript serves the same purpose as nodejs.

Want to give the player a fancy UI? Use HTML5js. Want to make a bunch of changes to tokens or maps? Use GraalVMjs.

It is good for the HTML5 javascript to be able to read and write to tokens and so on, just as it is good that GraalVMjs can create and send data to HTML5 contexts. But I don't think either one should be artifically gimped to match the intrinsic limitations of the other.

[cwisniew]

Rather than a REST API, we could enable XMLHttpRequests (for both lib:// and macro:// URIs). That is the first "tool" any reasonably experience javascript frontend dev is going to reach for when they realize that Frame5's are embedded web pages. And building a layer on top of them for arbitrary access and function calling would be fairly simple. Currently, they don't work though.

Isn't this just a REST API unless the suggestion is we don't create an actual API just enable the end point and allow users to write everything, that to me is kinda worst option, we dont offer an API for HTML5 controls and expect some framework author to create one and keep it maintained.

As for promises in GraalVM... I'm unconvinced that it will help as much as it will hurt. The Java implementation side is easy enough: just wrap all returns in a Thennable. This causes two problems.

First, it masks the fundamental differences between GraalVM javascript and HTML5 javascript. In the first case, the promise always returns immediately fulfilled, so users can just skip waiting on the promise, and won't get bit by that. In the second case, promises won't be immediately fulfilled. So code may not be 100% portable.

These are kind of my concern too we make things in GraalVM harder to do, on top of the above you hilighted it also makes certain javascripty things harder to do, something like the below becomes harder as getAllTokens() returns a promise and isVisible() returns a promise, it gets worse if its a function where the promise results in another promise rather than boolean like isVisible...

MapTool.tokens.getAllTokens().filter(t => t.isVisible())

potentially creating a requirement for us to now have a MapTool.tokens.getAllVisibleTokens() look at MTScript getTokens() to see the final consequences of that. Or going back to someone providing an API on top of what we have just to get around this which seems like creating work that doesn't need to be created...

Allowing the users to easily use built in functionality to filter the token list will lead to a much better API than requiring them to look up every option when they need it.

And then awaiting each call to foo. Which means wrapping all js UDFs in a non-async function just to async call the real function.

I think this becomes largely a non-problem if we consider the roles of GraalVM and HTML5 properly. HTML5 javascript is frontend, like the javascript running on a web page (because that is what it is). GraalVM javascript serves the same purpose as nodejs.

Want to give the player a fancy UI? Use HTML5js. Want to make a bunch of changes to tokens or maps? Use GraalVMjs.

If we are going to go this way I think I prefer really acting like they are different things and providing a rest API.
For example (once I submit the PR which fixes a bug)

If I have

I can use

<html>
<head>
	<link rel="stylesheet" href="./test.css?cachelib=false">
</head>
<body>
	<h1>This is a test</h1>
	<div id='test1'/>
	<script>
		document.getElementById('test1').innerHTML = '...This is a test!';
	</script>
	<script src="./test.js?cachelib=false"></script>
	<img src="asset://87f4e9bfa4f1f3db250b57b3599fa4e9"/>
	<a href="./test2.html">test2</a>
</body>

Even the navigation works for the test2 link, so for anything opened with html.frame5()/dialog5()/overlay() to webview it "looks like" it is just fetching stuff from any old web server. Notice how there is not a lib:// URI in there at all... Which means if someone writes a character sheet using above with a proper API there is no reason we could not serve that character sheet over a built in webserver so people could view it on their phones/tablets (yes granted its not as easy as waving a wand and saying its done but to the user it will be invisible once it happens). Which is why I was thinking about a rest API that way any lib URI starting with path of api (at the moment it has to be macro or property so no conflict there) gets shunted off to MapTool API.
Its kinda hacky but allows us to resolve both internal and external API access
Say we have a Lib:MyCharacterSheet
Internally that's lib://MyCharacterSheet

The JavaScript access the REST api via /api/...
have external access as something like http://192.168.1.222/lib/MyCharacterSheet

Now if we have an API request to /api/tokens/ that returns all tokens
internal this maps to lib://MyCharacterSheet/api/tokens/ (hence the requirement for the api path hack)
external this maps to http://192.168.1.222/api/tokens/
but the "web page" code doesn't need to know this and should work on both internal WebView and external browser.

Then you kinda get something similar to if you were developing an application where GraalVM code would be server side stuff you would do in something like node assuming we expose these calls as well as the standard MapTool API and I don't see why not (yeah yeah no imports or other node built in stuff but you know what I mean) and the HTML front end is something you could run in a browser.

While all of this could be also done with an API the internals would be so different it would essentially be a new API, and I am guessing most web developers would be content with a REST end point.

I think I have pretty much talked myself into non promise API for GraalVM, for HTML5 stuff REST based API + some JavaScript API on top of that to make the simple things (i.e. get/set property) simple by doing the REST call for you. This is not what I was wanting to talk myself into at all as its even more work... but probably best approach.

[perkinslr]

Isn't this just a REST API

REST has requirements relating to self-documenting. I think it's overkill in this case. That doesn't mean we don't expand the existing API from "call a macro" to do a few more things.

expect some framework author to create one and keep it maintained.

At some point, it is easier to write user-friendly and extensibe middleware in the target language than in the bound language. There is no reason, if it is easier to expand a narrow API into a full one, that that must be done by a framework dev. It would be easy enough to always include a <script> tag in HTML5 stuff, containing a chunk of middleware, bundled with maptool itself.

I don't know how hard it is to implement a full REST API in java, just something to think about, I guess.

---

I think I have pretty much talked myself into non promise API for GraalVM, for HTML5 stuff REST based API

The fact that it would be possible to open it up for viewing the pages outside maptool, or interacting via other languages is very compelling. If we didn't have GraalVM-js that can easily do lots of direct token manipulation, I think "faking" that in the html5 would be okay. Since we do have the "backend" version though, making the frontend behave like a normal frontend for the artistically talented frontend devs is probably the best way forward.

[cwisniew]

The fact that it would be possible to open it up for viewing the pages outside maptool, or interacting via other languages is very compelling. If we didn't have GraalVM-js that can easily do lots of direct token manipulation, I think "faking" that in the html5 would be okay. Since we do have the "backend" version though, making the frontend behave like a normal frontend for the artistically talented frontend devs is probably the best way forward.

You were supposed to be talking me out of it! not further into it!!!
But yes this is a good point, if we want to attract more people who do UI/enable those that do UI stuff all the time, then making it act in a familiar way is going to be the best thing we can do to do so...

[dark-ether]

we were discussing #3416 and we reached a conclusion that the js api shouldn't be a one to one reimplementation of the current macro functions as it could hinder future changes. but as it stands now the js api is still too incomplete so it was suggested a temporary "legacy" new javascript api which would contain most of the more used functions but wouldn't receive new features while the new js api is worked on, i think it would be useful to create a list of features the new api should have and which the legacy javascript api should include if any.