Error reporting to user: what we should do

[juliendorra]

This issue is for defining a good user flow (from code to UI) for reporting errors in code objects back to the users using writing them collectively.

I'll update this top comment to reflect the feedbacks, proposals and comments in this issue thread.

Status

What kind of errors?

• ProcessingJS syntax errors. They are now reported by the Renderer in the Javascript console.
• undefined errors. Frequent because of typos. They are in some case reported by the Renderer in the Javascript console in the Fix #73 : Catch codeObject rendering failure at playground full update #75 patch/branch but not always (see Undefined errors in draw() are not caught, no helpful log message #81)
• infinite loops
• Javascript syntax error, untouched by ProcessingJS. Ex: const bob = 76 ; let bob = 77;
• others?

Limiting errors before sending

• Some errors could be caught before sending by a hinter/linter, and reported in the editor.
• Some error are generated by typos, and could be limited by the use of autocompletion in the editor.

Good examples

• Khan Academy report errors in ProcessingJS in a friendly and useful way. https://www.khanacademy.org/computer-programming/new/pjs
  

• CodePen report errors and show them directly in the editor, below the affected line. https://codepen.io/
  

• LiveCodeLab is constantly evaluating the code, and thus catch error as the user type. Errors are reported in the top menu bar.
  

Existing issues

We now have 3 issues that are related to error and error reporting, covering it from different angles :

• Errors in programs should be reported Errors in programs should be reported #9
• Dealing with infinite loop, blocking errors, alert(""), XSS and other JavaScript annoyances Dealing with infinite loop, blocking errors, alert(""), XSS and other JavaScript annoyances #18
• Undefined errors in draw() are not caught, no helpful log message Undefined errors in draw() are not caught, no helpful log message #81

User flow goals

We catch errors

We catch and stop the effects of as many types of errors affecting users as possible.

This might lead to using sandboxing strategies, AST manipulation, code rewriting, redefining functions, as we can learn from the CodePen and JSbin experience.

So we need to prioritize and try and do the easiest things that gives us the more useful coverage first.

We channel errors back to the Programmer page

There several ways to do that:

• postMessage between frames.
  Cons: It won't work with preview off. It's another channel to manage in addition to webSockets. It won't work with another type of client (Like an iPad app for younger kids) tat is not using the browser or frames.
  Pros: Happens only in the programmer page, so maybe less moving parts.

• Playground Renderer send back the errors using webSocket.
  Cons: multiple Playgrounds will all send the same duplicate error to the server.
  Pros: Can debug a distant playground without a local preview. WebSocket standard for all communications between programmer and renderer. Any type of programmer client could listen to the bug reports, for example a dedicated bug console page for facilitator/teachers.

We point the user to the errors

We now need to direct the user to the source of the error. Possibilities from easiest to hardest to implement are:

• we highlight the buggy code objects in the list, for example showing it in blinking bold red :-)
• we pop a message to user currently editing the affected object with the error and it's position
• on send, we put alerts on the side of affected lines in the editor.
• on send, we put the insertion caret back where the error is located, and pop a local hint that there is an error here.
• during typing we evaluate the code at every keystroke and have a live message giving hint about the state of the code. That would make true live coding possible, with an auto-render in the Renderer (or an auto-send in the programmer) as soon as the code is proper (cf. LiveCodeLab or CodePen, where it can be turned off).

Implementation

To be determined as we try it! :-)

CodePen, Khan CS, LiveCodeLab all have docs, posts, and code that we can learn from.

[jonathanperret]

An excellent summary! 👍

[jonathanperret]

Routing errors back on the WebSocket seems to me the logical thing to do, but my brain always explodes when I try to think of the best way to handle multiple simultaneous renderers.

Have you seen Rollbar? Not our exact use case but it could provide some inspiration.

[juliendorra]

As for handling multiple (identical) error message we could:

• on receiving an error message from a Playground, the server compare it with the last error message received, stored in codeObject.lasterror
• if they are identical, the server simply ignores the received error
• if they are different it stores the received in codeObject.lasterror and send it to Programmers
• (Programmers do whatever they want with that message ;-)

---

• when a new code is received, codeObject.lasterror is emptied, before sending this code to Playgrounds
• that way if two similar errors are received in a row, but not from the same code, it's seen as a change

[juliendorra]

(Of course if we set it to null every time a new code is sent we will be always be comparing to either null or the same error message from other Playgrounds. Or is there edge cases where we would really compare an old message with a newer one?)

[jonathanperret]

I think it will be relatively easy to associate errors to their code objects. Keeping track of some kind of version identifier will be necessary to make sure only errors generated by the latest version are shown to the user (even then, the code may have changed since the last “go live”, but there’s only so much we can do).

What remains is that the multiple running instances of a single code object may encounter the same, or different errors while running. I guess we can just ignore duplicates and see what happens.

[juliendorra]

It would probably help to time stamp the code object server-side every time new code Ian received. That way the renderer can send the error back with the original time stamp, helping the server to disambiguate and be sure that it's from the last code for this code object
Of course a hash of they text would help too, probably in a complementary way.
That would also allow to report different errors from different browsers but coming from the same code, instead of just trashing them as duplicate or replacing the oldest with the newest.

[jonathanperret]

So, #95 should handle catching and attributing runtime errors, but matching them to a line number in the Processing source is going to be tricky. There isn't a line-per-line correspondence between the generated JS and the Processing code — even the order of the lines can change a lot when classes are involved. And the Processing.js compiler isn't smart enough to generate a source map, or even just source line markers.

I've toyed with adding line markers in the source before handing it to Processing.js: can't do it as comments since stripping comments is among the first things the compiler does. String literals are promising (as they are kept as-is in the generated source), but you can't insert them just anywhere, which would mean a first pass of analysis of the source to determine the safe places… 😓