IDOM Benchmarks

[Archmonger]

Would be a good idea to create a benchmark for IDOM.

Consider leveraging a similar schema used by Klen's ASGI benchmarks.

[rmorshea]

Ok, here are some of my thoughts on this...

What Won't Be Benchmarked

Unfortunately, standard benchmarking techniques that get applied to other Python web frameworks don't really make sense here. This is because they typically measure requests/second, which is meant to test web servers, and that's not what IDOM is.

IDOM can best be thought of as a framework for developing a full-stack application. Metrics which test web servers and UI frameworks will give misleading results because IDOM isn't bound to any particular technology stack and can thus be made to work with pretty much anything.

For example, if we were to use the standard benchmarks mentioned above it might suggest that IDOM is actually faster than other web servers (even though it uses them under the hood), because it has to deliver less content in the initial response, since the remainder of the content gets loaded later over a WebSocket.

What Should Be Benchmarked

In reality, the two metrics that matter for IDOM are:

1. Time to Interactive TTI - how long it takes for a user to be able to interact with the page
2. Time to Update TTU - how long it takes for the view to update after a user interacts with it

In both of these categories we should expect IDOM to perform worse than its peers because, in the case of TTI we're effectively loading HTML over a websocket, and in the case of TTU, instead of only sending the changes in underlying models, we're sending HTML diffs.

With all that said, IDOM offers a lot at the cost of some performance. The question we're trying to help people answer is whether that cost is worth the ease of use. The metrics mentioned above should help in making that determination.

What Should IDOM Be Compared To

I think the only way to compare IDOM will be to pit it against other full stack applications. That could mean comparing it to more hand crafted solutions like a simple FastAPI/ReactJS stack, as well as other frameworks that attempt to solve this problem like Streamlit or Voila.

I haven't done that much looking, but I've been unable to find an accepted way of benchmarking full-stack apps. This will probably mean that we need to construct our own benchmarks and, unfortunately, that will be a lot of work.

[MarcSkovMadsen]

Listening in as a comparison could interesting and potentially lead to improvements in HoloViz Panel.

FYI. @philippjfr

[MarcSkovMadsen]

Since IDOM is a full stack framework for Python it would probably also make sense to compare it to applications that use for example Transcrypt, Brython, Skulpt to create their applications this includes https://anvil.works/.

More closely related I would add Gradio, H2O wave, DataPane, Plotly Dash, Panel to the list of benchmarks.

[Archmonger]

I still would be interested in knowing if the TTI of a simple page developed entirely in IDOM compares to a relatively slow framework such as Django.

Mostly because I'd be willing to bet it's faster (since django-channels stack is lighter than Django core). Being faster than Django could be a decent selling point for adoption.

[rmorshea]

I think these are all really good points.

To get this moving, I think we need to brainstorm what app(s) should be used for these benchmarks. They needs to be simple enough to be implemented in just about any framework without too much effort or variance in the tech stack. For example, benchmarking with plots could end up being a test of the what plotting tool is being used under the hood rather than application framework behind it. With all that said, here's what I think should be tested:

Types of Updates:

1. Client-driven changes - for example, when a user causes a click event
2. Server-driven changes - the server pushes changes to the client

Types of Applications:

1. Simple tables - sorting on columns or updating individual elements
2. Synced inputs - a numerical slider and a text input that update each other
3. To do list - create, move, and delete items
4. Plots - maybe see how quickly the app can cycle through some data sets?

Any other ideas or improvements to the above?

[MarcSkovMadsen]

Some frameworks are Clients only driven as i understand. For example dash. But some High frequency streaming use case would be interesting. And also some load testing with a small and a large number of users ala Locust.