[GOAL] Benchmarking Workflows

[droumis]

Summary and Links

• benchmark: Benchmark speed of initial display and interaction

Key Benchmarking Metrics:

1. Latency to initial display (of anything useful)
2. Latency for interaction updates (pan, zoom, or scroll/scrub). The type of interaction that we want to prioritize depends on the workflow. For a stacked timeseries viewer, either a zoom out or pan is ideal. For an imagestack viewer, scrolling/scrubbing through the frames is ideal.

Benchmarking Dimensions/Parameters:

1. Dataset size (e.g. for stacked timeseries, number of channels or samples; for an imagestack, number of frames or frame size)
2. Underlying software changes, including a specific commit/version of any relevant package. (e.g. Param 1.X vs 2.X, or before and after a bug fix to HoloViews). This is the closest thing to what ASV usually would test for over time, but now for a whole set of relevant packages and for specific cherry-picked comparisons. This would require a log/env of commit/versions used per benchmarking run.
3. Workflow approach employed (e.g. stacked timeseries using HoloViews downsample options: LTTB vs MinMaxLTTB vs viewport. A second example is utilizing a numpy array vs pandas df vs xarray da as the data source for a holoviews element. A third example is using hvplot vs holoviews to produce a similar output. A fourth example is using Bokeh Canvas vs WebGL rendering). This would require a manual description about the benchmarking run.

Results handling

• The results of the benchmarking need to be reproducible - storing a copy of the benchmarks, info about the environment used, manual notes about the approach, info about the machine that it was run on.
• The timing results need to be in a format amenable to a comparison.. (e.g. show the latency to display as a function of the number of samples for the stacked timeseries workflow when employing no downsampling vs MinMaxLTTB downsampling)

Out of scope, future stuff:

• Incorporate into the CI
• Put benchmarking stuff in a separate repo (it's totally fine to do that now if you want, but not expected)

Tools for Benchmarking:

• Bokeh 'FigureView actual paint' messages that capture figure_id, state (start, end), and render count
• Playwright

Etc

Previous version of Benchmarking Goal