Automatic single-parameter sensitivity analysis

[EwoutH]

Automatic, one-line single-parameter sensitivity analysis (a.k.a. univariate or "one-factor-at-a-time" (OFAT) sensitivity analysis) would be a great addition for exploratory model testing and model validation. The idea is that we add a function to the EMAworkbench which can fully test sensitivities in a model with a single line of code. The function would work like this:

1. Request all variables and default values from model
2. Request all reporters (KPIs) from model
3. For each variables, start runs with -X% and +X% in that variable for the number of replications
4. Log the results
5. Normalize if specified
6. Return graphs and DataFrame as specified

The user can specify:

1. The number of replications
2. The number of time steps (= the measurement time)
3. The amount to vary (for example 5%, 10% or 20%)
4. The default scenario (values for each variable)
5. The variables to test for sensitivity (default=all)
6. The KPIs (reporters) to test against (default=all)
7. Normalization (none, normalize to base, normalize to base and calculate ratio)
8. The desired output data (DataFrame, graph, or both)

The output graph could like this:

No normalization (absolute)

Normalization (relative to base case)

I think the function could be called univariate_sensitivity() and a function call could look like this:

dataframes, graphs = univariate_sensitivity(model=Model, time_steps=60, replications=6, amount_to_vary=0.20)

I have a lot of code already from a recent agent-based modelling course, mainly run_experiments.py and process_experiments.ipynb

Open to feedback on how to improve this!

Edit: Maybe we can make it a Class, which has functions built-in to normalize and graph. That way you only have to run the runs once, and then can use the Class functions to get data and graphs from it. That would be a two-line solution but is more scalable and robust.

[steipatr]

Saltelli et al. (https://www.sciencedirect.com/science/article/pii/S1364815218302822) have strong opinions about one-factor-at-a-time sensitivity analysis, and claim it is only adequate if the model is provably linear. Just some food for thought.

Personally, I would love to see a smoother integration of the "bubble plot" introduced in https://pynetlogo.readthedocs.io/en/latest/_docs/SALib_ipyparallel.html with EMA and SALib, but that's a personal wish :)

[quaquel]

My thoughts exactly.

@steipatr you can make a separate feature request for the bubble plot.