Bootstrap resampling for estimating confidence intervals for static histories

[j-bryan]

In an effort to show the effect of uncertainty in the price time series on NPV for some model but without relying on a generative time series model, I had the idea to use bootstrap resampling to estimate the confidence interval for NPV using the historical data. I thought I'd open a discussion on the topic since it's not a feature currently implemented in HERON but may have limited utility to others.

This was my methodology:

1. Use the "RAVEN_sample_ID" column in the CSV file containing the historical price data to segment the data into 24-hour segments. For 1 year of data, this resulted in 365 segments. This segment length was chosen because the dispatch time scale of the storage components in our model is sub-daily.
2. Add a step to the inner.xml to save all of the "inner" realizations, which correspond to the NPV of the cash flows from each segment.
3. Use scipy.stats.bootstrap to estimate a confidence interval for the sum of a sampled group of these daily NPV values (total to an annual NPV).

A preliminary application of this methodology estimated a variance of NPV comparable to that estimated when using a clustered ARMA ROM. The estimated means were somewhat different, but that may be an artifact of the ARMA ROM fit. If there is interest, I can attach files with more info on my implementation.

I certainly don't advocate for this being the default workflow, but it shows there may be limited value for using generative time series models for electricity price when simulating models with a price-taker strategy. I believe the more interesting question is how to model an electricity market and its underlying uncertainties (e.g. uncertainty in renewables production, load, generator outages, fuel costs, etc.), rather than modeling price directly.

[PaulTalbot-INL]

This is an interesting idea. I certainly agree we're more interested in capturing the uncertain behavior of series than produce the series themselves.

An interesting point to consider; the intent of the synthetic histories is to do something more than bootstrapping; if functionally we're just using data to produce more similar data, we haven't made efficient use of synthetic histories, since we could just do UQ and modeling on historical data. If done right, synthetic histories are intended to produce realistic scenario data that expands the historical set based on trends and behaviors. If not done right, synthetic histories and bootstrapping are closer to the same process.

This deserves discussion.

[PaulTalbot-INL]

Second note, a collaborator at BYU proposed a strategy that involved more histogram/distribution-based UQ without the synthetic history approach, which might correlate strongly with your suggestion here.

[dylanjm]

I was thinking that what Jacob is proposing would approach a "simpler" method and fill that gap we were previously talking about for FORCE tools. This way a user could provide a static history and still quantify some level of uncertainty in NPV.

[dylanjm]

Another note to add, I think Jacob and I were previously talking about the opportunity to do some block bootstrapping approach across multiple years that you can randomly draw dispatch windows from. This way, you could incorporate some of the rare events that might happen across different years.

[j-bryan]

I totally agree with what you've both said. While you'll miss most of the interesting parts of the problem by bootstrapping, it does let you quantify some uncertainty with no modeling effort for the time series. In the narrow case of price-taker strategies in a static market, the distribution of economic metrics may be similar to results that could be obtained from more rigorous modeling, but many interesting questions fall outside of where this method would be applicable.

To give a little more context on why I tried this method at all, this was part of a sensitivity analysis. I wanted to see if variation in revenue was significant relative to other sources of uncertainty in the economics of a plant, such as uncertainty in capital costs, while avoiding any bias that using a clustered ROM for synthetic histories might introduce.

[PaulTalbot-INL]

Another note to add, I think Jacob and I were previously talking about the opportunity to do some block bootstrapping approach across multiple years that you can randomly draw dispatch windows from. This way, you could incorporate some of the rare events that might happen across different years.

Yeah, I think this is what BYU's approach was: build uncertainty via historic data and address it directly, rather than Monte Carlo-style blasting potential histories at it. This gets easier the more you know what parts of your uncertainty the economic metrics are most sensitive to, which again is something that could be done before and separate from the long-term optimization analysis. I would love to see this implemented as a "dispatch solver", although it also replaces the synth hists, not just the dispatch optimization.

[j-bryan]

Six months later now, I wanted to revisit this discussion. I was able to apply an approach similar to what I described earlier, using block bootstrap resampling on a detrended time series of hourly revenue (system dispatch times price time series) to estimate the uncertainty in revenue over a period of time. This isn't exactly the same as having a distribution of revenue (or NPV) derived from synthetic histories, but the variance in NPV obtained through the block bootstrap approach and sampling synthetic histories were comparable for the regions and time periods I studied.

This bootstrapping approach shouldn't replace synthetic histories, since the bootstrap analysis is limited to the historical data. However, the bootstrap approach offers a couple distinct advantages over synthetic histories:

• The dispatch optimization needs to be performed only on the historical data, meaning significantly less computational expense than synthetic history sampling.
• No time series modeling is required.

Considering these advantages, I think that the bootstrap approach could be useful as a litmus test for the sensitivity to time series variation when modeling an energy system. Researchers could apply the block bootstrap using just the historical data relevant to their model to determine if further time series modeling work is reasonable for their analysis. There are some systems and markets that are not very sensitive to time series variance, and this approach could offer a quick way to determine if this is the case relative to the other uncertainties present in the model.

[PaulTalbot-INL]

I think this would be a viable alternative workflow. Is this something you have done from modifying a RAVEN "inner" file, which modification could then be coded up in the template driver?