Elex 2770/2771 OLS and new quantile regression

[lennybronner]

Description

This is one PR for two tickets: elex-2770 and elex-2771.

Elex-2770 adds a bootstrap optimized ordinary least squares implementation to elex-solver.

Elex-2771 updates our quantile implementation to solve the dual problem in the non-regularized case. At the moment we keep our old cvxpy implementation for the case where we run the solver with regularization. Eventually we will want to replace this with the dual problem also.

Also updated the python version that the unit tests run for from 3.7/3.8 to 3.10/3.11.

Jira Ticket

https://arcpublishing.atlassian.net/browse/ELEX-2770
https://arcpublishing.atlassian.net/browse/ELEX-2771

Test Steps

tox works

[dmnapolitano]

Hi!

Thanks for submitting this PR! Code-wise, I think it looks great! 🎉

But can you help me understand these changes conceptually? 😅 Specifically:

1. The bootstrapping: is that just being used to create (better) prediction intervals for OLS predictions? Because if so, 🎉 🎉 🎉 Your paper and PR on elex-model just answered this question for me! 🎉
2. Also, where are we using OLS? Or is that work in progress? 😄
3. The updates to our quantile regression implementation to solve "the dual problem in the non-regularized case": can you say a bit more about the problem being solved here? Is it that you've tried fitting a quantile regression in the past without regularization and it failed? And by "failed" I don't mean "had terrible performance" but either threw an exception or model-fitting took a billion hours for no reason?

Thanks!

[lennybronner]

Hi!

Thanks for submitting this PR! Code-wise, I think it looks great! 🎉

But can you help me understand these changes conceptually? 😅 Specifically:

1. The bootstrapping: is that just being used to create (better) prediction intervals for OLS predictions? Because if so, 🎉 🎉 🎉 Your paper and PR on elex-model just answered this question for me! 🎉
2. Also, where are we using OLS? Or is that work in progress? 😄
3. The updates to our quantile regression implementation to solve "the dual problem in the non-regularized case": can you say a bit more about the problem being solved here? Is it that you've tried fitting a quantile regression in the past without regularization and it failed? And by "failed" I don't mean "had terrible performance" but either threw an exception or model-fitting took a billion hours for no reason?

Thanks!

Absolutely!

1. yay glad!
2. We're moving to OLS as the base for predictions in the bootstrap model, here
3. Oh yes absolutely. This massively speeds up the fitting. Also, this part isn't implemented yet, but it will eventually let us fit many quantile regressions simultaneously. So like one for many different quantiles (median, 10th, 90th etc.) with just one optimization problem. In the new bootstrap model there is a part where we fit 100 quantile regressions to estimate the empirical distribution of our errors, and that will also be sped up a lot when we do that.

[dmnapolitano]

Hi!
Thanks for submitting this PR! Code-wise, I think it looks great! 🎉
But can you help me understand these changes conceptually? 😅 Specifically:

1. The bootstrapping: is that just being used to create (better) prediction intervals for OLS predictions? Because if so, 🎉 🎉 🎉 Your paper and PR on elex-model just answered this question for me! 🎉
2. Also, where are we using OLS? Or is that work in progress? 😄
3. The updates to our quantile regression implementation to solve "the dual problem in the non-regularized case": can you say a bit more about the problem being solved here? Is it that you've tried fitting a quantile regression in the past without regularization and it failed? And by "failed" I don't mean "had terrible performance" but either threw an exception or model-fitting took a billion hours for no reason?

Thanks!

Absolutely!

1. yay glad!
2. We're moving to OLS as the base for predictions in the bootstrap model, here
3. Oh yes absolutely. This massively speeds up the fitting. Also, this part isn't implemented yet, but it will eventually let us fit many quantile regressions simultaneously. So like one for many different quantiles (median, 10th, 90th etc.) with just one optimization problem. In the new bootstrap model there is a part where we fit 100 quantile regressions to estimate the empirical distribution of our errors, and that will also be sped up a lot when we do that.

Whoa! This is awesome! 🎉

[dmnapolitano]

🎉 🎉