econometric underpinnings for standardizing ridge penalty by covariance in local linear forests

[kmoeltner]

ll_regression_forest and predict.ll_regression_forest give the option to use a "covariance-standardized ridge penalty" in the splitting step (via ll.split.weight.penalty) and / or the prediciton step (via ll.weight.penalty = TRUE).

I have a general idea how this might be implemented, but it would be reassuring to see the underlying econometrics.
Couldn't find anything to that effect in the grf documentation or the Friedberg et al. (2021) article.

Any pointers would be greatly appreciated. Thank you!

[rinafriedberg]

Hey @kmoeltner ! This is a good question, and you're correct that we didn't write any theory here. The general idea was to use a ridge regression but with a separate penalty for each covariate, and we chose that using the covariance matrix. It seems to work well in practice and would probably be an interesting thought experiment to detail more.

[kmoeltner]

Hi Rina, thank you so much for getting back. It also works very well in my application. I just want to show / understand the explicit econometrics behind this adjustment. In the generic case, we have penalty lambda*||theta(x0)||^2 (equ. 3 in your paper). What does this look like with cov-adjustment? Something like: lambda*(theta`*C *theta), where C is the covariance matrix of the features X? And is X the raw feature matrix, or centered on xo? Lastly, if we do the cov-adjustment, do theoretical guarantees still apply - I assume so, but just checking ... Thanks again for sharing all these excellent packages within GRF, and making them accessible to the broader community!