-
Notifications
You must be signed in to change notification settings - Fork 0
/
ConfInt_01_artificial.Rmd
90 lines (71 loc) · 4.86 KB
/
ConfInt_01_artificial.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
---
title: "Confidence intervals in Machine Learning Models"
output: html_document
---
# Generating confidence intervals
Compare confidence intervals to linear regression results, loess, and quantile random forest for very simple problem: 1-3 predictors with normal distributed availability and linear/quadratic response (plus some noise). Check which model generates which kind of confidence interval. Most important / key feature after experience with quantile randnom forest: for local models the confidence intervalls for extrapolation should get very wide.
Gaussian Process via GPflow is another candidate - which could be competitive to Keras-solution for model accuracy as well.
# Generate artificial datasets
To get some insight into the behavior of Error-Estrimates from Machine Learning Models I chose to generate artificcial datasets first.
```{r setup, echo=FALSE, message=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(data.table)
library(quantregForest)
```
## Noisy linear data
```{r LinearData, eval=FALSE, echo=FALSE}
plot(x, y_lin, main = "Loess")
lines(x_pred, y_loess_lin$fit)
lines(x_pred, y_loess_lin$fit + y_loess_lin$se.fit, lty = "dashed")
lines(x_pred, y_loess_lin$fit - y_loess_lin$se.fit, lty = "dashed")
```
## Gaussian for X and Y with interaction
A frequent observation for the yield of a manufacturing process is that there's an optimum value for a process parameter and reaches zero yield with deviations from that value. This observation is simulated her by a (kind of) normal distribution to describe the decrease of the process yield for 2 parameters. To bring in interaction between the two parameters I have chosen a term $e^{-(X-Y)^{-2}}$ : the more X and Y differ the more the Yield is reduced. A constant noise term is added with an standard deviation of 0.2 - plus a factor 0.45 to keep the sum of the arguments below 1 for most inputs. The Yield-data is restricted to be within [0,1] (pmin and pmax) - as an yield >1 or <0 doesn't make much sense. However, this also limits the noise impact for
The frequency of input data in X and Y direction is given by a normal distribution - that means most data around (0,0). Thus the uncertainty should also be lowest around there. The heatmap of the artificial data is shown below the code to generate it.
```{r plot_dualGauss}
lExample <- 1E3
set.seed(2019)
DTdualGauss <- data.table(X = rnorm(lExample),
Y = rnorm(lExample))
DTdualGauss[, Z := pmin(1, pmax(0, 0.45*(2^(-X^2) + 2^(-Y^2) - 2^(-(Y-X)^(-2)) + rnorm(lExample, 0, 0.2))))]
DTdualGauss.int <- with(DTdualGauss, akima::interp(X, Y, Z))
pin_org <- par("pin")
{
par(pin = c(3,3))
with(DTdualGauss.int, image(x, y, z))
par(pin = pin_org)
}
```
In the heatmap the high values around (0,0) with some alongation in the x=y direction (the interaction chosen here) are visible by the darker red colors.
## Quantile Regression Forest
Tree-based methods like the classic random forest are well established. One of which is the Quantile Regression Forest - it generates not only predictions for the mean but you can use quantiles for the prediction.
```{r plot_dualGaussModels}
testData <- data.table(X = -50:50 / 10, Y = 0)
testData[, Z_theor := pmin(1, pmax(0, 0.45*(2^(-X^2) + 2^(-Y^2) - 2^(-(Y-X)^(-2)))))]
# Generate Quantile Regression Forest and predict 1st decile, median, and 9th decile
QRF.DTdualGauss <- quantregForest(x=DTdualGauss[, list(X,Y)], y=DTdualGauss[, Z])
testData[, Z_Dec1 := predict(QRF.DTdualGauss, testData[, 1:2], 0.1)]
testData[, Z_med := predict(QRF.DTdualGauss, testData[, 1:2], 0.5)]
testData[, Z_Dec9 := predict(QRF.DTdualGauss, testData[, 1:2], 0.9)]
{ # Plot QuantRegForest-Output
plot(Z_theor~X, testData, ylim=c(0,1), type="l")
points(Z_med~X, testData)
with(testData, segments(X, Z_Dec1, X, Z_Dec9, col = gray(0.5)))
abline(v=c(min(DTdualGauss[round(5*(Y-mean(testData$Y)))==0, X]), max(DTdualGauss[round(5*(Y-mean(testData$Y)))==0, X])), lty="dashed")
}
loess.DTdualGauss <- loess(Z~X+Y, DTdualGauss, span = 0.25)
loess.pred.DTdualGauss <- predict(loess.DTdualGauss, testData[, 1:2], se = TRUE)
testData$Z_loess <- loess.pred.DTdualGauss$fit
# calculate limits for comparison: 2x StdError should be same order of magnitude as Decile
testData$Z_loess_upper <- loess.pred.DTdualGauss$fit + 2*loess.pred.DTdualGauss$se.fit
testData$Z_loess_lower <- loess.pred.DTdualGauss$fit - 2*loess.pred.DTdualGauss$se.fit
rm(loess.pred.DTdualGauss)
{ # Plot Loess-Output
plot(Z_theor~X, testData, ylim=c(0,1), type="l")
points(Z_loess~X, testData)
with(testData, segments(X, Z_loess_lower, X, Z_loess_upper, col = gray(0.5)))
abline(v=c(min(DTdualGauss[round(5*(Y-mean(testData$Y)))==0, X]), max(DTdualGauss[round(5*(Y-mean(testData$Y)))==0, X])), lty="dashed")
}
```
# Performance as compared to tensorflow
Use some standard-problems (chose the ones from RStudio-Keras website) to benchmark tprobability vs tensorflow. Is it worth the additional effort?