Session 2 - Regression and Regularization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Regression and Regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline \n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd \n",
    "import seaborn as sns \n",
    "import matplotlib.pyplot as plt \n",
    "\n",
    "sns.set_context('notebook')\n",
    "sns.set_style('whitegrid')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linear Regression\n",
    "\n",
    "_Fit_ a linear model to supervised data by adjusting a set of coefficients, $w$ to minimize the residual sum of squares between the observed response and the prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<dl style=\"float:left\">\n",
    "    <dt style=\"margin:12px 0\">Linear Model</dt>\n",
    "        <dd>$y=X\\beta+\\epsilon$</dd>\n",
    "    <dt style=\"margin:12px 0\">Objective Function</dt>\n",
    "        <dd>$min_w\\sum(Xw-y)^2$</dd>\n",
    "    <dt style=\"margin:12px 0\">Predictive Model</dt>\n",
    "        <dd>$\\hat{y}(w,x)=w_0+w_1x_1+...+w_px_p$</dd>\n",
    "</dl>\n",
    "\n",
    "<table style=\"float:left; margin-left:70px;\">\n",
    "    <thead>\n",
    "        <th>Notation</th>\n",
    "        <th>Description</th>\n",
    "    </thead>\n",
    "    <tbody>\n",
    "        <tr>\n",
    "            <td>$y$</td>\n",
    "            <td>observed value</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>$y$</td>\n",
    "            <td>observed value</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>$X$</td>\n",
    "            <td>values of features</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>$\\beta$</td>\n",
    "            <td>coefficients</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>$\\epsilon$</td>\n",
    "            <td>noise or randomness in observation</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>$w$</td>\n",
    "            <td>weights</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>$w_0$</td>\n",
    "            <td>adjusts the decision plane in target space</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>$\\hat{y}$</td>\n",
    "            <td>predicted value</td>\n",
    "        </tr>\n",
    "    </tbody>\n",
    "</table>\n",
    "\n",
    "<div style=\"clear: both\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Ordinary Least Squares\n",
    "\n",
    "![OLS](figures/ordinary_least_squares.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Normal Equation\n",
    "\n",
    "$w = (X^TX)^{-1}X^Ty = X^+y$\n",
    "\n",
    "One-step learning algorithm solved by linear algebra system of linear equations ($X^+y$ is called the pseudo-inverse of $X$). \n",
    "\n",
    "- No $\\alpha$ to select (more on this shortly) \n",
    "- No iteration, computed in one step \n",
    "- Slow if $n$ is large (e.g. $n\\geq10^4$)\n",
    "- Computation of $(X^TX)^{-1}$ is slow\n",
    "- $(X^TX)$ must be invertible "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Gradient Descent\n",
    "\n",
    "![Gradient Descent](figures/gradient_descent.png)\n",
    "\n",
    "$$h_\\theta = \\theta_0 + \\theta_1x$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Gradient Descent\n",
    "\n",
    "We _iteratively_ minimize our error by taking the derivative of the cost function to find the downward slope of $h_\\theta$ until we converge - e.g. there is no downward direction, we reach a suitable error threshold, or we reach a maximum number of steps. For linear regression these derivatives are as follows:\n",
    "\n",
    "$$\\theta_0 := \\theta_0 - \\alpha\\frac{1}{m}\\sum_{i=1}^m(h_\\theta(x^i)-y^i)$$\n",
    "$$\\theta_1 := \\theta_1 - \\alpha\\frac{1}{m}\\sum_{i=1}^m(h_\\theta(x^i)-y^i) \\cdot x^i$$\n",
    "\n",
    "- Guaranteed to converge \n",
    "- Works well with large $n$\n",
    "- Need to do many iterations \n",
    "- How to choose $\\alpha$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Learning Rate\n",
    "\n",
    "The hyperparameter $\\alpha$ in gradient descent is called the _learning rate_. It determines how fast down the error curve we descend. The larger $\\alpha$ is the faster we will minimize our cost function, however if $\\alpha$ is too large or too small, we might not be able to find the true minimum of the cost function. \n",
    "\n",
    "![Learning Rate](figures/learning_rate.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Polynomial Regression\n",
    "\n",
    "![Polynomial Regression](figures/polynomial_regression.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Polynomial Regression\n",
    "\n",
    "In order to do higher order polynomial regression, we can use linear models trained on nonlinear functions of data via a mapping, $\\phi$.\n",
    "\n",
    "- Speed of linear model computation\n",
    "- Fit a wider range of data or functions\n",
    "- But remember: polynomials aren’t the only functions to fit\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Consider the standard linear regression case:\n",
    "\n",
    "$$\\hat{y}(w,x) = w_0+\\sum_{i=0}^n(w_ix_i)$$\n",
    "\n",
    "The quadratic case (polynomial degree = 2) is:\n",
    "\n",
    "$$\\hat{y}(w,v,x) = w_0+\\sum_{i=0}^n(w_ix_i)+\\sum_{i=0}^n(v_ix_i^2)$$\n",
    "\n",
    "We can simplify this by defining a _mapping_, $\\phi$ that transforms our feature space:\n",
    "\n",
    "$$\\phi([x_0,...,x_n]) = [x_0,...,x_n,x_0^2,...,x_n^2]$$\n",
    "\n",
    "At which point we can apply our standard linear models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.preprocessing import PolynomialFeatures \n",
    "\n",
    "model = Pipeline([\n",
    "    ('quad', PolynomialFeatures(2)),\n",
    "    ('regr', LinearRegression()), \n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Residuals\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Residuals Plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Prediction Error Plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Coefficient of Determination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Multicollinearity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Rank 2D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Vector Norms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Ridge Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## LASSO Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ElasticNet Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Alpha Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Other Regression Models"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}