lecture_31_05_2022_opt_reg.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Basic setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Create anaconda environment\n",
    "<br>\n",
    "```bash\n",
    "conda create -n ml python=3.7.5 jupyter\n",
    "```\n",
    "Install fastai library\n",
    "<br>\n",
    "```bash\n",
    "conda install -c pytorch -c fastai fastai\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exponential moving average"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Define data as $(y_1, y_2, \\dots, y_n)$ peaks of function $f:X \\to Y$ where for some $(x_1, x_2, \\dots, x_n)$, $f(x_1) = y_1, f(x_2) = y_2, \\dots, f(x_n) = y_n$ and define\n",
    "<br>\n",
    "$$v_1 = (1 - \\beta)y_1,  \\\\\n",
    "v_2 = \\beta v_1 + (1 - \\beta)y_2 \\\\\n",
    "\\dots \\\\\n",
    "v_n = \\beta v_{n-1} + (1 - \\beta)y_n \\\\\n",
    "$$ \n",
    "</br>\n",
    "for $\\beta \\in [0, 1]$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/ema_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now lets try higher $\\beta$\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ema_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Try with lower $\\beta$\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ema_3.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Bias on early stage:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ema_4.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For early stages, worm up, better will be if we use, so called bias correction:\n",
    "$$\n",
    "v^{corr}_t =\\frac{v_t}{1 - \\beta^t}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Different loss functions and noisy gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Applied machine learning is hardly experiment base and needs many experiments while achieving the stable result. On the other hand, DL models need \"big\" data for training and full batch processing is almost never possible.\n",
    "<br>\n",
    "For instance $100000$ examples or even more than $1000000$ can't be feat in to the one batch in GPU memory and iteratively calculation is too slow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In batch gradient descent loss goes down per iteration, if not, than maybe learning rate is too big or other:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/bt_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "From the different point of view, gradients are pretty similar at each epoch and if we stack in local extrema or saddle point, we can stay there for long:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/bt_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Generally data is divided in mini-batches and model is trained on that mini-datasets with batch gradient descent:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/bt_3.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But to match noise can increase the training time significatly:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/bt_4.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Noisy gradient descent:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ng_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We need to somehow speedup the training time (the amount of experiments might be large):\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ng_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Loss function is often overcomplicated rather that MSE with convex landscape:\n",
    "\n",
    "For probability activations usually softmax is used:\n",
    "$$\n",
    "\\sigma(x) = \\frac{e^{x_j}}{\\sum_i{e^{x_i}}}\n",
    "$$\n",
    "<br>\n",
    "with cross entropy loss:\n",
    "$$\n",
    "C = -\\sum_{i}^{C}t_{i} log (s_{i}) = -log\\left ( \\frac{e^{s_{p}}}{\\sum_{j}^{C} e^{s_{j}}} \\right )\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Training deep learning models might use combination of many loss functions:\n",
    "\n",
    "- Object detection: bounding box regression plus (weighted) cross entropy for image classification.\n",
    "- Instance segmentation: bounding box regression plus (weighted) pixel binary classification plus (weighted) cross entropy for classification "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different losses combined for different tasks:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/loss_odcs_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Detect, segment and classify at once:\n",
    "\n",
    "Different losses combined for different tasks:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/loss_odcs_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand loss can have a different surfaces, because of many parameters, chance that many of them has the same direction is low:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/loss_ld_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss landscape and optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Non smooth surface:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ls_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Smooth surface:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ls_2.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Make step optimization (moving average):\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lsgd_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Make surface optimization (landscape):\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lsgd_2.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Gradient descent with momentum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall gradient descent algorithm, for some cost function $C$ and learning rate $\\alpha$ we do parameters (weights) update by:\n",
    "$$\n",
    "W^l_{i,j} = W^l_{i,j} - \\alpha \\frac{\\partial{C}}{\\partial{W^l_{i,j}}}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b^l_{j} = b^l_{j} - \\alpha \\frac{\\partial{C}}{\\partial{W^l_{j}}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If the cost function $C$ is defined, denote partial derivative with respect of weights: \n",
    "$$\n",
    "d{W} = \\nabla_{W}C\n",
    "$$\n",
    "<br>\n",
    "and with respect of biases:\n",
    "$$\n",
    "d{b} = \\nabla_{b}C\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So our gradient descent optimization can be written as:\n",
    "$$\n",
    "W^l_{i,j} = W^l_{i,j} - \\alpha \\partial{W^l_{i,j}}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b^l_{j} = b^l_{j} - \\alpha \\partial{b^l_{j}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or in general for batch or mini-batch gradient descent\n",
    "optimization can be written as:\n",
    "$$\n",
    "W = W - \\alpha d{W}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha d{b}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/gd_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For each iteration, compute $d W$ and $d b$ for the current mini-batch "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we have a picks and higher is oscillation slower the optimization is.\n",
    "We can use moving average in order to reduce horizontal variance and increase speed of optimization:\n",
    "$$\n",
    "V_{d W} = \\beta V_{d W} + (1 - \\beta)d W\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "V_{d b} = \\beta V_{d b} + (1 - \\beta)d b\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can use this averages in gradient descent optimization instead of direct gradients:\n",
    "$$\n",
    "W = W - \\alpha V_{d W}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha V_{d b}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/gd_mom_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RMSProp optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<b>Root Mean Square Prop</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For each iteration, compute $d W$ and $d b$ for the current mini-batch "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Calculate momentum with changes:\n",
    "$$\n",
    "S_{d W} = \\beta S_{d W} + (1 - \\beta)d W^2 \\text{ elementwise}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "S_{d b} = \\beta S_{d b} + (1 - \\beta)d b^2 \\text{ elementwise}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now update the weights and biases by:\n",
    "$$\n",
    "W = W - \\alpha \\frac{d W}{\\sqrt{S_{d W}}}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha \\frac{d b}{\\sqrt{S_{d b}}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/rms_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "RMSProp steps:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/rms_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here if $S_{d W}$ is large, it means that step will be forward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To avoid zero division error (if $\\sqrt{S_{d W}}$ or $\\sqrt{S_{d b}}$ is almost zero) we can add small $\\epsilon$ to the denominators:\n",
    "$$\n",
    "W = W - \\alpha \\frac{d W}{\\sqrt{S_{d W}} + \\epsilon}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha \\frac{d b}{\\sqrt{S_{d b}} + \\epsilon}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here $\\epsilon = 10^{-8}$ for instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Adam optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<b>Adaptive Moment Estimation </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For Adam optimization algorithm we will combine Momentum and RMSProp together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Set $V_{d W} = 0$, $S_{d W} = 0$, $V_{d b} = 0$, $S_{d b} = 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For each iteration, compute $d W$ and $d b$ for the current mini-batch "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First calculate momentums $V_{d W}$ and $V_{d b}$ with hyperparameter $\\beta_1$:\n",
    "$$\n",
    "V_{d W} = \\beta_1 V_{d W} + (1 - \\beta_1)d W\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "V_{d b} = \\beta_1 V_{d b} + (1 - \\beta_1)d b\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now calculate $S_{d W}$ and $S_{d b}$ with hyperparameter $\\beta_2$:\n",
    "$$\n",
    "S_{d W} = \\beta_2 S_{d W} + (1 - \\beta_2)d W^2 \\text{ elementwise}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "S_{d b} = \\beta_2 S_{d b} + (1 - \\beta_2)d b^2 \\text{ elementwise}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the current iteration, let's say $t$, calculate so called bias corrections for momentum:\n",
    "$$\n",
    "V^{corr}_{d W} = \\frac {V_{d W}}{1 - \\beta_1^t}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "V^{corr}_{d b} = \\frac{V_{d b}}{{1 - \\beta_1^t}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The same correction for RMSProp parameters:\n",
    "$$\n",
    "S^{corr}_{d W} = \\frac {S_{d W}}{1 - \\beta_2^t}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "S^{corr}_{d b} = \\frac{S_{d b}}{{1 - \\beta_2^t}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now lets update weights and biases:\n",
    "$$\n",
    "W = W - \\alpha \\frac{V^{corr}_{d W}}{\\sqrt{S^{corr}_{d W}} + \\epsilon}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha \\frac{V^{corr}_{d b}}{\\sqrt{S^{corr}_{d b}} + \\epsilon}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This algorithms combines the Momentum effect together with RMSProp optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We than have a number of hyperparameters:\n",
    "- The learning rate $\\alpha$\n",
    "- Momentum parameter $\\beta_1$ for derivatives $d W$ and $d b$ with common choice $0.9$\n",
    "- RMSProp parameter $\\beta_2$ for $d W^2$ and $d b^2$ squares with common choice $0.999$\n",
    "- $\\epsilon = 10^{-8}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Almost always $\\alpha$ is tunned and almost never other parameters, the values above was recommended by the authors of the Adam paper and I personally don't remember that other values gave me any positive effect in improvements of performance (optimization) if not negative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The $\\beta_1$ is considered for first moment and $\\beta_2$ for second moment and that's why optimizer has the name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Visualization of different optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Optimizers on surface:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/gds_1.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Optimizers on projection:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/gds_2.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning rate decay"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Near the extrema points, large learning rate might cause so called bouncing gradient:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lrdc_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In order to avoid this problem, we can decrease learning rate per iteration, or epochs:\n",
    "\n",
    "$$\n",
    "\\alpha = \\frac{1}{1 + dr \\cdot ep} \\cdot \\alpha_0\n",
    "$$\n",
    "<br>\n",
    "- $dr$ - is the decay rate\n",
    "- $ep$ - is the epoch number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or square root decay:\n",
    "\n",
    "$$\n",
    "\\alpha = \\frac{k}{\\sqrt{ep}} \\cdot \\alpha_0\n",
    "$$\n",
    "<br>\n",
    "- $k$ - is the another hyperparameter\n",
    "- $ep$ - is the epoch number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or mini-batch decay:\n",
    "\n",
    "$$\n",
    "\\alpha = \\frac{k}{\\sqrt{t}} \\cdot \\alpha_0\n",
    "$$\n",
    "<br>\n",
    "- $k$ - is the another hyperparameter\n",
    "- $t$ - mini-batch number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Stepwise decay:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lrdc_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Local optimum and saddle points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On early stages of deep learning, people thought that local optimum was the main problem of optimization. But for local optima we need that each \n",
    "$$\\frac{\\partial C}{\\partial W^l_{i,j}}$$ \n",
    "direction to be the same which for instance of $100000000$ or even $200000000$ parameters might have a $2^{-100000000}$ or $2^{-200000000}$ probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Local optima in lower dimensional case:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lo_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<b> Analysis in lower dimensions often does not generalize for higher dimensions </b>\n",
    "\n",
    "- The probability of local optima is low\n",
    "- But the probability of saddle points is high"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Local optima vs saddle points in higher dimensions:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lo_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Saddle points and plateaus:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/saddle_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main goal of optimization algorithms is to escape plateaus:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/saddle_2.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## One cycle policy CycleLR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In order to avoid saddle point trap, we can increase learning rate during the epoch and then decrease it with some linear or non linear (exponential) function:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/one_cycle_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One cycle policy with non linear learning rate change:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/one_cycle_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can use this technique to find initial learning rate before training:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lr_find_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/lr_find_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One cycle policy speeds up learning process significantly and reduces the chances to get in to the saddle point or bad extrema point trap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are many techniques of learning rate manipulation during the training:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/one_cycle_mx_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or even:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/one_cycle_mx_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<b>\n",
    "Any modification we make to a learning algorithm that is intended to reduce its generalization error but not it's training error\n",
    "</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There been many experiments for regularization of deep neural networks, to be more precise we split regularization techniques in two major approaches:\n",
    "- Cost functions penalty (soft influence)\n",
    "- Parameters regularization directly (hard penalization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For cost function norm regularization:\n",
    "\n",
    "$$\n",
    "\\tilde C(X, y, W, b) = C(X, y, W, b) + \\lambda \\Omega(W)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Biases learn from few data and are not regularized generally. They have a lower dimension than weights. Often regularize biases causes the underfitting because their low dimensionality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In some cases $\\lambda$ parameter is used per layer, but to avoid to many hyperparameters, we set the same parameter for each layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Weight decay - L2 regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets consider:\n",
    "$$\n",
    "\\tilde C(X, y, W, b) = C(X, y, W, b) + \\lambda \\Omega(W)\n",
    "$$\n",
    "with\n",
    "$$\n",
    "\\Omega(W) = \\frac{1}{2}||W||_2^2\n",
    "$$\n",
    "Here $W$ is considered as a vector of all weights from all layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In literature it is called as <b>Ridge regression</b> or <b>Tikhonov</b> regularization.\n",
    "<br>\n",
    "In deep learning it's called <b>Weight decay</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Study of gradient of the regularized function (assume that we don't have biases)\n",
    "$$\n",
    "\\tilde C(X, y, W) = \\frac{\\lambda}{2}W^TW + C(X, y, W)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With the corresponding parameter gradient:\n",
    "$$\n",
    "\\nabla_w C(X, y, W, b) = \\lambda w + \\nabla_w C(X, y, W, b)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then for gradient descent step:\n",
    "$$\n",
    "w \\leftarrow w - \\alpha (\\lambda w + \\nabla_w C(X, y, W, b)) = w - \\alpha \\lambda w - \\alpha \\nabla_w C(X, y, W, b)\n",
    "$$\n",
    "or\n",
    "$$\n",
    "w \\leftarrow (1 - \\alpha \\lambda)w - \\alpha \\nabla_w C(X, y, W, b)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As might be observed we have a weights shrink on each step by the constant factor $1 - \\alpha \\lambda$ before gradient update.\n",
    "<br>\n",
    "So weights are around the zero, they might be around any value (by the linear nature of neural networks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Analysis of weight decay (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/memes/technical.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For quadratic cost $C$ and $w^* = argmin_{w}(C(W))$ Taylor approximation gives:\n",
    "$$\n",
    "\\tilde C(w) \\approx C(w^*) + \\frac{1}{2}(w - w^*)^TH(w - w^*)\n",
    "$$\n",
    "Where $H$ is Hessian matrix of $C$ on the specific argument $W^*$, the first part of $\\nabla C(w^*) = 0$ series because of $w^*$ is minimal "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then for minimum of $\\tilde C(w)$ in $\\tilde w$ we have that:\n",
    "$$\n",
    "\\nabla_w \\tilde C(w) = H(w - w^*)\n",
    "$$\n",
    "should be $0$\n",
    "$$\n",
    "H(\\tilde w - w^*) = 0\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we add weight decay to $\\tilde C(W)$ and calculate it's gradient:\n",
    "$$\n",
    "\\lambda \\tilde w + H(\\tilde w - w^*) = 0\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "(H + \\lambda I)\\tilde w = H w^*\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "\\tilde w = (H + \\lambda I)^{-1}H w^*\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "According to the fact that the Hessian matrix of a convex function is positive semi-definite and (Schwarz's theorem) symmetric, thus we can decompose it as $H = Q \\Lambda Q^T$ of orthonormal matrices and eigenvalues:\n",
    "$$\n",
    "\\tilde w = (Q \\Lambda Q^T + I \\lambda)^{-1} Q \\Lambda Q^T w^* = (Q(\\Lambda + \\lambda I)Q^T)^{-1}Q \\Lambda Q^T w^* = \\\\\n",
    "Q(\\Lambda + \\lambda I)^{-1}\\Lambda Q^T w^*\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This means that for \n",
    "$$\n",
    "\\Lambda = \\begin{bmatrix}\n",
    "  l_1 & 0 & \\cdots & 0 \\\\\n",
    "  0 & l_2 & \\cdots & 0 \\\\\n",
    "  \\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "  0 & 0 & \\cdots & l_n\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Thus:\n",
    "$$\n",
    "(\\Lambda + \\lambda I)^{-1}\\Lambda = \n",
    "\\begin{bmatrix}\n",
    "  \\frac{l_1}{\\lambda + l_1} & 0 & \\cdots & 0 \\\\\n",
    "  0 & \\frac{l_2}{\\lambda + l_2} & \\cdots & 0 \\\\\n",
    "  \\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "  0 & 0 & \\cdots & \\frac{l_n}{\\lambda + l_n}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Each value is rescaled of factor $\\frac{l_i}{\\lambda + l_i}$ \n",
    "Along the directions where the eigenvalues of $H$ relatively large, for example,where $l_i \\gg \\lambda$, the effect of regularization is relatively small. Yet components with $l_i \\ll \\lambda$ will be shrunk to have nearly zero magnitude (Goodfellow at al)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Label smoothing (add noise to target, target augmentation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We will see how data augmentation can help model to generalize better, but there also exists target augmentation techniques. When target is a image (segmentation mask), bounding box, text, graph etc, it might be obvious. But for some times augmentation on probabilities also helps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When training classifier with softmax, instead of training on hard targets $0, 1, 2, \\dots, n$ we can use:\n",
    "$$\n",
    "l_i^{'} = (1- \\epsilon)l_i + \\frac{\\epsilon}{k}\n",
    "$$\n",
    "for some \"small\" $\\epsilon$ where $k$ is a total number of classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When label is hard, softmax, which returns probabilities, almost newer predicts hard labels and model continuous training, and increase weights, in contrary, soft label can reduce training time, and can serve as regularizer of weights. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Trained model with label smoothing is good for classification tasks but performs poorly as a teacher model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall ensemble methods, bagging, they perform better, because different models can not have the same error. On the other hand training the ensemble of deep neural networks is computationally expensive and might need significant enlargement of dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider the similar approach within the single neural network. Switch of activations per layer with probability $p^l$ in mini-batch, which can be considered as random sampling, bootstrapping,  and then average the cost function at the end.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dropout:\n",
    "<div>\n",
    "<img src=\"images/regs/drpt_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When we switch off the activation, we influence all the input weights in previous layers (less when layer is far enough)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dropout:\n",
    "<div>\n",
    "<img src=\"images/regs/drpt_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the end we should multiply activations on the inverse of switch off probability:\n",
    "$$\n",
    "a^l = \\frac{1}{p}a^l\n",
    "$$\n",
    "For instance if (as often is used) half of the layers where switched off, then activations should be: \n",
    "$$\n",
    "a^l = 2 a^l\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Most common implementation of dropout is the inverted dropout:\n",
    "<br>\n",
    "for probability keep-prob - $p = 0.5$ we keep alive only this amount of activations. We create a mask vector of the dimension of activation with randomly selected zeros and ones (with $p$ ones) and then multiply activation element by element on this vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or we can simply use inverted dropout with keep-prob argument (probability to keep active neurons) and divide activations on it:\n",
    "$$\n",
    "a^l = \\frac{a^l}{p}\n",
    "$$\n",
    "For instance if (as often is used) half of the layers where switched off, then keep-prob will be $p=0.5$ activations should be: \n",
    "$$\n",
    "a^l = \\frac{a^l}{0.5} = 2 a^l\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For instance:\n",
    "$$\n",
    "z^l = W^l a^{l-1} + b^l\n",
    "$$\n",
    "In order to approximate the expected value, we divide activations $a^{l-1}$ on keep-prob probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that we eliminate activation elements randomly and different per iteration or per mini-batch which makes dropout one step ensemble method\n",
    "<br>\n",
    "We don't use dropout during the test or validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note: We often vary keep-prob per layer, depending on the layer and number of units, for instance we won't apply dropout to the unit with one layer or with two layers with keep-prob 0.2 might not be wise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dropout can be considered as denoising quality of model (adding the random noise directly in the hidden units)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It is important to understand that a large portion of the power of dropout arises from the fact that the masking noise is applied to the hidden units. This can be seen as a form of highly intelligent, adaptive destruction of the information content of the input rather than destruction of the raw values of the input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For example, if the model learns a hidden unit $h_i$ that detects a face by finding the nose, then dropping $h_i$ corresponds to erasing the information that there is a nose in the image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The model must learn another $h_j$, that either redundantly encodes the presence of a nose or detects the face by another feature, such as the mouth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Traditional noise injection techniques that add unstructured noise at the input are not able to randomly erase the information about a nose from an image of a face unless the magnitude of the noise is so great that nearly all the information in the image is removed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Destroying extracted features rather than original values allows the destruction process to make use of all the knowledge about the input distribution that the model has acquired so far"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Goodfellow at al"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://www.youtube.com/watch?v=kAwF--GJ-ek\">Geoffry Hinton about dropouts</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data augmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Data augmentation helps model in better generalization. It's obvious that more balanced data always gives better results in statistical models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Augmentation should be different in different tasks, for instance for images, there exists random noise, lightning, blurring, rotation, random cropping, center cropping, channel flipping, etc "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Augmentations:\n",
    "<div>\n",
    "<img src=\"images/regs/aug_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Augmentations with labels:\n",
    "<div>\n",
    "<img src=\"images/regs/aug_2.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Augmentations big picture:\n",
    "<div>\n",
    "<img src=\"images/regs/aug_3.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Augmentations cropping:\n",
    "<div>\n",
    "<img src=\"images/regs/aug_4.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Sometimes augmentation depends on task and data even for images, for instance for OCR we can not flip \n",
    "<br>\n",
    "<b>d</b>\n",
    "in opposite case we'll get\n",
    "<br>\n",
    "<b>b</b>\n",
    "and vice-versa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or rotate in familiar MNIST dataset\n",
    "<br>\n",
    "<b>6</b>\n",
    "in opposite case we'll get\n",
    "<br>\n",
    "<b>9</b>\n",
    "and vice-versa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For sound recognition noise augmentation is used, for images there exists denoising autoencoders which learn to draw original image from augmented image (example of self-supervised generative model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Denoising autoencoders:\n",
    "<div>\n",
    "<img src=\"images/regs/dae_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Denoising autoencoder:\n",
    "<div>\n",
    "<img src=\"images/regs/dae_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For text switching words with synonyms, masking (can be considered as denoising autoencoder for language models) works as the state of the art for the language model pre-training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Text masking language model:\n",
    "<div>\n",
    "<img src=\"images/regs/mae_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Text masking:\n",
    "<div>\n",
    "<img src=\"images/regs/mae_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Early stopping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "During the training we use training set and validation set, first for training and second for observation. When both,  training and validation errors go down, no matter if with different speed and velocity, training is going well. But if training error goes down, but validation error starts to increase, this is the sign of overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Example of overfitting:\n",
    "<div>\n",
    "<img src=\"images/regs/ovft_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this case we can stop the training and use last stored parameters, while validation error was going down. In this case we need to store parameters once, every amount a time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Early stopping:\n",
    "<div>\n",
    "<img src=\"images/regs/ovft_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It has downside, that we should save weights after some period (checkpoints), but we won't use this weights in training and thus this procedure have minimal impact on training time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand we can consider early stopping as regularization with minimal effort."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Early stopping can be also considered as prevention of weights increase. Steps are limited so as the trajectory of weights. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Early stopping might happen on insufficient error. We should change the model architecture, augment data more, add more data, use label smoothing or other regularization, model might perform better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Early stopping might be used with other regularization techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Often many regularization techniques are used together in order to achieve the best result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One example of early stopping is learning rate finder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Weights initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are several weight initialization techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider the model:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/regs/wi_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Vanishing or exploding gradients:\n",
    "$$\n",
    "dw^l = dw^{l-1}dw^{l-2} \\dots dw^{1} \n",
    "$$\n",
    "so if each $dw^l$ is small $dw^l\\ll 1$ then gradient becomes near zero and if $dw^l \\gt 1$ then gradient might overflow "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Another case is mean and variance with random initialization:\n",
    "```python\n",
    "# random init\n",
    "w1 = torch.randn(784, 50) \n",
    "b1 = torch.randn(50)\n",
    "w2 = torch.randn(50, 10) \n",
    "b2 = torch.randn(10)\n",
    "w3 = torch.randn(10, 1) \n",
    "b3 = torch.randn(1)\n",
    "def linear(x, w, b):\n",
    "    return x@w + b\n",
    "def relu(x):\n",
    "    return x.clamp_min(0.)\n",
    "t1 = relu(linear(x_valid, w1, b1))\n",
    "t2 = relu(linear(t1, w2, b2))\n",
    "t3 = relu(linear(t2, w3, b3))\n",
    "print(t1.mean(), t1.std())\n",
    "print(t2.mean(), t2.std())\n",
    "print(t3.mean(), t3.std())\n",
    "############# output ##############\n",
    "tensor(13.0542) tensor(17.9457)\n",
    "tensor(93.5488) tensor(113.1659)\n",
    "tensor(336.6660) tensor(208.7496)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Xavier initialization:\n",
    "$$\n",
    "w^l = randn(N^l\\times M^l) \\cdot \\sqrt{\\frac{1}{N^{l-1}}}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b^l = 0\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Initialization depends on the activation function and the layer architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the ReLU activation function \"Kaiming Initialization\" weight initialization works:\n",
    "$$\n",
    "w^l = randn(N^l\\times M^l) \\cdot \\sqrt{\\frac{2}{N^{l-1}}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For tanh activation function better use $1$ instead of $2$ (Xavier initialization):\n",
    "$$\n",
    "w^l = randn(N^l\\times M^l) \\cdot \\sqrt{\\frac{1}{N^{l-1}}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Bengio at al:\n",
    "$$\n",
    "w^l = randn(N^l\\times M^l) \\cdot \\sqrt{\\frac{2}{N^l + N^{l-1}}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://arxiv.org/abs/1706.02515\">Self-Normalizing Neural Networks</a>\n",
    "<br>\n",
    "<a href=\"https://twitter.com/SELUAppendix/status/873882218774528003\" >Self-normalizing neural networks paper</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Features (data) normalization and standartization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For different range of data:\n",
    "- loss function might be asymmetric and cause the slower training. \n",
    "- One feature with large scale might change whole prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we normalize input features with mean and variance:\n",
    "$$\n",
    "\\mu = \\frac{1}{m}\\sum_{i = 1}^{m}X^{(i)}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "\\sigma^2 = \\frac{1}{m}\\sum_{i = 1}^{m}(X^{(i)} - \\mu)^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Normalized features:\n",
    "$$\n",
    "\\tilde X^{(i)} = \\frac{X^{(i)} - \\mu}{\\sigma^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/regs/fn_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then cost function will be more symmetric which reduces training time significantly "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/regs/fn_2.jpg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall the gradient descent:\n",
    "<div>\n",
    "<img src=\"images/regs/fn_3.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand changes in particular feature does not harm the prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/regs/fn_4.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note: Use $\\mu$ and $\\sigma^2$ calculated on training data for validation data and inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If the features have the similar scale, than the cost function will be symmetric, but normalizing in this case won't harm it at all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Batch normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In case of input variables, we normalize features with mean and variance in order to make the training faster, make the cost function more symmetric. We can normalize the outputs of each layer as well to reduce training time further. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Batch normalization gives us wider range for hyperparameter choise and works with \"bigger\" learning rates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In many models batch normalization is used before the activation functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets calculate mean and variance for each layer per batch:\n",
    "$$\n",
    "\\mu^l = \\frac{1}{m}\\sum_{i = 1}^{m}(Z^l)^{(i)}\n",
    "$$\n",
    "and\n",
    "$$\n",
    "(\\sigma^l)^2 = \\frac{1}{m}\\sum_{i = 1}^{m}((Z^l)^{(i)} - \\mu^l)^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now normalize each $Z^l$ as:\n",
    "$$\n",
    "(Z^l_{norm})^{(i)} = \\frac{(Z^l)^{(i)} - \\mu^l}{\\sqrt{(\\sigma^l)^2) + \\epsilon}}\n",
    "$$\n",
    "where $\\epsilon$ is a \"small\" number $10^{-8}$ in order to avoid zero division"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now set another trainable parameters $\\gamma^l$ and $\\beta^l$ as mean and variance:\n",
    "$$\n",
    "(\\tilde Z^l)^{(i)} = \\gamma^l (Z^l_{norm})^{(i)} + \\beta^l\n",
    "$$\n",
    "where $\\gamma$ and $\\beta$ are rearnable parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If $\\gamma^l = \\mu^l$ and $\\beta^l = \\sqrt{(\\sigma^l)^2) + \\epsilon}$ then:\n",
    "$$\n",
    " (\\tilde Z^l)^{(i)} = (Z^l)^{(i)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Why we need this additional parameters?\n",
    "<br>\n",
    "If we have activation function for instance sigmoid, then mean in $0$ and variance $1$ covers only near-linear part of sigmoid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/regs/bn_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can consider batch normalization as an addition to the activation function:\n",
    "$$\n",
    "{bn}^l : Z^l \\to \\tilde Z^l\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "\\tilde a^l = a^l \\circ {bn}^l\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So the $\\gamma^l$ and $\\beta^l$ might be learned as well:\n",
    "$$\n",
    "\\gamma^l = \\gamma^l - \\alpha d\\gamma^l \n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "\\beta^l = \\beta^l - \\alpha d\\beta^l \n",
    "$$\n",
    "or with other optimization algorithms (momentum, RMSProp, Adam) as well"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note: In practice we use mini-batches, and mean and variance for batch norms are calculated for each mini-batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The mean calculation cancels the biases:\n",
    "$$\n",
    "Z^{(i)} - \\mu = W a^{(i)} + b - \\frac{1}{m}\\sum_{i=1}^{m}(W a^{(i)} + b) = W a^{(i)} + b - b - \\frac{1}{m}\\sum_{i=1}^{m}W a^{(i)} = \\\\ \n",
    "W a^{(i)} - \\frac{1}{m}\\sum_{i=1}^{m}W a^{(i)}\n",
    "$$\n",
    "<br>\n",
    "so if we use batch normalization we can remove biases ($b^l = 0$) and rely on $\\beta^l$ parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Batch normalization during the inference (one example instead of batch):\n",
    "- Calculate exponential moving average of mean and variance of all the batches for each layer\n",
    "- Use this averages for normalize $Z^l$ (calculate $Z_{norm}^l$) for each layer\n",
    "- Use learned $\\gamma^l$ and $\\beta^l$ for batch normalization for each layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Why batch normalization works?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Covariant shifting:\n",
    "<div>\n",
    "<img src=\"images/regs/bn_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It reduces the shift of activation distribution per layer and generalizes better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Regularization effect:\n",
    "- Batch norm is calculated on mini-batches\n",
    "- Adds noise per layer as dropout if mini-batches are \"small\" enough\n",
    "- If mini-batches are \"large\" then regularization effect is small\n",
    "- Can be used with other regularizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With batch normalization:\n",
    "<div>\n",
    "<img src=\"images/regs/bn_3.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Without batch normalization:\n",
    "<div>\n",
    "<img src=\"images/regs/bn_4.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Without batch normalization:\n",
    "<div>\n",
    "<img src=\"images/regs/bn_5.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Small batches can be fixed by $\\epsilon$ or keeping moving averages of statistics, FastAI's running batch normalization with exponential moving averages, de-biasing and reducing the variance squeeze"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Why the batch normalization works anyway:\n",
    "<div>\n",
    "<img src=\"images/regs/bn_6.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Training with batch normalization:\n",
    "<div>\n",
    "<img src=\"images/regs/bn_7.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Covariance shift:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/regs/bn_8.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Persistence against noise:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/regs/bn_9.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Loss landscape smoothness:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/regs/bn_10.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In every \"big\" model with \"small\" batch sizes, especially when train starts with scratch, batch normalization won't work. It's also hard for RNN models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Weight normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of batch normalization which has a limitation with smaller batch size, sometimes it's practical to use weights normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The weights normalization:\n",
    "$$\n",
    "W = u \\frac{g}{||u||}\n",
    "$$\n",
    "<br>\n",
    "where $u$ and $g$ are the learnable parameters as in case of batch normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The purpose was to separate weights amount from directions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The use-case familiar to me is the super-resolution based on augmented UNet encoder and decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Mostly layer normalization is used in segmentation tasks, or in detection tasks unless we use pre-trained encoder, than layer normalization is used mostly in decoder because in such a \"big\" models, we can apply only \"small\" size of batches, such as $2$ or even $1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Layer normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the layer normalization, we normalize input layer-wise in height rather than in width"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/regs/ln_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Layer normalization also can be considered as an improvement over the batch-normalization, but still batch normalization is widely used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/regs/ln_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Other normalizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are other normalizations such as: instance normalization, group normalization, etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/regs/on_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But this normalization might be used in some architectures, but still are not as popular as batch normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/regs/on_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Many of them has a concrete realm where they work.  Layer norm in RNN models, instance norm for style-transfer, etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}