lecture_23_05_22_dnn.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Basic setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Create anaconda environment\n",
    "<br>\n",
    "```bash\n",
    "conda create -n ml python=3.7.5 jupyter\n",
    "```\n",
    "Install fastai library\n",
    "<br>\n",
    "```bash\n",
    "conda install -c pytorch -c fastai fastai\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Deep Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Linear Regression\n",
    "<br>\n",
    "$$\n",
    "F(x) = Wx + b\n",
    "$$\n",
    "<br>\n",
    "where:\n",
    "$\\begin{align}\n",
    "    W &= (W_{1}, W_{2}, \\dots, W_{m} \\in \\mathbb{R}^{m} (\\mathbb{R}^{m \\times 1})\n",
    "    \\text{, }\n",
    "    x &= \\begin{pmatrix}\n",
    "           x_{1} \\\\\n",
    "           x_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           x_{m}\n",
    "         \\end{pmatrix} \\in \\mathbb{R}^{1 \\times m}\n",
    "         \\text{ and }\n",
    "    b \\in \\mathbb{R}^{1}\n",
    "\\end{align}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Logistic Regression\n",
    "\n",
    "$$\n",
    "F(x) = \\sigma(Wx + b)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/dnn/logistic_classifier.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Neural networks can be considered as cascade or pipeline of linear classifiers or regressors. For instance:\n",
    "Let $X \\in \\mathbb{R}^m$ be an our data and $Y \\in \\mathbb{R}^n$ be a classes. Define $H \\in \\mathbb{R}^K$ and $\\phi_{i}:X \\to H_i$ is a linear function:\n",
    "<br>\n",
    "$$z_{i} = \\sum_{j=1}^kW_{i,j}x_j + b_i$$ or \n",
    "<br>\n",
    "$$z_i = W_ix + b$$\n",
    "<br>\n",
    "where \n",
    "$\\begin{align}\n",
    "    x &= \\begin{pmatrix}\n",
    "           x_{1} \\\\\n",
    "           x_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           x_{m}\n",
    "         \\end{pmatrix} \\in \\mathbb{R}^{1 \\times m}\n",
    "         \\text{, }\n",
    "    b &= \\begin{pmatrix}\n",
    "           b_{1} \\\\\n",
    "           b_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           b_{n}\n",
    "         \\end{pmatrix} \\in \\mathbb{R}^{1 \\times n}\n",
    "\\end{align}$ \n",
    "and $W_i = (W_{i,1}, W_{i,2}, \\dots, W_{i,m}) \\in \\mathbb{R}^{m \\times 1}$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/dnn_logistic.png\" alt=\"Logistic Classifier Deep Neural Network\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "f(x) = Wx + b\n",
    "$$\n",
    "<br>\n",
    "here \n",
    "$\\begin{align}\n",
    "    x &= \\begin{pmatrix}\n",
    "           x_{1} \\\\\n",
    "           x_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           x_{m}\n",
    "         \\end{pmatrix} \\in \\mathbb{R}^{1 \\times m}\n",
    "  \\text{,    }\n",
    "    b &= \\begin{pmatrix}\n",
    "           b_{1} \\\\\n",
    "           b_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           b_{n}\n",
    "         \\end{pmatrix} \\in \\mathbb{R}^{1 \\times n}\n",
    " \\text{ and }\n",
    "    W &= \\begin{pmatrix}\n",
    "           W_{1, 1}, W_{1, 2} \\dots W_{1, m} \\\\\n",
    "           W_{2, 1}, W_{2, 2} \\dots W_{2, m} \\\\\n",
    "           \\vdots \\\\\n",
    "           W_{n,1}, W_{n, 2} \\dots W_{n, m}\n",
    "         \\end{pmatrix} \\in \\mathbb{R}^{n \\times m}\n",
    " \\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now consider other mapping $\\sigma:H \\to A$ where $A \\in \\mathbb{R}^n$\n",
    "<br>\n",
    "$$\n",
    "\\begin{align}\n",
    "   \\sigma \\colon \\begin{pmatrix}\n",
    "           x_{1} \\\\\n",
    "           x_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           x_{m}\n",
    "         \\end{pmatrix}\n",
    "    \\mapsto \\begin{pmatrix}\n",
    "           \\sigma(x_{1}) \\\\\n",
    "           \\sigma(x_{2}) \\\\\n",
    "           \\vdots \\\\\n",
    "           \\sigma(x_{m})\n",
    "     \\end{pmatrix}\n",
    "     =& \\begin{pmatrix}\n",
    "           a_{1} \\\\\n",
    "           a_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           a_{m}\n",
    "     \\end{pmatrix}\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/deep-neural-network-1.jpg\" alt=\"Deep Neural Network\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\\sigma(x) = \\frac{1}{1+e^{-x}}$$\n",
    "Sigmoid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/sigmoid.png\" alt=\"Sigmoid\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\\sigma(x)=\\frac{1-e^{-x}}{1+e^{-x}}$$\n",
    "Tahn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/tanh.png\" alt=\"Tanh\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\\sigma(x) = max(0, x)$$\n",
    "ReLu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/relu.png\" alt=\"Relu\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "f(x) = \\begin{cases}\n",
    "    x & \\text{if } x > 0, \\\\\n",
    "    0.01x & \\text{otherwise}.\n",
    "\\end{cases}\n",
    "$$\n",
    "Leaky ReLu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/leaky_relu.png\" alt=\"Leaky ReLU\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why DNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Why deep neural networks?\n",
    "- Dimensionality reduction\n",
    "- Multi model (ensemble)\n",
    "- Features extractor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Still why should they work?\n",
    "- Needs more data\n",
    "- Computationaly expensive training and inference\n",
    "- Black box"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In case of kernel methods, linear regression, random forest or gradient boosting, there exists methods for analysis why model should work. But for DNN we don't have such a vivid imagination."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So, why we choose DNN anyway"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Universal Approximation Theorems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Theorem (The Universal Approximation Theorem):\n",
    "<br>\n",
    "For every $\\sigma:\\mathbb{R}\\to\\mathbb{R}$ bounded, and continuous function (called the activation function). Let $I_m$ denote the m-dimensional unit hypercube $[0,1]^m$ The space of real-valued continuous functions on \n",
    "$I_{m}$ is denoted by \n",
    "$C(I_{m})$. Then, given any $\\varepsilon >0$ and any function $f\\in C(I_{m})$, there exist an integer $N$, real constants $v_{i},b_{i}\\in \\mathbb {R}$ and real vectors $w_{i}\\in \\mathbb {R} ^{m}$ for $i=1,\\ldots ,N$, such that we may define:\n",
    "<br>\n",
    "$$\n",
    "F( x ) = \\sum_{i=1}^{N} v_i \\sigma \\left( w_i^T x + b_i\\right)\n",
    "$$\n",
    "<br>\n",
    "as an approximate realization of the function $f$; that is,\n",
    "<br>\n",
    "$$\n",
    "|F(x)-f(x)|<\\varepsilon\n",
    "$$\n",
    "<br>\n",
    "for all $x\\in I_{m}$. In other words, functions of the form $F(x)$ are dense in $\\displaystyle C(I_{m})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Theorem (The Universal Approximation Theorem for any Compact)\n",
    "<br>\n",
    "For every $\\sigma:\\mathbb{R}\\to\\mathbb{R}$ bounded, and continuous function (called the activation function). Let $K \\in \\mathbb{R}^m$ denote the any compact in $\\mathbb{R}^m$ The space of real-valued continuous functions on \n",
    "$K$ is denoted by \n",
    "$C(K)$. Then, given any $\\varepsilon >0$ and any function $f\\in C(K)$, there exist an integer $N$, real constants $v_{i},b_{i}\\in \\mathbb {R}$ and real vectors $w_{i}\\in \\mathbb {R} ^{m}$ for $i=1,\\ldots ,N$, such that we may define:\n",
    "<br>\n",
    "$$\n",
    "F( x ) = \\sum_{i=1}^{N} v_i \\sigma \\left( w_i^T x + b_i\\right)\n",
    "$$\n",
    "<br>\n",
    "as an approximate realization of the function $f$; that is,\n",
    "<br>\n",
    "$$\n",
    "|F(x)-f(x)|<\\varepsilon\n",
    "$$\n",
    "<br>\n",
    "for all $x\\in I_{m}$. In other words, functions of the form $F(x)$ are dense in $\\displaystyle C(K)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Theorem (Bounded case)\n",
    "<br>\n",
    "The universal approximation theorem for width-bounded networks can be expressed mathematically as follows:\n",
    "\n",
    "For any Lebesgue-integrable function \n",
    "$f:\\mathbb {R} ^{n}\\rightarrow \\mathbb {R}$ and any $\\epsilon >0$, there exists a fully-connected ReLU network \n",
    "$\\mathcal {A}$ with width $d_{m}\\leq {n+4}$, such that the function \n",
    "$F_{\\mathcal {A}}$ represented by this network satisfies\n",
    "<br>\n",
    "$$ \n",
    "\\int _{\\mathbb {R} ^{n}}\\left|f(x)-F_{\\mathcal {A}}(x)\\right|\\mathrm {d} x<\\epsilon\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Definitions and Notions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets define weights per layer $l$ as $W^l$:\n",
    "<br>\n",
    "$$\n",
    "\\begin{align}\n",
    "    W^l &= \\begin{pmatrix}\n",
    "           W_{1, 1}^l, W_{1, 2}^l \\dots W_{1, m^l}^l \\\\\n",
    "           W_{2, 1}^l, W_{2, 2}^l \\dots W_{2, m^l}^l \\\\\n",
    "           \\vdots \\\\\n",
    "           W_{n^l,1}^l, W_{n^l, 2}^l \\dots W_{n^l, m^l}^l\n",
    "         \\end{pmatrix} \\in \\mathbb{R}^{n^l \\times m^l}\n",
    " \\end{align}\n",
    "$$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "F(x) = \\sigma(W^{L-1}(\\dots \\sigma(W^2\\sigma(W^1x + b^1) + b^2)\\dots)) + b^{L-1}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We denote \n",
    "$$a^l = \\sigma(W^la^{l-1} + b^l)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "and\n",
    "<br>\n",
    "$$\n",
    "z^l = W^la^{l-1} + b^l\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or\n",
    "<br>\n",
    "$$\n",
    "f^l(a^{l-1}) = W^la^{l-1} +b^l\n",
    "$$\n",
    "<br>\n",
    "The linear function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have a $n^{L-1}$ (hyperparameter alarm) dimensional vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Different Architectures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Weights sharing:\n",
    "We can restrict some weights between layer to be equal during the training (they change equally)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/weight_sharing_1.jpg\" alt=\"Weights sharing\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/weight_sharing_2.png\" alt=\"Weights sharing in details\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Example of weights sharing is CNN layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual connections: Have skip connections between (among) layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/residual_1.jpeg\" alt=\"Residual\" height=\"400\" width=\"600\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/residual_2.png\" alt=\"Residual in details\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/residual_3.png\" alt=\"Residual multi connections\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/residual_4.png\" alt=\"Dense blocks\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Other operations over layers:\n",
    "UNet and Feature Pyramid Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/unet.png\" alt=\"UNet\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/fpn.png\" alt=\"FPN\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recurrent neural networks (RNN):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/rnn.png\" alt=\"RNN\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "LSTM, GRU Gates:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/LSTM.png\" alt=\"LSTM\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/gru.png\" alt=\"GRU\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss landscape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand loss can have a different surfaces, because of many parameters, chance that many of them has the same direction is low:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/loss_ld_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss landscape and optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Non smooth surface:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ls_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Smooth surface:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ls_2.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Make step optimization (moving average):\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lsgd_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Make surface optimization (landscape):\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lsgd_2.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Local Extrema vs Saddle Points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "According to the nature of neural network as a function, probability to get to the local extrema is very low:\n",
    "$$\n",
    "p = 2^{-L}\n",
    "$$\n",
    "\n",
    "Or when number of layers is high (in modern architectures):\n",
    "$$\n",
    "p = 2^{-160000000}\n",
    "$$\n",
    "\n",
    "Which is almost $0$, but probability to get to the suddle point is very very high.\n",
    "\n",
    "So our task is to get to the good enough (wide) plateau where loss satisfies our needs.\n",
    "\n",
    "Wide plateau means that model will be more robust and stable for changes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Because of mini-bach approach, think about wide plateau as a stable for mini-baches and for other changes.\n",
    "\n",
    "Function as the same for batch training, but for minibatch we have different but similar functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training Deep Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Reverse mode differentiation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Problem with $\\frac{\\partial L}{\\partial W_{i, j}^{l}}$ (or $\\frac{\\partial L}{\\partial b_{i}^{l}}$)\n",
    "Modern neural networks has more than 100000000 or even more than 200000000 parameters and hierarchical nature. i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can consider deep neural network as composition\n",
    "<br>\n",
    "$$\n",
    "F = \\sigma \\circ f^L \\circ \\sigma \\circ f^{L-1} \\circ \\dots \\circ \\sigma \\circ f^2 \\circ \\sigma \\circ f^1\n",
    "$$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "From the compositionality (and chain rule) we have to make the same multiplications multiply times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/composition1.jpg\" arc=\"Composition of functions\"  height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Reverse mode differentiation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/bp1.png\" arc=\"Composition of functions\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The same problem with DNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dnn/bp2.gif\" arc=\"Composition of functions\" height=\"600\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Backpropagation\n",
    "\n",
    "<a href=\"http://neuralnetworksanddeeplearning.com/\">Neural Networks and Deep Learning</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Hadamard Product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Elementwise product of two matrices $A, B \\in \\mathbb{R}^{n \\times m}$\n",
    "<br>\n",
    "$$\n",
    "C = A \\cdot B\n",
    "$$\n",
    "<br>\n",
    "where $C_{i, j} = A_{i, j}B_{i_j}$ and $C \\in \\mathbb{R}^{n \\times m}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "A = \n",
    "\\begin{pmatrix}\n",
    "           A_{1, 1}, A_{1, 2} \\dots A_{1, m} \\\\\n",
    "           A_{2, 1}, A_{2, 2} \\dots A_{2, m} \\\\\n",
    "           \\vdots \\\\\n",
    "           A_{n,1}, A_{n, 2} \\dots A_{n, m}\n",
    "\\end{pmatrix}\n",
    "\\text{, } \n",
    "B = \n",
    "\\begin{pmatrix}\n",
    "           B_{1, 1}, B_{1, 2} \\dots B_{1, m} \\\\\n",
    "           B_{2, 1}, B_{2, 2} \\dots B_{2, m} \\\\\n",
    "           \\vdots \\\\\n",
    "           B_{n,1}, B_{n, 2} \\dots B_{n, m}\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "A \\cdot B = \n",
    "\\begin{pmatrix}\n",
    "           A_{1, 1}B_{1, 1}, A_{1, 2}B_{1, 2} \\dots A_{1, m}B_{1, m} \\\\\n",
    "           A_{2, 1}B_{2, 1}, A_{2, 2}B_{2, 2} \\dots A_{2, m}B_{2, m} \\\\\n",
    "           \\vdots \\\\\n",
    "           A_{n,1}B_{n,1}, A_{n, 2}B_{n,2} \\dots A_{n, m}B_{n,m}\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Major Formulas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For formulas below assume that we fixed the input variable $x$ and treat or model and cost function as function of $W$ and $b$ variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In practice we make our error (gradient) calculation for each $x$ and then mean them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Later we will try to implement everything from scratch using only the NumPy library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's denote\n",
    "$$\n",
    "\\delta^l_j \\equiv \\frac{\\partial C}{\\partial z^l_j}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For $l = L$ we'll get\n",
    "$$\n",
    "\\delta^L_j = \\frac{\\partial C}{\\partial a^L_j} \\sigma'(z^L_j).\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider our $\\delta^L$ as a vector:\n",
    "$$\n",
    "\\delta^L = \\nabla_a C \\odot \\sigma'(z^L).\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now, each $\\delta^l$ is dependent on previous (backwards) calculated $\\delta^{l+1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "\\delta^l_j = \\frac{\\partial C}{\\partial z^l_j}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "\\delta^l_j = \\sum_k \\frac{\\partial C}{\\partial z^{l+1}_k} \\frac{\\partial z^{l+1}_k}{\\partial z^l_j}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "\\delta^l_j = \\sum_k \\frac{\\partial z^{l+1}_k}{\\partial z^l_j} \\delta^{l+1}_k\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "z^{l+1}_k = \\sum_j w^{l+1}_{kj} a^l_j +b^{l+1}_k = \\sum_j w^{l+1}_{kj} \\sigma(z^l_j) +b^{l+1}_k\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Differentiate this\n",
    "<br>\n",
    "$$\n",
    "\\frac{\\partial z^{l+1}_k}{\\partial z^l_j} = w^{l+1}_{kj} \\sigma'(z^l_j)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So put everithing together and we'll get\n",
    "<br>\n",
    "$$\n",
    "\\delta^l_j = \\sum_k w^{l+1}_{kj}  \\delta^{l+1}_k \\sigma'(z^l_j)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So for vectorized form\n",
    "<br>\n",
    "$$\n",
    "\\delta^l = (\\sum_k w^{l+1}_{k1}  \\delta^{l+1}_k \\sigma'(z^l_1), \\sum_k w^{l+1}_{k2}  \\delta^{l+1}_k \\sigma'(z^l_2), \\dots, \\sum_k w^{l+1}_{kp}  \\delta^{l+1}_k \\sigma'(z^l_p))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or on the other hand:\n",
    "$$\n",
    "\\delta^l = ((w^{l+1})^T \\delta^{l+1}) \\odot \\sigma'(z^l)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have something similar to feed forward here but in opposite direction "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's now turn to the our main point:\n",
    "How can we calculate $\\frac{\\partial C}{\\partial W^l_{i,j}}$ and $\\frac{\\partial C}{\\partial b^i}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Turns out:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial w^l_{jk}} = a^{l-1}_k \\delta^l_j\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "or vectorized form:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial w} = a_{\\rm in} \\delta_{\\rm out}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "and:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial b^l_j} = \\delta^l_j\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "or vectorized as well\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial b} = \\delta\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial C}{\\partial w^l_{jk}} = \\frac{\\partial C}{\\partial z^l_{j}}\\frac{\\partial z^l_k}{\\partial w^l_{jk}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial C}{\\partial z^l_{j}} = \\delta^l_j\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Proof for Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial z^l_k}{\\partial w^l_{jk}} = \\frac{\\partial (\\sum_{i=1}^{m^l}{a_i^{l-1}w_{ij}})}{\\partial w^l_{jk}} = a_k^{l-1}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have $\\frac{\\partial C}{\\partial z^l_{j}} = \\delta^l_j$ and $\\frac{\\partial z^l_k}{\\partial w^l_{jk}} = a_k^{l-1}$ by which we conclude:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial w^l_{jk}} = a^{l-1}_k \\delta^l_j\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Proof for Biases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial C}{\\partial b^l_j} = \\delta^l_j\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Try!!!!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Thank You"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}