Part 3 Gradient Descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align = 'center'> Neural Networks Demystified </h1>\n",
    "<h2 align = 'center'> Part 3: Gradient Descent </h2>\n",
    "\n",
    "\n",
    "<h4 align = 'center' > @stephencwelch </h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"400\"\n",
       "            height=\"300\"\n",
       "            src=\"https://www.youtube.com/embed/5u0jaA3qAGk\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x10354cbd0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('5u0jaA3qAGk')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align = 'center'> Variables </h3>\n",
    "\n",
    "|Code Symbol | Math Symbol | Definition | Dimensions\n",
    "| :-: | :-: | :-: | :-: |\n",
    "|X|$$X$$|Input Data, each row in an example| (numExamples, inputLayerSize)|\n",
    "|y |$$y$$|target data|(numExamples, outputLayerSize)|\n",
    "|W1 | $$W^{(1)}$$ | Layer 1 weights | (inputLayerSize, hiddenLayerSize) |\n",
    "|W2 | $$W^{(2)}$$ | Layer 2 weights | (hiddenLayerSize, outputLayerSize) |\n",
    "|z2 | $$z^{(2)}$$ | Layer 2 activation | (numExamples, hiddenLayerSize) |\n",
    "|a2 | $$a^{(2)}$$ | Layer 2 activity | (numExamples, hiddenLayerSize) |\n",
    "|z3 | $$z^{(3)}$$ | Layer 3 activation | (numExamples, outputLayerSize) |\n",
    "|J | $$J$$ | Cost | (1, outputLayerSize) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last time we built a neural network in python that made really bad predictions of your score on a test based on how many hours you slept and how many hours you studied the night before. This time we'll focus on the theory of making those predictions better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can initialize the network we built last time and pass in our normalized data, X, using our forward method, and have a look at our estimate of y, yHat."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "\n",
    "#Import code from last time:\n",
    "from partTwo import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "NN = Neural_Network()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "yHat = NN.forward(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.55850487],\n",
       "       [ 0.55993739],\n",
       "       [ 0.56372181]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yHat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.75],\n",
       "       [ 0.82],\n",
       "       [ 0.93]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1053532d0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF4xJREFUeJzt3X+M1PWdx/Hn20VBcBFbyHLK1rXWU7i0xZNTco12m/Za\nJLEkJ9FDu4WepuZyHtf+o9fL5dT0j4tp0xjrxcPTQhOSYvrjGqqAXkwnZ0xrSyulFiSSCvKj0i1F\nWNcuSPd9f+zsdhl3d2a/85md+bx5PZKN+535Mnw+vPDN7Gt+mbsjIiKxnNPsBYiISHoa7iIiAWm4\ni4gEpOEuIhKQhruISEAa7iIiAVUd7mb2DTM7Yma/nOCch83sVTP7hZldnXaJIiIyWbXcc18PLBvv\nSjNbDnzA3a8APg88mmhtIiJSUNXh7u7PA8cmOOXTwDfL574IzDGzjjTLExGRIlJ07pcAB0YdHwQW\nJLhdEREpKNUDqlZxrPc0EBFpomkJbuMQ0DnqeEH5sjOYmQa+iEgB7l55B7qqFPfcNwOfBTCzpcCb\n7n5krBPdPezXfffd1/Q1aH/am/YX76uoqvfczexbwEeBuWZ2ALgPOLc8rNe5+xYzW25me4F+4HOF\nV5Oxffv2NXsJDRV5f5H3Btrf2arqcHf3VTWcc3ea5YiISAp6hWoia9asafYSGiry/iLvDbS/s5XV\n0+lM6jcy86n6vUREojAzvEkPqApQKpWavYSGiry/yHuD1tqfmelrgq+UUjwVUkSkZvoJfmyph7tq\nGRGZMuWKodnLaEnj/dmolhERkREa7om0Uq/ZCJH3F3lvEH9/MjYNdxGRgNS5i8iUGatXXrRoCf39\njfs9Z82CXbu2N+43SCR1565ny4hIU/X3w7x5jRu+vb1LGnbbrUy1TCLRe83I+4u8N4i/vxS+8pWv\nsHLlyjMuW7t2LV/4wheatKL66Z67SHCrV98FtDd7GS2tp6eHBx54gOPHj3PhhRdy+vRpnnzySbZt\n29bspRWm4Z5Id3d3s5fQUJH3F3lvQ9obWntMxuuvp32hTirz58/nhhtu4Nvf/jZ33nkn27ZtY968\neVx99dXNXlphqmVERIDVq1ezceNGADZu3EhPT0+TV1QfDfdEoveakfcXeW8AAwN9zV5CFlasWMHO\nnTt5+eWXefrpp7n99tubvaS6aLiLiAAzZsxg5cqV3HbbbVx33XUsWLCg2Uuqi57nLhLcpZcuaZnO\n/Wc/a+3nub/wwgtcf/31rF+/ntWrVzduUWPQ89xFJJRWeoFRZ2cn559/PjfffHOzl1I31TKJRO9t\nI+8v8t5AnXutBgcH+drXvsaqVau44IILmr2cuumeu4ic9fr7++no6OCyyy7L+rnto6lzFwmu1Tt3\nGaL3cxcRkao03BOJ3ttG3l/kvYE697OVhruISEDq3EWCU+eeB3XuIiJSlYZ7ItF728j7i7w3UOd+\nttLz3EWkqZYsWkSj339g+65dhX95d3c3PT093HHHHSOXlUolenp6OHDgQNVfP5lzU9JwTyT6e4JH\n3l/kvQHMmNHiH9TR38/2efMadvNLenvr+vVmhllrvg/9RFTLiMhZr96P2Vu/fj2LFi1i9uzZXH75\n5Tz22GPA0Ctfb7zxRg4fPkx7ezuzZ8/mjTfeSL7+sWi4JxK9t428v8h7A3Xutejp6WHbtm0cP34c\nYORj9obfGbLaM3w6Ojp4+umnOXHiBOvXr+eLX/wiL730ErNmzWLbtm1cfPHF9PX1ceLECebPn9/w\n/YCGu4jIGR+zB5zxMXvuztq1a7noootGvm666aYzqprly5dz2WWXAXDDDTfwyU9+kueffx6o/g9D\no2i4JxK9t428v8h7gww69xYx3sfsmRlf//rXOXbs2MjXU089dcbQ3rp1K0uXLuW9730vF110EVu2\nbOHo0aNN2ccwDXcRESb3MXujB/vJkye5+eabueeee/jtb3/LsWPHWL58+cg5zXowVsM9kei9beT9\nRd4bqHOv1UQfszdRtXLq1ClOnTrF3LlzOeecc9i6dSvPPvvsyPUdHR0cPXqUEydONHT9lfRUSBFp\nrlmz6n66YrXbr9Xq1at5/PHHWb9+/RmXj3Xve/iy9vZ2Hn74YW655RZOnjzJTTfdxIoVK0bOu+qq\nq1i1ahXvf//7GRwcZNeuXVPyoGrV95Yxs2XAQ0Ab8Li7P1hx/YXARqCToX8svuruG8a4Hb23TMYa\n/TmX9ZjMZ2SejfTeMrV7/fXXWbhwIUeOHJnyT2Oa0s9QNbM24BHgE8Ah4Kdmttndd4867R+Bl939\nJjObC+wxs43ufnqyi5HW1d9PywyISr29S5q9BAkg2sfsVevcrwX2uvs+d38H2ASsqDhnEJhd/n42\ncPRsHOzqbfOl7KS/v5/Zs2fz3HPP8cADDzR7OUlU69wvAUa/IcJB4LqKcx4BfmBmh4F24JZ0yxMR\nabxZs2bx1ltvNXsZSVUb7rWUY8uAn7v7x8zscuB/zezD7v6uuwtr1qyhq6sLgDlz5rB48eKR5xgP\n33vK9Xj4slZZTyP219dXor29e+R7oGWO69lfd3d30/98G3k8Y0Z70/MZPpbqSqUSGzZsABiZl0VM\n+ICqmS0F7nf3ZeXjLwGDox9UNbOngP9w9xfKx88B97r79orb0gOqGWulB+Uq9fYuYf/+1lxbK2il\n7Fr9AdVmmuoP69gOXGFmXWZ2HnArsLninNcZesAVM+sArgR+PdmF5E69bb6UnUQ0YS3j7qfN7G7g\nGYaeCvmEu+82s7vK168DvgxsMLOdgAH3uPvvG7xuEclUjm+fmyN9hqrUpJV+tK+kWmZiyi5v+gxV\nEREZoeGeiHrbfCm7vEXPrygNdxGRgDTcE9F7gudL2eUten5FabiLiASk4Z5I9N4vcm+r7PIWPb+i\npvT93C+9tDXfvU9vGSsi0UzpcG/l59rWK3rvF7m3VXZ5i55fUaplREQC0nBPJHrvF7m3VXZ5i55f\nURruIiIBabgnEr33i9zbKru8Rc+vKA13EZGANNwTid77Re5tlV3eoudXlIa7iEhAGu6JRO/9Ive2\nyi5v0fMrSsNdRCQgDfdEovd+kXtbZZe36PkVpeEuIhKQhnsi0Xu/yL2tsstb9PyK0nAXEQlIwz2R\n6L1f5N5W2eUten5FabiLiASk4Z5I9N4vcm+r7PIWPb+iNNxFRALScE8keu8XubdVdnmLnl9RGu4i\nIgFpuCcSvfeL3Nsqu7xFz68oDXcRkYA03BOJ3vtF7m2VXd6i51eUhruISEAa7olE7/0i97bKLm/R\n8ytKw11EJCAN90Si936Re1tll7fo+RWl4S4iElDV4W5my8zsFTN71czuHeecbjN7ycxeNrNS8lVm\nIHrvF7m3VXZ5i55fUdMmutLM2oBHgE8Ah4Cfmtlmd9896pw5wH8Cn3L3g2Y2t5ELFhGR6qrdc78W\n2Ovu+9z9HWATsKLinNuA77r7QQB3/136Zba+6L1f5N5W2eUten5FVRvulwAHRh0fLF822hXAe8zs\nh2a23cx6Ui5QREQmb8JaBvAabuNc4C+BjwMzgR+Z2Y/d/dV6F5eT6L1f5N5W2eUten5FVRvuh4DO\nUcedDN17H+0A8Dt3/wPwBzP7P+DDwLuG+2uvrWH69C4A2trmMHPmYtrbuwHo6ysBNO14+Ee74b8o\nOj7zeGCgj76+Usvkpfwmd9zsfMY7HtbsP59WOi6VSmzYsAGArq4uijL38e+cm9k0YA9D98oPAz8B\nVlU8oHoVQw+6fgqYDrwI3Oruuypuy6+5ppYfBKZeb+8S9u/fXtdtlEql0PcgOjqupLNzT7OXMaZ6\n81N2zaP/96ozM9zdJvvrJrzn7u6nzexu4BmgDXjC3Xeb2V3l69e5+ytmtg3YCQwC/1052EVEZGpV\nq2Vw963A1orL1lUcfxX4atql5SXyPQeI3dsqu7xFz68ovUJVRCQgDfdEoj/XNvJzpZVd3qLnV5SG\nu4hIQBruiUTv/SL3tsoub9HzK6rqA6oiIo3y5uFfseTSS5u9jLHNmsX2Xfk+8U/DnTR/wfoGBmif\nMSPRikZpkb9grdzb1ptfw7KDlsivlbOb6YNsnzevrtso9fXR3Z7+p5Mlvb3Jb3Mqabijv2C5qze/\nRmUHyk+aR517Io0aDq0icm+r7PIWPb+iNNxFRALScE+k1Ne6vWYKrdzb1kvZ5S16fkVpuIuIBKTh\nnkj03i9yb6vs8hY9v6I03EVEAtJwTyR67xe5t1V2eYueX1Ea7iIiAWm4JxK994vc2yq7vEXPrygN\ndxGRgDTcE4ne+0XubZVd3qLnV5SGu4hIQBruiUTv/SL3tsoub9HzK0rDXUQkIA33RKL3fpF7W2WX\nt+j5FaXhLiISkIZ7ItF7v8i9rbLLW/T8itJwFxEJSMM9kei9X+TeVtnlLXp+RWm4i4gEpOGeSPTe\nL3Jvq+zyFj2/ojTcRUQC0nBPJHrvF7m3VXZ5i55fURruIiIBabgnEr33i9zbKru8Rc+vKA13EZGA\nNNwTid77Re5tlV3eoudXlIa7iEhAGu6JRO/9Ive2yi5v0fMrqupwN7NlZvaKmb1qZvdOcN5fmdlp\nM/vbtEsUEZHJmnC4m1kb8AiwDFgErDKzheOc9yCwDbAGrLPlRe/9Ive2yi5v0fMrqto992uBve6+\nz93fATYBK8Y475+A7wC9idcnIiIFVBvulwAHRh0fLF82wswuYWjgP1q+yJOtLiPRe7/Iva2yy1v0\n/IqqNtxrGdQPAf/i7s5QJXNW1jIiIq1kWpXrDwGdo447Gbr3Pto1wCYzA5gL3Ghm77j75sobe+21\nNUyf3gVAW9scZs5cTHt7NwB9fSWAph0P93bD9wIme/zQkSMsnjmz8K8f73hYqTS03u7u7qYcnzhx\nhBkzSi2TV8r8Rv9ZR8xvYKCv6fmMdzysFfPrGxj40/qmMK9SqcSGDRsA6OrqoigbusM9zpVm04A9\nwMeBw8BPgFXuvnuc89cDP3D3741xnV9zTWs2Nod2TOc3iz9Y122U+voa8uPhkt5etu/fn/x2J6uj\n40o6O/c0exljqje/RmUHrZFf5Owg/v97Zoa7T7oRmfCeu7ufNrO7gWeANuAJd99tZneVr19XaLUB\nRe/9Ive2yi5v0fMrqlotg7tvBbZWXDbmUHf3zyVal4iI1EGvUE0k+nNtIz9XWtnlLXp+RWm4i4gE\npOGeSPTeL3Jvq+zyFj2/ojTcRUQC0nBPJHrvF7m3VXZ5i55fURruIiIBabgnEr33i9zbKru8Rc+v\nKA13EZGANNwTid77Re5tlV3eoudXlIa7iEhAGu6JRO/9Ive2yi5v0fMrSsNdRCQgDfdEovd+kXtb\nZZe36PkVpeEuIhKQhnsi0Xu/yL2tsstb9PyK0nAXEQlIwz2R6L1f5N5W2eUten5FabiLiASk4Z5I\n9N4vcm+r7PIWPb+iNNxFRALScE8keu8XubdVdnmLnl9RGu4iIgFpuCcSvfeL3Nsqu7xFz68oDXcR\nkYA03BOJ3vtF7m2VXd6i51eUhruISEAa7olE7/0i97bKLm/R8ytKw11EJCAN90Si936Re1tll7fo\n+RWl4S4iEpCGeyLRe7/Iva2yy1v0/IrScBcRCUjDPZHovV/k3lbZ5S16fkVpuIuIBKThnkj03i9y\nb6vs8hY9v6I03EVEAqppuJvZMjN7xcxeNbN7x7j+djP7hZntNLMXzOxD6Zfa2qL3fpF7W2WXt+j5\nFVV1uJtZG/AIsAxYBKwys4UVp/0auMHdPwR8GXgs9UJFRKR2tdxzvxbY6+773P0dYBOwYvQJ7v4j\ndz9ePnwRWJB2ma0veu8XubdVdnmLnl9RtQz3S4ADo44Pli8bzx3AlnoWJSIi9ZlWwzle642Z2ceA\nvwc+Mtb1r722hunTuwBoa5vDzJmLaW/vBqCvrwTQtOPh3m74XsBkjx86coTFM2cW/vXjHQ8rlYbW\n293d3ZTjEyeOMGNGqWXySpnf6D/riPkNDPQ1PZ/xjoe1Yn59AwN/Wt8U5lUqldiwYQMAXV1dFGXu\nE89uM1sK3O/uy8rHXwIG3f3BivM+BHwPWObue8e4Hb/mmpr/nZhSh3ZM5zeLP1jXbZT6+hry4+GS\n3l6279+f/HYnq6PjSjo79zR7GWOqN79GZQetkV/k7CD+/3tmhrvbZH9dLbXMduAKM+sys/OAW4HN\nFb/5+xga7J8Za7CfDaL3fpF7W2WXt+j5FVW1lnH302Z2N/AM0AY84e67zeyu8vXrgH8HLgIeNTOA\nd9z92sYtW0REJlJL5467bwW2Vly2btT3dwJ3pl1aXhr5o30riPxcaWWXt+j5FaVXqIqIBKThnkj0\new6Re1tll7fo+RWl4S4iEpCGeyLR398icm+r7PIWPb+iNNxFRALScE8keu8XubdVdnmLnl9RGu4i\nIgFpuCcSvfeL3Nsqu7xFz68oDXcRkYA03BOJ3vtF7m2VXd6i51eUhruISEAa7olE7/0i97bKLm/R\n8ytKw11EJCAN90Si936Re1tll7fo+RWl4S4iEpCGeyLRe7/Iva2yy1v0/IrScBcRCUjDPZHovV/k\n3lbZ5S16fkVpuIuIBKThnkj03i9yb6vs8hY9v6I03EVEAtJwTyR67xe5t1V2eYueX1Ea7iIiAWm4\nJxK994vc2yq7vEXPrygNdxGRgDTcE4ne+0XubZVd3qLnV5SGu4hIQBruiUTv/SL3tsoub9HzK0rD\nXUQkIA33RKL3fpF7W2WXt+j5FaXhLiISkIZ7ItF7v8i9rbLLW/T8itJwFxEJSMM9kei9X+TeVtnl\nLXp+RWm4i4gEVHW4m9kyM3vFzF41s3vHOefh8vW/MLOr0y+z9UXv/SL3tsoub9HzK2rC4W5mbcAj\nwDJgEbDKzBZWnLMc+IC7XwF8Hni0QWttaTvefrvZS2ioU6fi7k/Z5S16fkVVu+d+LbDX3fe5+zvA\nJmBFxTmfBr4J4O4vAnPMrCP5Slvcm3/8Y7OX0FCDg3H3p+zyFj2/oqoN90uAA6OOD5Yvq3bOgvqX\nJiIiRVUb7l7j7VjBXxfGvpMnm72Ehjp9Ou7+lF3eoudXlLmPP4fNbClwv7svKx9/CRh09wdHnfNf\nQMndN5WPXwE+6u5HKm7rrBv4IiIpuHvlHeiqplW5fjtwhZl1AYeBW4FVFedsBu4GNpX/MXizcrAX\nXZyIiBQz4XB399NmdjfwDNAGPOHuu83srvL169x9i5ktN7O9QD/wuYavWkREJjRhLSMiInlK/grV\nyC96qrY3M+s2s+Nm9lL569+asc4izOwbZnbEzH45wTlZ5gbV95dzdgBm1mlmPzSzX5nZy2a2dpzz\nssywlv3lmqGZzTCzF81sR3lv949z3uSyc/dkXwxVN3uBLuBcYAewsOKc5cCW8vfXAT9OuYZGfdW4\nt25gc7PXWnB/1wNXA78c5/osc5vE/rLNrrz++cDi8vcXAHui/L83if1lmyEws/zfacCPgevqzS71\nPffIL3qqZW/w7qeFZsHdnweOTXBKrrkBNe0PMs0OwN3fcPcd5e/fAnYDF1eclm2GNe4PMs3Q3Ydf\nZnseQ3ceBytOmXR2qYd75Bc91bI3B/66/GPTFjNbNGWra7xcc6tVmOzKz267Gnix4qoQGU6wv2wz\nNLNzzGwHcAR41t1/WnHKpLOr9lTIyYr8oqda1vhzoNPd3zazG4HvA3/e2GVNqRxzq1WI7MzsAuA7\nwD+X7+G+65SK46wyrLK/bDN090FgsZldCPyPmf2Fu/+q4rRJZZf6nvshoHPUcSdD/8JMdM6C8mWt\nrure3L1v+Mcrd98KnGtm75m6JTZUrrnVJEJ2ZnYu8F1go7t/f4xTss6w2v4iZOjux4EfMvRmjaNN\nOrvUw33kRU9mdh5DL3raXHHOZuCzMPIK2DFf9NSCqu7NzDrMzMrfX8vQU01/P/VLbYhcc6tJ7tmV\n1/4EsMvdHxrntGwzrGV/uWZoZnPNbE75+/OBv2HoMYXRJp1d0lrGA7/oqZa9ASuBfzCz08DbwN81\nbcGTZGbfAj4KzDWzA8B9DD2wk3Vuw6rtj4yzK/sI8Blgp5m9VL7sX4H3QYgMq+6PfDP8M+CbNvQW\n6+cAT5azqmtu6kVMIiIB6WP2REQC0nAXEQlIw11EJCANdxGRgDTcRUQC0nAXEQlIw11EJCANdxGR\ngP4fVq+mNBBtk1kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105353310>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Compare estimate, yHat, to actually score\n",
    "bar([0,1,2], y, width = 0.35, alpha=0.8)\n",
    "bar([0.35,1.35,2.35],yHat, width = 0.35, color='r', alpha=0.8)\n",
    "grid(1)\n",
    "legend(['y', 'yHat'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right now our predictions are pretty inaccurate. To improve our model, we first need to quantify exactly how wrong our predictions are. We'll do this with a cost function. A cost function allows us to express exactly how wrong or \"costly\" our models is, given our examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to compute an overall cost is to take each error value, square it, and add these values together. Multiplying by one half will make things simpler down the road. Now that we have a cost, or job is to minimize it. When someone says they’re training a network, what they really mean is that they're minimizing a cost function. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "J = \\sum \\frac{1}{2}(y-\\hat{y})^2 \\tag{5}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OUR cost is a function of two things, our examples, and the weights on our synapses. We don't have much control of our data, so we'll minimize our cost by changing the weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conceptually, this is pretty simple concept. We have a collection of 9 individual weights, and we're saying that there is some combination of w's that will make our cost, J, as small as possible. When I first saw this problem in machine learning, I thought, I'll just try ALL THE WEIGHTS UNTIL I FIND THE BEST ONE! After all I have a computer!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enter the CURSE OF DIMENSIONALITY. Here's the problem. Let's pretend for a second that we only have 1 weight, instead of 9. To find the ideal value of our weight that will minimize our cost, we need to try a bunch of values for W, let's say we test 1000 values. That doesn't seem so bad, after all, my computer is pretty fast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "weightsToTry = np.linspace(-5,5,1000)\n",
    "costs = np.zeros(1000)\n",
    "\n",
    "startTime = time.clock()\n",
    "for i in range(1000):\n",
    "    NN.W1[0,0] = weightsToTry[i]\n",
    "    yHat = NN.forward(X)\n",
    "    costs[i] = 0.5*sum((y-yHat)**2)\n",
    "    \n",
    "endTime = time.clock()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.065137"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timeElapsed = endTime-startTime\n",
    "timeElapsed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes about 0.04 seconds to check 1000 different weight values for our neural network. Since we’ve computed the cost for a wide range values of W, we can just pick the one with the smallest cost, let that be our weight, and we’ve trained our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x105339fd0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFdW19/HvEsUJtR2IKCAthgRM1BYjQXFoFbHlGnEM\ncYhpM1zMFWMI1yhKlHuNA4ljcjMQh6BihDijF8SxoyIOqC0OTMZLBCKR1yhqEhVkvX/sgx57oKu7\nT3Wd2v37PE8/6apTdc5azwm9rL1q7zJ3R0REpNgGWQcgIiLlR8VBREQaUXEQEZFGVBxERKQRFQcR\nEWlExUFERBpJtTiYWY2ZLTCzxWZ2dhOvn2RmL5jZPDObbWa7Jz1XRETSY2nNczCzLsBCYCiwHHgG\nOMHd5xcdsw/wiruvMrMaYIK7D05yroiIpCfNK4dBwKvuvsTdVwNTgRHFB7j7HHdfVdh8CuiV9FwR\nEUlPmsWhJ7C0aHtZYV9zvgPMaOO5IiJSQhum+N6Jx6vM7CDg28CQ1p4rIiKll2ZxWA70LtruTbgC\n+IxCE/oaoMbd327luSoiIiJt4O62vtfTHFaaC/Qzs0oz6wqMBKYXH2BmOwF3ACe7+6utOXcdd4/2\n54ILLsg8BuWn/DpjfjHn5p7sv6lTu3Jw9zVmNhqYBXQBrnP3+WY2qvD6JOB8YGvgN2YGsNrdBzV3\nblqxlqslS5ZkHUKqlF++xZxfzLklleawEu4+E5jZYN+kot+/C3w36bkiItIxNEO6jNXW1mYdQqqU\nX77FnF/MuSWV2iS4jmBmnuf4RUSyYGZ4hg1paae6urqsQ0iV8su3mPOLObekVBxERKQRDSuJiHQy\nGlYSEZE2UXEoY7GPeyq/fIs5v5hzS0rFQUREGlHPQUSkk1HPQURE2kTFoYzFPu6p/PIt5vxizi0p\nFQcREWlEPQcRkU5GPQcREWkTFYcyFvu4p/LLt5jzizm3pFQcRESkEfUcREQ6GfUcRESkTVQcyljs\n457KL99izi/m3JJScRARkUbUcxAR6WTUcxARkTZRcShjsY97Kr98izm/mHNLKtXiYGY1ZrbAzBab\n2dlNvN7fzOaY2QdmNrbBa2PM7CUze9HM/mBmG6cZq4iIfCq1noOZdQEWAkOB5cAzwAnuPr/omO5A\nH+Ao4G13v7ywvyfwGDDA3T80s2nADHe/ocFndFjP4W9/g7lzYcgQqKjokI8UEUlF1j2HQcCr7r7E\n3VcDU4ERxQe4+0p3nwusbuL8DYHNzGxDYDNCgcnMG2/AFVdA796w557wgx/AbbeFoiEiEps0i0NP\nYGnR9rLCvha5+3LgcuB14K/AO+7+YMkjbIWqKnjoIXjrLfj1r6FXL5g8Gfr3Dz/f+x7cdBMsW1a6\nz4x93FP55VvM+cWcW1IbpvjebR7vMbOtgSOBSmAVcKuZneTuNzc8tra2lsrKSgAqKiqoqqqiuroa\n+PQLTmN7n31g0KA6xoyB7bar5tFH4dpr6xg9GnbcsZpDD4UePeqoqoLhw9v2efX19anFXw7byi/f\n27HnF9N2XV0dkydPBvjk72VL0uw5DAYmuHtNYXscsNbdJzZx7AXA+0U9h+OBw9z9u4XtbwKD3f30\nBueV3TyHtWvh+efhgQfCz1NPhWGoQw8NP3vvDRumWZJFRFqQdc9hLtDPzCrNrCswEpjezLENg/wL\nMNjMNjUzIzS1X0kv1NLZYAPYay8455wwDPXmmzB+PLz7Lpx2Gmy/PZx0EkydCm+/nXW0IiJNS604\nuPsaYDQwi/CHfZq7zzezUWY2CsDMepjZUmAMMN7MXjezbu7+NHAb8Bwwr/CWv0sr1jRtthkcdhhc\ndhm88EL4OeAAmDIF+vSBgw6Cyy+HRYsan7vusjBWyi/fYs4v5tySSnWAw91nAjMb7JtU9PsKoHcz\n504AJqQYXiZ69YJRo8LPP/8Zri7uvTcUic03h2OPheOPD0NRIiJZ0dpKZcIdnn023B57661h33HH\nhUKx115g6x0dFBFJLknPQcWhDLlDfX0oErfeCmvWhEJx4onhlloVChFpj6wb0tJGZmFYadiwOhYt\ngrvugq5d4ZhjYPfd4Wc/g+WZTgksjdjHdZVffsWcW1IqDmXODPbYAy66CP78Z/jVr2DxYthtt3Br\n7I03wvvvZx2liMRGw0o59a9/wfTpYVb244+H3sSoUfCVr2QdmYiUO/UcOok33oDrr4drroFttw1F\n4oQTYIstso5MRMqReg45l3Tcc4cd4LzzwrDTRRfBzJlhDsVpp8GLL6YbY3vEPq6r/PIr5tySUnGI\nSJcuUFMDd94ZisKOO4YJeEOHwv/+b1jaQ0QkCQ0rRe6jj2DaNLjyyjDp7oc/hFNOCTO3RaRzUs9B\nPuEOjz4aisTs2WHI6cwzYbvtso5MRDqaeg45V8pxTzM48MAwZ2LOnLAg4Be+AGedBStWlOxjWiX2\ncV3ll18x55aUikMn9PnPw6RJYRHADz+EXXeFM86A11/POjIRKRcaVhJWrAiPQL32Whg5En7yk9DM\nFpE4aVhJEunRIyzJsWgRdOsWZl+fc46eNyHSmak4lLGOHvfcbjv4+c/Don9vvRV6EpdeGu5ySkPs\n47rKL79izi0pFQdppHfvMNv68cfhueegXz+YPFnzJEQ6E/UcpEVPPx1ue12zBq6+GvbdN+uIRKQ9\nNM9BSmbtWrjlFjj77HBL7MSJ4al2IpI/akjnXDmNe26wAZx0EixYAH37hocOXXoprF7d9vcsp/zS\noPzyK+bcklJxkFbp1g0uvBCeeSbMuB44MEyqE5G4aFhJ2sw9PMZ0zBg48ki45BKoqMg6KhFpiYaV\nJFVm8PWvw8svh+1dd4U77sg2JhEpDRWHMpaXcc+KCvjNb8JVxDnnwMknJ5tAl5f82kr55VfMuSWV\nanEwsxozW2Bmi83s7CZe729mc8zsAzMb2+C1CjO7zczmm9krZjY4zVil/YYMCRPott02zLKeMSPr\niESkrVLrOZhZF2AhMBRYDjwDnODu84uO6Q70AY4C3nb3y4teuwH4k7tfb2YbApu7+6oGn6GeQ5l6\n5BH49rfhkEPCuk1bbpl1RCKyTtY9h0HAq+6+xN1XA1OBEcUHuPtKd58LfOaGSDPbCtjf3a8vHLem\nYWGQ8nbQQTBvXuhLDBwIc+dmHZGItEaaxaEnsLRoe1lhXxI7AyvN7Pdm9pyZXWNmne7ZZXkf99xi\ni7AMxyWXwPDh4QqieAmOvOfXEuWXXzHnltSGKb53e8Z7NgQGAqPd/Rkzuwo4Bzi/4YG1tbVUVlYC\nUFFRQVVVFdXV1cCnX3Bet+vr68sqnrZuH398NXvvDcOH1zFtGtxzTzWf+1w8+cX+/XXW/GLarqur\nY/LkyQCf/L1sSZo9h8HABHevKWyPA9a6+8Qmjr0AeH9dz8HMegBz3H3nwvZ+wDnufkSD89RzyJHV\nq+H88+HGG+Hmm6Hw/2ER6WBZ9xzmAv3MrNLMugIjgenNHPuZIN19BbDUzL5Q2DUUeDm1SKVDbLRR\nGGL6/e/hG98Iw0yq7SLlKbXi4O5rgNHALOAVYJq7zzezUWY2CsIVgpktBcYA483sdTPrVniLM4Cb\nzewFYHfg4rRiLVfrLgtjM2wYPPkk/Pa3dZx4IvzjH1lHlI5Yv791Ys4v5tySSrPngLvPBGY22Dep\n6PcVQO9mzn0B2DvN+CQ7lZXwy1+GlV4HD4Y77wzPthaR8qC1lSRT7mF29YQJoRdRU5N1RCLx0/Mc\nJDdmz4bjjoPzzoPRo7OORiRuWTekpZ1iH/cszm/IkFAgfv1rOOOM8NS5vOtM319sYs4tKRUHKRt9\n+8ITT8DChWEJ8HffzToikc5Lw0pSdlavDlcPs2fDvfdCnz5ZRyQSFw0rSS5ttFFoUp96Kuy3H7z0\nUtYRiXQ+Kg5lLPZxz/XlZwY/+hFMnAgHHwyPP95xcZVKZ/7+8i7m3JJScZCyduKJMGUKHH00TG9u\nfr2IlJx6DpILTz8dmtQXXxyeEyEibZek55DqDGmRUhk0CP70JzjsMHjrLTjrrKwjEombhpXKWOzj\nnq3N74tfDL2Ha6+F//7v8l+0T99ffsWcW1K6cpBc6dUrXEEccgh8+CH89KeheS0ipaWeg+TSypVh\nddeDD4bLLlOBEGkNzXOQaHXvDg89BI89FtZiKn78qIi0n4pDGYt93LO9+W2zDTzwANTXw6hR5Vcg\n9P3lV8y5JaXiILm21VYwaxYsWBCuIDTKKFIa6jlIFN59N/QgBg+GK69UD0JkfdRzkE5jyy3hvvtC\nD+Lss3UFIdJeKg5lLPZxz1LnV1ERehCzZsH555f0rdtE319+xZxbUprnIFHZZht48EGoroaNN4bx\n47OOSCSf1HOQKK1YAQceGO5i+tGPso5GpLxobSXptHr0CFcQ++0XriZqa7OOSCRf1HMoY7GPe6ad\nX+/eof8wbhzcfXeqH9UkfX/5FXNuSaVaHMysxswWmNliMzu7idf7m9kcM/vAzMY28XoXM3vezO5J\nM06JV//+cM898L3vgf69iySXWs/BzLoAC4GhwHLgGeAEd59fdEx3oA9wFPC2u1/e4D1+BOwFbOHu\nRzbxGeo5SCKPPAIjR4bbXQcOzDoakWxlPc9hEPCquy9x99XAVGBE8QHuvtLd5wKrG55sZr2A4cC1\ngKY0SbscdBD87ndwxBGwaFHW0YiUvzSLQ09gadH2ssK+pK4EzgLKbMWcjhP7uGdH53fUUXDRRWEm\n9bJl6X+evr/8ijm3pNK8W6nN4z1mdgTwprs/b2bV6zu2traWyspKACoqKqiqqqK6Opyy7gvO63Z9\nfX1ZxRNDfjvvDKefXk1NDVx6aR3dusWVX0dux55fTNt1dXVMnjwZ4JO/ly1Js+cwGJjg7jWF7XHA\nWnef2MSxFwDvr+s5mNnFwDeBNcAmwJbA7e5+SoPz1HOQVnOHH/4Q5s0LPYiNN846IpGOlXXPYS7Q\nz8wqzawrMBKY3syxnwnS3c91997uvjPwDeDhhoVBpK3M4IorYNttw/yHclvqW6QcpFYc3H0NMBqY\nBbwCTHP3+WY2ysxGAZhZDzNbCowBxpvZ62bWram3SyvOcrbusjBWWebXpQvcdBMsXQrnnJPOZ+j7\ny6+Yc0sq1RnS7j4TmNlg36Si31cAvVt4jz8Bf0olQOnUNt00TI4bMiRMmDvjjKwjEikfWltJOr0l\nS0KB+J//gaOPzjoakfRpbSWRBCorwyzqmhrYfnvYd9+sIxLJXos9BzO7Kck+Kb3Yxz3LKb+BA+HG\nG+GYY2DhwtK8Zznll4aY84s5t6SSNKS/XLxhZhsSlrQQiUpNDVxyCRx+OPztb1lHI5KtZnsOZnYu\nMA7YFPhX0Uurgd+5e0r3eCSnnoOkYcIEmDEjrMe0+eZZRyNSekl6Di02pM3s0nIoBE1RcZA0uMOp\np8Lf/w533hluexWJSakmwd27bu6BmX3TzK4wsz4liVDWK/Zxz3LNzyws0vevf8GZZ4Zi0Rblml+p\nxJxfzLkllaQ4/Ab4p5ntAfwIeA24MdWoRDLWtSvcdhs8+miYTS3S2SQZVnre3fcsrH+03N2vNbPn\n3D3zVfE1rCRpW7YM9tknFIjjj886GpHSKNU8h/cKzemTgf0LD/HZqBQBipS7Xr3g3nvh0ENhhx3C\nM6lFOoMkw0ojgQ+BbxeWu+gJ/DzVqASIf9wzL/ntsQdMmQLHHde6ORB5ya+tYs4v5tySarE4uPsb\nwM1AReE5Cx+4u3oO0qkMGwYXXwzDh2sOhHQOSXoOXydcKaxb/O4A4Cx3vzXl2FqknoN0tAsuCM+A\neOQR2GyzrKMRaZtSzXOYBwx19zcL292Bh9x995JF2kYqDtLR3MMzIFatgttv1xwIyadSzXMwYGXR\n9ls0eDiPpCP2cc885mcG11wD778fnia3vv82yWN+rRFzfjHnllSS4nAfMMvMas3sVGAGDZ7RINKZ\ndO0arhrq6uDKK7OORiQd61tbqR+wvbs/bmbHAkMKL70D/MHdX+2gGJulYSXJ0tKlYXlvzYGQvGlX\nz8HM/hcY5+7zGuzfHbjI3b9WskjbSMVBslZfH+5kuvPO8MAgkTxob89h+4aFAaCwb+f2Bicti33c\nM4b8qqrCs6iPPRYWLfrsazHktz4x5xdzbkmtrzhUrOe1TUodiEheHXYYXHRRmAPx5ptZRyNSGusb\nVpoKPOzuv2uw/3uEW1tHdkB866VhJSkn558P998PDz+sORBS3trbc+gB3Al8BDxb2L0XsDFwdGHm\ndKZUHKScuMO3vgXvvRdWdNUcCClX7eo5FNZR2hf4L2AJ8H/Af7n74HIoDJ1B7OOeseVnBtdeGybI\njRkDjzxSl3VIqYrt+ysWc25JrXeegwcPu/sv3P2X7v5waz/AzGrMbIGZLTazs5t4vb+ZzTGzD8xs\nbNH+3mb2iJm9bGYvmdkPWvvZIh2ta1e4444wtHTbbVlHI9J2LS6f0a43D8t7LwSGAsuBZ4AT3H1+\n0THdgT7AUcDb7n55YX8PoIe71xeeRPcscFSDczWsJGXp9dfDHIirrw53MomUk1Itn9Eeg4BX3X2J\nu68GpgIjig9w95XuPhdY3WD/CnevL/z+PjAf2DHleEVKYqed4J574PvfhyeeyDoakdZLuzj0BJYW\nbS8r7GsVM6sE9gSeKklUORH7uGfs+a1aVceNN4Yrh8WLs46m9GL+/mLOLakkT4Jrj3aP+RSGlG4D\nzixcQXxGbW0tlZWVAFRUVFBVVUV1dTXw6Rec1+36+vqyikf5tT6/qiq48MJqDj8cLrusjoqK8olP\n31/n2a6rq2Py5MkAn/y9bEnaPYfBwAR3rylsjwPWuvvEJo69AHh/Xc+hsG8j4F5gprtf1cQ56jlI\nLowfDw8+qDkQUh7KoecwF+hnZpVm1pXwyNHpzRz7mUDNzIDrgFeaKgwieXLhhdCvH5x8Mnz8cdbR\niLQs1eLg7muA0cAs4BVgmrvPN7NRZjYKwl1JZrYUGAOMN7PXC0NJQ4CTgYPM7PnCT02a8ZabdZeF\nsepM+ZnBddfBO+/A2LHNn5MnMX9/MeeWVNo9B9x9Jg2e/+Duk4p+XwH0buLUx0n/ykakw3QtzIHY\nbz/42c/gxz/OOiKR5qXac0ibeg6SR8uWwQEHhOJw2mlZRyOdUZKeQ+pXDiLyWb16heb0gQdCt26h\nDyFSbjRsU8ZiH/fszPn17RtWcD3rrPCgoDyK+fuLObekdOUgkpEBA2DGDKipCbe3HnZY1hGJfEo9\nB5GMPfEEjBgBt98eehEiaSuHeQ4i0oJ994VbboHjjoOnOtUCMVLOVBzKWOzjnsrvU0OHwuTJ8LWv\nwZNPphZSScX8/cWcW1IqDiJlYvjwUCCOPBLmzMk6Guns1HMQKTP33QennAJ33RWGnERKTT0HkRyq\nqYGbboKjjoLZs7OORjorFYcyFvu4p/Jr3mGHwZQpcPTR8NhjpYuplGL+/mLOLSkVB5EyNWwY/OEP\n4WFBM2e2fLxIKannIFLm5swJQ0y/+AWMHJl1NBIDra0kEoF99glrMdXUhCW/R43KOiLpDDSsVMZi\nH/dUfsntths8+ihMnAiXXlqyt22XmL+/mHNLSsVBJCd22QUefzw0qseOhbVrs45IYqaeg0jO/P3v\n4S6m7t3DLa+bbpp1RJI3mucgEqFttgnLfW+8MRx8MKxcmXVEEiMVhzIW+7in8mu7jTcOVw2HHBIa\n1osWpfZRzYr5+4s5t6R0t5JITm2wAfz0p1BZCfvvD9OmQXV11lFJLNRzEInAgw/CSSfBT34Cp58O\ntt7RZOnskvQcVBxEIvHaa2Gy3N57w69+BZtsknVEUq7UkM652Mc9lV9p9e0bnir37rtheOmvf033\n82L+/mLOLalUi4OZ1ZjZAjNbbGZnN/F6fzObY2YfmNnY1pwrIo116wZ//GN4JsTee4P+xklbpTas\nZGZdgIXAUGA58AxwgrvPLzqmO9AHOAp4290vT3pu4TgNK4k04/774VvfgtNOg/HjoUuXrCOScpH1\nsNIg4FV3X+Luq4GpwIjiA9x9pbvPBVa39lwRWb9hw+DZZ8PVw7BhsGJF1hFJnqRZHHoCS4u2lxX2\npX1uNGIf91R+6dtxx3An0/77w8CB8MADpXvvcsgvLTHnllSa8xzaM96T+Nza2loqKysBqKiooKqq\niurCzd7rvuC8btfX15dVPMovv/lNmABbblnHiSfCCSdUc+ml8PTT8eSn7fVv19XVMXnyZIBP/l62\nJM2ew2BggrvXFLbHAWvdfWITx14AvF/Uc0h0rnoOIq3z9tswejTMnQs33ghf/WrWEUkWsu45zAX6\nmVmlmXUFRgLTmzm2YZCtOVdEEtp6a7j5ZrjoIhgxIjSqP/oo66ikHKVWHNx9DTAamAW8Akxz9/lm\nNsrMRgGYWQ8zWwqMAcab2etm1q25c9OKtVytuyyMlfLLznHHQX09zJsHe+0VnjbXWuWcX3vFnFtS\nqa6t5O4zgZkN9k0q+n0F0DvpuSJSOj16wN13h3kRxx4bZldffDFUVGQdmZQDLZ8hIrz9NowbB/fc\nA1ddFa4stD5TvLS2koi0yuzZ4RnVO+wAV14JX/5y1hFJGrJuSEs7xT7uqfzKz5Ah8PzzYfmNgw8O\ndza99VbTx+Yxv6Rizi0pFQcR+YyNNoIzzoD5hVtA+veHq6/WXU2djYaVRGS9Xn4Z/vM/YcECmDAB\nTj5Z6zTlnXoOIlIyjz4K550XhpkuvBCOOUZN67xSzyHnYh/3VH75csABoUBccUWYRNe/fx133QVr\n12YdWenF9t21hYqDiCRmBjU1YfmNk04Kz7DebTe46SZY3XBtZck1DSuJSJu5h1VfL74Y/u//4Kyz\noLYWNt8868hkfTSsJCKpMoNDD4VHHoFbbglLgvfpE4rEkiVZRyftoeJQxmIf91R++dYwv332gbvu\ngqefDlcUX/kKHH10KBx5u8CP/btLQsVBREqqb1+47DL4y19Cf2L0aBgwAH72Mz2NLk/UcxCRVLnD\nE0/AddfBnXfCgQfCd74Dhx8OG6a69Kc0R/McRKSsvPceTJsG118fGtgjR8I3vhEeOqQ5Ex1HDemc\ni33cU/nlW1vy22IL+O53w5VEXV1YHvxb34JddoFzzw3PlyiH/96L/btLQsVBRDLxxS+G5TgWLIDb\nb4ePP4avfQ123TUsH/7kk3FOsMsLDSuJSNlYuzbc7XT33TB9Ovz976FgjBgBhxwCm2ySdYRxUM9B\nRHJt8eJQJO6+G154AfbfH4YNCz9f/KL6FG2lnkPOxT7uqfzyrSPy69cPxo4Nazq99lroT8ybF4pD\nnz6hf3Hrrc0/c6KtYv/uktCNZCKSC9tuC8cfH37cYeFCuP9+uOGGcGvsTjuFxQEPOCBcYfTsmXXE\n+aZhJRHJvTVroL4+XGE89lj42WqrUCT23z/cKjtggJ5DsY56DiLSKa1dG55k99hj8Pjjocn9xhuw\n114waBDsvXf435126px9C/Ucci72cU/ll2/lnN8GG8CXvgSnnQZTpsCiRWE5j3PPDVcUU6aEq4ke\nPeDf/i3snzYt3Fb78cflnVtHSbXnYGY1wFVAF+Bad5/YxDG/AA4H/gnUuvvzhf1jgO8ADrwInOru\nH6YZr4jEa5ttPr3TCULfYtkyePbZcCfUtGnhSXdvvAG9eoXhqD32gN13D3MvunfPNv6Oltqwkpl1\nARYCQ4HlwDPACe4+v+iY4cBodx9uZl8Frnb3wWbWE3gMGODuH5rZNGCGu9/Q4DM0rCQiJfXee/Di\ni+GuqBdeCD/z54d+Rf/+n/0ZMAAqK/PXy0gyrJTmlcMg4FV3X1IIZiowAphfdMyRwA0A7v6UmVWY\n2fZFsW1mZh8DmxEKjIhIqrbYAvbdN/ys4w4rV4YisWBB+HnoofC/K1bA5z8fbrvt2/fTn112Cbfb\ndu2aXS7tkWZx6AksLdpeBnw1wTE93f05M7sceB34FzDL3R9MMdayVFdXR3V1ddZhpEb55VvM+TXM\nzQw+97nwc+CBnz32n/8Mt9X++c9hLsa8eWH12ddeC8NWPXp8Wiz69g1XGjvtFArHDjuU78q0aYaV\ndLyn0aWNmW1NuKqoBFYBt5rZSe5+c8Nja2trqaysBKCiooKqqqpPvtR1TaW8btfX15dVPMpP+XWm\n/FqzveeesGpVHdttBz/+8aevf/wx7LxzNa+9BjNn1hWGq6p5/XVYtKiOVaugZ89qjjwSjjkmvfjq\n6uqYPHkywCd/L1uSZs9hMDDB3WsK2+OAtcVNaTP7LVDn7lML2wuAA4EDgMPc/buF/d8EBrv76Q0+\nQz0HEcmtjz6C5cvhww9DD6OjZH0r61ygn5lVmllXYCQwvcEx04FT4JNi8o67/40wnDTYzDY1MyM0\ntV9JMVYRkQ7XtSvsvHPHFoakUisO7r4GGA3MIvxhn+bu881slJmNKhwzA3jNzF4FJgH/Udj/FHAb\n8Bwwr/CWv0sr1nK17rIwVsov32LOL+bckkq1FeLuM4GZDfZNarA9uplzJwAT0opNRESap+UzREQ6\nmax7DiIiklMqDmUs9nFP5ZdvMecXc25JqTiIiEgj6jmIiHQy6jmIiEibqDiUsdjHPZVfvsWcX8y5\nJaXiICIijajnICLSyajnICIibaLiUMZiH/dUfvkWc34x55aUioOIiDSinoOISCejnoOIiLSJikMZ\ni33cU/nlW8z5xZxbUioOIiLSiHoOIiKdjHoOIiLSJioOZSz2cU/ll28x5xdzbkmpOIiISCPqOYiI\ndDLqOYiISJukWhzMrMbMFpjZYjM7u5ljflF4/QUz27Nof4WZ3WZm883sFTMbnGas5Sj2cU/ll28x\n5xdzbkmlVhzMrAvwP0ANsCtwgpkNaHDMcODz7t4P+HfgN0UvXw3McPcBwO7A/LRiLVf19fVZh5Aq\n5ZdvMecXc25JpXnlMAh41d2XuPtqYCowosExRwI3ALj7U0CFmW1vZlsB+7v79YXX1rj7qhRjLUvv\nvPNO1iGkSvnlW8z5xZxbUmkWh57A0qLtZYV9LR3TC9gZWGlmvzez58zsGjPbLMVYRUSkSJrFIelt\nRA075g6nudgUAAAFzElEQVRsCAwEfu3uA4F/AOeUMLZcWLJkSdYhpEr55VvM+cWcW1Kp3cpaaCBP\ncPeawvY4YK27Tyw65rdAnbtPLWwvAA4kFIw57r5zYf9+wDnufkSDz9B9rCIibdDSrawbpvjZc4F+\nZlYJ/BUYCZzQ4JjpwGhgaqGYvOPufwMws6Vm9gV3XwQMBV5u+AEtJSciIm2TWnFw9zVmNhqYBXQB\nrnP3+WY2qvD6JHefYWbDzexVwtDRqUVvcQZws5l1Bf7c4DUREUlRrmdIi4hIOqKYIW1mZxQmy71k\nZhNbPiN/zGysma01s22yjqWUzOznhe/uBTO7o3Abc64lmfyZV2bW28weMbOXC//efpB1TGkwsy5m\n9ryZ3ZN1LKWWdIJx7ouDmR1EmC+xu7t/Gbgs45BKzsx6A4cCf8k6lhTcD3zJ3fcAFgHjMo6nXZJM\n/sy51cAYd/8SMBg4PbL81jkTeIXkd13mSaIJxrkvDsD3gUsKE+1w95UZx5OGK4AfZx1EGtz9AXdf\nW9h8ijDPJc+STP7MLXdf4e71hd/fJ/xh2THbqErLzHoBw4FraXyrfa61ZoJxDMWhH3CAmT1pZnVm\n9pWsAyolMxsBLHP3eVnH0gG+DczIOoh2SjL5MwqFOxH3JBT1mFwJnAWsbenAHEo8wTjNW1lLxswe\nAHo08dJ5hBy2dvfBZrY38Eegb0fG114t5DcOGFZ8eIcEVULrye9cd7+ncMx5wEfu/ocODa70YhyG\naMTMugG3AWcWriCiYGZHAG+6+/NmVp11PClYN8F4tLs/Y2ZXESYYn9/UgWXP3Q9t7jUz+z5wR+G4\nZwpN223d/a0OC7CdmsvPzL5MqPQvmBmEIZdnzWyQu7/ZgSG2y/q+PwAzqyVcxh/SIQGlaznQu2i7\nN+HqIRpmthFwOzDF3e/KOp4S2xc4srAo6CbAlmZ2o7ufknFcpbKMMBLxTGH7NppZfSKGYaW7gIMB\nzOwLQNc8FYb1cfeX3H17d9+5MFt8GTAwT4WhJWZWQ7iEH+HuH2QdTwl8MvmzMEdnJGGyZxQs/FfK\ndcAr7n5V1vGUmruf6+69C//evgE8HFFhwN1XAEsLfyuhmQnGkJMrhxZcD1xvZi8CHwHRfJFNiHHI\n4pdAV+CBwtXRHHf/j2xDarvmJn9mHFYpDQFOBuaZ2fOFfePc/b4MY0pTjP/mEk0w1iQ4ERFpJIZh\nJRERKTEVBxERaUTFQUREGlFxEBGRRlQcRESkERUHERFpRMVBpAlmdqWZnVm0PcvMrinavtzMxjRz\n7n+Z2Xpne5vZBDMb28T+rQqz/kUypeIg0rTHCUspYGYbANsSluBeZx9gdlMnuvsF7v5QC+/f3ASj\nrYHcTgKUeKg4iDRtDqEAAHwJeAl4r/CglI2BAQCFlYDnmtl9ZtajsG+ymR1b+H144aEqc83sFw0e\nHrNr4cE5fzazMwr7LgV2KTxoJsoHV0k+xLB8hkjJuftfzWxN4UFL+xCKRc/C7+8SnmNwJWFNqP9n\nZiOBi4DvEK4K3Mw2AX5LWD//L2b2Bz69YjCgP1ANbAksNLNfA2cTHn60ZwelKtIkFQeR5j1BGFra\nl/DApZ6F31cRVl8dxqdrQnUB/lp07ro//q+5+7on+N0C/HvhdwfuLTwQ6C0zexPYnhwuyS5xUnEQ\nad5swkJzuwEvEh7i85+E4lAH9HT3fddzfsO+QsM//B8V/f4x+vcoZUQ9B5HmPQEcAbzlwdtABWFo\n6Rag+7qHs5vZRmZW3LB2YCHQ18z6FPaN5LPDSk15D9iitGmItJ6Kg0jzXiLcpfRk0b55wDuFZ5Uf\nB0w0s3rgeT5tYANQeD7FfwD3mdlcQq9i3fN6nSbuWCo8i2S2mb2ohrRkSUt2i6TIzDZ3938Ufv8V\nsMjdr844LJEW6cpBJF3fK9yW+jLhrqRJWQckkoSuHEREpBFdOYiISCMqDiIi0oiKg4iINKLiICIi\njag4iIhIIyoOIiLSyP8HyBw/RNRrVoUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1054055d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(weightsToTry, costs)\n",
    "grid(1)\n",
    "ylabel('Cost')\n",
    "xlabel('Weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you may be thinking that 0.04 seconds to train a network is not so bad, and we haven't even optimized anything yet. Plus, there are other, way faster languages than python our there. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we optimize through, let's consider the full complexity of the problem. Remember the 0.04 seconds required is only for one weight, and we have 9 total! Let's next consider 2 weights for a moment. To maintain the same precision we now need to check 1000 times 1000, or one million values. This is a lot of work, even for a fast computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "weightsToTry = np.linspace(-5,5,1000)\n",
    "costs = np.zeros((1000, 1000))\n",
    "\n",
    "startTime = time.clock()\n",
    "for i in range(1000):\n",
    "    for j in range(1000):\n",
    "        NN.W1[0,0] = weightsToTry[i]\n",
    "        NN.W1[0,1] = weightsToTry[j]\n",
    "        yHat = NN.forward(X)\n",
    "        costs[i, j] = 0.5*sum((y-yHat)**2)\n",
    "    \n",
    "endTime = time.clock()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "62.06971600000001"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timeElapsed = endTime-startTime\n",
    "timeElapsed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After our 1 million evaluations we’ve found our solution, but it took an agonizing 40 seconds! The real curse of dimensionality kicks in as we continue to add dimensions. Searching through three weights would take a billion evaluations, or 11 hours! Searching through all 9 weights we need for our simple network would take 1,268,391,679,350,583.5 years. (Over a quardrillion years). So for that reason, the \"just try everything\" or brute force optimization method is clearly not going to work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1268391679350583.5"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0.04*(1000**(9-1))/(3600*24*365)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's return to the 1-dimensional case and see if we can be more clever. Let's evaluate our cost function for a specific value of w. If w is 1.1 for example, we can run our cost function, and see that J is 2.8. Now we haven't learned much yet, but let's try to add a little information to what we already know. What if we could figure out which way was downhill? If we could, we would know whether to make W smaller or larger do decrease the cost. We could test the cost function immediately to the left and right of our test point and see which is smaller. This is called numerical estimation, and is sometimes a good approach, but for us, there's a faster way. Let's look at our equations so far."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "z^{(2)} = XW^{(1)} \\tag{1}\\\\\n",
    "$$\n",
    "$$\n",
    "a^{(2)} = f(z^{(2)}) \\tag{2}\\\\\n",
    "$$\n",
    "$$\n",
    "z^{(3)} = a^{(2)}W^{(2)} \\tag{3}\\\\\n",
    "$$\n",
    "$$\n",
    "\\hat{y} = f(z^{(3)}) \\tag{4}\\\\\n",
    "$$\n",
    "$$\n",
    "J = \\sum \\frac{1}{2}(y-\\hat{y})^2 \\tag{5}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 5 equations, but we can really think of them as one big equation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And since we have one big equation that uniquely determines our cost, J, from X, y, W1, and W2, we can use our good friend calculus to find what we're looking for. We want to know \"which way is downhill\", that is, what is the rate of change of J with respect to W, also known as the derivative. And in this case, since we’re just considering one weight at a time, the partial derivative. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can derive an expression for dJdW, that will give us the rate of change of J with respect to W, for any value of W! If dJdW is positive, then the cost function is going uphill. If dJdW is negative the cost function is going downhill. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can really speed things up. Since we know in which direction the cost decreases, we can save all that time we would have spent searching in the wrong direction. We can save even more computational time by iteratively taking steps downhill and stopping when the cost stops getting smaller. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method is known as gradient descent, and although it may not seem so impressive in one dimension, it is capable of incredible speedups in higher dimensions. In fact, in our final video, we’ll show that what would have taken 10^27 function evaluations with our brute force method will take less than 100 evaluations with gradient descent. Gradient descent allows us to find needles in very very very large haystacks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now before we celebrate too much here, there is a restriction. What if our cost function doesn't always go in the same direction? What if it goes up, then back down? The mathematical name for this is non-convex, and it could really throw off our gradient descent algorithm by getting it stuck in a local minima instead of our ideal global minima. One of the reasons we chose our cost function to be the sum of squared errors was to exploit the convex nature of quadratic equations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know that the graph of y equals x squared is a nice convex parabola and it turns out that higher dimensional versions are too!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another piece of the puzzle here is that depending on how we use our data, it might not matter if or cost function is convex or not. If we use our examples one at a time instead of all at once, sometimes it won't matter if our cost function is convex, we will still find a good solution. This is called stochastic gradient descent. So maybe we shouldn't be afraid of non-convex cost functions, as Neural Network wizard Yann Lecun says in his excellent talk \"Who is afraid on non-convex loss funtions?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Link to Yann's Talk:\n",
    "http://videolectures.net/eml07_lecun_wia/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The details of gradient descent are a deep topic for another day, for now we're going to do our gradient descent \"batch\" style, where we use all our example at once, and the way we've setup our cost function will keep things nice and convex. Next time we'll compute and code up our gradients!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}