Breaking Gradient Descent

Breaking Gradient Descent

Chapter 4/7. breaking Gradient Descent.ipynb

@@ -0,0 +1,99 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Learning is adjusting our weight to reduce the error to zero.\n",
+    "\n",
+    "The formular: error=((input*weight)-goal_pred)**2\n",
+    "\n",
+    "Derattive: it means how much does rod X move when tug on rod Y. Given a function, the derivative represents the direction and the amount that one variable changes if you change the other variable. \n",
+    "\n",
+    "The slope's sign gives us direction and the slope's steepness gives us amount.\n",
+    "\n",
+    "Bottom Line: In this book we are going to build nerual networks. A nerual network is really just one thing...a bunch of weights which we use to compute an error funtion. And for any error function(no matter how complicated),we can compute the relationship between any weight and the final error of the network. With this information, we can change each weight in our neural network to reduce our error down to 0... and that's exactly what we are going to do.\n",
+    "\n",
+    "This methos for learning(finding erroe minimums) is called Gradient Descent. This name should seem intuitive. We move in the weight value opposite the gradient value,which descend our error to 0. By opposite, I simply mean that we increase our weight when we have a negative gradient and vise versa. It is like gravity!\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Error: 0.04      Prediction: 1.0\n",
+      "Error: 0.36      Prediction: 0.2\n",
+      "Error: 3.24      Prediction: 2.6\n",
+      "Error: 29.16      Prediction: -4.6\n",
+      "Error: 262.44      Prediction: 17.0\n",
+      "Error: 2361.96      Prediction: -47.8\n",
+      "Error: 21257.64      Prediction: 146.6\n",
+      "Error: 191318.76      Prediction: -436.6\n",
+      "Error: 1721868.84      Prediction: 1313.0\n",
+      "Error: 15496819.56      Prediction: -3935.8\n",
+      "Error: 139471376.04      Prediction: 11810.6\n",
+      "Error: 1255242384.36      Prediction: -35428.6\n",
+      "Error: 11297181459.2      Prediction: 106289.0\n",
+      "Error: 1.01674633133e+11      Prediction: -318863.8\n",
+      "Error: 9.15071698198e+11      Prediction: 956594.6\n",
+      "Error: 8.23564528379e+12      Prediction: -2869780.6\n",
+      "Error: 7.41208075541e+13      Prediction: 8609345.0\n",
+      "Error: 6.67087267987e+14      Prediction: -25828031.8\n",
+      "Error: 6.00378541188e+15      Prediction: 77484098.6\n",
+      "Error: 5.40340687069e+16      Prediction: -232452292.6\n"
+     ]
+    }
+   ],
+   "source": [
+    "weight=0.5\n",
+    "goal_pred=0.8\n",
+    "inputs=2\n",
+    "\n",
+    "# Train the model\n",
+    "\n",
+    "for iteration in range(20):\n",
+    "    prediction=inputs*weight\n",
+    "    error=(prediction-goal_pred)**2\n",
+    "    weight_delta=inputs*(prediction-goal_pred)\n",
+    "    weight-=weight_delta\n",
+    "    \n",
+    "    print 'Error: '+str(error)+'      Prediction: '+str(prediction)\n",
+    "    \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You see, if we have a BIG input,then the prediction is very sensitive to changes in the weight since pred=input*weight. This can cause our network to overcorrect."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python [conda root]",
+   "language": "python",
+   "name": "conda-root-py"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}