exercise_510.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Minimizing the Loss\n",
    "\n",
    "Assuming the loss function to be $x^2+3$ and the neural network's weight to be $x$, you can find the weight $x$ which minimizes the loss by descending the gradient from a random starting value of $x$ (say $x = 2$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weight = tensor(2.)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import random\n",
    "\n",
    "# Let's say the weight = 2\n",
    "weight = torch.nn.Parameter(torch.Tensor([[2]]))\n",
    "\n",
    "print(\"Weight = \"+str(weight.data[0,0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume the weight to be 2.\n",
    "\n",
    "We can now perform the forward and backward passes below as many times as we want.\n",
    "\n",
    "The forward pass involves the computation of the loss from the training data and the current parameters.\n",
    "\n",
    "The backward pass is performed automatically by Pytorch when you call loss.backward().\n",
    "\n",
    "Pytorch calculates all the gradients with respect to the loss.\n",
    "\n",
    "These gradients are stored in each parameter's 'grad' member variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss for weight = tensor(2.) = tensor(7.)\n",
      "Gradient for weight = tensor(2.) = tensor(4.)\n",
      "The next iteration of the weight is tensor(1.9600)\n"
     ]
    }
   ],
   "source": [
    "# Forward pass = computing the loss from the parameters and input data\n",
    "\n",
    "# Let's first set the partial derivative of the loss with respect to the weight to 0.\n",
    "if weight.grad is not None:\n",
    "    weight.grad.data.zero_()\n",
    "\n",
    "# Toy loss function -> loss = weight^2 + 3\n",
    "loss = torch.mm(weight, weight) + 3\n",
    "\n",
    "print(\"Loss for weight = \"+str(weight.data[0,0])+\" = \"+str(loss.data[0,0]))\n",
    "\n",
    "# Backward pass ->  Pytorch does this automatically for you!\n",
    "\n",
    "loss.backward()\n",
    "\n",
    "gradient = weight.grad\n",
    "\n",
    "# The gradient should be 2 * weight\n",
    "print(\"Gradient for weight = \"+str(weight.data[0,0])+\" = \"+str(gradient.data[0,0]))\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "# Gradient descent = when you climb down the loss gradient by nudging the weights in a direction opposite the gradient.\n",
    "\n",
    "# Now weight = weight - lr * dloss/dweight\n",
    "weight.data = weight.data - learning_rate * gradient.data\n",
    "\n",
    "print(\"The next iteration of the weight is \"+str(weight.data[0,0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can run the above code as many times as you want.  The weight will keep moving down the gradient till it reaches the point where the loss is at a minimum.\n",
    "\n",
    "You can also use the code below to run this loop 100 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The loss is now 6.842\n",
      "\tThe gradient is now 3.92\n",
      "\t\tThe weight is now 1.921\n",
      "The loss is now 5.565\n",
      "\tThe gradient is now 3.203\n",
      "\t\tThe weight is now 1.569\n",
      "The loss is now 4.712\n",
      "\tThe gradient is now 2.617\n",
      "\t\tThe weight is now 1.282\n",
      "The loss is now 4.143\n",
      "\tThe gradient is now 2.138\n",
      "\t\tThe weight is now 1.048\n",
      "The loss is now 3.763\n",
      "\tThe gradient is now 1.747\n",
      "\t\tThe weight is now 0.856\n",
      "The loss is now 3.509\n",
      "\tThe gradient is now 1.428\n",
      "\t\tThe weight is now 0.699\n",
      "The loss is now 3.34\n",
      "\tThe gradient is now 1.166\n",
      "\t\tThe weight is now 0.572\n",
      "The loss is now 3.227\n",
      "\tThe gradient is now 0.953\n",
      "\t\tThe weight is now 0.467\n",
      "The loss is now 3.152\n",
      "\tThe gradient is now 0.779\n",
      "\t\tThe weight is now 0.382\n",
      "The loss is now 3.101\n",
      "\tThe gradient is now 0.636\n",
      "\t\tThe weight is now 0.312\n"
     ]
    }
   ],
   "source": [
    "for i in range(100):\n",
    "\n",
    "    if weight.grad is not None:\n",
    "        weight.grad.data.zero_()\n",
    "    \n",
    "    loss = torch.mm(weight, weight) + 3\n",
    "    \n",
    "    if i % 10 == 0:\n",
    "        print(\"The loss is now \"+str(round(loss.data[0,0].item(),3)))\n",
    "    \n",
    "    loss.backward()\n",
    "    \n",
    "    gradient = weight.grad\n",
    "    \n",
    "    if i % 10 == 0:\n",
    "        print(\"\\tThe gradient is now \"+str(round(gradient.data[0,0].item(),3)))\n",
    "        \n",
    "    learning_rate = 0.01\n",
    "    \n",
    "    weight.data = weight.data - learning_rate * gradient.data\n",
    "    \n",
    "    if i % 10 == 0:\n",
    "        print(\"\\t\\tThe weight is now \"+str(round(weight.data[0,0].item(),3)))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run it about 400 times, the weight should get really close to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}