Part 3 - Training Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Neural Networks\n",
    "\n",
    "The network we built in the previous part isn't so smart, it doesn't know anything about our handwritten digits. Neural networks with non-linear activations work like universal function approximators. There is some function that maps your input to the output. For example, images of handwritten digits to class probabilities. The power of neural networks is that we can train them to approximate this function, and basically any function given enough data and compute time.\n",
    "\n",
    "<img src=\"assets/function_approx.png\" width=500px>\n",
    "\n",
    "At first the network is naive, it doesn't know the function mapping the inputs to the outputs. We train the network by showing it examples of real data, then adjusting the network parameters such that it approximates this function.\n",
    "\n",
    "To find these parameters, we need to know how poorly the network is predicting the real outputs. For this we calculate a **loss function** (also called the cost), a measure of our prediction error. For example, the mean squared loss is often used in regression and binary classification problems\n",
    "\n",
    "$$\n",
    "\\ell = \\frac{1}{2n}\\sum_i^n{\\left(y_i - \\hat{y}_i\\right)^2}\n",
    "$$\n",
    "\n",
    "where $n$ is the number of training examples, $y_i$ are the true labels, and $\\hat{y}_i$ are the predicted labels.\n",
    "\n",
    "By minimizing this loss with respect to the network parameters, we can find configurations where the loss is at a minimum and the network is able to predict the correct labels with high accuracy. We find this minimum using a process called **gradient descent**. The gradient is the slope of the loss function and points in the direction of fastest change. To get to the minimum in the least amount of time, we then want to follow the gradient (downwards). You can think of this like descending a mountain by following the steepest slope to the base.\n",
    "\n",
    "<img src='assets/gradient_descent.png' width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation\n",
    "\n",
    "For single layer networks, gradient descent is simple to implement. However, it's more complicated for deeper, multilayer neural networks like the one we've built. Complicated enough that it took about 30 years before researchers figured out how to train multilayer networks, although it's straightforward once you learn about it. \n",
    "\n",
    "This is done through **backpropagation** which is really just an application of the chain rule from calculus. It's easiest to understand if we convert a two layer network into a graph representation.\n",
    "\n",
    "<img src='assets/w1_backprop_graph.png' width=400px>\n",
    "\n",
    "In the forward pass through the network, our data and operations go from right to left here. To train the weights with gradient descent, we propagate the gradient of the cost backwards through the network. Mathematically, this is really just calculating the gradient of the loss with respect to the weights using the chain rule.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\ell}{\\partial w_1} = \\frac{\\partial l_1}{\\partial w_1} \\frac{\\partial s}{\\partial l_1} \\frac{\\partial l_2}{\\partial s} \\frac{\\partial \\ell}{\\partial l_2}\n",
    "$$\n",
    "\n",
    "We update our weights using this gradient with some learning rate $\\alpha$. \n",
    "\n",
    "$$\n",
    "w^\\prime = w - \\alpha \\frac{\\partial \\ell}{\\partial w}\n",
    "$$\n",
    "\n",
    "The learning rate is set such that the weight update steps are small enough that the iterative method settles in a minimum.\n",
    "\n",
    "The first thing we need to do for training is define our loss function. In PyTorch, you'll usually see this as `criterion`. Here we're using softmax output, so we want to use `criterion = nn.CrossEntropyLoss()` as our loss. Later when training, you use `loss = criterion(output, targets)` to calculate the actual loss.\n",
    "\n",
    "We also need to define the optimizer we're using, SGD or Adam, or something along those lines. Here I'll just use SGD with `torch.optim.SGD`, passing in the network parameters and the learning rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autograd\n",
    "\n",
    "Torch provides a module, `autograd`, for automatically calculating the gradient of tensors. It does this by keeping track of operations performed on tensors. To make sure PyTorch keeps track of operations on a tensor and calculates the gradients, you need to set `requires_grad` on a tensor. You can do this at creation with the `requires_grad` keyword, or at any time with `x.requires_grad_(True)`.\n",
    "\n",
    "You can turn off gradients for a block of code with the `torch.no_grad()` content:\n",
    "```python\n",
    "x = torch.zeros(1, requires_grad=True)\n",
    ">>> with torch.no_grad():\n",
    "...     y = x * 2\n",
    ">>> y.requires_grad\n",
    "False\n",
    "```\n",
    "\n",
    "Also, you can turn on or off gradients altogether with `torch.set_grad_enabled(True|False)`.\n",
    "\n",
    "The gradients are computed with respect to some variable `z` with `z.backward()`. This does a backward pass through the operations that created `z`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "from collections import OrderedDict\n",
    "\n",
    "import numpy as np\n",
    "import time\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import helper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.randn(2,2, requires_grad=True)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x**2\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we can see the operation that created `y`, a power operation `PowBackward0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## grad_fn shows the function that generated this variable\n",
    "print(y.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The autgrad module keeps track of these operations and knows how to calculate the gradient for each one. In this way, it's able to calculate the gradients for a chain of operations, with respect to any one tensor. Let's reduce the tensor `y` to a scalar value, the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z = y.mean()\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the gradients for `x` and `y` but they are empty currently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the gradients, you need to run the `.backward` method on a Variable, `z` for example. This will calculate the gradient for `z` with respect to `x`\n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial x} = \\frac{\\partial}{\\partial x}\\left[\\frac{1}{n}\\sum_i^n x_i^2\\right] = \\frac{x}{2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z.backward()\n",
    "print(x.grad)\n",
    "print(x/2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These gradients calculations are particularly useful for neural networks. For training we need the gradients of the weights with respect to the cost. With PyTorch, we run data forward through the network to calculate the cost, then, go backwards to calculate the gradients with respect to the cost. Once we have the gradients we can make a gradient descent step. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the data and define the network\n",
    "\n",
    "The same as we saw in part 3, we'll load the MNIST dataset and define our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import datasets, transforms\n",
    "\n",
    "# Define a transform to normalize the data\n",
    "transform = transforms.Compose([transforms.ToTensor(),\n",
    "                              transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n",
    "                             ])\n",
    "# Download and load the training data\n",
    "trainset = datasets.MNIST('MNIST_data/', download=True, train=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll build a network with `nn.Sequential` here. Only difference from the last part is I'm not actually using softmax on the output, but instead just using the raw output from the last layer. This is because the output from softmax is a probability distribution. Often, the output will have values really close to zero or really close to one. Due to [inaccuracies with representing numbers as floating points](https://docs.python.org/3/tutorial/floatingpoint.html), computations with a softmax output can lose accuracy and become unstable. To get around this, we'll use the raw output, called the **logits**, to calculate the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparameters for our network\n",
    "input_size = 784\n",
    "hidden_sizes = [128, 64]\n",
    "output_size = 10\n",
    "\n",
    "# Build a feed-forward network\n",
    "model = nn.Sequential(OrderedDict([\n",
    "                      ('fc1', nn.Linear(input_size, hidden_sizes[0])),\n",
    "                      ('relu1', nn.ReLU()),\n",
    "                      ('fc2', nn.Linear(hidden_sizes[0], hidden_sizes[1])),\n",
    "                      ('relu2', nn.ReLU()),\n",
    "                      ('logits', nn.Linear(hidden_sizes[1], output_size))]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the network!\n",
    "\n",
    "The first thing we need to do for training is define our loss function. In PyTorch, you'll usually see this as `criterion`. Here we're using softmax output, so we want to use `criterion = nn.CrossEntropyLoss()` as our loss. Later when training, you use `loss = criterion(output, targets)` to calculate the actual loss.\n",
    "\n",
    "We also need to define the optimizer we're using, SGD or Adam, or something along those lines. Here I'll just use SGD with `torch.optim.SGD`, passing in the network parameters and the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's consider just one learning step before looping through all the data. The general process with PyTorch:\n",
    "\n",
    "* Make a forward pass through the network to get the logits \n",
    "* Use the logits to calculate the loss\n",
    "* Perform a backward pass through the network with `loss.backward()` to calculate the gradients\n",
    "* Take a step with the optimizer to update the weights\n",
    "\n",
    "Below I'll go through one training step and print out the weights and gradients so you can see how it changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Initial weights - ', model.fc1.weight)\n",
    "\n",
    "images, labels = next(iter(trainloader))\n",
    "images.resize_(64, 784)\n",
    "\n",
    "# Clear the gradients, do this because gradients are accumulated\n",
    "optimizer.zero_grad()\n",
    "\n",
    "# Forward pass, then backward pass, then update weights\n",
    "output = model.forward(images)\n",
    "loss = criterion(output, labels)\n",
    "loss.backward()\n",
    "print('Gradient -', model.fc1.weight.grad)\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Updated weights - ', model.fc1.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training for real\n",
    "\n",
    "Now we'll put this algorithm into a loop so we can go through all the images. This is fairly straightforward. We'll loop through the mini-batches in our dataset, pass the data through the network to calculate the losses, get the gradients, then run the optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.SGD(model.parameters(), lr=0.003)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 3\n",
    "print_every = 40\n",
    "steps = 0\n",
    "for e in range(epochs):\n",
    "    running_loss = 0\n",
    "    for images, labels in iter(trainloader):\n",
    "        steps += 1\n",
    "        # Flatten MNIST images into a 784 long vector\n",
    "        images.resize_(images.size()[0], 784)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # Forward and backward passes\n",
    "        output = model.forward(images)\n",
    "        loss = criterion(output, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        running_loss += loss.item()\n",
    "        \n",
    "        if steps % print_every == 0:\n",
    "            print(\"Epoch: {}/{}... \".format(e+1, epochs),\n",
    "                  \"Loss: {:.4f}\".format(running_loss/print_every))\n",
    "            \n",
    "            running_loss = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the network trained, we can check out it's predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images, labels = next(iter(trainloader))\n",
    "\n",
    "img = images[0].view(1, 784)\n",
    "# Turn off gradients to speed up this part\n",
    "with torch.no_grad():\n",
    "    logits = model.forward(img)\n",
    "\n",
    "# Output of the network are logits, need to take softmax for probabilities\n",
    "ps = F.softmax(logits, dim=1)\n",
    "helper.view_classify(img.view(1, 28, 28), ps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our network is brilliant. It can accurately predict the digits in our images. Next up you'll write the code for training a neural network on a more complex dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}