exercise_430.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss Function\n",
    "\n",
    "We can create a probability distrbution over the outputs by using a softmax function which is defined as:\n",
    "\n",
    "$$\\frac{e^{x}}{\\sum_{x'}e^{x'}}$$\n",
    "\n",
    "We can then use that probability distribution and knowledge of the correct categories to evaluate the classifier.\n",
    "\n",
    "A classifier with high confidence in an incorrect category is a poor classifier.  A classifier with low confidence in an incorrect category and high confidence in the correct category is a good classifier.\n",
    "\n",
    "We show below how the softmax function is applied to the output values to turn them into probabilities, and how a measure of the incorrectness, the cross-entropy loss, is computed from them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c1 and c2: \n",
      " 6.0000  3.9000\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "e to the power of c: \n",
      " 403.4288   49.4025\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "e to the power of c normalized: \n",
      " 0.8909  0.1091\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "softmax: Variable containing:\n",
      " 0.8909  0.1091\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "data = torch.Tensor([[6, 2, 1.9]])\n",
    "\n",
    "weights = torch.Tensor([[1, 0],[0, 1],[0, 1]])\n",
    "\n",
    "c = torch.mm(data, weights)\n",
    "print(\"c1 and c2: \" + str(c))\n",
    "\n",
    "exp = torch.exp(c)\n",
    "print(\"e to the power of c: \"+str(exp))\n",
    "\n",
    "soft = exp / torch.sum(exp)\n",
    "print(\"e to the power of c normalized: \"+str(soft))\n",
    "\n",
    "result = F.softmax(torch.autograd.Variable(c), dim=1)\n",
    "print(\"softmax: \" + str(result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convince yourself that the output of the 'softmax' function is the same as 'e to the power of c normalized'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "log(softmax): Variable containing:\n",
      "-0.1155 -2.2155\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "Loss: 0.11551952362060547\n"
     ]
    }
   ],
   "source": [
    "result = torch.log(result)\n",
    "print(\"log(softmax): \" + str(result))\n",
    "\n",
    "# The correct category\n",
    "target = torch.LongTensor(1)\n",
    "target[0] = 0 \n",
    "\n",
    "loss = F.nll_loss(result, torch.autograd.Variable(target))\n",
    "print(\"Loss: \"+str(loss.data[0]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pytorch also has a ready-made function to compute cross entropy directly from the neural network's outputs.\n",
    "Convince yourself that the loss calculated by either method is the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 0.11551953107118607\n"
     ]
    }
   ],
   "source": [
    "loss = F.cross_entropy(torch.autograd.Variable(c), torch.autograd.Variable(target))\n",
    "print(\"Loss: \"+str(loss.data[0]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try this for different values of input data.\n",
    "\n",
    "You could try them in the following order - in order of increasing difficulty\n",
    "\n",
    "$\\begin{bmatrix}6 & 2 & 1.9\\end{bmatrix}$\n",
    "$\\begin{bmatrix}5 & 2 & 1.9\\end{bmatrix}$\n",
    "$\\begin{bmatrix}4 & 2 & 1.9\\end{bmatrix}$\n",
    "\n",
    "You should see the loss increase as the input data points become more difficult to decide about (as $f_{1}$ and $f_{2} + f_{3}$ get increasingly closer.  Remember, we're still using the weight matrix for Toy Problem 2 and the decision that classifier has to take is whether $f_{1} < f_{2} + f_{3}$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}