exercise_480.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Converting Outputs to Probabilities Using the Logistic Function\n",
    "\n",
    "We don't always have to use the softmax function to create a probability distrbution over the outputs.\n",
    "\n",
    "A logistic function with normalization can also give us a good probability distribution from the output.\n",
    "\n",
    "The probability distribution that can be obtained from the normalized logistic function (also called the sigmoid function) is less extreme than that of the softmax.\n",
    "\n",
    "So, it might not work as well as a loss function, but oftentimes yields better thresholdable probabilities.\n",
    "\n",
    "Here we compute the cross-entropy loss using the normalized sigmoid function instead of the softmax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c1 and c2: tensor([[ 6.0000,  3.9000]])\n",
      "sigmoid: tensor([[ 0.9975,  0.9802]])\n",
      "normalized sigmoid: tensor([[ 0.5044,  0.4956]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "data = torch.Tensor([[6, 2, 1.9]])\n",
    "\n",
    "weights = torch.Tensor([[1, 0],[0, 1],[0, 1]])\n",
    "\n",
    "c = torch.mm(data, weights)\n",
    "print(\"c1 and c2: \" + str(c))\n",
    "\n",
    "result = F.sigmoid(torch.autograd.Variable(c))\n",
    "print(\"sigmoid: \" + str(result))\n",
    "\n",
    "# Normalize it\n",
    "\n",
    "result = result / torch.sum(result)\n",
    "print(\"normalized sigmoid: \" + str(result))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rest of the code remains the same as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "log(sigmoid): Variable containing:\n",
      "-0.6844 -0.7020\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "Loss: 0.6844037175178528\n"
     ]
    }
   ],
   "source": [
    "result = torch.log(result)\n",
    "print(\"log(sigmoid): \" + str(result))\n",
    "\n",
    "# The correct category\n",
    "target = torch.LongTensor(1)\n",
    "target[0] = 0 \n",
    "\n",
    "loss = F.nll_loss(result, torch.autograd.Variable(target))\n",
    "print(\"Loss: \"+str(loss.data.item()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the loss obtained using the sigmoid function with the loss from the softmax function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 0.11551953107118607\n"
     ]
    }
   ],
   "source": [
    "loss = F.cross_entropy(torch.autograd.Variable(c), torch.autograd.Variable(target))\n",
    "print(\"Loss: \"+str(loss.data.item()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the loss is lower for the softmax function reflecting the higher (extreme) confidence of the classifier in the category it has correctly selected.\n",
    "\n",
    "The confidence of the classifier and the loss are less extreme for the normalized sigmoid function, making it a rather useful function for thresholding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}