exercise_480.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Converting Outputs to Probabilities Using the Logistic Function\n",
    "\n",
    "We don't always have to create a probability distrbution over the outputs by using a softmax function.\n",
    "\n",
    "A logistic function with normalization can also give us a good probability distribution from the output.\n",
    "\n",
    "The probability distribution that can be obtained from the normalized logistic function (also called the sigmoid function) is less extreme than that of the softmax.\n",
    "\n",
    "So, it might not work as well as a loss function, but oftentimes yields better thresholdable probabilities.\n",
    "\n",
    "Here we compute the cross-entropy loss using the normalized sigmoid function instead of the softmax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c1 and c2: \n",
      " 6.0000  3.9000\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "sigmoid: Variable containing:\n",
      " 0.9975  0.9802\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "normalized sigmoid: Variable containing:\n",
      " 0.5044  0.4956\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "data = torch.Tensor([[6, 2, 1.9]])\n",
    "\n",
    "weights = torch.Tensor([[1, 0],[0, 1],[0, 1]])\n",
    "\n",
    "c = torch.mm(data, weights)\n",
    "print(\"c1 and c2: \" + str(c))\n",
    "\n",
    "result = F.sigmoid(torch.autograd.Variable(c))\n",
    "print(\"sigmoid: \" + str(result))\n",
    "\n",
    "# Normalize it\n",
    "\n",
    "result = result / torch.sum(result)\n",
    "print(\"normalized sigmoid: \" + str(result))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convince yourself that the output of the 'softmax' function is the same as 'e to the power of c normalized'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "log(sigmoid): Variable containing:\n",
      "-0.6844 -0.7020\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "Loss: 0.6844037175178528\n"
     ]
    }
   ],
   "source": [
    "result = torch.log(result)\n",
    "print(\"log(sigmoid): \" + str(result))\n",
    "\n",
    "# The correct category\n",
    "target = torch.LongTensor(1)\n",
    "target[0] = 0 \n",
    "\n",
    "loss = F.nll_loss(result, torch.autograd.Variable(target))\n",
    "print(\"Loss: \"+str(loss.data[0]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the loss obtained using the sigmoid function with the loss from the softmax function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 0.11551953107118607\n"
     ]
    }
   ],
   "source": [
    "loss = F.cross_entropy(torch.autograd.Variable(c), torch.autograd.Variable(target))\n",
    "print(\"Loss: \"+str(loss.data[0]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the loss is lower for the softmax function reflecting the higher (extreme) confidence of the classifier in the category it has correctly selected.\n",
    "\n",
    "The confidence of the classifier and the loss are less extreme for the normalized sigmoid function, making it a rather useful function for thresholding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}