baseline_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Baseline Model\n",
    "\n",
    "This is a simple baseline model that uses the relative frequency of a MeSH term's occurrence in a document's references in order to decide whether to apply that particular MeSH term to the document. MeSH term counts (and consequently relative frequencies) were determined by mining PMC full-texts to extract as many of a particular document's references as possible. MeSH terms were then counted for each of these references, summed, and divided by the total to give the relative frequency that each term was applied to the original document's references. This is all detailed in the notebook [data_aggregation_pipeline.ipynb](https://github.com/wigasper/FUSE/blob/master/data_aggregation_pipeline.ipynb), or in the original Python files in the [data-aggregation directory](https://github.com/wigasper/FUSE/tree/master/data-aggregation).\n",
    "\n",
    "For the model, I selected a series of threshold values starting at 0 and increasing up to 1. For each threshold, for each document in the dataset, if a term's frequency among the document's references exceeds the threshold, I predict that the term will be applied to the document.\n",
    "\n",
    "This model serves as a basis for evaluation for future, more complex, models.\n",
    "\n",
    "Throughout this process I've utilized lists and dictionaries as my primary data structures, and stored them on disk as JSON formatted files. This, in general, has turned out to be a good decision. A lot of this process requires millions of lookups on what would otherwise be an extremely large, sparse table. Using dictionaries rather than a matrix or Pandas DF speeds up this process immensely - this simple model takes less than a minute to run (faster than it takes just to load in the data using Pandas) and scales efficiently. It also takes up significantly less space on disk: the JSON file for the term frequency data is 83.5 MB, while storing as table in csv format takes up 448.8 MB. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "from sklearn.metrics import auc\n",
    "from matplotlib import pyplot\n",
    "\n",
    "# Load in term frequencies\n",
    "with open(\"./data/term_freqs.json\", \"r\") as handle:\n",
    "    term_freqs = json.load(handle)\n",
    "\n",
    "# Load in solution values\n",
    "with open(\"./data/baseline_solution.json\", \"r\") as handle:\n",
    "    solution = json.load(handle)\n",
    "\n",
    "# Dict to store predictions and lists to store evaluation metrics\n",
    "predictions = {}\n",
    "precisions = []\n",
    "recalls = []\n",
    "f1s = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next I generate a list of thresholds to test, starting at 0 and going to 1 in increments of .005."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "thresholds = [x * .005 for x in range(0,200)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the model for all thresholds, calculate evaluation metrics as each threshold is tested and record."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the model for all thresholds\n",
    "for thresh in thresholds:\n",
    "    # Predict\n",
    "    for doc in term_freqs:\n",
    "        predictions[doc[0]] = [key for key, val in doc[1].items() if val > thresh]\n",
    "        \n",
    "    # Get evaluation metrics\n",
    "    true_pos = 0\n",
    "    false_pos = 0\n",
    "    false_neg = 0\n",
    "    \n",
    "    for pmid in predictions:\n",
    "        true_pos += len([pred for pred in predictions[pmid] if pred in solution[pmid]])\n",
    "        false_pos += len([pred for pred in predictions[pmid] if pred not in solution[pmid]])\n",
    "        false_neg += len([sol for sol in solution[pmid] if sol not in predictions[pmid]])\n",
    "\n",
    "    # Avoid a ZeroDivisionError if precision and recall are both 0\n",
    "    if true_pos == 0:\n",
    "        precision = 0\n",
    "        recall = 0\n",
    "        f1 = 0\n",
    "    else:\n",
    "        precision = true_pos / (true_pos + false_pos)\n",
    "        recall = true_pos / (true_pos + false_neg)\n",
    "        f1 = (2 * precision * recall) / (precision + recall)\n",
    "    \n",
    "    precisions.append(precision)\n",
    "    recalls.append(recall)\n",
    "    f1s.append(f1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC:  0.3974255073464244\n"
     ]
    }
   ],
   "source": [
    "# Calculate AUC\n",
    "print(\"AUC: \", auc(recalls, precisions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl4VNX9x/H3NwkBRAWVoAhhkWIVcSUGRFEU2gJFqIIW3KAqm6i1WuvWKtr+3GprawUVFXEFXDEgSBVBEAEJolhQFBEF0RI3rCJCwvn9cSaQjWSAmdy5dz6v5+HJ3CWZ7yXhw8m5555jzjlERCRaMoIuQEREEk/hLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCIoK6g3bty4sWvVqlVQby8iEkqLFy/+wjmXU9N5gYV7q1atKCwsDOrtRURCycw+juc8dcuIiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgE1RjuZjbOzNab2X92cNzM7C4zW2lmS83smMSXKSIiOyOeoZDjgbuBR3ZwvCfQNvanI3BP7GNyjGpY5vWGpL2NiEiY1dhyd87NAb6q5pS+wCPOWwA0MrOmiSqwnLLBXtW2iIgAielzbwasKbO9NravEjMbamaFZlZYVFSUgLcWEZGqJCLcrYp9Va667Zwb65zLc87l5eTU+PSsiIjsokSE+1ogt8x2c2BdAr5uZRX72NXnLiJSpUSEewFwXmzUTCdgg3PuswR83ao1iLX4f/9B0t5CRCTsahwtY2YTgK5AYzNbC9wA1AFwzt0LTAN6ASuBjcBvklWsiIjEp8Zwd84NrOG4A0YmrCIREdltekJVRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkHpE+5r3oC5f/MfRUQiLivoAmrF6tfgkb6wtRgysuDXj8FPewZdlYhI0kQ/3D9dDE8N9sEO/uPEs+DQU6Hl8fDjd9C6C+TmB1qmiEgiRTfcP5oLs2+Gj+fDHvtBZjZsLYHMLDikN7z/b1j+vD83MxsGv6CAF5HIiGa4f7IQHukDbitYJvS7H7L3hNVzoVWslT77Nph9C+CgZDPMuA7OmwzZDYKuXkRkt0Xzhurr//TBXmrdEh/oXa7Y3jpvczJk1fPhn5EJa9+Aezr7Fr+ISMhFr+X+1Sr4YCZYBmC+y6VVl8rn5ebDoILtrfmSLfD8SHi4Nxw7xHfdrFu8vaUvIhIi0Qr3j+fDsxf6lnj/R+GLFdWHc25++WMjXodX/gwLxsCiB/y+rLowaIoCXkRCJa5uGTPrYWYrzGylmV1dxfEWZjbLzJaY2VIz65X4Umuw5g3f6t6wFrZuhj2blO+GiUf2HtDjFugwGHD+T/EmePFaWP9ukgoXEUm8GsPdzDKB0UBPoB0w0MzaVTjtj8CTzrmjgQHAmEQXWqOP5pQZ7rjVd7fsqqPOhqz6vmvHMuGzJTCmE4zvDcueg49f1wNRIpLS4umWyQdWOudWAZjZRKAvsLzMOQ7YO/a6IbAukUXGpV6j2IuMHfezx6tif/y+B8GSR2HROD9mHgDzN2QHFajLRkRSTjzh3gxYU2Z7LdCxwjmjgH+b2SVAA6B7QqrbGeuX+1A/4XL4SbfdD9yK/fEn/A46X+pvur49Ad9l8wMsnaRwF5GUE0+fu1Wxz1XYHgiMd841B3oBj5pZpa9tZkPNrNDMCouKina+2h0pKfYPJB3ySzj5muSFbUYm5J2/vcsGYNGD8OI1/klXEZEUEU+4rwVyy2w3p3K3ywXAkwDOuflAPaBxxS/knBvrnMtzzuXl5OTsWsVVWT0HNn4B7fsl7mvuSGmXzSl/hHOf82G/4B4Y3RFWTE/++4uIxCGecF8EtDWz1maWjb9hWlDhnE+AbgBmdig+3BPYNK/Bwnt9l8y2fvckK30gqs0p0PvvcP4MqLsXTBgAT54H//u8duoQEdmBGsPdOVcMXAzMAN7Fj4pZZmY3mVmf2GlXAEPM7G1gAjDYOVex6yY5PpwN78/wUwg8fkYwI1hadIRhc3xrfsWLcHc+/PtPMOcOjagRkUDE9RCTc24aMK3CvuvLvF4OHJ/Y0uKw5g1/g7NUyWY/wiWIG5xZ2XDilXDY6fDUb+D1u2L768KgqbrpKiK1Krxzy6xb4sedf7s2tiMBQyATYb820K4v2+5DF/8IM2+CLZsCLUtE0kt4w33NQt9SBz9ypU3X1Blz3rrL9knJLNP/NnFPZ/+glYhILQhvuOd23D4cMbMudE3iEMidtW1EzXVw/ot+VI0rgYdPhckjYeNXQVcoIhEX3onDDjwaDjgcvv8CzngodYK9VKVJyebDnNth3l3w/ot+DpvDzwCr6jECEZHdE96WO/humaZHpl6wVyV7D+g+Coa9Cvu0hGeHwGOnw7LJmqdGRBIuvC13gO+LwhHsZR1wOFzwkn+y9aXr4cNX0Dw1IpJo4W25by2BjV9CgwQ+6VpbMjKh41DoOCy2IzZPjZ5wFZEECW+4//CVX0qvQZOgK9l1h/zSz1NTOmxy0YN+4W4Rkd0U3nD/br3/2KDSFDbhUTqqptuf4PQHoGFzeOIMmH6VxsWLyG4Jb5/7Jwv9x41fBlvH7io7qubQU+HlG/xcOatfg34PQpNDgq1PREIpvC33uXf4j/++LjojTerUg563wVlP+snHxp7ku2pqaZoeEYmO8IZ76ZJ6JcW7t6ReKjr4F36x7pad4YXLYdI5evBJRHZKeMN929OpKTCfTDLstT+c/Qz8/P/8rJf3dIaF92lMvIjEJbx97s3y/NJ65z4b3bHhGRnQ+WI/V82Es2D6H/Bj4uvCoCnRvW4R2W3hbblv3QL7tk6PgGt6JBxzbmzDQfEmeO+FQEsSkdQW3nD/viicDzDtqjanxMbEl1m79YOXAy1JRFJXeMP9u//6gE+X/udtY+L/CP3GQaMW8Hg/P1d8SXHQ1YlIiglvuJdshs/fgYf7pFfAd7kCDu8HQ2bCMef5G6yP9NW6rSJSTnjDHQC3fWm9dFOnPvT5F/zqXlj3Jtx7AqyaHXRVIpIiQh7uKbK0XpCOGghDZkH9feGRX8HsW/2kaiKS1sId7kedpWlywU9RMHQWHPFrmH2Lnyf+u6KgqxKRAIU73Ft1UbCXym4Ap93ru2o+WeC7aVbPC7oqEQlI+MK9ZMv211N/mz43U+Nh5m+yXjjTh/3DvWHqZTDnDv09iaSZ8D2hWjbcS7b4m6lqvZd3QHu/nN+kc6DwIb8vq56eahVJI+FruWfWKfM6zW+mVqfuXtDqRLYtBFK8CeaP1gyTImki3OGum6nVa93Ft9gt00+0tnwyTBgA334WdGUikmTh65YpS8FevdKnWlfPhRadYd0S/0TrmI7Q4zY4coDvpxeRyAl3uEvNyq701PI4P1f88yNh8nBY9hyc+k/Yu2mwNYpIwoWvW0Z2z35tYPA06HErfDTHt+LfekJ98SIRE95wz8wOuoLwysiATiNgxDxo0g4mj4AnzoRv1wVdmYgkSHjD3TmN3d5d21rxt8FHc2F0J1jyuFrxIhEQV7ibWQ8zW2FmK83s6h2cc6aZLTezZWb2RGLLLKN0nPvWLek1I2SyZGRAp+G+Fb//YfD8RWrFi0RAjeFuZpnAaKAn0A4YaGbtKpzTFrgGON45dxhwWRJq9co9xJSmM0Imw35tYPAL0PN2WP1arBX/mFrxIiEVT8s9H1jpnFvlnNsMTAT6VjhnCDDaOfc1gHNufWLLLEMPMSVPRgZ0HOZb8Qe096NqHvgZvHSDfkMSCZl4wr0ZsKbM9trYvrIOBg42s3lmtsDMelT1hcxsqJkVmllhUdEuzlpYGu71GukhpmTZ9yAYNBU6jYRPF8G8f8D4XvDJwqArE5E4xRPuVT3lUvF39SygLdAVGAg8YGaNKn2Sc2Odc3nOubycnN1c/7RBjoI9mTIyoMF+/slW8N1hUy7VVMIiIRFPuK8FcstsNwcq3m1bCzzvnNvinPsIWIEP++TZtEFdBcnWqgtk1vXTF2RkwZcrYUwneHdK0JWJSA3iCfdFQFsza21m2cAAoKDCOZOBkwHMrDG+m2ZVIgvdpvSG6vfrNVom2UqnLzjlOvjNdBj+GjRs5mebfG44/PBN0BWKyA7UGO7OuWLgYmAG8C7wpHNumZndZGZ9YqfNAL40s+XALOBK59yXSam4eNP21yU/arRMspUuyp2bD00O9XPFn3QVLH0S7ukMH74SdIUiUoW45pZxzk0DplXYd32Z1w64PPYnucoOzXNbof5+SX9LKSOzDpx8rZ+j5rnh8OhpcOwQ+NmNfoEQEUkJ4XtC1W0ts5EBPyTnFwSpQbMOMGwOdLoIFt3vl/VTF5lIyghfuG+botYgq67GuQepTn3ocYsfNllSDON+AS+PguIfg65MJO2FL9xLtTxO49xTResu/sGno86G1+6E+0+Bz98JuiqRtBa+cN9a4j+2OE7Bnkrq7Q1974aBk+D7Ihh7sl+Yu6Q46MpE0lL4wr20z90yg61DqvbTHnDRAji0N7zyZ3ioB3yxMuiqRNJO+MK99OHYd56CwvGBViI7sMe+cMZ46PcgfPGBv9m68D7YurXGTxWRxAjvMntffwRTf+tf5w0OtBTZgcP7Q8vjoeASmP4HeG8qHDsUvnzf3whXt5pI0oQ33Eu9+7zCPZXt3RTOfgrefBimX+2X9sMgq55uiIskUQi7ZSo4tOLsw5JyzKDDYDj2wtgOB8U/wAf/DrIqkUgLb7g3aALHX6ZWe5i06wNZ9dk20eiiB2Hly4GWJBJV4Q3374v8TTo9FRkepRORdfsT9B0DezaBx/rBtCth88agqxOJlBD3ubvty+yp3zY8cvO3f7/a94OZN8KCMbBqNpw+Fg48OtDyRKIivC130DJ7YVennp++4NzJ8ON38EB3mPPX7Q+qicguC1fL/ZHTtr10wPuuBddPKwHm0/uIppx7XCt+2FzC4Icqd9X079CcM/Jy+er7zYx4bHGl4+d0asmpRx7Ium9+4HeT3qp0fEiXg+jebn8+LPqOa5+t/Gj9Jae05YS2jVm2bgM3TVle6fgfevyUDi33ZfHHX3H7iysqHb/+1HYcdmBDXvvgC/71ygeVjt98+uG0ydmTl5f/l/vnVp4q/85fH8WBjeoz5e11PLbg40rH7zmnA/s2yOapwjU8vXhtpePjf5NP/exMHp2/mqlLP6t0fNKw4wAYO+dDZr5bfoncenUyefh83xq/a+YHzFv5Rbnj++yRzb3ndgDgthff482Pvy53vGnDRvxjxDx44Qp45S+smPssdze6kvVZTQE4KKcBt5x+BADXPLuUVUXfl/v8dgfuzQ2nHgbAZROX8NmGTeWOH9NyH67qcQgAwx9dzNcbN5c7fvxPGnNpN7+2zKBxb7BpS/n/XLod2oShJ7YB4Nf3za/0d6OfvTD/7NXjHwP8b4s3TlnG8nXfljuerJ+90mtKpnC13D95vdxmy5KPAipEEm6PfaH/OJ5pPYrmxR9z+xcX0XXjjPJTPItI3MwF9I8nLy/PFRYW7twnPXIarCqzOMRBp8B5zyW2MAneN2tg8gh/P+WQ3nDqP6FB46CrEkkJZrbYOZdX03nharmffE312xINjXLhvAL4+V/8WPgxx8H7GhMvsjPCFe4Vl9TTEnvRlZEBnS+BobOhQQ48cQZM/R1s/r6mzxQRwhbum76tfluiZ//DYOgsH/SFD8G9XWBt5ZuSIlJeuML986XVb0s0ZdX1XTSDpvhVnh78Gcy+TXPFi1QjXOFecR4ZzSuTXkpXfGrfD2bf7Jf1+/LDoKsSSUnhCvflz1e/LdFXvxH0ux/6j4MvV/q54gsf0pBJkQrCFe4fv1b9tqSP9v3govl+KoOpl8GEAfDd+po/TyRNhCvcGzavflvSy94HwjnPQY9b4cNZfsjke9P8ZHJz/6ZJ5SSthWv6gc6/3b76Uum2pLeMDOg0Ag7qCs8OgYkDt6+vm5mtBUEkbYWr5f7Dl9VvS/pqcihc+Ipf1s+V+D8lP+pZCElb4Qr3+vtVvy3pLSsbuo+CzLp+2231Y+J//F+QVYkEIlzh/vnb1W+L5ObD4KnQ9Vq/QPf7031f/Iezgq5MpFaFK9wrjobQ6AipSm4+dL0K+j0I58/wi3E/+isouFRPNUvaCFe4U3Ess8Y2Sw1y82H4XDj+t7DkUd+KXzkz6KpEki6ucDezHma2wsxWmtnV1ZzX38ycmdU4HeUu2XP/6rdFqlKnPvzsJrjgJcjeAx47HQougU0bgq5MJGlqDHczywRGAz2BdsBAM2tXxXl7AZcCCxNd5DZHDtz+OiOr/LZITZrnwbC5cPxlsOSxWCv+5aCrEkmKeFru+cBK59wq59xmYCJQ1aQufwZuBzZVcSwxyo5XPriHxi/LzqtTD352I1zwMmTvCY/1g+cvViteIieecG8GrCmzvTa2bxszOxrIdc5NTWBtlRWO3/76vanlt0V2RvMOMGwOnPA7eOtx34r/4KWgqxJJmHjC3arYt+1OppllAHcCV9T4hcyGmlmhmRUWFRXFX2WpBWOq3xbZGXXq+XHxF74MdfeCx/vD5JHwwzdBVyay2+IJ97VAbpnt5sC6Mtt7Ae2B2Wa2GugEFFR1U9U5N9Y5l+ecy8vJydn5as2q3xbZFc1irfguV8DbE7Ssn0RCPOG+CGhrZq3NLBsYABSUHnTObXDONXbOtXLOtQIWAH2cczu5+nUcOo6ofltkV2XVhW7X+1Z8vYZ+Wb/JF6kVL6FVY7g754qBi4EZwLvAk865ZWZ2k5n1SXaB5eQNrn5bZHc1OwaGvQpdfg9vT4QxneD9GUFXJbLTzAW0yEFeXp4rLNzJxv2oRpR/cMlglFpWkiTrlvjW+/rlcORZ0ONmqL9P0FVJmjOzxc65Gp8l0hOqIjty4NEwdDaceCUsnQSjO8GK6UFXJRKXkIV7xRuouqEqSZZVF075Iwx5BfbYz6/49Oww2PhV0JWJVCtc4V6uC0ZdMlKLDjzKt+JPugr+87Tvi39vWtBViexQuMId/BwhANd+Gmwdkn6ysuHka30rvkETv+rTM0PUipeUFL5wFwla0yN9wHe9BpY9C6M7wnsvBF2VSDkKd5FdkZUNXa+GIbP87KQTz4JnLlQrXlJGuBbIFkk1TY+AobNg7t9hzu2w6lW/YDcOWnXR5HYSGIW7yO7KrONXfjqkFzw5CGbe6Pdn1YNBUxTwEgh1y4gkygGHx9YYiA3RLd4E8/4BW7cGWpakJ4W7SCIddJJvsVsmWIa/0fpQT1j/XtCVSZpRt4xIIuXmw6ACWD0XWp4AX30IM66Fe0/wc8d3ucJPNSySZAp3kUTLzd/ez96iI7T9uQ/4Obf7oZO9/wGtuwRbo0SeumVEkq1BYzh9LJz7HGwthod7w/MjNWxSkkrhLlJb2pwCI+b7BbrfmgCj8+GdpyGgmVkl2hTuIrUpew+/QPewV6FhLjxzgV/e7+vVQVcmEaNwFwnCAYf7VZ963AafLPBL+827C0qKg65MIkLhLhKUjEzoNBxGLoTWJ8FLf4L7T4ZP3wy6MokAhbtI0Bo2h4ET4MxH4Lv18EA3ePFa+PG7oCuTEFO4i6QCM2jX17fiOwyGBaO1fqvsFoW7SCqp3wh63wnnz4DsBvDEmfDUYPjff4OuTEJG4S6Silp0gmFz4eQ/+ikMRh8Li8drnhqJm8JdJFVlZcNJV8KI12H/w2HKb2H8L6FoRdCVSQgo3EVSXeO2MHgq9Lkb1i/389TMugWKfwy6MklhCneRMDCDY86Fiwv9jddXb/Uhv3pe0JVJilK4i4TJnjnQ7wE4+xk/X/z4XlBwKfzwddCVSYpRuIuEUdvucNEC6HwJLHkM7s6H/zyjeWpkG4W7SFhlN4Cf/8Wv4br3gfD0+X7o5DefBF2ZpACFu0jYNT0SLpwJv7jF98GP7gjzR2uemjSncBeJgswsOO4iGLkAWnXxi4M80A0+ezvoyiQgCneRKGnUAs6aBP0fgm/XwdiuMOM62Px90JVJLVO4i0SNGbQ/HS5+A445D+bfDaM7wQcvBV2Z1KK4wt3MepjZCjNbaWZXV3H8cjNbbmZLzWymmbVMfKkislPq7wOn/hN+M90vyv14f3/T9bv1QVcmtaDGcDezTGA00BNoBww0s3YVTlsC5DnnjgCeBm5PdKEisotadobhr0HXa+DdKXB3Hrz5iIZNRlw8Lfd8YKVzbpVzbjMwEehb9gTn3Czn3MbY5gKgeWLLFJHdklUXul4Nw+dBk8Og4BIY3xveeQbm/g3WvBF0hZJgWXGc0wxYU2Z7LdCxmvMvAKZXdcDMhgJDAVq0aBFniSKSMDkHw+AXYMmjMOMaeOZ8wHz4D5oCuflBVygJEk/L3arYV+Xvc2Z2DpAH/LWq4865sc65POdcXk5OTvxVikjiZGRAh0GQPyy2w/mpDBY9GGhZkljxhPtaILfMdnNgXcWTzKw7cB3Qxzmn6epEUt1Pe0JWfbAMwGDpRHj8TPjyw6ArkwSIp1tmEdDWzFoDnwIDgLPKnmBmRwP3AT2cc7oVLxIGufkwqABWz4XcTrDuTZh9q1/er/Ol0OVyP8WBhFKN4e6cKzazi4EZQCYwzjm3zMxuAgqdcwX4bpg9gafMDOAT51yfJNYtIomQm7+9n73V8XD4GfDS9TD3Dnh7Ivzi//wUw1ZV76yksnha7jjnpgHTKuy7vszr7gmuS0SCsNcBcPpYv0j3tCvhqUHQ+iTo9VfI+WnQ1clO0BOqIlJZy84w9FXodQd89hbc09lPY7Dp26Arkzgp3EWkaplZkD8ELnkTjhzopzG4+1hY+qQegAoBhbuIVK9BY+h7N1z4CuzdFJ4dAg/1hM/fCboyqYbCXUTi07yDD/hT74KiFXDfib5fXkv8pSSFu4jEr/QBqEsWQ94FsOgB+FcHP1fN1q1BVydlKNxFZOftsS/88g5/03W/tn6umge7w6eLg65MYhTuIrLrmh4B578Ip90H36yB+7tBwaXw/ZdBV5b2FO4isnvM4MgBvqvmuJGw5DH41zHwxv2wtSTo6tKWwl1EEqPe3v6J1hHzfIt+2u9h7EnwyYKgK0tLCncRSawmh8J5BXDGeNj4FYz7BTw3HP7336ArSysKdxFJPDM47DS4eBGccDm887QfVTN/NJRsCbq6tKBwF5HkyW4A3W+AixZAi44w41q49wT4aE7QlUWewl1Ekq/xT+Dsp2HABNjyAzx8Kjw1GDasDbqyyFK4i0jtMINDesHIhdD1Wlgx3c9VM/dvUKz1fRJN4S4itatOfeh6FYx8A9qcAjNvgjHHwQcvB11ZpCjcRSQY+7SEAY/DOc/47cf7wYSz4OvVgZYVFQp3EQnWT7rDRfOh+yhYNRtGd4RZt/i+edllCncRCV5WXTjhd37o5CG/hFdvhdH58O5UzR2/ixTuIpI6GjaD/uNg0FSo0wAmnQ2P94cvVgZdWego3EUk9bTuAsPnQo9bYc0bMKYTvDwKfvwu6MpCI64FskVEal1mHeg0Atr3g5dugNfuhLcnQd5gsCz/H0BuftBVpiyFu4iktj2bwGn3QIfBMHkEzLrZ78+qB4OmKOB3QN0yIhIOLTrCUWcB5rdLtsDquYGWlMoU7iISHq1P9C12y4TMbGjVJeiKUpa6ZUQkPHLzYVCBb7G3Up97dRTuIhIuufkK9TioW0ZEJIIU7iIiEaRwFxGJIIW7iEgExRXuZtbDzFaY2Uozu7qK43XNbFLs+EIza5XoQkVEJH41hruZZQKjgZ5AO2CgmbWrcNoFwNfOuZ8AdwK3JbpQERGJXzwt93xgpXNulXNuMzAR6FvhnL7Aw7HXTwPdzMwSV2YZL13vP958YFK+vIhIFMQT7s2ANWW218b2VXmOc64Y2ADsl4gCyxnVsPptEREB4gv3qlrgFWfPj+cczGyomRWaWWFRUVE89YmIyC6IJ9zXArlltpsD63Z0jpllAQ2Bryp+IefcWOdcnnMuLycnZ9cqFhGRGsUT7ouAtmbW2syygQFAQYVzCoBBsdf9gVecS8LaWKM2VL8tIiJAHHPLOOeKzexiYAaQCYxzzi0zs5uAQudcAfAg8KiZrcS32AckrWIFuohIjeKaOMw5Nw2YVmHf9WVebwLOSGxpIiKyq/SEqohIBCncRUQiSOEuIhJBCncRkQhSuIuIRJAlYzh6XG9sVgR8vIuf3hj4IoHlhIGuOT3omtPD7lxzS+dcjU+BBhbuu8PMCp1zeUHXUZt0zelB15weauOa1S0jIhJBCncRkQgKa7iPDbqAAOia04OuOT0k/ZpD2ecuIiLVC2vLXUREqpHS4Z6OC3PHcc2Xm9lyM1tqZjPNrGUQdSZSTddc5rz+ZubMLPQjK+K5ZjM7M/a9XmZmT9R2jYkWx892CzObZWZLYj/fvYKoM1HMbJyZrTez/+zguJnZXbG/j6VmdkxCC3DOpeQf/PTCHwIHAdnA20C7CudcBNwbez0AmBR03bVwzScDe8Rej0iHa46dtxcwB1gA5AVddy18n9sCS4B9YttNgq67Fq55LDAi9rodsDrounfzmk8EjgH+s4PjvYDp+JXsOgELE/n+qdxyT62FuWtHjdfsnJvlnNsY21yAXxkrzOL5PgP8Gbgd2FSbxSVJPNc8BBjtnPsawDm3vpZrTLR4rtkBe8deN6Tyim+h4pybQxUr0pXRF3jEeQuARmbWNFHvn8rhnjoLc9eeeK65rAvw//OHWY3XbGZHA7nOuam1WVgSxfN9Phg42MzmmdkCM+tRa9UlRzzXPAo4x8zW4tePuKR2SgvMzv573ylxLdYRkIQtzB0icV+PmZ0D5AEnJbWi5Kv2ms0sA7gTGFxbBdWCeL7PWfiuma74387mmll759w3Sa4tWeK55oHAeOfc38zsOPzqbu2dc1uTX14gkppfqdxyT9jC3CESzzVjZt2B64A+zrkfa6m2ZKnpmvcC2gOzzWw1vm+yIOQ3VeP92X7eObfFOfcRsAIf9mEVzzWnUJeRAAABIElEQVRfADwJ4JybD9TDz8ESVXH9e99VqRzuqbMwd+2p8ZpjXRT34YM97P2wUMM1O+c2OOcaO+daOeda4e8z9HHOFQZTbkLE87M9GX/zHDNrjO+mWVWrVSZWPNf8CdANwMwOxYd7Ua1WWbsKgPNio2Y6ARucc58l7KsHfUe5hrvNvYD38XfZr4vtuwn/jxv8N/8pYCXwBnBQ0DXXwjW/DPwXeCv2pyDompN9zRXOnU3IR8vE+X024O/AcuAdYEDQNdfCNbcD5uFH0rwF/DzomnfzeicAnwFb8K30C4DhwPAy3+PRsb+PdxL9c60nVEVEIiiVu2VERGQXKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiaD/B4/kQ0FNWvMfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot PR curve\n",
    "pyplot.plot([0, 1], [0.5, 0.5], linestyle=\"--\")\n",
    "pyplot.plot(recalls, precisions, marker=\".\")\n",
    "pyplot.savefig(\"../pr_curve.png\")\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately recall never hits 1.0 because documents often have terms applied to them that were not applied to any of their references. I try to improve on this in future models by incorporating semantic similarity of terms into the predictive process.\n",
    "\n",
    "Let's see what the highest F1 score is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max F1: 0.44532596971611266 at threshold: 0.015\n"
     ]
    }
   ],
   "source": [
    "print(\"Max F1:\", str(max(f1s)), \"at threshold:\", str(thresholds[f1s.index(max(f1s))]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}