N-gram tagger exercise.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import brown\n",
    "import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##A Simple Baseline Tagger##\n",
    "Keep in mind that the brown corpus is already tagged.  The simplest possible tagger assigns the **most likely** tag to each token. This establishes a baseline tagger.  So let's use the data we have to figure out what the most likely tag for English is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tags = [tag for (word, tag) in brown.tagged_words(categories='news')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use FreqDist and max() to find out which tag is the **most likely tag** for English according to the Brown corpus by counting how many tags have been assigned to the words in this corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'NN'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.FreqDist(tags).max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know empirically which is the most likely tag for English, we can make a baseline tagger that automatically assigns the most likely tag when we don't know what else to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('what', 'NN'), ('will', 'NN'), ('this', 'NN'), ('silly', 'NN'), ('tagger', 'NN'), ('do', 'NN'), ('?', 'NN')]\n"
     ]
    }
   ],
   "source": [
    "default_tagger = nltk.DefaultTagger('NN')\n",
    "raw = r'''what will this silly tagger do?'''\n",
    "tokens = nltk.word_tokenize(raw)\n",
    "print (default_tagger.tag(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Train a Unigram Tagger From Pre-Tagged Text##\n",
    "Now train a unigram tagger on the news portion of the Brown corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('what', 'WDT'), ('will', 'MD'), ('this', 'DT'), ('silly', 'JJ'), ('tagger', None), ('do', 'DO'), ('?', '.')]\n"
     ]
    }
   ],
   "source": [
    "brown_tagged_sents = brown.tagged_sents(categories='news')\n",
    "unigram_tagger = nltk.UnigramTagger(brown_tagged_sents)\n",
    "print (unigram_tagger.tag(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Separate Training From Testing Data###\n",
    "But really we need to separate training and testing data.  We can use the handy python string slicing operator to do this *really* easily.  Here we divide into 90% training and 10% testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('what', 'WDT'), ('will', 'MD'), ('this', 'DT'), ('silly', 'JJ'), ('tagger', None), ('do', 'DO'), ('?', '.')]\n"
     ]
    }
   ],
   "source": [
    "def create_data_sets(tagged_sents):\n",
    "    size = int(len(tagged_sents) * 0.9)\n",
    "    train_sents = tagged_sents[:size]\n",
    "    test_sents = tagged_sents[size:]\n",
    "    return train_sents, test_sents\n",
    "\n",
    "sample_sents = brown.tagged_sents(categories='news')\n",
    "train_sents, test_sents = create_data_sets(sample_sents)\n",
    "\n",
    "unigram_tagger = nltk.UnigramTagger(train_sents)\n",
    "print (unigram_tagger.tag(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Evaluation Metric###\n",
    "NLTK's tagger has a handy evaluation function built right in!  It automatically compares the output of your tagger with the tags assigned to the Brown corpus.  The score shown below is the average across the entire test collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.812\n"
     ]
    }
   ],
   "source": [
    "print (\"%0.3f\" % unigram_tagger.evaluate(test_sents))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Question###\n",
    "What is this evaluation metric measuring?\n",
    "* Answer: \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which tags did it get wrong?  If you want to see what the gold standard tags were vs. what the tagger produced, here is some code to do it (written by Jason Ost, MIMS from 2014).  The first column is the word, the second is the tag from the gold standard, and the third is what the algorithm assigned.  (The last element is a little tricky: the tagger's tag() function expects a list of words as input, so you have to enclose \"w\" in square brackets, and it returns a list of tagged words (as two-element tuples), so you have to grab the second element of the first tuple, which is the predicted tag.  This works because the unigram tagger looks at each word in isolation.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('For', 'IN', 'IN'),\n",
       " ('18', 'CD', 'CD'),\n",
       " ('months', 'NNS', 'NNS'),\n",
       " (',', ',', ','),\n",
       " ('Hamilton', 'NP', None),\n",
       " ('Holmes', 'NP', None),\n",
       " (',', ',', ','),\n",
       " ('19', 'CD', 'CD'),\n",
       " (',', ',', ','),\n",
       " ('and', 'CC', 'CC'),\n",
       " ('Charlayne', 'NP', None),\n",
       " ('Hunter', 'NP', 'NP-TL'),\n",
       " (',', ',', ','),\n",
       " ('18', 'CD', 'CD'),\n",
       " (',', ',', ','),\n",
       " ('had', 'HVD', 'HVD'),\n",
       " ('tried', 'VBN', 'VBN'),\n",
       " ('to', 'TO', 'TO'),\n",
       " ('get', 'VB', 'VB'),\n",
       " ('into', 'IN', 'IN'),\n",
       " ('the', 'AT', 'AT'),\n",
       " ('university', 'NN', 'NN'),\n",
       " ('.', '.', '.')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(w, t, unigram_tagger.tag([w])[0][1]) for w, t in test_sents[3]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Negro', 'NP', 'NP'),\n",
       " ('lawyers', 'NNS', 'NNS'),\n",
       " ('dug', 'VBD', 'VBN'),\n",
       " ('into', 'IN', 'IN'),\n",
       " ('the', 'AT', 'AT'),\n",
       " ('records', 'NNS', 'NNS'),\n",
       " ('of', 'IN', 'IN'),\n",
       " ('300', 'CD', 'CD'),\n",
       " ('white', 'JJ', 'JJ'),\n",
       " ('students', 'NNS', 'NNS'),\n",
       " (',', ',', ','),\n",
       " ('found', 'VBD', 'VBN'),\n",
       " ('that', 'CS', 'CS'),\n",
       " ('many', 'AP', 'AP'),\n",
       " ('were', 'BED', 'BED'),\n",
       " ('hardly', 'RB', 'RB'),\n",
       " ('interviewed', 'VBN', 'VBD'),\n",
       " ('at', 'IN', 'IN'),\n",
       " ('all', 'ABN', 'ABN'),\n",
       " ('--', '--', '--'),\n",
       " ('and', 'CC', 'CC'),\n",
       " ('few', 'AP', 'AP'),\n",
       " ('had', 'HVD', 'HVD'),\n",
       " ('academic', 'JJ', 'JJ'),\n",
       " ('records', 'NNS', 'NNS'),\n",
       " ('as', 'QL', 'CS'),\n",
       " ('good', 'JJ', 'JJ'),\n",
       " ('as', 'CS', 'CS'),\n",
       " ('Hamilton', 'NP', None),\n",
       " ('Holmes', 'NP', None),\n",
       " ('.', '.', '.')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(w, t, unigram_tagger.tag([w])[0][1]) for w, t in test_sents[16]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Train an N-Gram Tagger With Backoff ##\n",
    "\n",
    "Below is code for a bigram tagger with backoff.  When it encounters a token, it first\n",
    "1. Tries tagging the token with the bigram tagger.\n",
    "2. If the bigram tagger is unable to find a tag for the token, tries the unigram tagger.\n",
    "3. If the unigram tagger is also unable to find a tag, uses the default tagger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.845\n"
     ]
    }
   ],
   "source": [
    "def build_backoff_tagger (train_sents):\n",
    "    t0 = nltk.DefaultTagger('NN')\n",
    "    t1 = nltk.UnigramTagger(train_sents, backoff=t0)\n",
    "    t2 = nltk.BigramTagger(train_sents, backoff=t1)\n",
    "    return t2\n",
    "ngram_tagger = build_backoff_tagger(train_sents)\n",
    "bigram_tagger = ngram_tagger\n",
    "print (\"%0.3f\" % ngram_tagger.evaluate(test_sents))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to compare the output of your trained bigram tagger to the gold standard, here is code for that as well, again courtesy Jason Ost.  This a little more complicated, since you need to give the tagger code not only the current word, but also the one before it, unless it's the first word in a sentence, in which case it supplies some padding words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Negro', 'NP', 'NP'),\n",
       " ('lawyers', 'NNS', 'NNS'),\n",
       " ('dug', 'VBD', 'VBN'),\n",
       " ('into', 'IN', 'IN'),\n",
       " ('the', 'AT', 'AT'),\n",
       " ('records', 'NNS', 'NNS'),\n",
       " ('of', 'IN', 'IN'),\n",
       " ('300', 'CD', 'CD'),\n",
       " ('white', 'JJ', 'JJ'),\n",
       " ('students', 'NNS', 'NNS'),\n",
       " (',', ',', ','),\n",
       " ('found', 'VBD', 'VBD'),\n",
       " ('that', 'CS', 'CS'),\n",
       " ('many', 'AP', 'AP'),\n",
       " ('were', 'BED', 'BED'),\n",
       " ('hardly', 'RB', 'RB'),\n",
       " ('interviewed', 'VBN', 'VBD'),\n",
       " ('at', 'IN', 'IN'),\n",
       " ('all', 'ABN', 'ABN'),\n",
       " ('--', '--', '--'),\n",
       " ('and', 'CC', 'CC'),\n",
       " ('few', 'AP', 'AP'),\n",
       " ('had', 'HVD', 'HVD'),\n",
       " ('academic', 'JJ', 'JJ'),\n",
       " ('records', 'NNS', 'NNS'),\n",
       " ('as', 'QL', 'CS'),\n",
       " ('good', 'JJ', 'JJ'),\n",
       " ('as', 'CS', 'CS'),\n",
       " ('Hamilton', 'NP', 'NN'),\n",
       " ('Holmes', 'NP', 'NN'),\n",
       " ('.', '.', '.')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(w2, t2, bigram_tagger.tag([w1,w2])[1][1]) \n",
    " for (w1, t1), (w2, t2) in nltk.bigrams(test_sents[16], pad_left=True, pad_symbol=(None, None))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EXERCISE: Train and Evaluate a Trigram Tagger ##\n",
    "\n",
    "Modify build_backoff_tagger() to build a backoff trigram tagger.  Evaluate the results.  How does it do compared to the previous backoff tagger?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.843\n"
     ]
    }
   ],
   "source": [
    "def build_backoff_tagger (train_sents):\n",
    "    t0 = nltk.DefaultTagger('NN')\n",
    "    t1 = nltk.UnigramTagger(train_sents, backoff=t0)\n",
    "    t2 = nltk.BigramTagger(train_sents, backoff=t1)\n",
    "    t3 = nltk.TrigramTagger(train_sents, backoff=t2)\n",
    "    return t3\n",
    "trigram_tagger = build_backoff_tagger(train_sents)\n",
    "print (\"%0.3f\" % trigram_tagger.evaluate(test_sents))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EXERCISE: Train a Simplified Tagger ##\n",
    "Train and evaluate a bigram backoff tagger like the one above but using the universal Brown tagset (or make a tagset of your own by discarding all but the first character of each tag name). This tagger has fewer distinctions to make but more ambiguity.  Evaluate its performance.  How does it compare to the previous tagger?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.846\n"
     ]
    }
   ],
   "source": [
    "train, test = create_data_sets(brown.tagged_sents(categories='news', tagset='universal'))\n",
    "                                           \n",
    "ut = build_backoff_tagger(train)\n",
    "print (\"%0.3f\" % ut.evaluate(test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Negro', 'NOUN', 'NOUN'),\n",
       " ('lawyers', 'NOUN', 'NOUN'),\n",
       " ('dug', 'VERB', 'VERB'),\n",
       " ('into', 'ADP', 'ADP'),\n",
       " ('the', 'DET', 'DET'),\n",
       " ('records', 'NOUN', 'NOUN'),\n",
       " ('of', 'ADP', 'ADP'),\n",
       " ('300', 'NUM', 'NUM'),\n",
       " ('white', 'ADJ', 'ADJ'),\n",
       " ('students', 'NOUN', 'NOUN'),\n",
       " (',', '.', '.'),\n",
       " ('found', 'VERB', 'VERB'),\n",
       " ('that', 'ADP', 'ADP'),\n",
       " ('many', 'ADJ', 'ADJ'),\n",
       " ('were', 'VERB', 'VERB'),\n",
       " ('hardly', 'ADV', 'ADV'),\n",
       " ('interviewed', 'VERB', 'VERB'),\n",
       " ('at', 'ADP', 'ADP'),\n",
       " ('all', 'PRT', 'PRT'),\n",
       " ('--', '.', '.'),\n",
       " ('and', 'CONJ', 'CONJ'),\n",
       " ('few', 'ADJ', 'ADJ'),\n",
       " ('had', 'VERB', 'VERB'),\n",
       " ('academic', 'ADJ', 'ADJ'),\n",
       " ('records', 'NOUN', 'NOUN'),\n",
       " ('as', 'ADV', 'ADP'),\n",
       " ('good', 'ADJ', 'ADJ'),\n",
       " ('as', 'ADP', 'ADP'),\n",
       " ('Hamilton', 'NOUN', 'NN'),\n",
       " ('Holmes', 'NOUN', 'NN'),\n",
       " ('.', '.', '.')]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(w2, t2, ut.tag([w1,w2])[1][1]) \n",
    " for (w1, t1), (w2, t2) in nltk.bigrams(test[16], pad_left=True, pad_symbol=(None, None))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating a Tagger by Looking at Tags that Follow Tags ##\n",
    "(For this exercise, use your regular tagger, not the simplified one.)  The word **to** is frequently confused; it can be helpful to inspect the context it occurs in.  This code shows how to view the frequency of the tag that *follows* the word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tags that follow the target word and tag to and TO\n",
      "  VB   NN   AT   BE   DO  PP$   JJ  PPO   NP   HV  NNS NN-TL   DT   CD  VBN JJ-TL   CS  VBG  WDT   `` \n",
      " 361  220  155   83   28   28   25   18   18   12    9    7    7    6    6    6    6    6    5    5 \n",
      "Tags that follow the target word and tag to and IN\n",
      "  VB   NN   AT   BE   NP   CD   JJ  PPO  NNS   DT   DO   HV  PP$  ABN   `` NN-TL  DTS  DTI  VBD NNS-TL \n",
      " 142  114  103   29   14   13    9    9    9    6    5    4    4    3    3    3    3    3    3    2 \n"
     ]
    }
   ],
   "source": [
    "def examine_tag_contexts(tagger, target_word, target_tag):\n",
    "    test_sents = [tagger.tag(sent) for sent in brown.sents(categories='editorial')]\n",
    "    tags = [b[1] for test_sent in test_sents \n",
    "            for (a,b) in nltk.bigrams(test_sent)\n",
    "            if a[0] == target_word and a[1] == target_tag]\n",
    "    fd = nltk.FreqDist(tags)\n",
    "    print (\"Tags that follow the target word and tag \" + target_word + \" and \" + target_tag)\n",
    "    fd.tabulate(20)\n",
    "examine_tag_contexts(ngram_tagger, 'to', 'TO')\n",
    "examine_tag_contexts(ngram_tagger, 'to', 'IN')                                               "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.3.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}