linear-svm-classification-of-sentiment-in-tweets.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "3fb90d46-4cf7-edf1-f149-6fdd49ecabb2"
   },
   "outputs": [],
   "source": [
    "# Before we begin, we supress deprecation warnings resulting from nltk on Kaggle\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "2595fd17-1db0-1448-d3c9-4bfd1bc33644"
   },
   "source": [
    "# Linear SVM classification of sentiment in tweets about airlines\n",
    "\n",
    "This notebook describes an attempt to classify tweets by sentiment. It describes the initial data exploration, as well as implementation of a linear one-vs-rest Support-Vector-Machine (SVM) classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "1302dbd6-79d2-0a40-ddeb-7231a2f8e3e8"
   },
   "source": [
    "## What is in the dataset?\n",
    "\n",
    "It's always good to start by exploring the data that we have available. To do this we load the raw csv file using [Pandas][1] and check what the columns are.\n",
    "\n",
    "  [1]: http://pandas.pydata.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "f5f1a8d6-2465-1f81-50ca-173e7d15d79c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['airline',\n",
       " 'review date',\n",
       " 'Type Of Traveller',\n",
       " 'seat type',\n",
       " 'Seat Comfort',\n",
       " 'Cabin Staff Service',\n",
       " 'Food & Beverages',\n",
       " 'Inflight Entertainment',\n",
       " 'Ground Service',\n",
       " 'Wifi & Connectivity',\n",
       " 'Value For Money',\n",
       " 'airline_sentiment',\n",
       " 'Recommended',\n",
       " 'overal rating(10)',\n",
       " 'title',\n",
       " 'review']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "tweets = pd.read_csv(\"C:\\\\Users\\\\Win 1809 UEFI\\\\Desktop\\\\0.csv\")\n",
    "list(tweets.columns.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "39182328-e7fa-1a61-723b-3cbf2517751c"
   },
   "source": [
    "We want to be able to determine the sentiment of a tweet without any other information but the tweet text itself, hence the 'text' column is our focus. Using the text we are going to try and predict 'airline_sentiment'.\n",
    "\n",
    "First we take a look at what a typical record looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_cell_guid": "56171d3a-313c-a946-fb1e-8c86cc3d1966"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline</th>\n",
       "      <th>review date</th>\n",
       "      <th>Type Of Traveller</th>\n",
       "      <th>seat type</th>\n",
       "      <th>Seat Comfort</th>\n",
       "      <th>Cabin Staff Service</th>\n",
       "      <th>Food &amp; Beverages</th>\n",
       "      <th>Inflight Entertainment</th>\n",
       "      <th>Ground Service</th>\n",
       "      <th>Wifi &amp; Connectivity</th>\n",
       "      <th>Value For Money</th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>Recommended</th>\n",
       "      <th>overal rating(10)</th>\n",
       "      <th>title</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>air arabia</td>\n",
       "      <td>2019</td>\n",
       "      <td>Solo Leisure</td>\n",
       "      <td>Economy Class</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5</td>\n",
       "      <td>Negative</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>I will fly them again</td>\n",
       "      <td>I flew from Beirut to Sharjah on Air Arabia. C...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>air arabia</td>\n",
       "      <td>2019</td>\n",
       "      <td>Solo Leisure</td>\n",
       "      <td>Economy Class</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Negative</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>200 riyals as fine for the baggage</td>\n",
       "      <td>I was flying from Hyderabad to Bahrain via Sha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>air arabia</td>\n",
       "      <td>2019</td>\n",
       "      <td>Family Leisure</td>\n",
       "      <td>Economy Class</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Negative</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Absolute disaster</td>\n",
       "      <td>Not sure how this airline is allowed to legall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>air arabia</td>\n",
       "      <td>2019</td>\n",
       "      <td>Couple Leisure</td>\n",
       "      <td>Economy Class</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Negative</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>nobody let us know about cancellation</td>\n",
       "      <td>We bought two tickets from Prague to Kathmandu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>air arabia</td>\n",
       "      <td>2019</td>\n",
       "      <td>Solo Leisure</td>\n",
       "      <td>Economy Class</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "      <td>Negative</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Never using Air Arabia again</td>\n",
       "      <td>Worst experience. I am a student and i was tra...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      airline  review date Type Of Traveller      seat type  Seat Comfort  \\\n",
       "0  air arabia         2019      Solo Leisure  Economy Class           5.0   \n",
       "1  air arabia         2019      Solo Leisure  Economy Class           1.0   \n",
       "2  air arabia         2019    Family Leisure  Economy Class           1.0   \n",
       "3  air arabia         2019    Couple Leisure  Economy Class           NaN   \n",
       "4  air arabia         2019      Solo Leisure  Economy Class           3.0   \n",
       "\n",
       "   Cabin Staff Service  Food & Beverages  Inflight Entertainment  \\\n",
       "0                  5.0               5.0                     5.0   \n",
       "1                  1.0               NaN                     NaN   \n",
       "2                  1.0               NaN                     NaN   \n",
       "3                  NaN               NaN                     NaN   \n",
       "4                  3.0               3.0                     1.0   \n",
       "\n",
       "   Ground Service  Wifi & Connectivity  Value For Money airline_sentiment  \\\n",
       "0             5.0                  NaN                5          Negative   \n",
       "1             1.0                  NaN                1          Negative   \n",
       "2             1.0                  NaN                1          Negative   \n",
       "3             NaN                  NaN                1          Negative   \n",
       "4             1.0                  1.0                3          Negative   \n",
       "\n",
       "   Recommended  overal rating(10)                                  title  \\\n",
       "0            1                 10                  I will fly them again   \n",
       "1            0                  1     200 riyals as fine for the baggage   \n",
       "2            0                  1                      Absolute disaster   \n",
       "3            0                  1  nobody let us know about cancellation   \n",
       "4            0                  1           Never using Air Arabia again   \n",
       "\n",
       "                                              review  \n",
       "0  I flew from Beirut to Sharjah on Air Arabia. C...  \n",
       "1  I was flying from Hyderabad to Bahrain via Sha...  \n",
       "2  Not sure how this airline is allowed to legall...  \n",
       "3  We bought two tickets from Prague to Kathmandu...  \n",
       "4  Worst experience. I am a student and i was tra...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "cb5ee880-d138-4e49-7f09-e902a5318785"
   },
   "source": [
    "Now lets take a look at what sentiments have been found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_cell_guid": "fbdaecb5-81f1-50c3-4ec7-a34a8262a5cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Negative    2386\n",
      "Positive    1373\n",
      "Neutral     1095\n",
      "Name: airline_sentiment, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "sentiment_counts = tweets.airline_sentiment.value_counts()\n",
    "number_of_tweets = tweets.review.count()\n",
    "print(sentiment_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "92085794-bb90-cc78-1e96-f9e7f3f11f84"
   },
   "source": [
    "It turns out that our dataset is unbalanced with significantly more negative than positive tweets. We will focus on the issue of identifying negative tweets, and hence treat neutral and positive as one class. It's good to keep in mind that, while a terrible classifier, if we always guessed a tweet was negative we'd be right 62.7% of the time (9178 of 14640). That clearly wouldn't be a very useful classifier, but worth to remember."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "e4530f8b-ce34-a6c5-21e1-85db7e173211"
   },
   "source": [
    "# What characterizes text of different sentiments?\n",
    "\n",
    "While we still haven't decided what classification method to use, it's useful to get an idea of how the different texts look. This might be an \"old school\" approach in the age of deep learning, but lets indulge ourselves nevertheless. \n",
    "\n",
    "To explore the data we apply some crude preprocessing. We will tokenize and lemmatize using [Python NLTK][1], and transform to lower case. As words mostly matter in context we'll look at bi-grams instead of just individual tokens.\n",
    "\n",
    "As a way to simplify later inspection of results we will store all processing of data together with it's original form. This means we will extend the Pandas dataframe into which we imported the raw data with new columns as we go along.\n",
    "\n",
    "### Preprocessing\n",
    "Note that we remove the first two tokens as they always contain \"@ airline_name\". We begin by defining our normalization function.\n",
    "\n",
    "\n",
    "  [1]: http://www.nltk.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_cell_guid": "03b28e2b-38d0-b478-856d-25ac7f052dff"
   },
   "outputs": [],
   "source": [
    "import re, nltk\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words('english'))\n",
    "wordnet_lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "def normalizer(tweet):\n",
    "    only_letters = re.sub(\"[^a-zA-Z]\", \" \",tweet) \n",
    "    tokens = nltk.word_tokenize(only_letters)[2:]\n",
    "    lower_case = [l.lower() for l in tokens]\n",
    "    filtered_result = list(filter(lambda l: l not in stop_words, lower_case))\n",
    "    lemmas = [wordnet_lemmatizer.lemmatize(t) for t in filtered_result]\n",
    "    return lemmas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_cell_guid": "e14cfc54-490d-ce20-4031-cd385a6999d0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['text', 'airline', 'like']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalizer(\"Here is text about an airline I like.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_cell_guid": "33e1be47-7c09-948b-101d-380ccaeb4196"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>normalized_tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I flew from Beirut to Sharjah on Air Arabia. Check in was easy as I did it on line. I showed my boarding pass on my phone to get through security. At the gate I changed my e boarding pass to a paper ticket as that was required. Although they loaded a lot of cargo we took off on time. I had three seats to my self and was able to sleep. I was given water and dates which were very nice. I will fly them again.</td>\n",
       "      <td>[beirut, sharjah, air, arabia, check, easy, line, showed, boarding, pas, phone, get, security, gate, changed, e, boarding, pas, paper, ticket, required, although, loaded, lot, cargo, took, time, three, seat, self, able, sleep, given, water, date, nice, fly]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I was flying from Hyderabad to Bahrain via Sharjah,while I was transit to Bahrain at Sharjah airport the ground staff of weighed hand bag &amp; laptop was carrying on cabin then demanded to pay 200 riyals as fine for the baggage. I had no chance to debate with them because of their horrible attitude towards passengers. I suggest you don't waste your valuable time &amp; money by flying with Air Arabia for their worst hospitality &amp; cheap flight tickets.</td>\n",
       "      <td>[flying, hyderabad, bahrain, via, sharjah, transit, bahrain, sharjah, airport, ground, staff, weighed, hand, bag, laptop, carrying, cabin, demanded, pay, riyal, fine, baggage, chance, debate, horrible, attitude, towards, passenger, suggest, waste, valuable, time, money, flying, air, arabia, worst, hospitality, cheap, flight, ticket]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not sure how this airline is allowed to legally operate?? They cancelled flights with no notice and cost us over £500 in additional costs to get to destination. They offered no reason, no compensation and ignored most of the communications between myself and them. Absolute disaster. Never again. They need to be banned from operating within the EU.</td>\n",
       "      <td>[airline, allowed, legally, operate, cancelled, flight, notice, cost, u, additional, cost, get, destination, offered, reason, compensation, ignored, communication, absolute, disaster, never, need, banned, operating, within, eu]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>We bought two tickets from Prague to Kathmandu via Sharjah in November 2018. We were supposed to leave April 13, 2019, Ten days before the flight we tried to add some food and to change baggage allowance through internet browser. It must be said that their website does not work optimal in any of the internet browsers. We managed to change baggage allowance but no food. From that reason we log out and tried to add food later but we realized we don´t have access to our PNR anymore. We had still ten days to departure so we tried to contact their customer service via e-mail. Till now we do not have any answer. We contacted sales department in Prague by phone and also personally. They proved our PNR with conclusion we had cancelled both tickets in internet browser ourselves and the flight is full. We are still in shock. Two weeks of holiday, year of preparations for our big way to Himalayas, another friend waiting us in Kathmandu, services paid in Nepal are gone. Mainly, nobody let us know about tickets cancellation. If I wasn´t interested about my PNR, I wouldn´t know yet my tickets are cancelled. As proof they just reproduced conclusion of their IT Department in Sharjah. You have no chance to get in touch with somebody from their central agency in Sharjah. Mainly I would like to point out that I do not get any document confirming ticket cancellation. Welcome in modern times full of IT technologies where human being is a poor individual and those big fish use it for making money</td>\n",
       "      <td>[two, ticket, prague, kathmandu, via, sharjah, november, supposed, leave, april, ten, day, flight, tried, add, food, change, baggage, allowance, internet, browser, must, said, website, work, optimal, internet, browser, managed, change, baggage, allowance, food, reason, log, tried, add, food, later, realized, access, pnr, anymore, still, ten, day, departure, tried, contact, customer, service, via, e, mail, till, answer, contacted, sale, department, prague, phone, also, personally, proved, pnr, conclusion, cancelled, ticket, internet, browser, flight, full, still, shock, two, week, holiday, year, preparation, big, way, himalaya, another, friend, waiting, u, kathmandu, service, paid, nepal, gone, mainly, nobody, let, u, know, ticket, cancellation, interested, pnr, ...]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Worst experience. I am a student and i was travelling from Mumbai to Tbilisi,I was allowed to carry 10 kg hand carriage. So i had my 10 kgs and my laptop bag. In Mumbai Airport it wasn't a issue they let me pass. Once I've reached Sharjah for transit they started weighing my laptop bag and took my ticket away and asked me to pay fine. Never had such a worst experience. Never using Air Arabia again.</td>\n",
       "      <td>[student, travelling, mumbai, tbilisi, allowed, carry, kg, hand, carriage, kg, laptop, bag, mumbai, airport, issue, let, pas, reached, sharjah, transit, started, weighing, laptop, bag, took, ticket, away, asked, pay, fine, never, worst, experience, never, using, air, arabia]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        review  \\\n",
       "0  I flew from Beirut to Sharjah on Air Arabia. Check in was easy as I did it on line. I showed my boarding pass on my phone to get through security. At the gate I changed my e boarding pass to a paper ticket as that was required. Although they loaded a lot of cargo we took off on time. I had three seats to my self and was able to sleep. I was given water and dates which were very nice. I will fly them again.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \n",
       "1  I was flying from Hyderabad to Bahrain via Sharjah,while I was transit to Bahrain at Sharjah airport the ground staff of weighed hand bag & laptop was carrying on cabin then demanded to pay 200 riyals as fine for the baggage. I had no chance to debate with them because of their horrible attitude towards passengers. I suggest you don't waste your valuable time & money by flying with Air Arabia for their worst hospitality & cheap flight tickets.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
       "2  Not sure how this airline is allowed to legally operate?? They cancelled flights with no notice and cost us over £500 in additional costs to get to destination. They offered no reason, no compensation and ignored most of the communications between myself and them. Absolute disaster. Never again. They need to be banned from operating within the EU.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "3  We bought two tickets from Prague to Kathmandu via Sharjah in November 2018. We were supposed to leave April 13, 2019, Ten days before the flight we tried to add some food and to change baggage allowance through internet browser. It must be said that their website does not work optimal in any of the internet browsers. We managed to change baggage allowance but no food. From that reason we log out and tried to add food later but we realized we don´t have access to our PNR anymore. We had still ten days to departure so we tried to contact their customer service via e-mail. Till now we do not have any answer. We contacted sales department in Prague by phone and also personally. They proved our PNR with conclusion we had cancelled both tickets in internet browser ourselves and the flight is full. We are still in shock. Two weeks of holiday, year of preparations for our big way to Himalayas, another friend waiting us in Kathmandu, services paid in Nepal are gone. Mainly, nobody let us know about tickets cancellation. If I wasn´t interested about my PNR, I wouldn´t know yet my tickets are cancelled. As proof they just reproduced conclusion of their IT Department in Sharjah. You have no chance to get in touch with somebody from their central agency in Sharjah. Mainly I would like to point out that I do not get any document confirming ticket cancellation. Welcome in modern times full of IT technologies where human being is a poor individual and those big fish use it for making money   \n",
       "4  Worst experience. I am a student and i was travelling from Mumbai to Tbilisi,I was allowed to carry 10 kg hand carriage. So i had my 10 kgs and my laptop bag. In Mumbai Airport it wasn't a issue they let me pass. Once I've reached Sharjah for transit they started weighing my laptop bag and took my ticket away and asked me to pay fine. Never had such a worst experience. Never using Air Arabia again.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           normalized_tweet  \n",
       "0  [beirut, sharjah, air, arabia, check, easy, line, showed, boarding, pas, phone, get, security, gate, changed, e, boarding, pas, paper, ticket, required, although, loaded, lot, cargo, took, time, three, seat, self, able, sleep, given, water, date, nice, fly]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
       "1  [flying, hyderabad, bahrain, via, sharjah, transit, bahrain, sharjah, airport, ground, staff, weighed, hand, bag, laptop, carrying, cabin, demanded, pay, riyal, fine, baggage, chance, debate, horrible, attitude, towards, passenger, suggest, waste, valuable, time, money, flying, air, arabia, worst, hospitality, cheap, flight, ticket]                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n",
       "2  [airline, allowed, legally, operate, cancelled, flight, notice, cost, u, additional, cost, get, destination, offered, reason, compensation, ignored, communication, absolute, disaster, never, need, banned, operating, within, eu]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       \n",
       "3  [two, ticket, prague, kathmandu, via, sharjah, november, supposed, leave, april, ten, day, flight, tried, add, food, change, baggage, allowance, internet, browser, must, said, website, work, optimal, internet, browser, managed, change, baggage, allowance, food, reason, log, tried, add, food, later, realized, access, pnr, anymore, still, ten, day, departure, tried, contact, customer, service, via, e, mail, till, answer, contacted, sale, department, prague, phone, also, personally, proved, pnr, conclusion, cancelled, ticket, internet, browser, flight, full, still, shock, two, week, holiday, year, preparation, big, way, himalaya, another, friend, waiting, u, kathmandu, service, paid, nepal, gone, mainly, nobody, let, u, know, ticket, cancellation, interested, pnr, ...]  \n",
       "4  [student, travelling, mumbai, tbilisi, allowed, carry, kg, hand, carriage, kg, laptop, bag, mumbai, airport, issue, let, pas, reached, sharjah, transit, started, weighing, laptop, bag, took, ticket, away, asked, pay, fine, never, worst, experience, never, using, air, arabia]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', -1) # Setting this so we can see the full content of cells\n",
    "tweets['normalized_tweet'] = tweets.review.apply(normalizer)\n",
    "tweets[['review','normalized_tweet']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_cell_guid": "5e83d39c-c35d-dc3f-1d24-013810053bc9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>grams</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[beirut sharjah, sharjah air, air arabia, arabia check, check easy, easy line, line showed, showed boarding, boarding pas, pas phone, phone get, get security, security gate, gate changed, changed e, e boarding, boarding pas, pas paper, paper ticket, ticket required, required although, although loaded, loaded lot, lot cargo, cargo took, took time, time three, three seat, seat self, self able, able sleep, sleep given, given water, water date, date nice, nice fly, beirut sharjah air, sharjah air arabia, air arabia check, arabia check easy, check easy line, easy line showed, line showed boarding, showed boarding pas, boarding pas phone, pas phone get, phone get security, get security gate, security gate changed, gate changed e, changed e boarding, e boarding pas, boarding pas paper, pas paper ticket, paper ticket required, ticket required although, required although loaded, although loaded lot, loaded lot cargo, lot cargo took, cargo took time, took time three, time three seat, three seat self, seat self able, self able sleep, able sleep given, sleep given water, given water date, water date nice, date nice fly]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[flying hyderabad, hyderabad bahrain, bahrain via, via sharjah, sharjah transit, transit bahrain, bahrain sharjah, sharjah airport, airport ground, ground staff, staff weighed, weighed hand, hand bag, bag laptop, laptop carrying, carrying cabin, cabin demanded, demanded pay, pay riyal, riyal fine, fine baggage, baggage chance, chance debate, debate horrible, horrible attitude, attitude towards, towards passenger, passenger suggest, suggest waste, waste valuable, valuable time, time money, money flying, flying air, air arabia, arabia worst, worst hospitality, hospitality cheap, cheap flight, flight ticket, flying hyderabad bahrain, hyderabad bahrain via, bahrain via sharjah, via sharjah transit, sharjah transit bahrain, transit bahrain sharjah, bahrain sharjah airport, sharjah airport ground, airport ground staff, ground staff weighed, staff weighed hand, weighed hand bag, hand bag laptop, bag laptop carrying, laptop carrying cabin, carrying cabin demanded, cabin demanded pay, demanded pay riyal, pay riyal fine, riyal fine baggage, fine baggage chance, baggage chance debate, chance debate horrible, debate horrible attitude, horrible attitude towards, attitude towards passenger, towards passenger suggest, passenger suggest waste, suggest waste valuable, waste valuable time, valuable time money, time money flying, money flying air, flying air arabia, air arabia worst, arabia worst hospitality, worst hospitality cheap, hospitality cheap flight, cheap flight ticket]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[airline allowed, allowed legally, legally operate, operate cancelled, cancelled flight, flight notice, notice cost, cost u, u additional, additional cost, cost get, get destination, destination offered, offered reason, reason compensation, compensation ignored, ignored communication, communication absolute, absolute disaster, disaster never, never need, need banned, banned operating, operating within, within eu, airline allowed legally, allowed legally operate, legally operate cancelled, operate cancelled flight, cancelled flight notice, flight notice cost, notice cost u, cost u additional, u additional cost, additional cost get, cost get destination, get destination offered, destination offered reason, offered reason compensation, reason compensation ignored, compensation ignored communication, ignored communication absolute, communication absolute disaster, absolute disaster never, disaster never need, never need banned, need banned operating, banned operating within, operating within eu]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[two ticket, ticket prague, prague kathmandu, kathmandu via, via sharjah, sharjah november, november supposed, supposed leave, leave april, april ten, ten day, day flight, flight tried, tried add, add food, food change, change baggage, baggage allowance, allowance internet, internet browser, browser must, must said, said website, website work, work optimal, optimal internet, internet browser, browser managed, managed change, change baggage, baggage allowance, allowance food, food reason, reason log, log tried, tried add, add food, food later, later realized, realized access, access pnr, pnr anymore, anymore still, still ten, ten day, day departure, departure tried, tried contact, contact customer, customer service, service via, via e, e mail, mail till, till answer, answer contacted, contacted sale, sale department, department prague, prague phone, phone also, also personally, personally proved, proved pnr, pnr conclusion, conclusion cancelled, cancelled ticket, ticket internet, internet browser, browser flight, flight full, full still, still shock, shock two, two week, week holiday, holiday year, year preparation, preparation big, big way, way himalaya, himalaya another, another friend, friend waiting, waiting u, u kathmandu, kathmandu service, service paid, paid nepal, nepal gone, gone mainly, mainly nobody, nobody let, let u, u know, know ticket, ticket cancellation, cancellation interested, interested pnr, pnr know, ...]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[student travelling, travelling mumbai, mumbai tbilisi, tbilisi allowed, allowed carry, carry kg, kg hand, hand carriage, carriage kg, kg laptop, laptop bag, bag mumbai, mumbai airport, airport issue, issue let, let pas, pas reached, reached sharjah, sharjah transit, transit started, started weighing, weighing laptop, laptop bag, bag took, took ticket, ticket away, away asked, asked pay, pay fine, fine never, never worst, worst experience, experience never, never using, using air, air arabia, student travelling mumbai, travelling mumbai tbilisi, mumbai tbilisi allowed, tbilisi allowed carry, allowed carry kg, carry kg hand, kg hand carriage, hand carriage kg, carriage kg laptop, kg laptop bag, laptop bag mumbai, bag mumbai airport, mumbai airport issue, airport issue let, issue let pas, let pas reached, pas reached sharjah, reached sharjah transit, sharjah transit started, transit started weighing, started weighing laptop, weighing laptop bag, laptop bag took, bag took ticket, took ticket away, ticket away asked, away asked pay, asked pay fine, pay fine never, fine never worst, never worst experience, worst experience never, experience never using, never using air, using air arabia]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           grams\n",
       "0  [beirut sharjah, sharjah air, air arabia, arabia check, check easy, easy line, line showed, showed boarding, boarding pas, pas phone, phone get, get security, security gate, gate changed, changed e, e boarding, boarding pas, pas paper, paper ticket, ticket required, required although, although loaded, loaded lot, lot cargo, cargo took, took time, time three, three seat, seat self, self able, able sleep, sleep given, given water, water date, date nice, nice fly, beirut sharjah air, sharjah air arabia, air arabia check, arabia check easy, check easy line, easy line showed, line showed boarding, showed boarding pas, boarding pas phone, pas phone get, phone get security, get security gate, security gate changed, gate changed e, changed e boarding, e boarding pas, boarding pas paper, pas paper ticket, paper ticket required, ticket required although, required although loaded, although loaded lot, loaded lot cargo, lot cargo took, cargo took time, took time three, time three seat, three seat self, seat self able, self able sleep, able sleep given, sleep given water, given water date, water date nice, date nice fly]                                                                                                                                                                                                                                                                                                                                                                        \n",
       "1  [flying hyderabad, hyderabad bahrain, bahrain via, via sharjah, sharjah transit, transit bahrain, bahrain sharjah, sharjah airport, airport ground, ground staff, staff weighed, weighed hand, hand bag, bag laptop, laptop carrying, carrying cabin, cabin demanded, demanded pay, pay riyal, riyal fine, fine baggage, baggage chance, chance debate, debate horrible, horrible attitude, attitude towards, towards passenger, passenger suggest, suggest waste, waste valuable, valuable time, time money, money flying, flying air, air arabia, arabia worst, worst hospitality, hospitality cheap, cheap flight, flight ticket, flying hyderabad bahrain, hyderabad bahrain via, bahrain via sharjah, via sharjah transit, sharjah transit bahrain, transit bahrain sharjah, bahrain sharjah airport, sharjah airport ground, airport ground staff, ground staff weighed, staff weighed hand, weighed hand bag, hand bag laptop, bag laptop carrying, laptop carrying cabin, carrying cabin demanded, cabin demanded pay, demanded pay riyal, pay riyal fine, riyal fine baggage, fine baggage chance, baggage chance debate, chance debate horrible, debate horrible attitude, horrible attitude towards, attitude towards passenger, towards passenger suggest, passenger suggest waste, suggest waste valuable, waste valuable time, valuable time money, time money flying, money flying air, flying air arabia, air arabia worst, arabia worst hospitality, worst hospitality cheap, hospitality cheap flight, cheap flight ticket]\n",
       "2  [airline allowed, allowed legally, legally operate, operate cancelled, cancelled flight, flight notice, notice cost, cost u, u additional, additional cost, cost get, get destination, destination offered, offered reason, reason compensation, compensation ignored, ignored communication, communication absolute, absolute disaster, disaster never, never need, need banned, banned operating, operating within, within eu, airline allowed legally, allowed legally operate, legally operate cancelled, operate cancelled flight, cancelled flight notice, flight notice cost, notice cost u, cost u additional, u additional cost, additional cost get, cost get destination, get destination offered, destination offered reason, offered reason compensation, reason compensation ignored, compensation ignored communication, ignored communication absolute, communication absolute disaster, absolute disaster never, disaster never need, never need banned, need banned operating, banned operating within, operating within eu]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
       "3  [two ticket, ticket prague, prague kathmandu, kathmandu via, via sharjah, sharjah november, november supposed, supposed leave, leave april, april ten, ten day, day flight, flight tried, tried add, add food, food change, change baggage, baggage allowance, allowance internet, internet browser, browser must, must said, said website, website work, work optimal, optimal internet, internet browser, browser managed, managed change, change baggage, baggage allowance, allowance food, food reason, reason log, log tried, tried add, add food, food later, later realized, realized access, access pnr, pnr anymore, anymore still, still ten, ten day, day departure, departure tried, tried contact, contact customer, customer service, service via, via e, e mail, mail till, till answer, answer contacted, contacted sale, sale department, department prague, prague phone, phone also, also personally, personally proved, proved pnr, pnr conclusion, conclusion cancelled, cancelled ticket, ticket internet, internet browser, browser flight, flight full, full still, still shock, shock two, two week, week holiday, holiday year, year preparation, preparation big, big way, way himalaya, himalaya another, another friend, friend waiting, waiting u, u kathmandu, kathmandu service, service paid, paid nepal, nepal gone, gone mainly, mainly nobody, nobody let, let u, u know, know ticket, ticket cancellation, cancellation interested, interested pnr, pnr know, ...]                                     \n",
       "4  [student travelling, travelling mumbai, mumbai tbilisi, tbilisi allowed, allowed carry, carry kg, kg hand, hand carriage, carriage kg, kg laptop, laptop bag, bag mumbai, mumbai airport, airport issue, issue let, let pas, pas reached, reached sharjah, sharjah transit, transit started, started weighing, weighing laptop, laptop bag, bag took, took ticket, ticket away, away asked, asked pay, pay fine, fine never, never worst, worst experience, experience never, never using, using air, air arabia, student travelling mumbai, travelling mumbai tbilisi, mumbai tbilisi allowed, tbilisi allowed carry, allowed carry kg, carry kg hand, kg hand carriage, hand carriage kg, carriage kg laptop, kg laptop bag, laptop bag mumbai, bag mumbai airport, mumbai airport issue, airport issue let, issue let pas, let pas reached, pas reached sharjah, reached sharjah transit, sharjah transit started, transit started weighing, started weighing laptop, weighing laptop bag, laptop bag took, bag took ticket, took ticket away, ticket away asked, away asked pay, asked pay fine, pay fine never, fine never worst, never worst experience, worst experience never, experience never using, never using air, using air arabia]                                                                                                                                                                                                                                                                                            "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk import ngrams\n",
    "def ngrams(input_list):\n",
    "    #onegrams = input_list\n",
    "    bigrams = [' '.join(t) for t in list(zip(input_list, input_list[1:]))]\n",
    "    trigrams = [' '.join(t) for t in list(zip(input_list, input_list[1:], input_list[2:]))]\n",
    "    return bigrams+trigrams\n",
    "tweets['grams'] = tweets.normalized_tweet.apply(ngrams)\n",
    "tweets[['grams']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "c0b51587-a300-8f90-cd30-f44791540f96"
   },
   "source": [
    "And now some counting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_cell_guid": "3a4c4888-2be9-54d6-9b5b-9b8b336cdfa8"
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "def count_words(input):\n",
    "    cnt = collections.Counter()\n",
    "    for row in input:\n",
    "        for word in row:\n",
    "            cnt[word] += 1\n",
    "    return cnt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "_cell_guid": "c0395d2b-61cb-213e-6c38-e476770a9028"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets[(tweets.airline_sentiment == 'negative')][['grams']].apply(count_words)['grams']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "cdca1e1c-eb02-239f-eb98-dc16e32dbcde"
   },
   "source": [
    "We can already tell there's a pattern here. Sentences like \"cancelled flight\", \"late flight\", \"booking problems\",  \"delayed flight\" stand out clearly. Lets check the positive tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "_cell_guid": "aa00dfb3-81e0-980b-cfd2-e56454a612c4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets[(tweets.airline_sentiment == 'positive')][['grams']].apply(count_words)['grams']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "e11015ce-3f91-7708-36ab-1ccad7143cfb"
   },
   "source": [
    "Some more good looking patterns here. We can however see that with 3-grams clear patterns are rare. \"great customer service\" occurs 12 times in 2362 positive responses, which really doesn't say much in general. \n",
    "\n",
    "Satisfied that our data looks possible to work with begin to construct our first classifier.\n",
    "\n",
    "# Linear SVM classifier\n",
    "We will build a simple, linear Support-Vector-Machine (SVM) classifier. The classifier will take into account each unique word present in the sentence, as well as all consecutive words. To make this representation useful for our SVM classifier we transform each sentence into a vector. The vector is of the same length as our vocabulary, i.e. the list of all words observed in our training data, with each word representing an entry in the vector. If a particular word is present, that entry in the vector is 1, otherwise 0.\n",
    "\n",
    "To create these vectors we use the CountVectorizer from [sklearn][1]. \n",
    "\n",
    "\n",
    "  [1]: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "13b5d690-eb25-6243-7a40-c06cfa3bf825"
   },
   "source": [
    "## Preparing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "_cell_guid": "f16bfa38-fb30-aff1-d53a-54d23b6a4322"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.sparse import hstack\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "count_vectorizer = CountVectorizer(ngram_range=(1,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "_cell_guid": "3a964c4c-c16e-6592-e68e-3f6ef1197fdc"
   },
   "outputs": [],
   "source": [
    "vectorized_data = count_vectorizer.fit_transform(tweets.review)\n",
    "indexed_data = hstack((np.array(range(0,vectorized_data.shape[0]))[:,None], vectorized_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "_cell_guid": "457bdfd2-685d-bbc1-699f-1954108dd1d5"
   },
   "outputs": [],
   "source": [
    "def sentiment2target(sentiment):\n",
    "    return {\n",
    "        'Negative': 0,\n",
    "        'Neutral': 1,\n",
    "        'Positive' : 2\n",
    "    }[sentiment]\n",
    "targets = tweets.airline_sentiment.apply(sentiment2target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "f215daf3-d2b9-134b-a371-98ef9bc3c4e7"
   },
   "source": [
    "To check performance of our classifier we want to split our data in to train and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "_cell_guid": "c99298bd-84e9-d31e-58d1-00dbbdfa83e1"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "data_train, data_test, targets_train, targets_test = train_test_split(indexed_data, targets, test_size=0.4, random_state=0)\n",
    "data_train_index = data_train[:,0]\n",
    "data_train = data_train[:,1:]\n",
    "data_test_index = data_test[:,0]\n",
    "data_test = data_test[:,1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "936edaf6-9b95-44f7-fc0f-80f0699b66aa"
   },
   "source": [
    "## Fitting the classifier\n",
    "\n",
    "We're now ready to fit the classifier to our data. We'll spend more time on hyper parameter tuning later, so for now we just pick some reasonable guesses. Note here that we use the OneVsRestClassifier. This allows us to get the probability distribution over all three classes. Behind the scenes we actually create three classifiers. Each of these classifiers determines the probability that the datapoint belongs to it's corresponding class, or any of the other classes. Hence the name OneVsRest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "_cell_guid": "ed3e70df-a983-fb02-1b47-0499bda60850"
   },
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "clf = OneVsRestClassifier(svm.SVC(gamma=0.01, C=100., probability=True, class_weight='balanced', kernel='linear'))\n",
    "clf_output = clf.fit(data_train, targets_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "3f5a37b6-e56f-036b-e44e-9ec4814cd8d7"
   },
   "source": [
    "## Evaluation of results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "_cell_guid": "1af23c8b-7c12-63f5-260b-9f34d1285064"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5417095777548918"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(data_test, targets_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "6f1e98e8-f816-5e9a-f0f9-957798dfa8c1"
   },
   "source": [
    "It's most likely possible to achieve a higher score with more tuning, or a more advanced approach. Lets check on how it does on a couple of sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "_cell_guid": "c8a2059d-c320-c958-613e-0085c791f128"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.44742314, 0.21497132, 0.33760554],\n",
       "       [0.38933424, 0.21293726, 0.3977285 ],\n",
       "       [0.46183854, 0.1960791 , 0.34208236],\n",
       "       [0.52294389, 0.20500897, 0.27204714],\n",
       "       [0.50038688, 0.19455051, 0.30506261],\n",
       "       [0.45184917, 0.19888444, 0.34926638],\n",
       "       [0.45500853, 0.20756098, 0.33743049]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = count_vectorizer.transform([\n",
    "    \"What a great airline, the trip was a pleasure!\",\n",
    "    \"My issue was quickly resolved after calling customer support. Thanks!\",\n",
    "    \"What the hell! My flight was cancelled again. This sucks!\",\n",
    "    \"Service was awful. I'll never fly with you again.\",\n",
    "    \"You fuckers lost my luggage. Never again!\",\n",
    "    \"I have mixed feelings about airlines. I don't know what I think.\",\n",
    "    \"\"\n",
    "])\n",
    "clf.predict_proba(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "74935e17-0463-78c9-c8be-e2ba71fdfe21"
   },
   "source": [
    "So while results aren't very impressive overall, we can see that it's doing a good job on these obvious sentences. \n",
    "\n",
    "## What is hard for the classifier?\n",
    "\n",
    "It's interesting to know which sentences are hard. To find out, lets apply the classifier to all our test sentences and sort by the marginal probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "179f152a-dd8b-13ac-3b3a-649b16f0bcf2"
   },
   "source": [
    "Here are some of the hardest sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "_cell_guid": "c94ef2e0-bd5d-bfb9-16ac-90459b0d29e0"
   },
   "outputs": [],
   "source": [
    "predictions_on_test_data = clf.predict_proba(data_test)\n",
    "index = np.transpose(np.array([range(0,len(predictions_on_test_data))]))\n",
    "indexed_predictions = np.concatenate((predictions_on_test_data, index), axis=1).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "_cell_guid": "a1346d01-0e5d-8f01-b364-1523c8665203"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Check-in and boarding in Cape Town were smooth. The flight to Dubai on a B777 was pleasant, with a comfortable seat which had adequate storage, and excellent food. The service was efficient but rather formal. The IFE on this flight was quite clunky. At Dubai we were bussed to the terminal, but as there was a dedicated Business Class bus this process was quite painless. Both the terminal itself and the Emirates lounge were very busy just after midnight. The second flight on an A380 was a big step up in luxury, with a more spacious seat, much more storage, more friendly cabin crew and very smooth IFE. Something went wrong with my vegetarian meal order, but it was cheerfully fixed. All in all a good experience.',\n",
       " ' Tallin to Singapore via Istanbul. It was my first time flying with Turkish on long haul flight and I was expecting a good experience. The food was bad on all the flights we had. The A330-300 from Istanbul to Singapore seemed old. The touch screens were not working normally, only with the remote that made navigation uncomfortable. The in-flight entertainment was really good though. It would have been nice if cabin crew would have offered drinks and snacks during the flight in addition to the dinner and breakfast served. On the way back from Singapore to Istanbul we were seated in the isle and it was just so cold, I was sitting with a hat, scarf, sweater, cardigan, blanket on me and my wool coat way over my head and I still got sore throat. ',\n",
       " 'Boarding was smooth and take off on time, because of a typhoon hitting India we had to take a different longer route so the flight took more than 9 hours. We had the new QSuites in business class, they are comfortable but lack storage so I had to put my shoes in the overhead. The food was poor, I choose the Arabic Mezza, they used to offer a full mezza previously, now it is only three selections with a tasteless taboula and hummus that lacked lemon and olive oil. The entertainment system was nice in the movie selection but poor on TV, the toilets are huge on Qatar Airways and very well stocked. Very friendly and attentive cabin staff.',\n",
       " \"This was the 2nd leg of our journey from Gatwick to Ho Chi Minh City with Emirates, the first having been on a A380. We arrived in the morning local time and left in the morning local time. The food on the first leg was ok - not brilliant, but not awful just average. When we got on the next flight we were served a seafood lunch, at 9 am (ish) local time or 6 am as per our body clocks. Needless to say it remained untouched. Can't fathom why a lunch was served so early in the day. The bus from terminal to plane was extraordinarily long and seemed to take for ever but was most definitely at least 20-25 mins. The aircraft itself was definitely very dated and the seats far from the best. All in all it was a distinctly average experience and isn't likely to encourage us to use Emirates in the future. Having said all of that it was relatively cheap for Business class so I guess in this instance you sort of get what you pay for. (Best food to date was on ANA, best seats on Qantas A380)\",\n",
       " \"  Vilnius to Istanbul. Turkish Airlines excelled in cabin service, catering, aircraft and seat comfort. The departure was on time,  aircraft was new (with sky interior lightning) and very clean. There weren't too many passengers. Therefore, there was a free middle seat between me and the other passenger. The seat pitch was fine. In the beginning, I got a chocolate praline. Then we got earbuds (bad quality and shallow sound but it is economy so I am not going to complain) and I managed to watch a full movie during the flight (the screen was pretty large and had good resolution). The food on this short-haul flight was tasty and it included a hot meal (which is rare on European flights) and a complimentary beverage (I recommend Ayran yoghurt and homemade lemonade but you can also take alcoholic drinks). The flight attendants offered tea and coffee. The flight attendants provided top notch service. Overall it was an exceptional experience.\",\n",
       " 'The standard has declined drastically. The previous time I had flown with Emirates before this was in 2011 and it was fantastic, especially the food and service on board. Now, it is disappointing. The food selection was horrible. Snacks and drinks very sparse. The cabin crew took a long time to serve our meals. The Emirates lounge in Kuala Lumpur was not given to Economy passengers for free, unlike in 2011. The inflight entertainment was okay. Some headsets do not work. Flights were not punctual. There was a slight delay in all the flights throughout my journey with Emirates this time. Seat selection was not given for free (like some other airlines) for Economy Saver.',\n",
       " 'Manchester to Bangkok via Doha return, and a superb trip with Qatar Airways. Check in at Manchester business desk took only a few minutes then on to the Escape lounge which was quite full, perhaps one day Qatar will have their own lounge in Manchester and maybe a complimentary Chauffeur service. Boarded and a member of cabin crew asked about drinks while others introduced themselves, always a nice touch, took off virtually on time. A330 is a nice aircraft, small business class section with comfortable seats, decent selection of movies, food very good as usual on Qatar. Arrived Hamad International a few minutes ahead of time and disembarked into airport, no bus. As I had a few hours to spare went to the Al Mourjan lounge until gate for Bangkok advised. Bangkok flight on the A380, some of the departure gates designed with this aircraft in mind as separate business class gate and upstairs lounge to board direct to upper section. Much bigger business section with 48 seats, very comfortable but strange seats no arm rest of the left hand side until you reclined the seat. Good flight, again managed to sleep for a short time before arriving at Bangkok. Return flights just as good despite an 8 hours transit in Doha (more about that in the lounge review) I always get messed up with the different Airbus A330 models but the one from Doha to Manchester was different from the first flight, looked a little older and smaller. I can’t praise the Qatar crews on all four flights enough, brilliant service with maybe the best food on any airline. The only thing that spoilt a perfect trip was that between Bangkok and Manchester my case was damaged in three different areas, it looked like to main damage was done by the wheels of another case.',\n",
       " \"\\xa0Flight to Doha from Istanbul was fine. The seat configuration on this flight was 2x2 in business class and there were 12 business class seats on A321 aircraft. My headset was dysfunctional. After a mundane meal service, I dozed off. We got to Hamad International Airport and proceeded to the Al Mourjan Lounge. The lounge was pretty packed, we were lucky enough to find a table at the restaurant and ordered the fillet steak, disappointing that the steak was overdone. Service was great in the Al Mourjan Lounge. We transferred to the area near the family rooms to take a shower. The lounge staff informed us it was the final call for boarding at our gate so we rushed to find out that boarding haven't commenced yet and there was no separate boarding area for business class. Right after take off, I fell into a slumber. Three hours later, a female cabin crew approached us and asked for our meal preference. I have to commend her for attentively paying attention to what we needed. She was at our beck and call so for that, she deserves proper credits. Cabin crew service was exceptionally five star class. Despite that, they need to step it up when it comes to food selection and quality at lounges and boarding service.\",\n",
       " 'Doha to Barcelona. Clean, spacious seating and friendly service of the crew, the attention to detail was stand out. I had many worried when the plane takes off since my mother was sick. My mother got sick from the claustrophobia and fear of heights for a long time flight But I told this fact to crew, she asked to the passengers the excuse and she took a step for my mother can lie down. In addition, she frequently checked my mother’s condition, and she asked if my mother needs. It might be small kindness because of her duty, but for my family, it was really impressive service. Although the many time goes by, I’d like to compliment the crew at the time.',\n",
       " 'Seat 32K on Turkish Airlines A330-300 from Istanbul to New York. Several rounds of security interrogations for USA bound flights at Istanbul Ataturk were bothersome, but staff polite and spoke English. The boarding process had separate lines for groups A/B/C, in addition to business and elites, etc.  Flight attendants in my section, especially female staff members, remained cheerful and attentive throughout the flight. Food was very good. Smoked salmon with celeriac remoulade, salad, grilled beef kofte with bulgur pilaf, dessert, etc. Bread and drinks were offered along with food service. Refills given if requested. Well-stocked galley snacks. Pre-landing second meal service had a hot entree. Their headphones are of better quality now. Toilets remained in decent condition. Announcements could be better. ']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def marginal(p):\n",
    "    top2 = p.argsort()[::-1]\n",
    "    return abs(p[top2[0]]-p[top2[1]])\n",
    "margin = sorted(list(map(lambda p : [marginal(np.array(p[0:3])),p[3]], indexed_predictions)), key=lambda p : p[0])\n",
    "list(map(lambda p : tweets.iloc[data_test_index[int(p[1])].toarray()[0][0]].review, margin[0:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "9d28561f-5767-29f8-a49c-85e191b63c4b"
   },
   "source": [
    "and their probability distributions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "_cell_guid": "fa092618-003a-c2e5-1228-a95aa18337ef"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([0.34523002, 0.30963503, 0.34513496]),\n",
       " array([0.37329894, 0.25320246, 0.3734986 ]),\n",
       " array([0.38429004, 0.3839978 , 0.23171216]),\n",
       " array([0.36435664, 0.27066729, 0.36497607]),\n",
       " array([0.36213288, 0.27645426, 0.36141285]),\n",
       " array([0.38178277, 0.23568127, 0.38253595]),\n",
       " array([0.34533388, 0.31020216, 0.34446396]),\n",
       " array([0.41374828, 0.41285583, 0.17339588]),\n",
       " array([0.40528169, 0.18790892, 0.40680939]),\n",
       " array([0.35714661, 0.28378483, 0.35906856])]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(map(lambda p : predictions_on_test_data[int(p[1])], margin[0:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "3d8c9fdd-8e68-0fd1-3f06-bfaaeba4b79f"
   },
   "source": [
    "How about the easiest sentences?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "_cell_guid": "c2468ac1-b7fb-889c-50a3-e5d7af0269d5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['  Kilaminjaro to Chicago via Istanbul. My problems began the day before the flight, I had booked a Turkish Airlines award business class ticket using United Mileageplus miles months earlier, but the day before noticed I couldn\\'t select my seat and part of my flights were missing from the itinerary. I called Turkish Airlines to confirm everything was but for some reason I couldn\\'t select my seat, so to arrive to the airport early. I arrive the next day to the airport 3 hours early in Kiliminjaro airport in Tanzania and it turns out there is a massive problem with my ticket. A Turkish airlines agent later said it was \"it was award ticket and NO REC on your onward boarding pass\"\" which I have no idea what it means but the agent was smashing away at the keyboard for hours trying to get me on the flight to the point they had to hold the flight to Istanbul. At the very last minute I was finally I was able to board and the Turkish airlines agent said I shouldn\\'t have any more problems, but once I get to Istanbul I need to get my next boarding pass and that my bag would arrive just fine to Chicago (both of these turned out to be false) Already having a bad day,I was informed that for the duration of the flight I would only receive economy service and meals despite booking a business class ticket. Once I arrived to Istanbul, more problems began as the transfer desks agents were extremely unhelpful just ping ponging me from one desk to the other with no agent willing to print my boarding pass or ticket. As it got closed to boarding time I told a transfer desk agent I\\'m not leaving until they print me my pass which I paid for and I was bound to miss my flight! In the end a desk agent in Istanbul wrote me ticket by hand 30 minutes before the flight was set to take off and told me to run! This ticket got me past security and it seemed everything was ok until I got to the gate and once again they denied me boarding, while they grabbed my passport and the agents again smashing on a computer keys trying to get me boarded. Then they tried to claim that my ticket was only economy until I showed them the receipt, at this point they were once again, holding the next plane up for me. At the very last second they finally let me board the flight to Chicago. Once arriving to Chicago I was once again greeted with more bad news. My bag had not made it so I asked them to ship the bag to Mexico City (where I live). They said I would need to pick it up at the airport \"the next day.\" Well turns out the next day was a whopping 5 days later. When the bag arrived it was completely destroyed. The nightmare doesn\\'t end there. I figured they would be quick to provide compensation over such an incident, especially over an expensive long distance business class ticket but was I wrong. Once I began the process I sent them all relevant details including pictures of my damaged bag, and receipts totalling about $500 for various purchases I had to make for the 5 days I didn\\'t have my bag. Then they said that they couldn\\'t pay me anything for my damaged bag because they needed a \"repair invoice\" when they originally said a financial request will suffice in lieu. At that point I had already thrown my damaged baggage away. ',\n",
       " \"\\xa0I was scheduled to fly yesterday from Atlanta to Chennai via Doha. After all check-in, security clearance process, I was informed that the flight is cancelled while boarding. They didn't send any notifications to mobile or email. They said we need to rebook for next available flight and asked us to check out all luggages. I checked out and was waited at the booking counter line for 2 hours. For the entire Atlanta International airport, they had only one booking counter open and one booking agent available. After one hour of waiting, I got frustrated and called their customer service while staying on the line. I was on call-wait status for 50 minutes before getting hold of someone in their customer service. Customer service agent couldn't rebook me on next day flight unless I'm ready to do a 3 stopper instead of 1 stopper that I booked. I got the same itinerary 2 days later and asked him to book it. He couldn't book me because I was already checked-in and they needed me checked out from airport side. He said he will do all that and call + email me later with confirmation. I never received any call or email till today. I called their customer service again and was able to get hold of someone after 30 minutes wait. I got the same answer today as well. They have asked me to call them again after 4 hours. Not sure how long I should be on hold to get hold of them. And their website still shows the flight status as delayed and not as cancelled. I still didn't receive any delay/cancel notification from them. This is one of the worst experience I had. They are completely understaffed and don't infrastructure in place to handle delays/cancellations.\",\n",
       " '\\xa0I travelled with Emirates Back and fourth to Bali during the Christmas holiday. My experience of the flight was ok, but my experience with the transport of baggage and the customer service was the worst I have ever experienced. I go on a surf trip 3-4 times a year and I always pack my board the same way with allot of bubble wrapper, towels, pillows and a good board bag and I have never had any problems. On the trip down when I unpacked my board I had a few cracks that I had to fix. I called Emirates, and a very rude customer service member said that someone would call me up, of course no one did. But lesson learned. On my way back I packed my board even more carefully and added a fragile sign on the bag. To my chock when I packed up my board on arrival in Amsterdam, it was completely destroyed and rendered it useless. I called Emirates who refuse to take this matter on the telephone but told me they wont cover anything but I could send them a mail. Which I did. In the email I pointed to according to the Montreal convention they are responsible for any damage they cause on the baggage. The response was very simple. \"we are sorry for the broken board but we can and wont do anything about it, and we will not answer any more enquiries regarding this matter\" I have emailed them many times after that but the refuse to answer. My friend travelling to Copenhagen also got his board destroyed and same horrible treatment by Emirates. That’s why I write this review, I want to warn everybody to travel with Emirates! As long as you pay and shut up there is no problem but when they have caused a problem they wont do anything about it. If you Google similar matters you will see that I am not alone in being mistreated by emirates customer service. I will never travel with Emirates again. 0/10 points.',\n",
       " 'We were planning a trip to Perth to see our daughter, tragically my wife fell ill, diagnosed terminal within a few days. Just a few weeks later on the day the flight was leaving, after an Emirates flight reminder, she died in just 7 short weeks. During her illness I tried to cancel the flight and use the money to fly my daughter back. Although I thought I had purchased one return ticket with one set of conditions that allowed cancellation, the return flight unknown to me had its own set of conditions that precluded any refund. After many emails and phone calls I was not able to cancel the flight and get a refund. After family help, with no help from Emirates, we got enough money together get my daughter home, tragically too late. During that time Emirates thanked me for my understanding and refused to help quoting terms and conditions of the return ticket and refused to consider changing a ticket to allow my daughter to fly back. They would retain my money and provide no help. During my wife’s illness MacMillan Cancer Support contacted Emirates on several occasions and attempted to help get my daughter home using the tickets or a refund on the purchased tickets. The same reply was given to MacMillan Cancer Support as was given to me, despite them talking to senior staff. Emirates final email thanked me for my understanding in these difficult times, saying they could do nothing to help. MacMillan Cancer Support wrote to Emirates to express their concerns over the lack of compassion and support, to-date there has been no reply. If you fly with Emirates pray you will not need their help because they will thank you for your understanding and take your money and leave you alone to sort it out.',\n",
       " 'I booked a round trip flight from London to New York JFK with Kuwait Airways. On the first leg of the trip, the flight was delayed for 7 hours without any explanation for the delay. Although this was a bit annoying and unexpected, the airline did provide a hotel and meals, which I appreciated. My major grievance with this airline was the experience for my return flight. Upon arrival at the airport 3 hours before my scheduled departure I was told that the flight had been cancelled and that I had been rebooked on another airline. However, when I went to the new airline to check in, I was told that they did not know what I was talking about, that they had no booking for me, and that Kuwait Airways had not contacted them. I found this to be unprofessional on the part of Kuwait Airways. After literally 2 hours of back and forth and waiting in long queues, the new airline did not accept my rebooking since they were overbooked themselves, and Kuwait Airways essentially told me that they could not rebook me for that day and they could not tell me when they would be able to rebook me, and that I should wait in a hotel they were providing until they called or emailed me with my rebooking information. I found this unacceptable and asked to speak to the supervisor who was unapologetic, unsympathetic, and quite nonchalant about the situation. He eventually gave me a list of customer relations and reservations numbers for me to call and air my grievances. I returned to the airport the next morning and could not find a representative from Kuwait Airways to speak to, furthermore, none of the numbers I received worked. They either rang out or rang to a voicemail that stated that the customer services representative was on holiday and would return on a date that was earlier than the actual date, meaning that he should have been back from holiday already! I could not contact Kuwait Airways, and did not hear from them for 2 days. I eventually bought a return ticket on another airline since I had to be back in the UK in time for other travel arrangements. I did not hear from Kuwait Airways until I was already back in the UK 3 days after the flight was supposed to leave JFK. Since then, I have contacted them by email and phone to request a refund for the return flight that was not rebooked and have been unsuccessful. I know Kuwait Airways is cheap in comparison to others, but the hassle I experienced was not worth it. I would have rather paid more to be guaranteed that I would be able to travel on time and to have a better customer service experience. In the long run, choosing Kuwait Airways eventually cost me more since I ended up having to buy another return ticket anyway. Use this airline at your own risk but I do not recommend them.',\n",
       " '\\xa0I was flying from Prague to Singapore via Dubai and I purposefully chose to fly with Emirates to try a new carrier on that route as it was recommended to me by a friend who had a great experience with them in the past. However, on the day of the flight, I was unpleasantly surprised to find out that the 5.5hr flight from PRG to DXB both ways was serviced by flydubai. As the flight number was starting with EK like all Emirates flights do, and as I did not see any warning about it while purchasing the ticket online, I automatically assumed the whole trip would be with Emirates. Oh boy was I shocked when I found out - I felt totally deceived! And to clarify, I had 3 other options I could have chosen from in the same price category and flight time for that route, but I chose this \"Emirates\" one to give it a try (as I have flown the same route before with Turkish, Qatar, Lufthansa/SIA) - so yes, the excuse by flydubai for all their mediocre and below standard service as their \"efforts to keep prices low\" is just completely wrong - it\\'s not cheap at all. And on a side note, a lady I met in Dubai that was taking the same flight back with flydubai felt exactly the same way and said she would never fly with them again and that she also did not know it was going to be with them either - she was outraged. So, what was the flight like? A complete dread and discomfort, which I would certainly not expect for such a long flight. If it was 1-1.5 hour flight, it would be perhaps acceptable, but certainly not for this long flights - I am shocked that they are not banned from operating such long flights given that they are simply not suited for them. In short, there was much lesser legroom and smaller seats compared to the Emirates ones, no food or drinks provided on the whole trip (unless for a few exceptions - I was one of them and I received a 1.5dl of water in a plastic box and a veggie lasagne and that was it - and it\\'s not like I couldn\\'t have bought something from their onboard shop - the thing is that I was so disgruntled with their services that I did not want to contribute them any more money. And btw I would much rather pay those few extra bucks at the point of buying the ticket for the comfort of having everything included in the price - again, I felt like I was cheated once more, because I do expect much better services on long flights). All the entertainment was paid (again, I would much rather pay those few bucks beforehand to have it included) and no wifi. It was all just a pain. Another huge downside about the experience was the flydubai\\'s distant and extremely low quality terminal 2 in Dubai, where all their flights fly from. Not only that it took 45mins to get to the main Emirates terminal on my way to SG, but it also did not feel safe and the service was really bad - we took some shabby buses that seemed to be for workers at the airport and the drivers were not too compelled to work and to drive us there, so it took another 20mins just waiting on them. Also, on my way back, I was looking forward to doing some last-minute souvenir shopping for my family in the modern part of the airport - and nope, that part of the airport was not possible to access if you were leaving from T2, so I went directly there and it was just bad - one large dirty hallway, and small duty free. Never again - not them, nor Emirates - for the fear that part of the flight would be with flydubai again.',\n",
       " 'I can’t describe how upset I am by being stuck on Fl Dubai plane for 5 hours with no food, drink, in-flight entertainment or internet. I did book Moscow to Hanoi via Dubai specifically to fly with Emirates, that cost me 650USD one way, compared to Aeroflot 650USD/round trip which is direct, modern and safe anyway. I paid twice as much just to have an experience with Emirates. Instead of Emirates plane I found myself in an outdated shabby unsafe little aircraft. I never been send to a low-cost instead of a full service. How is that even possible a company like Emirates co-shares with a lower cost. By co-sharing I mean they stuck me on a lower cost, send one meal, 1 cup of free water and let me carry 30kg of luggage. There was no choice of food and the suggested free dinner, only one option, was not suitable for me, no free snacks available no nothing. I had to buy water and peanuts. I booked a ticket with Emirates for my husband London-Dubai-Hanoi round trip and he had an amazing flight despite his flight was longer and the price was cheaper. I don’t book planes which sell water and sandwiches and I would never in a conscious mind choose Fly Dubai for a 5 hours flight, seats were uncomfortable even for my 5 feet. It was freezing cold the entire time and no blankets available, I couldn’t even hear the announcement it was just cracking noise. The whole experience felt like a charter flight in 90s, I never saw flight attendants just sleeping on the back row on a 5 hours flight before, our departure was also delayed because of late arrival. I couldn’t even charge my phone up to 60% in 5 hours, the equipment was that old! I have noticed that my flight is operated by FlyDubai on behalf of Emirates on my ticket but I have no idea what FlyDubai is and didn’t expect such a con being pulled off by Emirates. FlyDubai is not the same level as Emirates, if I wanted to fly with FlyDubai, in my case Emirates is a middle-man profiting by using the brand to sell cheap flights at high price. I complained to the customer service afterwards but they just blamed me on not researching what FlyDubai is. So I guess they are aware of what they are doing but hope that people will be just tricked and book a ticket because of Emirates brand on it. Second flight Dubai-Hanoi was also pretty awful, just got a sandwich, croissant and an omelette in 7 hours flight, was pretty uneatable, the plane was very shabby as well. And at the check-in Emirates even tried to put me on standby because the plane was overbooked, even though I paid for my tickets 2 months in advance. guess Emirates don’t care about their customers as much as they did.',\n",
       " '  In September 2016 I had a horrible experience with Turkish Airlines. My flight (TK-476) was on 17/09 from Ekaterinburg (Russia) to Barcelona via Istanbul. The time of departure was supposed to be at 7.30 - arriving time to Istanbul at 10.15 and the departure to BCN at 14.40 - so having 4.25 hours would be more than enough to change the plane in Istanbul. Unfortunately the Turkish Airlines delayed the flight in Russia for more than 3 hours. I can understand that there might be some technical problems or something but in that case the company is supposed to \"think\" of its passengers and be more client-oriented which I haven\\'t noticed. The representatives didn\\'t give any explanations regarding the reasons of delay, time of the flight, food, etc. When we finally arrived to Istanbul we still had enough time to catch the plane to BCN (we had 1 hour before the flight) and  they could delay the plane for 15-20 min to let us in (I was not the only passenger - we were like 20 people who found themselves in this situation).  The only thing they said was \"sorry, you have to wait for the next plane\" (additional 5 hours of waiting). They ignored us and our questions. After that I asked for food and I was given some sort of a coupon which was possible to use only in some places in food court. The cheapest ones, as I realized later. So the only thing I managed to have using this coupon was a bottle of water. Therefore sitting in Istanbul airport waiting for the plane for 5 hours I had to spend my money on normal food in a normal place. For the information: before that in Russia I was waiting for 3 hours and nobody informed me about breakfast so again I had to spend my money on food there. My advice to Turkish Airlines:  teach your employees how to communicate with clients and to be more client-oriented. ',\n",
       " '  Kiev to Male via Istanbul. Two days lost and headaches all the way to our wedding. Horrible experience from start to finish with Turkish Airlines on our wedding/honeymoon trip from Kiev to Maldives. Sorry for the lengthy post. I will try to make it as concise as possible. Website and Mobile App issues: After creating my online profile, I needed to update my name to include my middle name, as it shows on my passport. Once my tickets were issued from Expedia, Turkish Airlines website and App couldn\\'t link the ticket to my Miles&Smiles profile because of the name discrepancy. It took several calls and emails to support, and even now it still shows my first name and middle name combined. Couldn\\'t update my rewards number: I mistakenly had the wrong Miles&Smiles number on my Expedia account when I purchased the tickets (copy/pasted without the last digit). Turkish airlines couldn\\'t update my ticket with the correct rewards number. Sure, it was my mistake, but it seemed like a simple request to update my ticket. Flight schedule change: Our flight times from IST to MLE on April 30th change from arriving at 11:50am to arriving 12:15pm, which was not a big deal, but logging into their website and App the system asked me to confirm the change, cancel the flight, or change the flight. There was no option to confirm the change. I couldn\\'t change my flight (which was free of charge). Every time I\\'d select the new flight arriving in MLE at 12:15, the system would say it couldn\\'t find a flight with that time. I made several calls to Turkish Airlines support, and they never helped me clarify if I was confirmed on the changed flight. The best they could do was tell me to contact Expedia and have them re-issue tickets. Between Turkish Airlines and Expedia support, I must have made a half dozen calls regarding this. By the end, Turkish airlines best response was that \"everything should be fine\"\\x9d. All I wanted to know is that everything IS fine\"\\x9d. Cancelled flight from KBP to IST: As we were about to board our flight from KBP to IST on April 29th at 9:30pm, we were told at the gate that our flight was cancelled. \"• It took 6 hours to stand in line for new tickets. Calling customer support, the guy was rude and told me to just wait in line. When calling Expedia\\'s Gold customer support to get me a new flight, they were unable to assist because the Turkish Airlines flight was not showing an updated status, and that the flight was still on-time. Expedia could have helped me get new tickets, but they just needed to confirm the flight was cancelled. Expedia even called Turkish Airlines to get updates, and Turkish Airlines was unable to give them the cancelled status. So, after several calls to Turkish airlines and Expedia, we just continued to wait in line. We eventually got new tickets to IST, which left Kiev at 10am the next day (April 30th) and arrived in IST at 12pm. However, our connection from IST to MLE was at 1:00am that morning on April 30th, arriving in MLE at 12:15pm. The next flight from IST to MLE was at 4:15am on May 1st arriving in MLE at 2:20pm. \"• Turkish Airlines did put us up in a hotel for a few hours, since our new departure wasn\\'t unto 4:15am the next day. But, the customer service representatives in IST airport were all rude and lacked any empathy. The hotel (Ibis Hotel) was a 1-hour drive from the airport The room was dirty. Sheets stained. Hair and bead trimmings in bathroom I had to pay for a visa ($30, not a big deal)',\n",
       " \"Very spacious legroom for economy but frequent Late flights. IFE had a small selection of movies, cabin Crew extremely rude and unprofessional. Ground Staff rude and unhelpful. Transfer experience at Kuwait is confusing and difficult. Kuwait Airport lacks many facilities common in most international airports Review: I initially booked this flight because it was the cheapest option to Dubai, around $60 cheaper than the competition; big mistake. I did not have a single flight with Kuwait, either on my way to Dubai or back that was on time. From JFK, my flight was delayed for 3 hours, causing me to miss my connection to Dubai. They put us in a Hotel on Airport property, which, although had a good view of the apron, was a sub-par hotel. The ground staff do not know how to do their job correctly. They are rude, and the airport lacks proper signage. You are bussed around from terminal to terminal with no real staff presence to guide you. We almost missed our flight to JFK because the gate was labeled Mumbai, so we didn't know that it was intended for New York. Kuwait Airport doesn't even seem qualified to be a domestic airport, lacking proper signage and two distant terminals which you're bussed around in. Ground crew does not care about you, and show no interest or courtesy when dealing with you. The same can be said with the cabin crew. They are rude, and do not know how to treat passengers. The meal timings are horrible, as they leave many passengers extremely hungry for long hours on the plane, and water service is extremely selective. You'd think an airline knows when to give out water and food. They also gave us no details about our stop in Shannon on our way back to JFK and could not be seen for much of the flight. The only good part about the flight was the legroom, which was spacious. However, that is not a reason for booing Kuwait over any other competent airline. Do not book Kuwait, they don't care about customers and lack the competency to even communicate basic necessary facts to customers.\"]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(map(lambda p : tweets.iloc[data_test_index[int(p[1])].toarray()[0][0]].review, margin[-10:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "d2387867-ac36-8219-0d71-fb0f578d4913"
   },
   "source": [
    "and their probability distributions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "_cell_guid": "80f2aff9-3273-caab-b9a4-959aebc9d69d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([0.83927641, 0.15528079, 0.0054428 ]),\n",
       " array([0.80588326, 0.11993946, 0.07417727]),\n",
       " array([0.83598591, 0.14973393, 0.01428016]),\n",
       " array([0.80991653, 0.1116305 , 0.07845297]),\n",
       " array([0.84558568, 0.13788772, 0.0165266 ]),\n",
       " array([0.83249298, 0.11428938, 0.05321764]),\n",
       " array([0.83156732, 0.11326797, 0.05516471]),\n",
       " array([0.83910267, 0.11167507, 0.04922226]),\n",
       " array([0.85328033, 0.08139333, 0.06532634]),\n",
       " array([0.8747255 , 0.09478096, 0.03049353])]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(map(lambda p : predictions_on_test_data[int(p[1])], margin[-10:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "254f79d4-feee-706a-a2a0-a602069be3cc"
   },
   "source": [
    "Looks like all of the easiest sentences are negative. What is the distribution of certainty across all sentences?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "_cell_guid": "48fe3201-eb49-9617-eeb5-fca45beb657f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHuFJREFUeJzt3XmcHVWZ//HPF2LYl0AaJJsNGBdERSciKoNsMwMoEkdgyKAmGs1PZVzGXXEG3EZ0/AHuTgaUgAgBXIgIKEsibsEJu6hohJBEAmkgYVWZwDN/nNOkcjndfe9N3yXd3/frdV9dy6mq59a9Xc89p6pOKSIwMzOrtVmnAzAzs+7kBGFmZkVOEGZmVuQEYWZmRU4QZmZW5ARhZmZFThCjiKSPSjpjmNYVkp45HOtqYJufknSvpLslTZH0sKTNByh7sqRvtTO+wUiaJelng8y/TNLMdsa0KZK0SNJb8vCg+7Sw7DJJh7YuupFnTKcDGI0kLQMmABMi4t7K9BuBFwK7R8Sy4d5uRPzHcK+zXSRNBt4HPCMiVufJ23YwpGEVEYfXU05SAFMjYmmLQ+oYSQL+CPwlIvbqUAwjfj/XwzWIzrkDmNE/Iun5wFbNrkzSSE/2zwDuqyQHG2Zd9B06ANgF2EPSSzodzGjmBNE55wBvrIzPBM6uFpD0Kkk3SHpQ0gpJJ1fm9eZmntmSlgNX5+lvlHSnpPsk/Vu1Wl1tdqksP1PS8tx0c2Jl/ftK+qWktZJWSfqypLH1vDFJO0n6pqS7JK2R9P3KvLdKWirpfkkLJE2ozAtJb5P0h7zcV5QcClwBTMjNSmdV4h+Tl91d0k8kPSTpCmB8TUz7SfpFfj83STqwMm+RpE9K+nle/seSxlfm719ZdoWkWXn6FpI+n/ffPZK+LmnQJJ/Lr5F0h6TDK9OrTSfPzO/lgfy5zM/Tr8nFb8r74Z/q2Kd/L+m2vK6v5vVWm2h+Luk0SfcDJ0vaU9LV+ftzr6RzJe1YWd8ySR+QdLOkRySdKWlXpSayhyRdKWncoF+Qoc0ELgYuzcNNkfSGyv/CiTXzBvx+l/azpHGSLpHUlz+/SyRNaja2TUZE+NXmF7AMOBS4DXgusDmwgvQrOYDeXO5A4PmkRP4C4B5gep7Xm8ueDWxDqn3sBTwM7A+MBT4P/C9waF7mZOBbNcv/d172hcBfgefm+X8D7EdqhuwFfgu8p/IeAnjmAO/vh8B8YBzwNOCVefrBwL3Ai4EtgC8B19Ss8xJgR2AK0AccVtkXKytl++Mfk8d/CZya13sA8FDlvU4E7gOOyPvy7/J4T56/iNSk8ay8LxYBp+R5U/K6ZuT3sjOwT553OrAA2AnYDvgB8JkB9sms/Fm8NX/ebwfuAlSJ4S15+DzgxBzrlsD+A+33wfYpKUk+CPxj/hzfnWN4SyWmdcA78/ytgGfm/bMF0ANcA5xe891dDOya9+tq4HrgRXmZq4GTNuJ/Y+sc8xHA6/J7G1uZX91Ps4CfDbCe/v+FA3Jcp+b32v+/0ND3O3/ur8vxbQdcCHy/08eSlh+rOh3AaHyxPkF8DPgMcBjpF/IYKgmisNzpwGl5uDeX3aMy/9+B8yrjWwOPMXiCmFQp/yvguAG2/R7ge5XxYoIAdgOeAMYV5p0JfK4yvm0+YPVW1lk9GF4AfDgPH8gACYJ0EF8HbFOZ/+3Ke/0QcE5NLD8CZubhRcDHKvPeAVyehz9Sfd+VMgIeAfasTHsZcMcA+28WsLTmswng6ZUY+g98ZwNzq5/NQPt9sH1KqqH+sibmFWx4gF0+xHd1OnBDzXf3+Mr4d4CvVcbfyUYcOIHXk34YjCEd2NcCr63MX0R9CeLfgfMr49tQ+V9o9vtdmb8PsKbZ97mpvNzE1FnnAP9M+qKfXTtT0kslLczV2geAt1HTdEL6h+83oToeEY+SfikP5u7K8KPkE7+SnpWr0XdLehD4j8K2SyYD90fEmsK8CcCdlfgezvFNHCqeIUwg/bM+Upl2Z2X4GcAxuTlhraS1pFrWbnVsdzKpdlGrh3SQv66yzsvz9IE8uY382UD5/X2QdDD/laRbJb15kHUOtk9rvw8BrKxZvvr9QdIuks6X9Kf8uX+Lp37u91SG/1wYL35muRnq4fw6foD3MxO4ICLWRcRfge/SXDNT7Xt/hMr/QqPfb0lbS/qv3GT1IKlmtaMGuIpupHCC6KCIuJN0svoI0j9CrW+TmjAmR8QOwNdJB44NVlMZXgU82S6a28N3bjK8rwG/I13JsT3w0cK2S1YAO1XbrSvuIh2s++PbJsf3pyZj7LcKGJfX129KTUznRMSOldc2EXFKHeteAexZmH4v6WD4vMo6d4iIjb6yKiLujoi3RsQE4P8BX9XAlxQPtk9rvw+qjvdvrmb8M3naC/Ln/nrq+9yHFBGHR8S2+XVu7fzcpn8w8Pp84L4bOBo4onpOqE6rSMm9f91bs+H/QqPf7/cBzwZemssf0L/qBuPapDhBdN5s4OCaX7/9tiP9Gv+LpH1JtY3BXAQcKenl+YTbx2n+C7wdqS34YUnPIbWZDykiVgGXkQ5q4yQ9TVL/P9O3gTdJ2kfSFqRfbdfGRl7SmxPtEuDjksZK2h84slLkW6T98g+SNpe0paQD6zzJeC5wqKRjJY2RtLOkfSLiCdL5m9Mk7QIgaaKkf9iY95LXc0wltjWkA/bjefweYI9K8cH26Q+B50uarnQy/wTg6UNsfjtS2/1aSROBD2zs+2nAG4Dfkw7E++TXs0i1nhmDLFdyEfBqpQsMxgKfYMPj3VDf79r9vB3pB8FaSTsBJzUYzybJCaLDIuKPEbFkgNnvAD4h6SFSm+oFQ6zrVlIb8PmkX1APkU4i/rWJ0N5PSkgPkQ6E8xtY9g2kdvDf5e2/J8d3FfBvpHbrVaRf5sc1EVvJPwMvBe4n/fM+2WQXESuAo0i/EvtItYIPUMf3PyKWk2p478vr7r9XBdK5jaXA4tzscCXp4LaxXgJcK+lhUg3y3RFxR553MjAvN2sdO9g+jXSPzTHA50jNK3uREulg34ePk054P0BKMKWabavMBL6aa1BPvkg154aamfL/wgmkBLqKlGirzWtDfb9PprKfSef/tiLVHBeTmhNHvP4rKGwEkrQt6STf1MoBxkYpSZuRDpLHR8TCTsdj3c81iBFG0pH5hNo2pMtcbyFdeWKjUG5W2zE3P/W3sy/ucFi2iXCCGHmOIp24vAuYSrps1dXE0etlpKuw7iWdl5keEX/ubEi2qXATk5mZFbkGYWZmRd3SOVdTxo8fH729vZ0Ow8xsk3LdddfdGxGD3dQJbOIJore3lyVLBrpC1MzMSiTdOXQpNzGZmdkAnCDMzKzICcLMzIqcIMzMrMgJwszMipwgzMysqGUJQtI3JK2W9OvCvPcrPU94fB6XpC8qPVf3ZkkvblVcZmZWn1bWIM4iPUpzA5Imk555u7wy+XBSv0FTgTmkh3mYmVkHtSxBRMQ1pP7za51GeqRitROoo4CzI1lMepTfboVlzcysTdp6J7Wk1wB/ioib0tMPnzSRDZ+NuzJPW1VYxxxSLYMpU6bUzh5VDjqosfIL/QQAM2tA205S52fCnkh6MtpTZhemFbuZjYi5ETEtIqb19AzZlYiZmTWpnTWIPYHdgf7awyTg+vys5ZVUHjCe593VxtjMzKxG22oQEXFLROwSEb0R0UtKCi/Oz5xdALwxX820H/BARDylecnMzNqnZTUISecBBwLjJa0EToqIMwcofinpwfBLgUeBN7UqLqtfo+c4wOc5zEaSliWIiJgxxPzeynAAJ7QqFjMza5zvpDYzsyInCDMzK3KCMDOzIicIMzMr2qSfSW2NaeaqJDMbvVyDMDOzIicIMzMrcoIwM7MiJwgzMyvySeou4pPIZtZNXIMwM7OiUVuDcEd0ZmaDcw3CzMyKnCDMzKxo1DYxWXfwc7XNupdrEGZmVuQEYWZmRU4QZmZW5ARhZmZFPkndAJ9QNbPRpGUJQtI3gFcDqyNi7zztP4EjgceAPwJvioi1ed5HgNnA48C7IuJHrYrNWsfdhZiNHK1sYjoLOKxm2hXA3hHxAuD3wEcAJO0FHAc8Ly/zVUmbtzA2MzMbQssSRERcA9xfM+3HEbEujy4GJuXho4DzI+KvEXEHsBTYt1WxmZnZ0Dp5kvrNwGV5eCKwojJvZZ72FJLmSFoiaUlfX1+LQzQzG706kiAknQisA87tn1QoFqVlI2JuREyLiGk9PT2tCtHMbNRr+1VMkmaSTl4fEhH9SWAlMLlSbBJwV7tjMzOz9dqaICQdBnwIeGVEPFqZtQD4tqRTgQnAVOBX7YzNNg2+1NisfVp5met5wIHAeEkrgZNIVy1tAVwhCWBxRLwtIm6VdAHwG1LT0wkR8XirYjMzs6G1LEFExIzC5DMHKf9p4NOtisfMzBrjrjbMzKzICcLMzIqcIMzMrMgJwszMipwgzMysyN19t5B7NjWzTZlrEGZmVuQEYWZmRU4QZmZW5ARhZmZFThBmZlbkBGFmZkVOEGZmVuQEYWZmRb5RzkY0P2DIrHlOEGYVzdz97qRiI5WbmMzMrMgJwszMipwgzMysyAnCzMyKnCDMzKyoZQlC0jckrZb068q0nSRdIekP+e+4PF2SvihpqaSbJb24VXGZmVl9WlmDOAs4rGbah4GrImIqcFUeBzgcmJpfc4CvtTAuMzOrQ8sSRERcA9xfM/koYF4engdMr0w/O5LFwI6SdmtVbGZmNrR2n4PYNSJWAeS/u+TpE4EVlXIr87SnkDRH0hJJS/r6+loarJnZaNYtJ6lVmBalghExNyKmRcS0np6eFodlZjZ6tTtB3NPfdJT/rs7TVwKTK+UmAXe1OTYzM6tod4JYAMzMwzOBiyvT35ivZtoPeKC/KcrMzDqjZZ31SToPOBAYL2klcBJwCnCBpNnAcuCYXPxS4AhgKfAo8KZWxWVmZvVpWYKIiBkDzDqkUDaAE1oVi5mZNa5bTlKbmVmXcYIwM7MiJwgzMytygjAzsyInCDMzK3KCMDOzIicIMzMrcoIwM7MiJwgzMytygjAzsyInCDMzK3KCMDOzIicIMzMrcoIwM7MiJwgzMytygjAzsyInCDMzK3KCMDOzIicIMzMrcoIwM7OijiQISf8q6VZJv5Z0nqQtJe0u6VpJf5A0X9LYTsRmZmZJXQlC0ivqmVbnuiYC7wKmRcTewObAccBngdMiYiqwBpjdzPrNzGx41FuD+FKd0+o1BthK0hhga2AVcDBwUZ4/D5i+Ees3M7ONNGawmZJeBrwc6JH03sqs7Um//BsWEX+S9HlgOfBn4MfAdcDaiFiXi60EJg4Q0xxgDsCUKVOaCcHMzOowVA1iLLAtKZFsV3k9CBzdzAYljQOOAnYHJgDbAIcXikZp+YiYGxHTImJaT09PMyGYmVkdBq1BRMRPgJ9IOisi7hymbR4K3BERfQCSvkuqpewoaUyuRUwC7hqm7ZmZWRMGTRAVW0iaC/RWl4mIg5vY5nJgP0lbk5qYDgGWAAtJtZLzgZnAxU2s28zMhkm9CeJC4OvAGcDjG7PBiLhW0kXA9cA64AZgLvBD4HxJn8rTztyY7ZiZ2capN0Gsi4ivDddGI+Ik4KSaybcD+w7XNsza5aCDWrv+hQtbu36zgdR7mesPJL1D0m6Sdup/tTQyMzPrqHprEDPz3w9UpgWwx/CGY2Zm3aKuBBERu7c6EDMz6y51JQhJbyxNj4izhzccMzPrFvU2Mb2kMrwl6dLU6wEnCDOzEareJqZ3Vscl7QCc05KIzMysKzTb3fejwNThDMTMzLpLvecgfsD6vpE2B54LXNCqoMzMrPPqPQfx+crwOuDOiFjZgnjMzKxL1NXElDvt+x2pJ9dxwGOtDMrMzDqv3ifKHQv8CjgGOBa4VlJT3X2bmdmmod4mphOBl0TEagBJPcCVrH8CnJmZjTD1XsW0WX9yyO5rYFkzM9sE1VuDuFzSj4Dz8vg/AZe2JiQzM+sGQz2T+pnArhHxAUn/COwPCPglcG4b4jOzBjXa/bi7E7eBDNVMdDrwEEBEfDci3hsR/0qqPZze6uDMzKxzhkoQvRFxc+3EiFhCevyomZmNUEMliC0HmbfVcAZiZmbdZagE8T+S3lo7UdJs4LrWhGRmZt1gqKuY3gN8T9LxrE8I04CxwGtbGZiZmXXWoAkiIu4BXi7pIGDvPPmHEXF1yyMzM7OOqvd5EAuBYbsYTtKOwBmkpBPAm4HbgPmkk9/LgGMjYs1wbdPMzBrTqbuhvwBcHhHPAV4I/Bb4MHBVREwFrsrjZmbWIW1PEJK2Bw4AzgSIiMciYi1wFDAvF5sHTG93bGZmtl4nahB7AH3ANyXdIOkMSduQ7theBZD/7lJaWNIcSUskLenr62tf1GZmo0wnEsQY4MXA1yLiRcAjNNCcFBFzI2JaREzr6elpVYxmZqNeJxLESmBlRFybxy8iJYx7JO0GkP+uHmB5MzNrg7YniIi4G1gh6dl50iHAb4AFwMw8bSZwcbtjMzOz9ert7nu4vRM4V9JY4HbgTaRkdUG+S3s56el1ZqNeo72zmg2XjiSIiLiRdEd2rUPaHYuZmZX5qXBmZlbkBGFmZkVOEGZmVtSpk9Rm1iX8iFIbiGsQZmZW5ARhZmZFThBmZlbkcxBm1pBmbtzzeYtNk2sQZmZW5ARhZmZFThBmZlbkBGFmZkVOEGZmVuQEYWZmRU4QZmZW5ARhZmZFThBmZlbkBGFmZkVOEGZmVuQEYWZmRR1LEJI2l3SDpEvy+O6SrpX0B0nzJY3tVGxmZtbZGsS7gd9Wxj8LnBYRU4E1wOyORGVmZkCHEoSkScCrgDPyuICDgYtykXnA9E7EZmZmSadqEKcDHwSeyOM7A2sjYl0eXwlM7ERgZmaWtD1BSHo1sDoirqtOLhSNAZafI2mJpCV9fX0tidHMzDpTg3gF8BpJy4DzSU1LpwM7Sup/wt0k4K7SwhExNyKmRcS0np6edsRrZjYqtT1BRMRHImJSRPQCxwFXR8TxwELg6FxsJnBxu2MzM7P1uuk+iA8B75W0lHRO4swOx2NmNqqNGbpI60TEImBRHr4d2LeT8ZiZ2XrdVIMwM7Mu4gRhZmZFThBmZlbkBGFmZkVOEGZmVuQEYWZmRU4QZmZW5ARhZmZFThBmZlbkBGFmZkVOEGZmVuQEYWZmRU4QZmZW5ARhZmZFThBmZlbkBGFmZkVOEGZmVuQEYWZmRU4QZmZW5ARhZmZFThBmZlY0pt0blDQZOBt4OvAEMDciviBpJ2A+0AssA46NiDXtjs/Mht9BBzVWfuHC1sRhjelEDWId8L6IeC6wH3CCpL2ADwNXRcRU4Ko8bmZmHdL2BBERqyLi+jz8EPBbYCJwFDAvF5sHTG93bGZmtl5Hz0FI6gVeBFwL7BoRqyAlEWCXAZaZI2mJpCV9fX3tCtXMbNTpWIKQtC3wHeA9EfFgvctFxNyImBYR03p6eloXoJnZKNeRBCHpaaTkcG5EfDdPvkfSbnn+bsDqTsRmZmZJ2xOEJAFnAr+NiFMrsxYAM/PwTODidsdmZmbrtf0yV+AVwBuAWyTdmKd9FDgFuEDSbGA5cEwHYjMzs6ztCSIifgZogNmHtDMWMzMbWCdqEGZmg2r1jXW+ca8+7mrDzMyKXIMws01eozUCq49rEGZmVuQEYWZmRU4QZmZW5ARhZmZFThBmZlbkBGFmZkVOEGZmVuQEYWZmRU4QZmZW5ARhZmZFThBmZlbkBGFmZkVOEGZmVuQEYWZmRU4QZmZW5ARhZmZFfmCQmdkwa+YBRt34WFPXIMzMrKjrahCSDgO+AGwOnBERp3Q4JDMb5drxSNNGt9GOGkdX1SAkbQ58BTgc2AuYIWmvzkZlZjY6dVWCAPYFlkbE7RHxGHA+cFSHYzIzG5W6rYlpIrCiMr4SeGm1gKQ5wJw8+rCk25rc1njg3iaXbTXH1hzH1rxujs+xFUhDFhkstmfUs41uSxCltxwbjETMBeZu9IakJRExbWPX0wqOrTmOrXndHJ9ja85wxNZtTUwrgcmV8UnAXR2KxcxsVOu2BPE/wFRJu0saCxwHLOhwTGZmo1JXNTFFxDpJ/wL8iHSZ6zci4tYWbW6jm6layLE1x7E1r5vjc2zN2fim+IgYupSZmY063dbEZGZmXcIJwszMikZ8gpB0mKTbJC2V9OHC/C0kzc/zr5XU20WxHSDpeknrJB3drrjqjO29kn4j6WZJV0mq67rqNsX2Nkm3SLpR0s/aeTf+ULFVyh0tKSS17RLJOvbbLEl9eb/dKOkt3RJbLnNs/s7dKunb3RKbpNMq++z3ktZ2UWxTJC2UdEP+Xz2ioQ1ExIh9kU50/xHYAxgL3ATsVVPmHcDX8/BxwPwuiq0XeAFwNnB0l+23g4Ct8/Dbu2y/bV8Zfg1webfElsttB1wDLAamdUtswCzgy+36njUY21TgBmBcHt+lW2KrKf9O0sU1XREb6UT12/PwXsCyRrYx0msQ9XTdcRQwLw9fBBwi1XGPYhtii4hlEXEz8EQb4mk0toUR8WgeXUy6Z6VbYnuwMroNNTdbdjK27JPA54C/tCmuRmLrhHpieyvwlYhYAxARq7sotqoZwHltiay+2ALYPg/vQIP3lY30BFHqumPiQGUiYh3wALBzl8TWKY3GNhu4rKURrVdXbJJOkPRH0oH4Xd0Sm6QXAZMj4pI2xdSv3s/0dbkp4iJJkwvzW6Ge2J4FPEvSzyUtzr0+d0tsAORm1t2Bq9sQF9QX28nA6yWtBC4l1XDqNtITxJBdd9RZphU6td161B2bpNcD04D/bGlElU0Wpj0ltoj4SkTsCXwI+FjLo0oGjU3SZsBpwPvaFE9VPfvtB0BvRLwAuJL1NetWqye2MaRmpgNJv9LPkLRji+OCxv5PjwMuiojHWxhPVT2xzQDOiohJwBHAOfl7WJeRniDq6brjyTKSxpCqYfd3SWydUldskg4FTgReExF/7abYKs4Hprc0ovWGim07YG9gkaRlwH7AgjadqB5yv0XEfZXP8b+Bv2lDXHXFlstcHBH/GxF3ALeREkY3xNbvONrXvAT1xTYbuAAgIn4JbEnqxK8+7TiZ0qkX6VfH7aRqX/9JnOfVlDmBDU9SX9AtsVXKnkV7T1LXs99eRDpBNrULP9OpleEjgSXdEltN+UW07yR1Pfttt8rwa4HFXRTbYcC8PDye1LSyczfElss9G1hGvvm4i/bbZcCsPPxcUgKpO8a2vJFOvkjVqt/ng9mJedonSL96IWXUC4GlwK+APbootpeQfiU8AtwH3NpFsV0J3APcmF8Luii2LwC35rgWDnaQbndsNWXbliDq3G+fyfvtprzfntNFsQk4FfgNcAtwXLfElsdPBk5pV0wN7Le9gJ/nz/RG4O8bWb+72jAzs6KRfg7CzMya5ARhZmZFThBmZlbkBGFmZkVOEGZmVuQEYV0n93J6TmV8TO5ldNi6p5B0RrO9vOZeT788XLEU1t+Texa+QdLfSrq0dNewpJMlvb9VcZh11SNHzbJHgL0lbRURfwb+DvhTIyuQNCZS31pFEdG2rqybcAjwu4iYmcd/2slgbPRyDcK61WXAq/LwBj1kStpX0i/yL+xfSHp2nj5L0oWSfgD8WNJmkr6anx9wSf4lfnQuu6i/iwtJD0v6tKSbckdwu+bpR1Z+yV/ZP30gkraV9M38LIqbJb0uT5+Rp/1a0mcr5Z+yXUn7kDoYPCI/X2ArScskjc/LnJj7/7+SdPdu/7r2lHS5pOsk/VTSc/L0syR9Me+n21V5roikD+a4bpJ0ymDrsVGq3Xf++eXXUC/gYdJzMC4i3el+I6mTtkvy/O2BMXn4UOA7eXgW6c7znfL40aQeLDcDng6sIXdZQuUuZlIHZ0fm4c8BH8vD41j/3Pa3AP+/sp2nPDcB+CxwemV8HDABWA70kGrsVwPTh9juBusndeEwntQ30i3A1nkfLAXen8tcRe5iBHgpcHUePovUU8BmpLtql+bphwO/YP0zPXYabD1+jc6Xm5isK0XEzUpP95tBOshX7QDMkzSVdJB9WmXeFRHR39ni/sCFEfEEcLekhQNs7jGg//zGdaQmLUidn82XtBupr5s7hgj7UFJ/Xv3vYY2kA4BFEdEHIOlc4ADg+4NsdyB/C3wv8nM4JC3If7cFXg5cWHmUyRaV5b6f98FvKrWgQ4Fv9q8rIu6vYz02yjhBWDdbAHyeVHuoPqPjk8DCiHhtTiKLKvMeqQzX++Cn/42I/j5nHmf9/8WXgFMjYoGkA0n97QxG1Ned/FDbHUypb5zNgLURsc8Ay1R72lXlb+26hlqPjTI+B2Hd7BvAJyLilprpO7D+pPWsQZb/GekBOJvlX84HNrj96nZmDlYw+zHwL/0jksYB1wKvlDRe0uakGtFPGoyj3zXAa/N5ie1IPdUS6Ql6d0g6Jm9Xkl5YR6xvlrR1XmanJtdjI5gThHWtiFgZEV8ozPoc8BlJPyc9l3cg3yGdk/g18F+kg/UDDYRwMqm55afAvXWU/xQwLp+Mvgk4KCJWAR8h9Y56E3B9RFzcQAxPiojrgfmkczLfYcOrm44HZuft3soQjxONiMtJNbQlkm4E+i+XbWg9NrK5N1cb0SRtGxEPS9qZ1J37KyLi7k7HZbYp8DkIG+kuyTeZjQU+6eRgVj/XIMzMrMjnIMzMrMgJwszMipwgzMysyAnCzMyKnCDMzKzo/wBoRBisCIHI/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "marginal_probs = list(map(lambda p : p[0], margin))\n",
    "n, bins, patches = plt.hist(marginal_probs, 25, facecolor='blue', alpha=0.75)\n",
    "plt.title('Marginal confidence histogram - All data')\n",
    "plt.ylabel('Count')\n",
    "plt.xlabel('Marginal confidence')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "c3e7841f-d705-d9bc-3dc9-e2117cea3b26"
   },
   "source": [
    "Lets break it down by positive and negative sentiment to see if one is harder than the other.\n",
    "\n",
    "### Positive data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "_cell_guid": "1709f0eb-febe-0161-f1df-ac9585ff580d"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHTZJREFUeJzt3Xm8HFWd9/HPl8SwL4EEZDUsEWFcwImICkoAHZYHiCMwIDMmMyzj6Kg8IyAKM4PLI8jwAPOo6IOgBGQHgYCKsiTiBhh2ERgQwiKBBEiEgCMEf/PHOZdU2r7dfW9udd97z/f9et3XreVU1a+rq+tXp073KUUEZmZWrpV6HYCZmfWWE4GZWeGcCMzMCudEYGZWOCcCM7PCORGYmRXOiWAYkvQ5SWcN0bpC0lZDsa4BbPNLkp6R9JSkzSQtkTSmn7InSPpuN+NrRdIMST9rMf+HkqZ3M6YSSNpZ0gMt5rc8joYwjkn5MzO2zu0MN04ELUiaJ+llSRMapt+ZD5ZJdWw3Ir4cEYfVse66SdoU+DSwbUS8PiIei4g1IuLVXsc2FCJiz4iY2a5cLxJwN+XX92I+Of9O0qkrcpKOiJ9GxNaV9c+TtHtl/rA7jiTtIumJXscxFJwI2nsEOLhvRNJbgFUHu7ICrjTeADwbEQt6HchoNYyOobdFxBrAbsCHgcN7HI8NkhNBe+cBH6mMTwfOrRaQtLekOyQ9L+lxSSdU5vVVNQ+V9BhwY57+EUmPSnpW0r9Wr4Cqt0sqy0+X9Fi+5XJcZf07SPqlpMWS5kv6mqRxnbwwSetK+o6kJyUtknRlZd7hkh6S9JykWZI2qswLSR+V9GBe7utKdgeuAzbKV4rnNFa1JW0u6SeSXpB0HdBY29pR0i/y67lL0i6VeXMkfVHSz/PyP67W1iTtVFn2cUkz8vSVJZ2S99/Tkr4pqWUyz+UXSXpE0p4NMRyWh7fKr+X3+X25OE+/KRe/K++Hv+lgn35A0gN5XWfk9fZtZ0Z+zadJeg44QdKWkm7Mx88zks6XtE5lffMkHS3pbqUr97MlbaB0a+sFSddLGt/yAOlQRNwP/BR4c972Nnk/LZZ0r6R9K3HtJek3OYbfSToqT3/t6lrSecBmwNV5/x1TPY4kHSRpbsP79b8lzcrDHb/fksbkss9IehjYu2H+30u6L8f7sKR/zNNXB37IsmN9iaSNtAKfx56KCP/18wfMA3YHHgC2AcYAj5OuegOYlMvtAryFlFjfCjwNTMvzJuWy5wKrk2oT2wJLgJ2AccApwCvA7nmZE4DvNiz/rbzs24A/Atvk+X8J7AiMzWXvA46svIYAturn9X0fuBgYD7wOeF+evivwDPB2YGXgq8BNDeu8BliH9IFdCOxR2RdPVMr2xT82j/8SODWv973AC5XXujHwLLBX3pfvz+MT8/w5wG+BN+Z9MQc4Kc/bLK/r4Pxa1gO2y/NOB2YB6wJrAlcDJ/azT2bk9+Lw/H7/E/AkoEoMh+XhC4HjcqyrADv1t99b7VNSMnwe+Ov8Pn4qx3BYJaalwCfy/FWBrfL+WRmYCNwEnN5w7N4MbJD36wLgdmD7vMyNwL+vwGfjtddHOp6fAg7N+/4h4HOkY3vX/L5sncvOB3bOw+OBt/dz3Mwjfx4ajyNgtbzOyZX5vwIOGsT7/VHgfmDTXH42yx+vewNbAgLeB7zUX8ydfB6H61/PAxjOfyxLBMcDJwJ7kK54x1JJBE2WOx04LQ/3HcBbVOb/G3BhZXw14GVaJ4JNKuVv7Tvom2z7SOCKynjTRABsCPwJGN9k3tnAyZXxNUgnpkmVdVZPepcAx+bhxg909QO8GemEtnpl/gWV1/oZ4LyGWH4ETM/Dc4DjK/M+Blybhz9bfd2VMgJeBLasTHsX8Eg/+28G8FDDexPA6ysx9J2gzwXOrL43/e33VvuUVOP8ZUPMj7N8IniszbE6Dbij4dg9pDJ+OfCNyvgngCtX4LMRpOS1iJScv0RKiDuTksJKlbIXAifk4ceAfwTWalhf43Ezj34SQR7/LvBveXgyKTGsNoj3+0bgo5XxD1S306T8lcCnmsXcyedxuP751lBnziPdA51Bw20hAEnvlDRb0kJJvyddZUxoKPZ4ZXij6nhEvES68m3lqcrwS6QTCZLeKOkapW/oPA98ucm2m9kUeC4iFjWZtxHwaCW+JTm+jdvF08ZGwKKIeLEy7dHK8BuAA3K1erGkxaRa04YdbHdT0gmp0UTSCeK2yjqvzdP789o28nsDzV/fMaQTz635Fsg/tFhnq33aeDwE0NgIWT1+kLS+pIvy7ZXnSSfGxvf96crwH5qMN33P8u2jvtsdh7R4TW+PiPERsWVEHB8Rf+p7LXm4z6MsO3Y+RKrxPZpvf72rxfpbuYBlbXcfJiW1lxj4+73cvmf54xFJe0q6Od/OW5xj7/fztQKfx55yIuhARDxKajTeC/hekyIXkKqim0bE2sA3SSeI5VZTGZ4PbNI3ku9frjfI8L5BqtpOjoi1SFXyxm038ziwbvW+csWTpJNyX3yr5/h+N8gY+8wHxuf19dmsIabzImKdyt/qEXFSB+t+nFSFb/QM6aT3F5V1rh2pkXOFRMRTEXF4RGxEuso9Q/1/U6jVPm08HlQd79tcw/iJedpb8/v+t3T2vrcV6ZtRa+S/8we4+JPAppKq55bNyMdORPwqIvYD1iddXV/SXxhttvNjYIKk7UgJ4YI8faDv93zSRUQ1ViC1NZBqUqcAG0TEOsAPWLafm8U42M9jTzkRdO5QYNeGq9k+a5Kurv9b0g6kK5RWLgP2kfTu3JD0eQZ/sKxJqqIvkfQm0j3ttiJiPqmx6wxJ4yW9TtJ78+wLgL+XtF3+MHwZuCUi5g0yxr5tPgrMBT4vaZyknYB9KkW+S9ovf5Ub8VbJjYiNJ8Vmzgd2l3RgblBcT9J2+cr0W8BpktYHkLSxpL9akdeS13NAJbZFpBND39cbnwa2qBRvtU+/D7xF0jSlRvWPA69vs/k1Se1MiyVtDBy9oq9niNxCujVzTD6mdiG9xxfl9/wQSWtHxCuk47a/r4M27r/lRMRS0ufoP0j39q/L0wf6fl8CfFLSJkqN58dW5o0jtacsBJYqfWngAw0xridp7cq0QX0ee82JoEMR8duImNvP7I8BX5D0Aun+f39XOX3rupd0j/Yi0hXJC6TGvD8OIrSjSInnBdIH4OIBLPt3pPvU9+ftH5njuwH4V9LV0HzSlfZBg4itmQ8D7wSeA/6dyq22iHgc2I90FbWQdJV/NB0cpxHxGKnG9um87jtJDeuQ2h4eAm7O1fXrga2brWeA3gHcImkJqUb4qYh4JM87AZiZb08c2GqfRsQzwAHAyaTbRduSEmar4+HzpIbn35MSSbOaatdFxMvAvsCepKvzM4CPRPpmEaRjbl5+Hz5Kqsk0cyJwfN5/R/VT5gJSG96lOTH0Gcj7/S1SO9RdpMb01/ZjRLwAfJL0eV5EOnZnVebfT2r/eDjHuREr9nnsmb5vQlgPSVoDWEyqTj7SrryNbvm2yhOkxt7ZvY7HRj/XCHpE0j6SVsv3ik8B7iF9U8IKlG+HrZNvG/XdV765x2FZIZwIemc/UsPak6Svvx0Urp6V7F2kbz09Q7qnPi0i/tDbkKwUvjVkZlY41wjMzAo3XDqvamnChAkxadKkXodhZjai3Hbbbc9ERKsfTwIjJBFMmjSJuXP7++ammZk1I+nR9qV8a8jMrHhOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgnAjOzwjkRmJkVzonAzKxwI+KXxaPN1JlTB1R+9nR3SW9m9ak1EUiaR3pSz6vA0oiYImld0lN7JpH63z+wnweom5lZF3Tj1tDUiNguIqbk8WOBGyJiMnADyz8j1MzMuqwXbQT7ATPz8ExgWg9iMDOzrO5EEMCPJd0m6Yg8bYOImA+Q/6/fbEFJR0iaK2nuwoULaw7TzKxcdTcWvycinpS0PnCdpPs7XTAizgTOBJgyZYofo2ZmVpNaawQR8WT+vwC4AtgBeFrShgD5/4I6YzAzs9ZqSwSSVpe0Zt8w8AHg18AsYHouNh24qq4YzMysvTpvDW0AXCGpbzsXRMS1kn4FXCLpUOAx4IAaYzAzszZqSwQR8TDwtibTnwV2q2u7ZmY2MO5iwsyscE4EZmaFcyIwMyucE4GZWeGcCMzMCudEYGZWOCcCM7PCORGYmRXOicDMrHBOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgnAjOzwjkRmJkVzonAzKxwTgRmZoVzIjAzK5wTgZlZ4ZwIzMwK50RgZlY4JwIzs8I5EZiZFc6JwMyscE4EZmaFcyIwMyucE4GZWeGcCMzMCudEYGZWOCcCM7PC1Z4IJI2RdIeka/L45pJukfSgpIsljas7BjMz6183agSfAu6rjH8FOC0iJgOLgEO7EIOZmfWj1kQgaRNgb+CsPC5gV+CyXGQmMK3OGMzMrLWxNa//dOAYYM08vh6wOCKW5vEngI2bLSjpCOAIgM0226zmMJeZOnPqgMrPnj67pkiWGWhMA9WN12Bmw1dtNQJJ/wtYEBG3VSc3KRrNlo+IMyNiSkRMmThxYi0xmplZvTWC9wD7StoLWAVYi1RDWEfS2Fwr2AR4ssYYzMysjdpqBBHx2YjYJCImAQcBN0bEIcBsYP9cbDpwVV0xmJlZe3W3ETTzGeAiSV8C7gDOrnNjdd9fNzMb6bqSCCJiDjAnDz8M7NCN7ZqZWXv+ZbGZWeGcCMzMCudEYGZWOCcCM7PCORGYmRXOicDMrHBOBGZmhevFD8pGlRJ/sNaN1+yO8My6xzUCM7PCORGYmRXOicDMrHBuI7Bh2c4xHB8QZDZauUZgZlY4JwIzs8I5EZiZFc6JwMyscE4EZmaFcyIwMyucE4GZWeGcCMzMCudEYGZWOCcCM7PCORGYmRXOicDMrHBOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgnAjOzwjkRmJkVzonAzKxwtSUCSatIulXSXZLulfT5PH1zSbdIelDSxZLG1RWDmZm1V2eN4I/ArhHxNmA7YA9JOwJfAU6LiMnAIuDQGmMwM7M2aksEkSzJo6/LfwHsClyWp88EptUVg5mZtVdrG4GkMZLuBBYA1wG/BRZHxNJc5Alg436WPULSXElzFy5cWGeYZmZF6ygRSHpPJ9MaRcSrEbEdsAmwA7BNs2L9LHtmREyJiCkTJ07sJEwzMxuETmsEX+1wWlMRsRiYA+wIrCNpbJ61CfBkp+sxM7OhN7bVTEnvAt4NTJT0L5VZawFj2iw7EXglIhZLWhXYndRQPBvYH7gImA5cNfjwzcxsRbVMBMA4YI1cbs3K9OdJJ/NWNgRmShpDqnlcEhHXSPoNcJGkLwF3AGcPKnIzMxsSLRNBRPwE+ImkcyLi0YGsOCLuBrZvMv1hUnuBmZkNA+1qBH1WlnQmMKm6TETsWkdQZmbWPZ0mgkuBbwJnAa/WF46ZmXVbp4lgaUR8o9ZIzMysJzr9+ujVkj4maUNJ6/b91RqZmZl1Rac1gun5/9GVaQFsMbThmJlZt3WUCCJi87oDMTOz3ugoEUj6SLPpEXHu0IZjZmbd1umtoXdUhlcBdgNuB5wIzMxGuE5vDX2iOi5pbeC8WiIyM7OuGmw31C8Bk4cyEDMz641O2wiuZll30WNI3UlfUldQZmbWPZ22EZxSGV4KPBoRT9QQj5mZdVlHt4Zy53P3k3ogHQ+8XGdQZmbWPZ0+oexA4FbgAOBA4BZJ7bqhNjOzEaDTW0PHAe+IiAXw2kNnrmfZQ+jNzGyE6vRbQyv1JYHs2QEsa2Zmw1inNYJrJf0IuDCP/w3wg3pCMjOzbmr3zOKtgA0i4mhJfw3sBAj4JXB+F+IzM7OatasRnA58DiAivgd8D0DSlDxvn1qjMxsmps6cOqDys6fPrikSs6HX7j7/pPzs4eVExFzSYyvNzGyEa5cIVmkxb9WhDMTMzHqjXSL4laTDGydKOhS4rZ6QzMysm9q1ERwJXCHpEJad+KcA44AP1hmY2UD4Hr7Z4LVMBBHxNPBuSVOBN+fJ34+IG2uPzMzMuqLT5xHMBnwJZWY2CvnXwWZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgnAjOzwtWWCCRtKmm2pPsk3SvpU3n6upKuk/Rg/j++rhjMzKy9OmsES4FPR8Q2wI7AxyVtCxwL3BARk4Eb8riZmfVIbYkgIuZHxO15+AXgPmBjYD9gZi42E5hWVwxmZtZeV9oIJE0CtgduIT3oZj6kZAGs340YzMysudoTgaQ1gMuBIyPi+QEsd4SkuZLmLly4sL4AzcwKV2sikPQ6UhI4Pz/hDOBpSRvm+RsCC5otGxFnRsSUiJgyceLEOsM0Mytand8aEnA2cF9EnFqZNQuYnoenA1fVFYOZmbXXUe+jg/Qe4O+AeyTdmad9DjgJuCQ/3OYx4IAaYzAzszZqSwQR8TNA/czera7tmpnZwPiXxWZmhXMiMDMrnBOBmVnh6mwsNivW1JlTB1R+9nQ/CdZ6xzUCM7PCORGYmRXOicDMrHBuI7AiDfQevtlo5hqBmVnhnAjMzArnRGBmVjgnAjOzwjkRmJkVzonAzKxwTgRmZoVzIjAzK5wTgZlZ4ZwIzMwK50RgZlY49zVkZk35mQrlcI3AzKxwTgRmZoVzIjAzK5wTgZlZ4ZwIzMwK50RgZlY4JwIzs8I5EZiZFc6JwMyscE4EZmaFcyIwMyuc+xoyGwYG2q8PuG8fGzq11QgkfVvSAkm/rkxbV9J1kh7M/8fXtX0zM+tMnbeGzgH2aJh2LHBDREwGbsjjZmbWQ7Ulgoi4CXiuYfJ+wMw8PBOYVtf2zcysM91uLN4gIuYD5P/r91dQ0hGS5kqau3Dhwq4FaGZWmmH7raGIODMipkTElIkTJ/Y6HDOzUavbieBpSRsC5P8Lurx9MzNr0O1EMAuYnoenA1d1eftmZtagzq+PXgj8Etha0hOSDgVOAt4v6UHg/XnczMx6qLYflEXEwf3M2q2ubZqZ2cAN28ZiMzPrDicCM7PCORGYmRXOicDMrHBOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgnAjOzwjkRmJkVzonAzKxwTgRmZoVzIjAzK1xt3VCbWb2mzpw6oPKzp8+uKRIb6VwjMDMrnBOBmVnhnAjMzArnRGBmVjg3FpsVYqCNy1YO1wjMzArnRGBmVjgnAjOzwrmNwMx6YjBtFv5RXD1cIzAzK5wTgZlZ4ZwIzMwK5zYCMxsS3fidgjvaq4drBGZmhXMiMDMrnBOBmVnh3EZgZqPWcGtTGG7x9OlJjUDSHpIekPSQpGN7EYOZmSVdTwSSxgBfB/YEtgUOlrRtt+MwM7OkFzWCHYCHIuLhiHgZuAjYrwdxmJkZvWkj2Bh4vDL+BPDOxkKSjgCOyKNLJD0wyO1NAJ4Z5LK9MJLiHUmxwsiKdyTFCqMkXs1QD0LpX45nRfbtGzop1ItE0GxPx59NiDgTOHOFNybNjYgpK7qebhlJ8Y6kWGFkxTuSYgXHW6duxNqLW0NPAJtWxjcBnuxBHGZmRm8Swa+AyZI2lzQOOAiY1YM4zMyMHtwaioilkv4Z+BEwBvh2RNxb4yZX+PZSl42keEdSrDCy4h1JsYLjrVPtsSriz27Pm5lZQdzFhJlZ4ZwIzMwKN2oSQbtuKyStLOniPP8WSZO6H+VrsbSL9b2Sbpe0VNL+vYixIZ528f6LpN9IulvSDZI6+u5yXTqI96OS7pF0p6Sf9fKX7Z12tyJpf0khqadfeexg386QtDDv2zslHdaLOHMsbfetpAPzsXuvpAu6HWNDLO327WmV/fpfkhYP2cYjYsT/kRqdfwtsAYwD7gK2bSjzMeCbefgg4OJhHOsk4K3AucD+I2DfTgVWy8P/1Kt9O4B416oM7wtcO1xjzeXWBG4CbgamDPN9OwP4Wq9iHGCsk4E7gPF5fP3hHG9D+U+QvmgzJNsfLTWCTrqt2A+YmYcvA3aT1IufEbaNNSLmRcTdwJ96EF+jTuKdHREv5dGbSb8N6ZVO4n2+Mro6TX7Q2CWddrfyReBk4L+7GVwTI6l7mE5iPRz4ekQsAoiIBV2OsWqg+/Zg4MKh2vhoSQTNuq3YuL8yEbEU+D2wXlei6yeOrFmsw8lA4z0U+GGtEbXWUbySPi7pt6QT7Ce7FFujtrFK2h7YNCKu6WZg/ej0WPhQvk14maRNm8zvhk5ifSPwRkk/l3SzpD26Ft2f6/hzlm+9bg7cOFQbHy2JoJNuKzrq2qILhkscneo4Xkl/C0wB/qPWiFrrtAuTr0fElsBngONrj6q5lrFKWgk4Dfh01yJqrZN9ezUwKSLeClzPslp4t3US61jS7aFdSFfYZ0lap+a4+jOQ88JBwGUR8epQbXy0JIJOuq14rYykscDawHNdia6fOLLh3sVGR/FK2h04Dtg3Iv7YpdiaGej+vQiYVmtE/WsX65rAm4E5kuYBOwKzethg3HbfRsSzlff/W8Bfdim2Rp2eE66KiFci4hHgAVJi6IWBHLcHMYS3hYBR01g8FniYVF3qa2j5i4YyH2f5xuJLhmuslbLn0PvG4k727fakhq7JI+RYmFwZ3geYO1xjbSg/h942FneybzesDH8QuHkYx7oHMDMPTyDdmllvuMaby20NzCP/GHjItt+rg6qGHbkX8F/5hHRcnvYF0hUqwCrApcBDwK3AFsM41neQrhBeBJ4F7h3m+/Z64Gngzvw3a5jH+5/AvTnW2a1Ovr2OtaFsTxNBh/v2xLxv78r79k3DOFYBpwK/Ae4BDhrO+zaPnwCcNNTbdhcTZmaFGy1tBGZmNkhOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGA9kXvSPK8yPjb3WjlkXSlIOmuwPYvmXjS/NlSxNFn/xNwL7h2Sdpb0g2a/apV0gqSj6orDDHrwqEqz7EXgzZJWjYg/AO8HfjeQFUgaG6nfqKYiomddIHdgN+D+iJiex3/ay2CsbK4RWC/9ENg7Dy/Xm6KkHST9Il8x/0LS1nn6DEmXSroa+LGklSSdkfuTvyZfWe+fy87p645B0hJJ/0fSXbmDsQ3y9H0qV+bX903vj6Q1JH0nP8/gbkkfytMPztN+LekrlfJ/tl1J25E6u9sr9y2/qqR5kibkZY7L/dJfT/olad+6tpR0raTbJP1U0pvy9HMk/b+8nx5W5RkWko7Jcd0l6aRW67GC9fKXdP4r9w9YQnrmwmWkX33fSer865o8fy1gbB7eHbg8D88g/ep63Ty+P/AD0kXN64FF5G45qPwSl9SB1z55+GTg+Dw8nmXP7j4M+L+V7fxZv/rAV4DTK+PjgY2Ax4CJpFr2jcC0Nttdbv2kbgMmkPrmuQdYLe+Dh4CjcpkbyN1jAO8EbszD55B+Nb8SsC2pO2OAPYFfsOxZEeu2Wo//yv3zrSHrmYi4W+lJcQeTTuZVawMzJU0mnUxfV5l3XUT0dRi4E3BpRPwJeErS7H429zLQ1/5wG+lWFKTOvS6WtCGpj5dH2oS9O6mvqr7XsEjSe4E5EbEQQNL5wHuBK1tstz87A1dEfr6DpFn5/xrAu4FLK4/RWLmy3JV5H/ymUqvZHfhO37oi4rkO1mMFciKwXpsFnEKqDVSfD/FFYHZEfDAnizmVeS9Whjt9uNArEdHXn8qrLDv2vwqcGhGzJO1C6sulFdFZF+fttttKs35fVgIWR8R2/SxT7fFVlf+N62q3HiuQ2wis174NfCEi7mmYvjbLGo9ntFj+Z6QHoayUr4R3GeD2q9uZ3qpg9mPgn/tGJI0HbgHeJ2mCpDGkGs5PBhhHn5uAD+Z2gzVJvaMS6alqj0g6IG9Xkt7WQaz/IGm1vMy6g1yPjXJOBNZTEfFERPxnk1knAydK+jnpea79uZzUZvBr4P+TTsq/H0AIJ5Buk/wUeKaD8l8CxudG4buAqRExH/gsqbfNu4DbI+KqAcTwmoi4HbiY1GZyOct/m+gQ4NC83Xtp85jIiLiWVOOaK+lOoO9rqANaj41+7n3URjxJa0TEEknrkboYf09EPNXruMxGCrcR2GhwTf4x1jjgi04CZgPjGoGZWeHcRmBmVjgnAjOzwjkRmJkVzonAzKxwTgRmZoX7H2gT9cHSTC4MAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "positive_test_data = list(filter(lambda row : row[0]==2, hstack((targets_test[:,None], data_test)).toarray()))\n",
    "positive_probs = clf.predict_proba(list(map(lambda r : r[1:], positive_test_data)))\n",
    "marginal_positive_probs = list(map(lambda p : marginal(p), positive_probs))\n",
    "n, bins, patches = plt.hist(marginal_positive_probs, 25, facecolor='green', alpha=0.75)\n",
    "plt.title('Marginal confidence histogram - Positive data')\n",
    "plt.ylabel('Count')\n",
    "plt.xlabel('Marginal confidence')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "98258e2a-fbd7-9e18-b43c-1f2330124e67"
   },
   "source": [
    "### Neutral data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "_cell_guid": "eea28df4-f4a2-0b9b-2547-173935fe551c"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHfdJREFUeJzt3XmYXFWd//H3B8JOhARaDEuMAi6MS3BaREUEjA7gwwAziDAuZAaM+8iIjCio4Ir+UHBcJ4omIDu4RESUJRFRgUkwASIqCGENIUACCSga+P7+OKflpqiuutXpW9Xd9/N6nnr67udbp27fb917bp2riMDMzOprvV4HYGZmveVEYGZWc04EZmY150RgZlZzTgRmZjXnRGBmVnNOBCOUpI9K+vYwbSsk7TQc2+qgzE9LekDSfZImS1otaf1Blj1R0ve6GV8rkqZLurrF/J9KOqKbMdnQSFoiaVrJZVt+7mOZE0EbeUf6q6StG6YvzAfYKVWUGxGfjYijqth21STtABwD7BIRz4qIOyNi84h4otexDYeI2C8iZrdbrhcJuJvy+7tR0nqFaZ+WNGsYtj0lb3/cum6rCiPty8u6ciIo53bg8IERSS8GNhnqxkbqzj2Mng08GBH39zqQsWoE7UPbAof1ouARVAejnhNBOWcCby+MHwGcUVxA0hsl/VbSI5LuknRiYd7At5sjJd0JXJmnv13SHZIelPSx4mls8RtHYf0jJN2ZL7kcX9j+bpJ+I2mlpKWSvippwzJvTNJESd+VdK+kFZJ+WJj3Dkm3SnpI0hxJ2xbmhaR3Sbolr/c1JdOAy4Bt8+WgWY3f7iQ9R9IvJK2SdBnQeLa1u6Rf5/ezSNJehXnzJH1K0q/y+j8vnq1J2qOw7l2SpufpG0k6JdffMknflNQymeflV0i6XdJ+DTEclYd3yu/l4fy5nJenX5UXX5Tr4c0l6vQNkv6Qt/X1vN2Bcqbn93yqpIeAEyXtKOnKvP88IOksSVsWtrdE0rGSbpD0qKTTJW2jdGlrlaTLJU1ouYO09wXgJA1yUG7zWa512UZrf8seqL+Vuf5eOZQ6aEXSVvkzeETSdcCODfO/nPehRyQtkPSaPH1f4KPAm3Nsi/L0f5d0c67b2yS9s0wcI0JE+NXiBSwBpgF/AF4IrA/cRfrWG8CUvNxewItJyfUlwDLgoDxvSl72DGAz0tnELsBqYA9gQ+AU4G/AtLzOicD3Gtb/Vl73pcDjwAvz/H8EdgfG5WVvBo4uvIcAdhrk/f0EOA+YAGwAvDZP3wd4AHgZsBHwFeCqhm1eDGwJTAaWA/sW6uLuwrID8Y/L478BvpS3uyewqvBetwMeBPbPdfn6PN6X588D/gQ8L9fFPODkPG9y3tbh+b1sBUzN804D5gATgfHAj4HPDVIn0/Nn8Y78eb8buBdQIYaj8vA5wPE51o2BPQar91Z1SkqGjwD/kj/HD+QYjirEtAZ4f56/CbBTrp+NgD7SwfO0hn33GmCbXK/3A9cDu+Z1rgQ+sQ7/GwHsDCwoxPlpYFbJz3IJeX9vsc+Pa/hchlIH0waJ/1zgfNL/5IuAe4CrC/PfmvehcaRLnfcBGzfGWlj+jaRkIuC1wGPAy3p9DCv1WfY6gJH+4qlEcALwOWBf0jfecRQSQZP1TgNOzcMDO/VzC/M/DpxTGN8U+CutE8H2heWvAw4bpOyjgR8UxpsmAmAS8CQwocm804EvFMY3Jx2YphS2WTzonQ8cl4f3YpBEQDpYrwE2K8w/u/BePwyc2RDLz4Aj8vA84ITCvPcAl+bhjxTfd2EZAY8COxamvRK4fZD6mw7c2vDZBPCsQgwDB74zgJnFz2awem9Vp6Qzzt80xHwXayeCO9vsqwcBv23Yd99SGL8I+EZh/P3AD9fhfyNIB+L9gTtJB+NiImj3WS6h80QwlDp4WiIgJfi/AS8oTPsshUTQZJ0VwEsbY22x/A+BDwy1frv58qWh8s4E/o20M57ROFPSKyTNlbRc0sPAu2i45EH6xx6wbXE8Ih4jfVtq5b7C8GOkAwmSnifpYqU7dB4h7dCNZTezA/BQRKxoMm9b4I5CfKtzfNu1i6eNbYEVEfFoYdodheFnA2/KlxJWSlpJOmuaVKLcHUhnC436SAfzBYVtXpqnD+bvZeTPBpq/v/8mHbSvk7RY0n+02GarOm3cHwK4u2H94v6DpGdKOlfSPflz/x5P/9yXFYb/3GS86WeWLx+tzq+3tHhPRMQlpEQwo2FWmc+yU0Opg2b6SF9Mitsr7odIOiZf6nk4x75Fq21L2k/SNfmy30pSgiwTS885EZQUEXeQGo33B77fZJGzSZcedoiILYBvkg4Qa22mMLwU2H5gJF+v3mqI4X0D+D2wc0Q8g3T9srHsZu4CJg5yTfVe0j/yQHyb5fjuGWKMA5YCE/L2BkxuiOnMiNiy8NosIk4use27aLjOmz1AOuj9Q2GbW0REmcTVUkTcFxHviIhtgXcCX9fgdwq1qtPG/UHF8YHiGsY/l6e9JH/ub6Xc595WpDujNs+vs0qscgLpEtmmhWntPstHG5Z/VjGEwUJrGB9qHSwnnZnuUJj29/0wtwd8GDiUdMa8JfBwYdtrxSFpI9IZ1ynANnn5S0rG0nNOBJ05Etin4dvsgPGkb9d/kbQb6eyhlQuBAyS9Sqlh9ySGvtOMJ11fXi3pBaRr2m1FxFLgp6SD1wRJG0jaM88+G/h3SVPzTv5Z4NqIWDLEGAfKvAOYT2pg3FDSHsABhUW+R6qXf5K0vqSNJe0lqfGg2MxZwDRJh0oalxsDp0bEk6T2lVMlPRNA0naS/mld3kvezpsKsa0gHSAGbpNdBjy3sHirOv0J8GJJB+WG1/ey9oGxmfGkdqaVkrYDjl3X9zNUETEPuJF0I8WAdp/lQuCwvN/1A4cU1l1OumxZrL9mhlQHkW5l/j6pwXlTSbs0xD6elCiWA+MkfRx4RmH+MmCKnrp1dkPSpbHlwBqlmwveUCaWkcCJoAMR8aeImD/I7PcAn5S0inT9//w221pMukZ7Lunb4CpSY97jQwjtQ6TEs4p0wDuvg3XfRrpW+vtc/tE5viuAj5G+5SwlfdMertsE/w14BfAQ8AkKl9oi4i7gQNJZzXLSt8pjKbGvRsSdpDO2Y/K2F5Ia1iF9u7sVuCZfQrgceP4wvJeXA9dKWk06I/xARNye550IzM6XRQ5tVacR8QDwJtJdOA+SbiaYT+v94SRSw/PDpETS7Ey1m04gNcYDpT7Lj5HqYAXpvZxdWPcx4DPAr3L97T5ImetSB+8jXRq7D5gFfLcw72ekL0l/JF0y+gtrX0a6IP99UNL1EbEK+E/S//0K0j4+p4NYemrgLgjrMUmbAytJl3dub7e8jW35m+bdpMbeub2Ox8Y2nxH0kKQD8mnpZqRrizeS7nKwGsqXULbMl40G2nmu6XFYVgNOBL11IKkB8V7S/diHhU/R6uyVpLueHiC1mxwUEX/ubUhWB740ZGZWcz4jMDOruVHRadPWW28dU6ZM6XUYZmajyoIFCx6IiFY/nARGSSKYMmUK8+cPdtemmZk1I+mO9kv50pCZWe05EZiZ1ZwTgZlZzTkRmJnVnBOBmVnNORGYmdWcE4GZWc05EZiZ1ZwTgZlZzY2KXxZbZ/beu7Pl57q3e7Na8xmBmVnNVZYI8vNJr5O0SNJiSSfl6bMk3S5pYX5NrSoGMzNrr8pLQ4+THvS+WtIGwNWSfprnHRsRF1ZYtpmZlVRZIshP2lqdRzfILz8Fx8xshKm0sVjS+sACYCfgaxFxraR3A5+R9HHgCuC4iHi8ybozgBkAkydPrjJM61CnjdHgBmmzkazSxuKIeCIipgLbA7tJehHwEeAFwMuBicCHB1l3ZkT0R0R/X1/b5yqYmdkQdeWuoYhYCcwD9o2IpZE8DnwX2K0bMZiZWXNV3jXUJ2nLPLwJMA34vaRJeZqAg4CbqorBzMzaq7KNYBIwO7cTrAecHxEXS7pSUh8gYCHwrgpjMDOzNqq8a+gGYNcm0/epqkwzM+ucf1lsZlZzTgRmZjXnRGBmVnNOBGZmNedEYGZWc04EZmY1N+YfTOOHtLQ3lL6DzGzs8BmBmVnNORGYmdWcE4GZWc05EZiZ1ZwTgZlZzY35u4Y65advmVnd+IzAzKzmnAjMzGrOicDMrOacCMzMas6NxTYmuCsRs6HzGYGZWc05EZiZ1VxliUDSxpKuk7RI0mJJJ+Xpz5F0raRbJJ0nacOqYjAzs/aqPCN4HNgnIl4KTAX2lbQ78Hng1IjYGVgBHFlhDGZm1kZliSCS1Xl0g/wKYB/gwjx9NnBQVTGYmVl7lbYRSFpf0kLgfuAy4E/AyohYkxe5G9hukHVnSJovaf7y5curDNPMrNYqTQQR8URETAW2B3YDXthssUHWnRkR/RHR39fXV2WYZma11pW7hiJiJTAP2B3YUtLA7xe2B+7tRgxmZtZclXcN9UnaMg9vAkwDbgbmAofkxY4AflRVDGZm1l6VvyyeBMyWtD4p4ZwfERdL+h1wrqRPA78FTq8wBjMza6OyRBARNwC7Npl+G6m9wMzMRgD3NTQKDOVhOWZmZbmLCTOzmnMiMDOrOScCM7OacyIwM6s5JwIzs5pzIjAzqzknAjOzmnMiMDOrOScCM7OacyIwM6s5dzExDDrtAmLu3GriMDMbCp8RmJnVnBOBmVnNORGYmdWcE4GZWc05EZiZ1ZwTgZlZzTkRmJnVnBOBmVnNVZYIJO0gaa6kmyUtlvSBPP1ESfdIWphf+1cVg5mZtVflL4vXAMdExPWSxgMLJF2W550aEadUWLaZmZVUWSKIiKXA0jy8StLNwHZVlWdmZkPTlTYCSVOAXYFr86T3SbpB0nckTRhknRmS5kuav3z58m6EaWZWS5UnAkmbAxcBR0fEI8A3gB2BqaQzhi82Wy8iZkZEf0T09/X1VR2mmVltVZoIJG1ASgJnRcT3ASJiWUQ8ERFPAt8CdqsyBjMza63Ku4YEnA7cHBFfKkyfVFjsYOCmqmIwM7P2qrxr6NXA24AbJS3M0z4KHC5pKhDAEuCdFcZgZmZtVHnX0NWAmsy6pKoyzcysc35CmXWFn+I2/FynNlzcxYSZWc05EZiZ1ZwTgZlZzTkRmJnVnBuLbUTqtCHUzIbOZwRmZjXnRGBmVnNOBGZmNedEYGZWc04EZmY150RgZlZzTgRmZjXnRGBmVnNOBGZmNedEYGZWc04EZmY1V6qvIUmvjohftZtm5bgfHTMbScqeEXyl5DQzMxtlWp4RSHol8CqgT9IHC7OeAaxfZWBmZtYd7c4INgQ2JyWM8YXXI8AhrVaUtIOkuZJulrRY0gfy9ImSLpN0S/47Yd3fhpmZDVXLM4KI+AXwC0mzIuKODre9BjgmIq6XNB5YIOkyYDpwRUScLOk44Djgw0OI3czMhkHZB9NsJGkmMKW4TkTsM9gKEbEUWJqHV0m6GdgOOBDYKy82G5iHE4GZWc+UTQQXAN8Evg080WkhkqYAuwLXAtvkJEFELJX0zEHWmQHMAJg8eXKnRZoNq07v9Jo7t5o4zKpQNhGsiYhvDKUASZsDFwFHR8QjkkqtFxEzgZkA/f39MZSyzcysvbK3j/5Y0nskTcqNvRMlTWy3kqQNSEngrIj4fp68TNKkPH8ScP+QIjczs2FR9ozgiPz32MK0AJ472ApKX/1PB26OiC8VZs3J2zs5//1R6WjNzGzYlUoEEfGcIWz71cDbgBslLczTPkpKAOdLOhK4E3jTELZtZmbDpGwXE29vNj0izhhsnYi4GhisQeB1Zco1M7Pqlb009PLC8MakA/n1wKCJwMzMRoeyl4beXxyXtAVwZiURmZlZVw21G+rHgJ2HMxAzM+uNsm0EPybdJQSps7kXAudXFZSZmXVP2TaCUwrDa4A7IuLuCuIxM7MuK9tG8AtJ2/BUo/Et1YVkVr2R+HCgkRiT1UOpNgJJhwLXke75PxS4VlLLbqjNzGx0KHtp6Hjg5RFxP4CkPuBy4MKqAjMzs+4oe9fQegNJIHuwg3XNzGwEK3tGcKmknwHn5PE3A5dUE5KZmXVTu2cW70R6fsCxkv4F2IPUbcRvgLO6EJ+ZmVWs3eWd04BVABHx/Yj4YET8F+ls4LSqgzMzs+q1SwRTIuKGxokRMZ/02EozMxvl2iWCjVvM22Q4AzEzs95olwj+T9I7GifmZwksqCYkMzPrpnZ3DR0N/EDSW3jqwN8PbAgcXGVgZmbWHS0TQUQsA14laW/gRXnyTyLiysojMzOzrijb19BcYG7FsZiZWQ/418FmZjVXWSKQ9B1J90u6qTDtREn3SFqYX/tXVb6ZmZVT5RnBLGDfJtNPjYip+eVuKszMeqyyRBARVwEPVbV9MzMbHr1oI3ifpBvypaMJPSjfzMwKup0IvgHsCEwFlgJfHGxBSTMkzZc0f/ny5d2Kz8ysdrqaCCJiWUQ8ERFPAt8Cdmux7MyI6I+I/r6+vu4FaWZWM11NBJImFUYPBm4abFkzM+uOsg+m6Zikc4C9gK0l3Q18AthL0lQggCXAO6sq38zMyqksEUTE4U0mn15VeWZmNjT+ZbGZWc05EZiZ1ZwTgZlZzTkRmJnVnBOBmVnNORGYmdWcE4GZWc05EZiZ1ZwTgZlZzTkRmJnVnBOBmVnNORGYmdWcE4GZWc05EZiZ1ZwTgZlZzTkRmJnVnBOBmVnNORGYmdWcE4GZWc05EZiZ1VxliUDSdyTdL+mmwrSJki6TdEv+O6Gq8s3MrJwqzwhmAfs2TDsOuCIidgauyONmZtZDlSWCiLgKeKhh8oHA7Dw8GzioqvLNzKyccV0ub5uIWAoQEUslPXOwBSXNAGYATJ48uUvhmY1de+/d2fJz51YTh408I7axOCJmRkR/RPT39fX1OhwzszGr24lgmaRJAPnv/V0u38zMGnQ7EcwBjsjDRwA/6nL5ZmbWoMrbR88BfgM8X9Ldko4ETgZeL+kW4PV53MzMeqiyxuKIOHyQWa+rqkwzM+tct+8aMquFTu/QGYl8l1F9jNi7hszMrDucCMzMas6JwMys5pwIzMxqzonAzKzmnAjMzGrOicDMrOacCMzMas6JwMys5pwIzMxqzonAzKzmnAjMzGrOicDMrOacCMzMas6JwMys5pwIzMxqzonAzKzmnAjMzGrOicDMrOZ68sxiSUuAVcATwJqI6O9FHGZm1tuH1+8dEQ/0sHwzM8OXhszMaq9XZwQB/FxSAP8bETMbF5A0A5gBMHny5C6HZ2ad2nvvzpafO7eaOKxzvTojeHVEvAzYD3ivpD0bF4iImRHRHxH9fX193Y/QzKwmepIIIuLe/Pd+4AfAbr2Iw8zMepAIJG0mafzAMPAG4KZux2FmZkkv2gi2AX4gaaD8syPi0h7EYWZm9CARRMRtwEu7Xa6ZmTXXy98RmJl1xHcmVcO/IzAzqzknAjOzmnMiMDOrOScCM7OacyIwM6s53zVkZpZVfVdSp9sfShlD4TMCM7OacyIwM6s5JwIzs5pzIjAzqzk3FptZTwyl4XQkljEW+IzAzKzmnAjMzGrOicDMrOacCMzMas6JwMys5pwIzMxqzonAzKzmnAjMzGquJ4lA0r6S/iDpVknH9SIGMzNLup4IJK0PfA3YD9gFOFzSLt2Ow8zMkl6cEewG3BoRt0XEX4FzgQN7EIeZmdGbvoa2A+4qjN8NvKJxIUkzgBl5dLWkPwyxvK2BB4a4bq+MtphHW7ww+mIebfHC6Iu543iliiIpX0a7mJ9dpoxeJIJmbyueNiFiJjBznQuT5kdE/7pup5tGW8yjLV4YfTGPtnhh9MU82uKF4Yu5F5eG7gZ2KIxvD9zbgzjMzIzeJIL/A3aW9BxJGwKHAXN6EIeZmdGDS0MRsUbS+4CfAesD34mIxRUWuc6Xl3pgtMU82uKF0RfzaIsXRl/Moy1eGKaYFfG0y/NmZlYj/mWxmVnNORGYmdXcmEkE7bqtkLSRpPPy/GslTel+lGvF0y7ePSVdL2mNpEN6EWOjEjF/UNLvJN0g6QpJpe5hrkqJeN8l6UZJCyVdPRJ+4V62+xVJh0gKST293bFEHU+XtDzX8UJJR/UizoaY2taxpEPzvrxY0tndjrEhlnZ1fGqhfv8oaWXHhUTEqH+RGp3/BDwX2BBYBOzSsMx7gG/m4cOA80Z4vFOAlwBnAIeMkjreG9g0D797FNTxMwrD/wxcOtLrOC83HrgKuAboH8nxAtOBr/ayXocQ887Ab4EJefyZIznehuXfT7oBp6NyxsoZQZluKw4EZufhC4HXSd34XWBTbeONiCURcQPwZC8CbKJMzHMj4rE8eg3pNyK9UibeRwqjm9Hkh41dVrb7lU8BXwD+0s3gmhiN3cWUifkdwNciYgVARNzf5RiLOq3jw4FzOi1krCSCZt1WbDfYMhGxBngY2Kor0T1dmXhHmk5jPhL4aaURtVYqXknvlfQn0oH1P7sU22DaxixpV2CHiLi4m4ENouw+8a/5cuGFknZoMr+bysT8POB5kn4l6RpJ+3Ytuqcr/X+XL8U+B7iy00LGSiIo021Fqa4tumQkxVJW6ZglvRXoB/5fpRG1VrYrk69FxI7Ah4ETKo+qtZYxS1oPOBU4pmsRtVamjn8MTImIlwCX89RZea+UiXkc6fLQXqRv2N+WtGXFcQ2mk2PFYcCFEfFEp4WMlURQptuKvy8jaRywBfBQV6J7utHYzUapmCVNA44H/jkiHu9SbM10WsfnAgdVGlF77WIeD7wImCdpCbA7MKeHDcZt6zgiHizsB98C/rFLsQ2m7LHiRxHxt4i4HfgDKTH0Qif78WEM4bIQMGYai8cBt5FOiwYaVP6hYZn3snZj8fkjOd7CsrMYGY3FZep4V1LD1s6jJN6dC8MHAPNHeswNy8+jt43FZep4UmH4YOCakV7HwL7A7Dy8NenSzFYjNd683POBJeQfCXdcTi8/lGGusP2BP+YD0fF52idJ30wBNgYuAG4FrgOeO8LjfTnp28CjwIPA4lFQx5cDy4CF+TVnhMf7ZWBxjnVuq4PuSIm5YdmeJoKSdfy5XMeLch2/YKTXMelyzJeA3wE3AoeN5Hjz+InAyUMtw11MmJnV3FhpIzAzsyFyIjAzqzknAjOzmnMiMDOrOScCM7OacyKwnsm9Z55ZGB+Xe6octu4TJH17qL2K5p4zvzpcsTTZfl/uCfe3kl4j6ZJmv2CVdKKkD1UVh1nXH1VpVvAo8CJJm0TEn4HXA/d0sgFJ4yL1HdVURPS82+MWXgf8PiKOyOO/7GUwVl8+I7Be+ynwxjy8Vs+JknaT9Ov8jfnXkp6fp0+XdIGkHwM/l7SepK/nvuMvzt+sD8nLzhvogkHSakmfkbQodya2TZ5+QOGb+eUD0wcjaXNJ383PMrhB0r/m6YfnaTdJ+nxh+aeVK2kqqaO7/XM/8ptIWiJp67zO8bkP+stJvxod2NaOki6VtEDSLyW9IE+fJel/cj3dpsIzLCT9d45rkaSTW23HaqrXv/Lzq74vYDXpmQsXkn75vZDU0dfFef4zgHF5eBpwUR6eTvrV9cQ8fghwCemLzbOAFeRuOSj8+pbUWdcBefgLwAl5eAJPPb/7KOCLhXKe1pc+8HngtML4BGBb4E6gj3SmfSVwUJty19o+qYuArUn98dwIbJrr4FbgQ3mZK8hdYwCvAK7Mw7NIv5xfD9iF1HUxwH7Ar3nqORETW23Hr3q+fGnIeioiblB6WtzhpIN50RbAbEk7kw6mGxTmXRYRA50G7gFcEBFPAvdJmjtIcX8FBtofFpAuRUHqyOs8SZNI/bnc3ibsaaT+qgbewwpJewLzImI5gKSzgD2BH7YodzCvAX4Q+dkOkubkv5sDrwIuKDxKY6PCej/MdfC7wlnNNOC7A9uKiIdKbMdqxonARoI5wCmks4HiMyI+BcyNiINzsphXmPdoYbjsA4b+FhEDfao8wVP7/1eAL0XEHEl7kfptaUWU6+a8XbmtNOv7ZT1gZURMHWSdYm+vKvxt3Fa77VjNuI3ARoLvAJ+MiBsbpm/BU43H01usfzXp4Sfr5W/Ce3VYfrGcI1otmP0ceN/AiKQJwLXAayVtLWl90hnOLzqMY8BVwMG53WA8qWdUIj1R7XZJb8rlStJLS8T6H5I2zetMHOJ2bAxzIrCei4i7I+LLTWZ9AficpF+Rnt06mItIbQY3Af9LOig/3EEIJ5Iuk/wSeKDE8p8GJuRG4UXA3hGxFPgIqYfNRcD1EfGjDmL4u4i4HjiP1GZyEWvfTfQW4Mhc7mLaPBoyIi4lnXHNl7QQGLgNtaPt2Njm3kdtTJC0eUSslrQVqZvxV0fEfb2Oy2w0cBuBjRUX5x9jbQh8yknArDyfEZiZ1ZzbCMzMas6JwMys5pwIzMxqzonAzKzmnAjMzGru/wMB6EDT5GP3oQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "positive_test_data = list(filter(lambda row : row[0]==1, hstack((targets_test[:,None], data_test)).toarray()))\n",
    "positive_probs = clf.predict_proba(list(map(lambda r : r[1:], positive_test_data)))\n",
    "marginal_positive_probs = list(map(lambda p : marginal(p), positive_probs))\n",
    "n, bins, patches = plt.hist(marginal_positive_probs, 25, facecolor='blue', alpha=0.75)\n",
    "plt.title('Marginal confidence histogram - Neutral data')\n",
    "plt.ylabel('Count')\n",
    "plt.xlabel('Marginal confidence')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "c43fb9ec-8751-f1e6-24a0-b77f18dacefd"
   },
   "source": [
    "### Negative data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "_cell_guid": "52fda712-b5c7-7bb2-0333-68a888547226"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHqJJREFUeJzt3Xu8XFV5//HPNwnhGkgCB0yAGNCIYhVoI4JSyiVaRJFYgYKoSQ2kXkq1XlFojcqvgPUnUG8UQRPulygSqKIQElG5aIBwxxogQCQkARIgYMXg0z/WOmQzzDkzc5I9Mzn7+3695jX7vp99mXlmrb33GkUEZmZWXUM6HYCZmXWWE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFORF0AUlfkHT2elpWSHr1+lhWC+s8SdLjkh6TNE7SaklD+5h2hqTz2xlffyRNlfTLfsb/RNKUdsZkjUm6W9J+bVjPYkmTyl5Ppw3rdADdRNJiYCwwNiIeLwxfCOwG7BQRi9f3eiPi39f3MttF0o7Ap4BXRsTyPHiLDoa0XkXEO5qZTlIAEyJiUckhdUTevruA3SLiz3nYScAOETG15HXPBJZExIm9wyLi9WWucyA25HPAJYKXexA4qrdH0huATQe6MEmDPdm+EniikARsPeuic2gscGSng7D1z4ng5c4DPljonwKcW5xA0jsl3SbpaUmPSJpRGDc+V89Mk/QwcF0e/kFJD0l6QtK/FoucxeqSwvxTJD2cq1xOKCx/T0k3Slolaamkb0oa3syGSRot6fuSHpW0UtKPCuOOlbRI0pOS5kgaWxgXkj4s6Xd5vm8pmQRcA4zN1UEzC/EPy/PuJOnnkp6RdA2wTU1Me0m6IW/P7cXivqT5kr4i6Vd5/p9J2qYwfp/CvI9ImpqHbyzpa3n/LZN0pqR+k3mefqWkByW9ozB8vqRjcver87Y8lY/LJXn49Xny2/N++Psm9unbJf02L+vbebm965mat/k0SU8CMyS9StJ1+fx5XNIFkkYWlrdY0mck3SHpWUnnSNpOqWrrGUnXShrV7wnS2FeBL6mPxNTgWO4k6fpCLN9SoYpQ0mVKVYtP5elen4dPB44GPpv37ZWF7Z0kaaykP0gaXVjWHnkfbZT7PyTp3nx8fyrplX1toKQPaO3n9ISacX1+9uqdA5JGSbpK0oq87qsk7dDSHm+XiPArv4DFwCTgt8DrgKHAI6RfvQGMz9PtB7yBlEjfCCwDJudx4/O05wKbk0oTuwKrgX2A4cDXgD8Bk/I8M4Dza+b/bp53N+CPwOvy+L8C9iJV640H7gU+UdiGAF7dx/b9N3AJMArYCPibPPwA4HHgL4GNgW8A19cs8ypgJDAOWAEcVNgXSwrT9sY/LPffCHw9L3df4JnCtm4PPAEcnPfl23J/Tx4/H7gfeE3eF/OBU/K4cXlZR+Vt2RrYPY87HZgDjAZGAFcCJ/exT6bmY3FsPt4fAR4FVIjhmNx9EXBCjnUTYJ++9nt/+5SUDJ8G/i4fx4/nGI4pxLQGOC6P3xR4dd4/GwM9wPXA6TXn7k3Adnm/LgduBfbI81wHfHEdPhsBTABuKcR5EjCzyWN5I+m8H076HDzdex7k8R/Kx2rjfPwWFsbNBE6q91nN3dcBxxbG/QdwZu6eDCwifZ6HAScCN/Sxjb2f031zHF/Px6F3PS199kjn5HuBzfK2XQb8qNPfc3W3vdMBdNOLtYngROBk4CDSL95hFBJBnflOB07L3ePztDsXxv8bcFGhfzPgefpPBDsUpv81cGQf6/4EcHlfJ2Nh+Bjgz8CoOuPOAb5a6N+C9MU0vrDM4pfepcDxuXs/+kgEpC/rNcDmhfEXFrb1c8B5NbH8FJiSu+cDJxbGfRS4Ond/vrjdhWkEPAu8qjBsb+DBPvbfVGBRzbEJ4BWFGHq/+M4Fzioem772e3/7lFTivLEm5kd4aSJ4uMG5Ohm4rebcPbrQ/wPgO4X+41iHL6He7SN90T9M+qIsJoI+j2XhPNisMO58ComgZr6ReX1b5f6Z9J8IjgGuq9mX++b+nwDTCvMNAZ4jXdOqXe+/ARcX+jen8Dkd6GevMH53YOVAj0GZL1cN1Xce8D7SB/Lc2pGS3ixpXi7yPQV8mJoqD9LJ2GtssT8iniP9WurPY4Xu58gXYCW9JhcxH5P0NPDvddZdz47AkxGxss64scBDhfhW5/i2bxRPA2NJJ/6zhWEPFbpfCRyei9qrJK0i/Voc08R6dySVFmr1kL7Mbyks8+o8vC8vriMfG6i/fZ8lfdH8WumulQ/1s8z+9mnt+RDAkpr5i+cPkraVdLGk3+fjfj4vP+7LCt1/qNNf95jl6qPV+XV0P9tERPyYlAim14zq71iOJZ17zxWmf3H7JA2VdIqk+/O2Lc6jmjmvAWYDe+eqt31JX8i/KMR1RiGmJ0nHcPs6y6k9Ls9S+Jy2+tmTtJmk/8pVTU+TSnEj1ccddZ3kRFBHRDxEumh8MPDDOpNcSKp62DEitgLOJJ1cL1lMoXsp8GLdoFJ99dYDDO87wH2kuxO2BL5QZ931PAKMLtYrFzxK+sD0xrd5ju/3A4yx11JgVF5er3E1MZ0XESMLr80j4pQmlv0I8Ko6wx8nfem9vrDMrSJine9kiojHIuLYiBgL/CPwbfV9q25/+7T2fFCxv3d1Nf0n52FvzMf9/TR33BuKiHdExBb5dUETs5xIqiLbrDCsv2O5lHTuFaffsdD9PuBQUml8K1KpCdZuX+2+qI1/FfAz4Ii8rItycu2N6x9r4to0Im6os6ilxbhyvMXPaaufvU8BuwBvztPvW7NdXcOJoG/TgANqfs32GkH6hfO/kvYknXz9mQ0cIukt+eLSlxj4yTCCVL+6WtJrSXXaDUXEUlIx+dv5ItZGknpPzAuBf5C0u6SNSb90bo51vFU2J9QFpAuMwyXtAxxSmOR80n752/yrcBNJ+zV5Qe0CYJKkIyQNk7S1pN0j3dr4XeA0SdsCSNpe0t+uy7bk5RxeiG0l6Qvqhdy/DNi5MHl/+/S/gTdImqx04fVjwCsarH4Eqf56laTtgc+s6/YMVETMB+4kVfv06vNYFs6DGfk82JuXngcjSNfBniAll9rbqWv3bT0Xkqrc3pu7e50JfF5rLz5vJenwPpYxG3iX0k0Iw4Ev89LvyEafvdo4R5B+lKzKF7O/2GAbOsaJoA8RcX9ELOhj9EeBL0t6hlSveGmDZd1NqqO9mPSr4xnSxbw/DiC0T5MSzzOkL7xLWpj3A6R66vvy+j+R45sL/CupXnkp6Zf2+rpN8H3Am0lF8i9SqGqLiEdIvwS/QLoA/QjpC67heRkRD5NKbJ/Ky+591gNSffUi4KZcJL+W9MtsXb0JuFnSalKJ8OMR8WAeNwOYlasgjuhvn0Z6RuVw0l04T5AuUi6g//PhS6QLz0+REkm9kmo7nUi6GA80dSyPJl2reYJ0beES1m7vuaRqtN8D95AuehedA+ya9+2PqG8O6WL2soi4vRDX5cCpwMX5XLgLqPtsSP6cfoyUSJaSkn2xyq7RZ28GhXOAdO1wU1Ip9SZSFWVX0toSlLWLpC2AVaQi5oONprfBTdIQ0hfO0RExr9PxtIPSrbf3RUTX/kquEpcI2kTSIfni0eak2+juZO1FMauYXIUyMlcb9dY11/4SHjQkvUnpWYghkg4ilR76+nVvbeZE0D6Hki4gPkoqwh4ZLo5V2d6ku54eJ9WXT46IP3Q2pFK9gnQr7mrgP4GPRMRtHY3IXuSqITOzinOJwMys4rqlMat+bbPNNjF+/PhOh2FmtkG55ZZbHo+I/h6mBDaQRDB+/HgWLOjrTk4zM6tH0kONp3LVkJlZ5TkRmJlVnBOBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFbRBPFps1tP/+rU0/rxLN/ps1xSUCM7OKcyIwM6s4Vw1Zd2q1qsfMBswlAjOzinMiMDOrOFcNWTX5LiOzF7lEYGZWcU4EZmYV50RgZlZxTgRmZhXnRGBmVnGlJgJJIyXNlnSfpHsl7S1ptKRrJP0uv48qMwYzM+tf2SWCM4CrI+K1wG7AvcDxwNyImADMzf1mZtYhpSUCSVsC+wLnAETE8xGxCjgUmJUnmwVMLisGMzNrrMwSwc7ACuD7km6TdLakzYHtImIpQH7ftsQYzMysgTKfLB4G/CVwXETcLOkMWqgGkjQdmA4wbty4ciK0xE/ZmlVamSWCJcCSiLg5988mJYZlksYA5Pfl9WaOiLMiYmJETOzp6SkxTDOzaistEUTEY8AjknbJgw4E7gHmAFPysCnAFWXFYGZmjZXd6NxxwAWShgMPAP9ASj6XSpoGPAwcXnIMZmbWj1ITQUQsBCbWGXVgmes1M7Pm+cliM7OK8/8RWPn8t5NmXc0lAjOzinMiMDOrOCcCM7OKcyIwM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OKcyIwM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OK8/8RbAhabc9/3rxy4jCzQcklAjOzinMiMDOrOFcNWev815Nmg4pLBGZmFedEYGZWcaVWDUlaDDwDvACsiYiJkkYDlwDjgcXAERGxssw4zMysb+0oEewfEbtHxMTcfzwwNyImAHNzv5mZdUgnqoYOBWbl7lnA5A7EYGZmWdmJIICfSbpF0vQ8bLuIWAqQ37ctOQYzM+tH2bePvjUiHpW0LXCNpPuanTEnjukA48aNKyu+wcm3d5pZC0otEUTEo/l9OXA5sCewTNIYgPy+vI95z4qIiRExsaenp8wwzcwqrbREIGlzSSN6u4G3A3cBc4ApebIpwBVlxWBmZo2VWTW0HXC5pN71XBgRV0v6DXCppGnAw8DhJcZgZmYNlJYIIuIBYLc6w58ADixrvWZm1ho/WWxmVnFudK4TfFePmXURlwjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqbvDfNVT2HTrz5pW7fDOzkrlEYGZWcU4EZmYVN/irhszWh1arGNtRZehqT1tPXCIwM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OK8+2j68r/LWD1dOPtpq0aDNtgTXGJwMys4pwIzMwqzonAzKzinAjMzCrOicDMrOJKv2tI0lBgAfD7iHiXpJ2Ai4HRwK3AByLi+bLjMOtqvvvMOqgdJYKPA/cW+k8FTouICcBKYFobYjAzsz6Umggk7QC8Ezg79ws4AJidJ5kFTC4zBjMz61/ZJYLTgc8Cf879WwOrImJN7l8CbF9vRknTJS2QtGDFihUlh2lmVl2lJQJJ7wKWR8QtxcF1Jo1680fEWRExMSIm9vT0lBKjmZmVe7H4rcC7JR0MbAJsSSohjJQ0LJcKdgAeLTEGMzNroLQSQUR8PiJ2iIjxwJHAdRFxNDAPOCxPNgW4oqwYzMyssU48R/A54JOSFpGuGZzTgRjMzCxrS+ujETEfmJ+7HwD2bMd6zcysMT9ZbGZWcU0lAklvbWaYmZlteJotEXyjyWFmZraB6fcagaS9gbcAPZI+WRi1JTC0zMDMzKw9Gl0sHg5skacbURj+NGtvATUzsw1Yv4kgIn4O/FzSzIh4qE0xmZlZGzV7++jGks4CxhfniYgDygjKzMzap9lEcBlwJqkV0RfKC8fMzNqt2USwJiK+U2okZmbWEc3ePnqlpI9KGiNpdO+r1MjMzKwtmi0RTMnvnykMC2Dn9RuOmZm1W1OJICJ2KjsQMzPrjKYSgaQP1hseEeeu33DMzKzdmq0aelOhexPgQOBWwInAzGwD12zV0HHFfklbAeeVEpGZmbXVQJuhfg6YsD4DMTOzzmj2GsGVrP2T+aHA64BLywrKzMzap9lrBF8rdK8BHoqIJSXEY2ZmbdZU1VBufO4+Uguko4DnywzKzMzap9l/KDsC+DVwOHAEcLMkN0NtZjYINFs1dALwpohYDiCpB7gWmF1WYGZm1h7N3jU0pDcJZE+0MK+ZmXWxZksEV0v6KXBR7v974MflhGRmZu3U6D+LXw1sFxGfkfR3wD6AgBuBCxrMuwlwPbBxXs/siPiipJ2Ai4HRpKeTPxARvvhsZtYhjap3TgeeAYiIH0bEJyPiX0ilgdMbzPtH4ICI2A3YHThI0l7AqcBpETEBWAlMW5cNMDOzddMoEYyPiDtqB0bEAtLfVvYpktW5d6P8CuAA1l5kngVMbiVgMzNbvxolgk36Gbdpo4VLGippIbAcuAa4H1gVEWvyJEuA7fuYd7qkBZIWrFixotGqzMxsgBolgt9IOrZ2oKRpwC2NFh4RL0TE7sAOwJ6kpileNlkf854VERMjYmJPT0+jVZmZ2QA1umvoE8Dlko5m7Rf/RGA48J5mVxIRqyTNB/YCRkoalksFOwCPthy1mZmtN/2WCCJiWUS8BfgSsDi/vhQRe0fEY/3NK6lH0sjcvSkwCbgXmAf0PpU8BbhiXTbAzMzWTbP/RzCP9AXeijHALElDSQnn0oi4StI9wMWSTgJuA85pcblmZrYeNftAWcvy3UZ71Bn+AOl6gZmZdQE3E2FmVnFOBGZmFedEYGZWcU4EZmYV50RgZlZxTgRmZhXnRGBmVnFOBGZmFedEYGZWcU4EZmYV50RgZlZxTgRmZhXnRGBmVnFOBGZmFedEYGZWcU4EZmYV50RgZlZxTgRmZhXnRGBmVnFOBGZmFedEYGZWcU4EZmYV50RgZlZxpSUCSTtKmifpXkl3S/p4Hj5a0jWSfpffR5UVg5mZNVZmiWAN8KmIeB2wF/AxSbsCxwNzI2ICMDf3m5lZh5SWCCJiaUTcmrufAe4FtgcOBWblyWYBk8uKwczMGmvLNQJJ44E9gJuB7SJiKaRkAWzbxzzTJS2QtGDFihXtCNPMrJJKTwSStgB+AHwiIp5udr6IOCsiJkbExJ6envICNDOruFITgaSNSEnggoj4YR68TNKYPH4MsLzMGMzMrH9l3jUk4Bzg3oj4emHUHGBK7p4CXFFWDGZm1tiwEpf9VuADwJ2SFuZhXwBOAS6VNA14GDi8xBjMzKyB0hJBRPwSUB+jDyxrvWZm1ho/WWxmVnFOBGZmFedEYGZWcU4EZmYVV+ZdQ2ZWJfvv39r08+aVE4e1zCUCM7OKcyIwM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OKcyIwM6s4P1BmZhsOP7RWCpcIzMwqzonAzKziXDVkZp3RajVPO9ZR0aoklwjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqzonAzKziSksEkr4nabmkuwrDRku6RtLv8vuostZvZmbNKbNEMBM4qGbY8cDciJgAzM39ZmbWQaUlgoi4HniyZvChwKzcPQuYXNb6zcysOe2+RrBdRCwFyO/b9jWhpOmSFkhasGLFirYFaGZWNV17sTgizoqIiRExsaenp9PhmJkNWu1OBMskjQHI78vbvH4zM6vR7kQwB5iSu6cAV7R5/WZmVqPM20cvAm4EdpG0RNI04BTgbZJ+B7wt95uZWQeV1gx1RBzVx6gDy1qnmZm1rmsvFpuZWXs4EZiZVZwTgZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVXGl/TGNmNujtv39r08+bV04c68glAjOzinMiMDOrOFcNmZm1S5dWJblEYGZWcU4EZmYV50RgZlZxHblGIOkg4AxgKHB2RJzSiTjMzF6i1Tr8QaLtJQJJQ4FvAe8AdgWOkrRru+MwM7OkE1VDewKLIuKBiHgeuBg4tANxmJkZnaka2h54pNC/BHhz7USSpgPTc+9qSb8d4Pq2AR4f4Lxlc2wD49gGxrENTOdikxpN0Si2Vzazmk4kgnpbFi8bEHEWcNY6r0xaEBET13U5ZXBsA+PYBsaxDUwVYutE1dASYMdC/w7Aox2Iw8zM6Ewi+A0wQdJOkoYDRwJzOhCHmZnRgaqhiFgj6Z+An5JuH/1eRNxd4irXuXqpRI5tYBzbwDi2gRn0sSniZdXzZmZWIX6y2Mys4pwIzMwqbtAkAkkHSfqtpEWSjq8zfmNJl+TxN0sa30Wx7SvpVklrJB3WrriajO2Tku6RdIekuZKaui+5TbF9WNKdkhZK+mU7n1BvFFthusMkhaS23X7YxH6bKmlF3m8LJR3TLbHlaY7I59zdki7sltgknVbYZ/8jaVUXxTZO0jxJt+XP6sEtrSAiNvgX6aLz/cDOwHDgdmDXmmk+CpyZu48ELumi2MYDbwTOBQ7rsv22P7BZ7v5Il+23LQvd7wau7pbY8nQjgOuBm4CJ3RIbMBX4ZrvOsxZjmwDcBozK/dt2S2w10x9HutGlK2IjXTT+SO7eFVjcyjoGS4mgmWYrDgVm5e7ZwIFS48f22hFbRCyOiDuAP7chnlZjmxcRz+Xem0jPfXRLbE8XejenzoOJnYot+wrwVeB/2xRXK7F1QjOxHQt8KyJWAkTE8i6Krego4KK2RNZcbAFsmbu3osVnswZLIqjXbMX2fU0TEWuAp4CtuyS2Tmk1tmnAT0qNaK2mYpP0MUn3k75w/7lbYpO0B7BjRFzVpph6NXtM35urEGZL2rHO+DI0E9trgNdI+pWkm3JLxd0SGwC5enQn4Lo2xAXNxTYDeL+kJcCPSSWWpg2WRNBMsxVNNW1Rgk6ttxlNxybp/cBE4D9KjaiwyjrD6jVF8q2IeBXwOeDE0qNK+o1N0hDgNOBTbYqnqJn9diUwPiLeCFzL2pJy2ZqJbRipemg/0q/usyWNLDkuaO1zeiQwOyJeKDGeomZiOwqYGRE7AAcD5+XzsCmDJRE002zFi9NIGkYqPj3ZJbF1SlOxSZoEnAC8OyL+2E2xFVwMTC41orUaxTYC+AtgvqTFwF7AnDZdMG643yLiicJx/C7wV22Iq6nY8jRXRMSfIuJB4LekxNANsfU6kvZVC0FzsU0DLgWIiBuBTUgN0jWnHRc72nAxZRjwAKm41nsx5fU103yMl14svrRbYitMO5P2XixuZr/tQbpQNaELj+mEQvchwIJuia1m+vm072JxM/ttTKH7PcBNXRTbQcCs3L0NqUpk626ILU+3C7CY/DBuF+23nwBTc/frSImi6RjbsiFt2lkHA/+Tv7ROyMO+TPoVCylDXgYsAn4N7NxFsb2JlPWfBZ4A7u6i2K4FlgEL82tOF8V2BnB3jmtef1/G7Y6tZtq2JYIm99vJeb/dnvfba7soNgFfB+4B7gSO7JbYcv8M4JR2xdTCftsV+FU+pguBt7eyfDcxYWZWcYPlGoGZmQ2QE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBdURukfO8Qv+w3CLmemuSQdLZA22RNLfQ+c31FUud5ffkVnBvk/TXkn5c7wlaSTMkfbqsOMygA39VaZY9C/yFpE0j4g/A24Dft7IAScMitRtVV0S0rXnlATgQuC8ipuT+X3QyGKs2lwisk34CvDN3v6Q1R0l7Sroh/2K+QdIuefhUSZdJuhL4maQhkr6d266/Kv+yPixPO7+3WQdJqyX9P0m358bMtsvDDyn8Mr+2d3hfJG0h6fv5fxDukPTePPyoPOwuSacWpn/ZeiXtTmok7+Dctv2mkhZL2ibPc0Jue/5a0pOsvct6laSrJd0i6ReSXpuHz5T0n3k/PaDCf1pI+myO63ZJp/S3HKuwdj8h55dfEQGwmvQfDLNJT30vJDU0dlUevyUwLHdPAn6Qu6eSnsIenfsPI7W2OAR4BbCS3EwHhSd6SY10HZK7vwqcmLtHsfa/u48B/n9hPS9rsx84FTi90D8KGAs8DPSQStnXAZMbrPclyyc1W7ANqd2fO4HN8j5YBHw6TzOX3KwG8Gbgutw9k/TU/BDSE6aL8vB3ADew9v8kRve3HL+q+3LVkHVMRNyh9E9xR5G+zIu2AmZJmkD6Mt2oMO6aiOhtMHAf4LKI+DPwmKR5fazueaD3+sMtpKooSA14XSJpDKkdlwcbhD2J1FZV7zaslLQvMD8iVgBIugDYF/hRP+vty18Dl0f+DwhJc/L7FsBbgMsKf6OxcWG+H+V9cE+hVDMJ+H7vsiLiySaWYxXkRGCdNgf4Gqk0UPx/iK8A8yLiPTlZzC+Me7bQ3eyfC/0pInrbU3mBtef+N4CvR8QcSfuR2pLpj2iuifNG6+1PvXZfhgCrImL3PuYptgqrwnvtshotxyrI1wis074HfDki7qwZvhVrLx5P7Wf+X5L+ZGVI/iW8X4vrL65nSn8TZj8D/qm3R9Io4GbgbyRtI2koqYTz8xbj6HU98J583WAEqVVVIv0b24OSDs/rlaTdmoj1Q5I2y/OMHuBybJBzIrCOioglEXFGnVFfBU6W9CvSf7b25QekawZ3Af9F+lJ+qoUQZpCqSX4BPN7E9CcBo/JF4duB/SNiKfB5UkuetwO3RsQVLcTwooi4FbiEdM3kB7z0bqKjgWl5vXfT4C8oI+JqUolrgaSFQO9tqC0txwY/tz5qGzxJW0TEaklbk5oYf2tEPNbpuMw2FL5GYIPBVflhrOHAV5wEzFrjEoGZWcX5GoGZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnF/R+QnSioc0P4VgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "negative_test_data = list(filter(lambda row : row[0]==0, hstack((targets_test[:,None], data_test)).toarray()))\n",
    "negative_probs = clf.predict_proba(list(map(lambda r : r[1:], negative_test_data)))\n",
    "marginal_negative_probs = list(map(lambda p : marginal(p), negative_probs))\n",
    "n, bins, patches = plt.hist(marginal_negative_probs, 25, facecolor='red', alpha=0.75)\n",
    "plt.title('Marginal confidence histogram - Negative data')\n",
    "plt.ylabel('Count')\n",
    "plt.xlabel('Marginal confidence')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "90217d44-58a8-782f-5b68-9ee7032aa9c7"
   },
   "source": [
    "Clearly the positive data is much harder for the classifier. This makes sense since there's a lot less of it. An important challenge in building a better classifier will then be how to handle positive data.\n",
    "\n",
    "A more advanced classifier is described here."
   ]
  }
 ],
 "metadata": {
  "_change_revision": 132,
  "_is_fork": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}