notes_4.4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<hr width=60%>\n",
    "\n",
    "# Notes Week 4, Day 4 { &#1114; }\n",
    "### 2019 June 27 \n",
    "#### Cameron.Cn | &#26607;&#26234;&#26126;\n",
    "\n",
    "**Made a data dashboard today!!** | https://public.tableau.com/profile/cameron.r.cumberland#!/vizhome/nasa-meteors/WorldMapSearch\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Interesting Notes\n",
    "https://openai.com/blog/adversarial-example-research/\n",
    "\n",
    "Explaining and Harnessing Adversarial Examples in AI | https://arxiv.org/pdf/1412.6572.pdf\n",
    "\n",
    "Practical Black-Box Attacks against Neural Nets | https://arxiv.org/abs/1602.02697\n",
    "\n",
    "OpenSource \"Clever Hans\" Python code to adversarially test AIs | https://github.com/tensorflow/cleverhans\n",
    "\n",
    "AI Discovers Unicorns living in the Andes, speaking English | https://openai.com/blog/better-language-models/\n",
    "\n",
    "Recipes from pictures | https://arxiv.org/pdf/1812.06164.pdf\n",
    "\n",
    "---\n",
    "**Learn to Build your own Image Recognition AI with TensorFlow** | https://www.tensorflow.org/tutorials/keras/basic_classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## <center> Working on Data Cleaning (Some Linked Review)\n",
    "Dealing with Outliers (3 Simple methods)\n",
    "+ https://www.kdnuggets.com/2017/01/3-methods-deal-outliers.html\n",
    "\n",
    "How to remove Outliers by Std.Dev\n",
    "+ https://www.kdnuggets.com/2017/02/removing-outliers-standard-deviation-python.html\n",
    "\n",
    "Detecting Outliers\n",
    "+ https://stats.stackexchange.com/questions/38001/detecting-outliers-using-standard-deviations\n",
    "\n",
    "Python for Oceanographers\n",
    "+ https://ocefpaf.github.io/python4oceanographers/blog/2015/03/16/outlier_detection/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fname</th>\n",
       "      <th>lname</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>exercise</th>\n",
       "      <th>hours</th>\n",
       "      <th>grade</th>\n",
       "      <th>address</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Marcia</td>\n",
       "      <td>Pugh</td>\n",
       "      <td>female</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>82.4</td>\n",
       "      <td>9253 Richardson Road, Matawan, NJ 07747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kadeem</td>\n",
       "      <td>Morrison</td>\n",
       "      <td>male</td>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>78.2</td>\n",
       "      <td>33 Spring Dr., Taunton, MA 02780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nash</td>\n",
       "      <td>Powell</td>\n",
       "      <td>male</td>\n",
       "      <td>18</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>79.3</td>\n",
       "      <td>41 Hill Avenue, Mentor, OH 44060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Noelani</td>\n",
       "      <td>Wagner</td>\n",
       "      <td>female</td>\n",
       "      <td>14</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>83.2</td>\n",
       "      <td>8839 Marshall St., Miami, FL 33125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Noelani</td>\n",
       "      <td>Cherry</td>\n",
       "      <td>female</td>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>87.4</td>\n",
       "      <td>8304 Charles Rd., Lewis Center, OH 43035</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     fname     lname  gender  age  exercise  hours  grade  \\\n",
       "0   Marcia      Pugh  female   17         3     10   82.4   \n",
       "1   Kadeem  Morrison    male   18         4      4   78.2   \n",
       "2     Nash    Powell    male   18         5      9   79.3   \n",
       "3  Noelani    Wagner  female   14         2      7   83.2   \n",
       "4  Noelani    Cherry  female   18         4     15   87.4   \n",
       "\n",
       "                                    address  \n",
       "0   9253 Richardson Road, Matawan, NJ 07747  \n",
       "1          33 Spring Dr., Taunton, MA 02780  \n",
       "2          41 Hill Avenue, Mentor, OH 44060  \n",
       "3        8839 Marshall St., Miami, FL 33125  \n",
       "4  8304 Charles Rd., Lewis Center, OH 43035  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "location = \"./Module_0/datasets/gradedata.csv\"\n",
    "df = pd.read_csv(location)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning by Standard Deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fname</th>\n",
       "      <th>lname</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>exercise</th>\n",
       "      <th>hours</th>\n",
       "      <th>grade</th>\n",
       "      <th>address</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Marcia</td>\n",
       "      <td>Pugh</td>\n",
       "      <td>female</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>82.4</td>\n",
       "      <td>9253 Richardson Road, Matawan, NJ 07747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kadeem</td>\n",
       "      <td>Morrison</td>\n",
       "      <td>male</td>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>78.2</td>\n",
       "      <td>33 Spring Dr., Taunton, MA 02780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nash</td>\n",
       "      <td>Powell</td>\n",
       "      <td>male</td>\n",
       "      <td>18</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>79.3</td>\n",
       "      <td>41 Hill Avenue, Mentor, OH 44060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Noelani</td>\n",
       "      <td>Wagner</td>\n",
       "      <td>female</td>\n",
       "      <td>14</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>83.2</td>\n",
       "      <td>8839 Marshall St., Miami, FL 33125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Noelani</td>\n",
       "      <td>Cherry</td>\n",
       "      <td>female</td>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>87.4</td>\n",
       "      <td>8304 Charles Rd., Lewis Center, OH 43035</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     fname     lname  gender  age  exercise  hours  grade  \\\n",
       "0   Marcia      Pugh  female   17         3     10   82.4   \n",
       "1   Kadeem  Morrison    male   18         4      4   78.2   \n",
       "2     Nash    Powell    male   18         5      9   79.3   \n",
       "3  Noelani    Wagner  female   14         2      7   83.2   \n",
       "4  Noelani    Cherry  female   18         4     15   87.4   \n",
       "\n",
       "                                    address  \n",
       "0   9253 Richardson Road, Matawan, NJ 07747  \n",
       "1          33 Spring Dr., Taunton, MA 02780  \n",
       "2          41 Hill Avenue, Mentor, OH 44060  \n",
       "3        8839 Marshall St., Miami, FL 33125  \n",
       "4  8304 Charles Rd., Lewis Center, OH 43035  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#still using gradedata.csv\n",
    "#check df.head() if needed\n",
    "\n",
    "#Standard Deviation Method\n",
    "\n",
    "meangrade = df['grade'].mean()\n",
    "stdgrade = df['grade'].std()\n",
    "toprange = meangrade + stdgrade * 1.96  # why is it being done at 1.96 and not another number?\n",
    "botrange = meangrade - stdgrade * 1.96  \n",
    "\n",
    "# the forumla for such an activity is:  (Mean + 2*SD) and any points below (Mean - 2*SD) before plotting the frequencies.\n",
    "# https://www.kdnuggets.com/2017/02/removing-outliers-standard-deviation-python.html\n",
    "\n",
    "newdf = df.copy() #to not mess up the original df\n",
    "newdf = newdf.drop(newdf[newdf['grade'] > toprange].index)\n",
    "newdf = newdf.drop(newdf[newdf['grade'] < botrange].index)\n",
    "\n",
    "newdf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning by Quartile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fname</th>\n",
       "      <th>lname</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>exercise</th>\n",
       "      <th>hours</th>\n",
       "      <th>grade</th>\n",
       "      <th>address</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Marcia</td>\n",
       "      <td>Pugh</td>\n",
       "      <td>female</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>82.4</td>\n",
       "      <td>9253 Richardson Road, Matawan, NJ 07747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kadeem</td>\n",
       "      <td>Morrison</td>\n",
       "      <td>male</td>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>78.2</td>\n",
       "      <td>33 Spring Dr., Taunton, MA 02780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nash</td>\n",
       "      <td>Powell</td>\n",
       "      <td>male</td>\n",
       "      <td>18</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>79.3</td>\n",
       "      <td>41 Hill Avenue, Mentor, OH 44060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Noelani</td>\n",
       "      <td>Wagner</td>\n",
       "      <td>female</td>\n",
       "      <td>14</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>83.2</td>\n",
       "      <td>8839 Marshall St., Miami, FL 33125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Noelani</td>\n",
       "      <td>Cherry</td>\n",
       "      <td>female</td>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>87.4</td>\n",
       "      <td>8304 Charles Rd., Lewis Center, OH 43035</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     fname     lname  gender  age  exercise  hours  grade  \\\n",
       "0   Marcia      Pugh  female   17         3     10   82.4   \n",
       "1   Kadeem  Morrison    male   18         4      4   78.2   \n",
       "2     Nash    Powell    male   18         5      9   79.3   \n",
       "3  Noelani    Wagner  female   14         2      7   83.2   \n",
       "4  Noelani    Cherry  female   18         4     15   87.4   \n",
       "\n",
       "                                    address  \n",
       "0   9253 Richardson Road, Matawan, NJ 07747  \n",
       "1          33 Spring Dr., Taunton, MA 02780  \n",
       "2          41 Hill Avenue, Mentor, OH 44060  \n",
       "3        8839 Marshall St., Miami, FL 33125  \n",
       "4  8304 Charles Rd., Lewis Center, OH 43035  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Interquartile Range Method\n",
    "\n",
    "q1 = df['grade'].quantile(.25)\n",
    "q3 = df['grade'].quantile(.75)\n",
    "iqr = q3-q1\n",
    "toprange = q3 + iqr * 1.5\n",
    "botrange = q1 - iqr * 1.5\n",
    "\n",
    "newdf2 = df.copy()\n",
    "newdf2 = newdf.drop(newdf[newdf['grade'] > toprange].index)\n",
    "newdf2 = newdf.drop(newdf[newdf['grade'] < botrange].index)\n",
    "\n",
    "newdf2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## <center>  Feature Scaling\n",
    "Guide found at | https://github.com/Cameron-Cn/AdvDataCourse-2019/blob/master/FeatureScaling.ipynb\n",
    "    \n",
    "Project at | "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}