Chapter 09 - Hyper-Parameter Tuning with Cross-Validation.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as mpl\n",
    "\n",
    "from collections import defaultdict\n",
    "from functools import reduce\n",
    "from path import Path\n",
    "from pprint import pprint\n",
    "\n",
    "%matplotlib inline\n",
    "mpl.style.use('ggplot')\n",
    "mpl.rcParams['figure.figsize'] = 16,6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "from scipy.stats import rv_continuous, kstest\n",
    "from cv import PurgedKFold\n",
    "\n",
    "# Code from Chapter 9\n",
    "\n",
    "class TheNewPipe(Pipeline):\n",
    "    def fit(self, X, y, sample_weight=None, **fit_params):\n",
    "        if sample_weight is not None:\n",
    "            fit_params[self.steps[-1][0] + '__sample_weight'] = sample_weight\n",
    "        return super(TheNewPipe, self).fit(X, y, **fit_params)\n",
    "\n",
    "def clfHyperFit(feat, lbl, t1, pipe_clf, param_grid, cv=3, bagging=[0, None, 1.0],\n",
    "                rndSearchIter=0, n_jobs=-1, pctEmbargo=0, **fit_params):\n",
    "    if set(lbl.values) == {0, 1}:\n",
    "        scoring = 'f1' # f1 for meta-labeling\n",
    "    else:\n",
    "        scoring = 'neg_log_loss' # symmetric towards all classes\n",
    "    \n",
    "    # 1) hyperparameter searching, on train data\n",
    "    inner_cv = PurgedKFold(n_splits=cv, t1=t1, pctEmbargo=pctEmbargo)\n",
    "    if rndSearchIter == 0:\n",
    "        gs = GridSearchCV(estimator=pipe_clf, param_grid=param_grid, scoring=scoring, cv=inner_cv, n_jobs=n_jobs, iid=False)\n",
    "    else:\n",
    "        gs = RandomizedSearchCV(estimator=pipe_clf, param_distributions=param_grid, scoring=scoring, cv=inner_cv, n_jobs=n_jobs, iid=False, n_iter=rndSearchIter)\n",
    "    gs = gs.fit(feat, lbl, **fit_params).best_estimator_\n",
    "    # 2) fit validated model on the entirety of the data\n",
    "    if bagging[1] > 0:\n",
    "        gs = BaggingClassifier(bare_estimator=TheNewPipe(gs.steps), n_estimators=int(bagging[0]), max_samples=float(bagging[1]),\n",
    "                              max_features=float(bagging[2]), n_jobs=n_jobs)\n",
    "        gs = gs.fit(feat, lbl, sample_weight=fit_params[gs.base_estimator.steps[-1][0] + '__sample_weight'])\n",
    "        gs = Pipeline([('bag', gs)])\n",
    "    return gs\n",
    "        \n",
    "class logUniform_gen(rv_continuous):\n",
    "    # random numbers log-uniformly distributed between 1 and e\n",
    "    def _cdf(self, x):\n",
    "        return np.log(x / self.a) / np.log(self.b / self.a)\n",
    "    \n",
    "def logUniform(a=1, b=np.exp(1)):\n",
    "    return logUniform_gen(a=a, b=b, name='logUniform')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.1a\n",
    "\n",
    "Using the function `getTestData` from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise. \n",
    "\n",
    "Use `GridSearchCV` on 10-fold-CV to find the `C, gamma` optimal hyper-parameters on a SVC with RBF kernel, where `param_grid={'C': [1E-2, 1E-1, 1, 10, 100], 'gamma': [1E-2, 1E-1, 1, 10, 100]}` and the scoring function is `neg_log_loss`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "from feature_imp import getTestData\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "param_grid = {'C': [1e-2, 1e-1, 1, 10, 100], 'gamma': [1e-2, 1e-1, 1, 10, 100]}\n",
    "\n",
    "testing = False\n",
    "n_samples = 1000 if testing else 10000\n",
    "n_splits = 3 if testing else 10\n",
    "\n",
    "trnsX, cont = getTestData(n_features=10, n_informative=5, n_redundant=0, n_samples=n_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pipe_clf = SVC(probability=True)\n",
    "\n",
    "inner_cv = PurgedKFold(n_splits=n_splits, t1=cont.index.to_series())\n",
    "gs1 = GridSearchCV(estimator=pipe_clf, param_grid=param_grid, scoring='neg_log_loss', cv=inner_cv,\n",
    "                   n_jobs=-1, iid=False, return_train_score=True)\n",
    "\n",
    "gs1 = gs1.fit(X=trnsX, y=cont['bin'])\n",
    "gs1_results = pd.DataFrame(gs1.cv_results_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.1b\n",
    "\n",
    "Using the function `getTestData` from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise. \n",
    "\n",
    "How many nodes are there in the grid?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.1c\n",
    "\n",
    "Using the function `getTestData` from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise. \n",
    "\n",
    "How many fits did it take to find the optimal solution?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It took 25 fits\n"
     ]
    }
   ],
   "source": [
    "print(\"It took %s fits\" % len(gs1_results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.1d\n",
    "\n",
    "Using the function `getTestData` from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise. \n",
    "\n",
    "How long did it take to find this solution?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It took 328 seconds.\n"
     ]
    }
   ],
   "source": [
    "print(\"It took {:.0f} seconds.\".format(gs1_results['mean_fit_time'].sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.1e\n",
    "\n",
    "Using the function `getTestData` from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise. \n",
    "\n",
    "How can you access the optimal result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=10, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma=0.1, kernel='rbf',\n",
       "  max_iter=-1, probability=True, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "be1 = gs1.best_estimator_\n",
    "be1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.1f\n",
    "\n",
    "Using the function `getTestData` from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise. \n",
    "\n",
    "What is the CV score of the optimal parameter combination?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>16</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean_test_score</th>\n",
       "      <td>-0.288744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>params</th>\n",
       "      <td>{'C': 10, 'gamma': 0.1}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      16\n",
       "mean_test_score                -0.288744\n",
       "params           {'C': 10, 'gamma': 0.1}"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best1_idx = gs1_results['mean_test_score'].idxmax()\n",
    "best1 = gs1_results['mean_test_score'].max()\n",
    "gs1_results.iloc[best1_idx][['mean_test_score', 'params']].to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The CV score is -0.289\n"
     ]
    }
   ],
   "source": [
    "print(\"The CV score is {:.3f}\".format(best1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.1g\n",
    "\n",
    "Using the function `getTestData` from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise. \n",
    "\n",
    "How can you pass sample weights to the SVC?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.2a\n",
    "\n",
    "Using the same dataset from exercise 1,\n",
    "\n",
    "Use `RandomizedSearchCV` on 10-fold-CV to find the `C, gamma` optimal hyper-parameters on a SVC with RBF kernel, where `param_distributions={'C': logUniform(a=1E-2, b=1E2), 'gamma': logUniform(a=1E-2, b=1E2)}, n_iter=25` and the scoring function is `neg_log_loss`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "inner_cv = PurgedKFold(n_splits=n_splits, t1=cont.index.to_series())\n",
    "param_distributions = {'C': logUniform(a=1e-2, b=1e2), 'gamma': logUniform(a=1e-2, b=1e2)}\n",
    "n_iter = 25\n",
    "gs2 = RandomizedSearchCV(estimator=pipe_clf, param_distributions=param_distributions, scoring='neg_log_loss',\n",
    "                         cv=inner_cv, n_jobs=-1, iid=False, n_iter=n_iter, return_train_score=True)\n",
    "\n",
    "gs2 = gs2.fit(X=trnsX, y=cont['bin'])\n",
    "gs2_results = pd.DataFrame(gs2.cv_results_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.2b\n",
    "\n",
    "Using the same dataset from exercise 1,\n",
    "\n",
    "How long did it take to find this solution?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It took 328 seconds.\n"
     ]
    }
   ],
   "source": [
    "print(\"It took {:.0f} seconds.\".format(gs1_results['mean_fit_time'].sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.2c\n",
    "\n",
    "Using the same dataset from exercise 1,\n",
    "\n",
    "Is the optimal parameter combination similar to the one found in exercise 1?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=10.7875109732391, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma=0.07549547952136182,\n",
       "  kernel='rbf', max_iter=-1, probability=True, random_state=None,\n",
       "  shrinking=True, tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "be2 = gs2.best_estimator_\n",
    "be2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.2d\n",
    "\n",
    "Using the same dataset from exercise 1,\n",
    "\n",
    "What is the CV score of the optimal parameter combination? How does it compare to the CV score from exercise 1?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The CV score is -0.282, and therefore higher than -0.289 from the first exercise.\n"
     ]
    }
   ],
   "source": [
    "best2 = gs2_results['mean_test_score'].max()\n",
    "print(\"The CV score is {:.3f}, and therefore higher than {:.3f} from the first exercise.\".format(best2, best1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.3a\n",
    "\n",
    "From exercise 1,\n",
    "\n",
    "Compute the Sharpe ratio of the resulting in-sample forecasts, from point 1.a "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Sharpe ratio is 0.85.\n"
     ]
    }
   ],
   "source": [
    "def sharpe(r):\n",
    "    return r.mean() / r.std()\n",
    "\n",
    "predictions1 = be1.predict(trnsX)\n",
    "bin_returns = cont['bin'] * 2 - 1\n",
    "\n",
    "print(\"The Sharpe ratio is {:.2f}.\".format(sharpe(predictions1 * bin_returns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.3b\n",
    "\n",
    "From exercise 1,\n",
    "\n",
    "Repeat point 1.a, this time with `accuracy` as the scoring function. Compute the in-sample forecasts derived from the hyper-tuned parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Sharpe ratio is 0.85.\n"
     ]
    }
   ],
   "source": [
    "inner_cv = PurgedKFold(n_splits=10, t1=cont.index.to_series())\n",
    "gs3 = GridSearchCV(estimator=pipe_clf, param_grid=param_grid, scoring='accuracy', cv=inner_cv,\n",
    "                   n_jobs=-1, iid=False, return_train_score=True)\n",
    "\n",
    "gs3 = gs3.fit(X=trnsX, y=cont['bin'])\n",
    "gs3_results = pd.DataFrame(gs3.cv_results_)\n",
    "be3 = gs3.best_estimator_\n",
    "\n",
    "predictions3 = be3.predict(trnsX)\n",
    "\n",
    "print(\"The Sharpe ratio is {:.2f}.\".format(sharpe(predictions3 * bin_returns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.3c\n",
    "\n",
    "What scoring method leads to higher (in-sample) Sharpe ratio?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: In this instance GridSearchCV with either accuracy or neg_log_loss picks the same set of parameters.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.4a\n",
    "\n",
    "From exercise 2,\n",
    "\n",
    "Compute the Sharpe ratio of the resulting in-sample forecasts, from point 2.a "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Sharpe ratio is 0.81.\n"
     ]
    }
   ],
   "source": [
    "predictions2 = be2.predict(trnsX)\n",
    "\n",
    "print(\"The Sharpe ratio is {:.2f}.\".format(sharpe(predictions2 * bin_returns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.4b\n",
    "\n",
    "From exercise 2,\n",
    "\n",
    "Repeat point 2.a, this time with `accuracy` as the scoring function. Compute the in-sample forecasts derived from the hyper-tuned parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Sharpe ratio is 0.78.\n"
     ]
    }
   ],
   "source": [
    "gs4 = RandomizedSearchCV(estimator=pipe_clf, param_distributions=param_distributions, scoring='accuracy', cv=inner_cv, n_jobs=-1, iid=False, n_iter=n_iter)\n",
    "\n",
    "gs4 = gs4.fit(X=trnsX, y=cont['bin'])\n",
    "be4 = gs4.best_estimator_\n",
    "\n",
    "predictions4 = be4.predict(trnsX)\n",
    "\n",
    "sharpe(predictions4 * bin_returns)\n",
    "print(\"The Sharpe ratio is {:.2f}.\".format(sharpe(predictions4 * bin_returns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.4c\n",
    "\n",
    "From exercise 2,\n",
    "\n",
    "What scoring method leads to higher (in-sample) Sharpe ratio?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: For randomized search, negative log-loss leads to higher in-sample Sharpe ratio.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.5a\n",
    "\n",
    "Read the definition of log loss, $L[Y,P]$.\n",
    "\n",
    "Why is the scoring function `neg_log_loss` defined as the negative log loss, $-L[Y,P]$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: Because for most it's more intuitive to maximize a scoring function.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.5b\n",
    "\n",
    "Read the definition of log loss, $L[Y,P]$.\n",
    "\n",
    "What would be the outcome of maximizing the log loss, rather than the negitive log loss?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: I'd expect this to select for the model with the least predictive power.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.6\n",
    "\n",
    "Consider an investment strategy that sizes its bets equally, regardless of the forecast's confidence. In this case, what is the more appropriate scoring function for hyper-parameter tuning, accuracy or cross-entropy loss?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: Accuracy accounts equally for erronous predictions with high or low probabilities while Log loss computes the log-likelihood of the classifier given the true label, which takes predictions' probabilities into account.**\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}