Python codes..ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "9a02f0d4-28b6-4ffe-9411-7de921443255",
    "_uuid": "3b442fc985940c2f992c2ab74e89e42d2937987e"
   },
   "source": [
    "\n",
    "\n",
    "**Understanding the data:**\n",
    "\n",
    "Let us import the dataset and have a sneak peak at what kind of data is present inside.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "a1ca8b83-fb5c-4c5b-80ac-a796ef30cb43",
    "_uuid": "a5c491f5b30e222e796a6c9252ffc25241418bde",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn import model_selection, preprocessing, metrics\n",
    "import xgboost as xgb\n",
    "import lightgbm as lgb\n",
    "from haversine import haversine\n",
    "color = sns.color_palette()\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "pd.options.mode.chained_assignment = None  # default='warn'\n",
    "pd.set_option('display.max_columns', 500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "a0988646-8018-4f27-8ad2-280bffe35484",
    "_uuid": "e99a048fed1beeb78a3d51a6248600f75318bf89"
   },
   "source": [
    "Reading the dataset into pandas dataframe and looking at the top few rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "b44c4969-632f-4cbd-a45b-15e56558a216",
    "_uuid": "f5540243de6ee69edfbca9f24516bfab73e39e8f"
   },
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('../input/nyc-taxi-trip-duration/train.csv', parse_dates=['pickup_datetime'])\n",
    "test_df = pd.read_csv('../input/nyc-taxi-trip-duration/test.csv', parse_dates=['pickup_datetime'])\n",
    "print(\"Train dataframe shape : \",train_df.shape)\n",
    "print(\"Test dataframe shape : \", test_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "3fe43711-80d6-44e7-a62c-4d4fe22e0d10",
    "_uuid": "5e647522cb09d03b353ef1dc79ac15459953edf4"
   },
   "outputs": [],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "3e40d0f5-4997-48aa-952e-13463d253b51",
    "_uuid": "34619f804ca63363e4039118e4f09769a7b2548a"
   },
   "outputs": [],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "8666afd1-ba50-4942-8b41-af019666bf47",
    "_uuid": "30038ba73a09e2c07b11b43b4bd69e138bc96bfd"
   },
   "source": [
    "The columns are self-explanatory and two columns 'dropoff_datetime' and 'trip_duration' are not present in the test set. \n",
    "\n",
    "'trip_duration' is the column to predict and Root Mean Square Logarithmic Error is our error metric. So let us look at the log distribution of the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "ed20cd36-79f5-4ead-9e63-b22d1e443488",
    "_uuid": "016564f3deda81afe1fa43b14faaed41cfe31cde"
   },
   "outputs": [],
   "source": [
    "train_df['log_trip_duration'] = np.log1p(train_df['trip_duration'].values)\n",
    "\n",
    "plt.figure(figsize=(8,6))\n",
    "plt.scatter(range(train_df.shape[0]), np.sort(train_df.log_trip_duration.values))\n",
    "plt.xlabel('index', fontsize=12)\n",
    "plt.ylabel('log_trip_duration', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "0679cc00-b022-4105-ba09-84ba4c4e1c54",
    "_uuid": "924d164d7128b07648734c843319a9ca438e0d4b"
   },
   "source": [
    "Looks like there are few outiers in the data. Let us check  the actual count of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "fed5432c-2eee-44d5-a795-0db672c73f7e",
    "_uuid": "6b0d0ca5061cef911172b0e6e45f39e5660ced1f"
   },
   "outputs": [],
   "source": [
    "(train_df['log_trip_duration'] > 12).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "dd312e03-c4bc-421e-961e-9bef74ef1208",
    "_uuid": "9cc84ed972bc1a00bdd63841666e7a51b5e0edce"
   },
   "source": [
    "I think 4 is a smaller value and let us not worry about it now (but probably need to check later if needed).\n",
    "\n",
    "Next step is to check whether there are any null values in the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "5d84cfea-e81a-45e4-9a8e-35e183df4500",
    "_uuid": "91138503734a0c80322af1d133411fa456f5e948"
   },
   "outputs": [],
   "source": [
    "null_count_df = train_df.isnull().sum(axis=0).reset_index()\n",
    "null_count_df.columns = ['col_name', 'null_count']\n",
    "null_count_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "a69dd2c0-0988-4b88-bcce-516127a19788",
    "_uuid": "b659f9b4a95d2386c49c169177f5ac9a409ef97f"
   },
   "outputs": [],
   "source": [
    "null_count_df = test_df.isnull().sum(axis=0).reset_index()\n",
    "null_count_df.columns = ['col_name', 'null_count']\n",
    "null_count_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "d0a0b2ff-37b6-4c0b-981c-edfd3a6f6f8e",
    "_uuid": "6d9cb1b6885480175e0f20921760b4de91abcfa7"
   },
   "source": [
    "There are no missing values :)\n",
    "\n",
    "**Validation Strategy:**\n",
    "\n",
    "Validation strategy is very important because without a proper validation starategy, it will be very hard to evaluate the models against each other. \n",
    "\n",
    "Since dates are given as part of the dataset, it is essential to check whether the train and test datasets are from the same time period or different time period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "0fcda763-ca87-48ba-b6f4-102f37b6a5f3",
    "_uuid": "a82c97ae9cce1f1a7a84bad3356122ed00fc778b"
   },
   "outputs": [],
   "source": [
    "train_df['pickup_date'] = train_df['pickup_datetime'].dt.date\n",
    "test_df['pickup_date'] = test_df['pickup_datetime'].dt.date\n",
    "\n",
    "cnt_srs = train_df['pickup_date'].value_counts()\n",
    "plt.figure(figsize=(12,4))\n",
    "ax = plt.subplot(111)\n",
    "ax.bar(cnt_srs.index, cnt_srs.values, alpha=0.8)\n",
    "ax.xaxis_date()\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "e2268da6-6c24-4b60-90e0-dabd50894823",
    "_uuid": "1e35fce4c861bd514421c58885160ec65c80cfd8"
   },
   "outputs": [],
   "source": [
    "cnt_srs = test_df['pickup_date'].value_counts()\n",
    "plt.figure(figsize=(12,4))\n",
    "ax = plt.subplot(111)\n",
    "ax.bar(cnt_srs.index, cnt_srs.values, alpha=0.8)\n",
    "ax.xaxis_date()\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "906c18f2-67f8-4cee-8fdc-d9bcaaf0d00c",
    "_uuid": "ed8ccab4827f7a7ee3b54da694e2b68876e50eff"
   },
   "source": [
    "Wow. The distributions are very similar and so we could potentially use the K-fold cross validation for our dataset. Please note that if the train and test datasets are from different time frames, kindly use time based validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "8108e117-7668-4115-94a1-c1aed2e5e6b9",
    "_uuid": "7500f2fc0589cfbc431059b95e24211e92a2bd7e"
   },
   "source": [
    "**Baseline Model:**\n",
    "\n",
    "Now that we have got an idea about the dataset, let us buid a baseline model using xgboost and check the performance. \n",
    "\n",
    "We could create few basic variables from datetime column and convert the store_and_forward_flag to numeric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "7a66ff11-380e-4565-b7f3-13d7de26f6f9",
    "_uuid": "1b8237c2d3379eae6231069a7ebb9032de08762d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# day of the month #\n",
    "train_df['pickup_day'] = train_df['pickup_datetime'].dt.day\n",
    "test_df['pickup_day'] = test_df['pickup_datetime'].dt.day\n",
    "\n",
    "# month of the year #\n",
    "train_df['pickup_month'] = train_df['pickup_datetime'].dt.month\n",
    "test_df['pickup_month'] = test_df['pickup_datetime'].dt.month\n",
    "\n",
    "# hour of the day #\n",
    "train_df['pickup_hour'] = train_df['pickup_datetime'].dt.hour\n",
    "test_df['pickup_hour'] = test_df['pickup_datetime'].dt.hour\n",
    "\n",
    "# Week of year #\n",
    "train_df[\"week_of_year\"] = train_df[\"pickup_datetime\"].dt.weekofyear\n",
    "test_df[\"week_of_year\"] = test_df[\"pickup_datetime\"].dt.weekofyear\n",
    "\n",
    "# Day of week #\n",
    "train_df[\"day_of_week\"] = train_df[\"pickup_datetime\"].dt.weekday\n",
    "test_df[\"day_of_week\"] = test_df[\"pickup_datetime\"].dt.weekday\n",
    "\n",
    "# Convert to numeric #\n",
    "map_dict = {'N':0, 'Y':1}\n",
    "train_df['store_and_fwd_flag'] = train_df['store_and_fwd_flag'].map(map_dict)\n",
    "test_df['store_and_fwd_flag'] = test_df['store_and_fwd_flag'].map(map_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "e01cf615-64e9-4904-a72d-bd8e034e4932",
    "_uuid": "fd173e82a8f7f8498af57d367486fc0004303fbc",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop off the variables which are not needed #\n",
    "cols_to_drop = ['id', 'pickup_datetime', 'pickup_date']\n",
    "train_id = train_df['id'].values\n",
    "test_id = test_df['id'].values\n",
    "train_y = train_df.log_trip_duration.values\n",
    "train_X = train_df.drop(cols_to_drop + ['dropoff_datetime', 'trip_duration', 'log_trip_duration'], axis=1)\n",
    "test_X = test_df.drop(cols_to_drop, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "4ff320c5-2473-4083-a848-577fe13e66ba",
    "_uuid": "1af7fd1ae917d937c29cef84677af663ea4dec00"
   },
   "source": [
    "Let us write a helper function to run the xgboost model and light gbm model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "129e6b38-058a-47e6-a582-3a3d49aa6d5c",
    "_uuid": "36ac0b129246a8b1b8a94081b893b52942cffa10",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def runXGB(train_X, train_y, val_X, val_y, test_X, eta=0.05, max_depth=5, min_child_weight=1, subsample=0.8, colsample=0.7, num_rounds=8000, early_stopping_rounds=50, seed_val=2017):\n",
    "    params = {}\n",
    "    params[\"objective\"] = \"reg:linear\"\n",
    "    params['eval_metric'] = \"rmse\"\n",
    "    params[\"eta\"] = eta\n",
    "    params[\"min_child_weight\"] = min_child_weight\n",
    "    params[\"subsample\"] = subsample\n",
    "    params[\"colsample_bytree\"] = colsample\n",
    "    params[\"silent\"] = 1\n",
    "    params[\"max_depth\"] = max_depth\n",
    "    params[\"seed\"] = seed_val\n",
    "    params[\"nthread\"] = -1\n",
    "\n",
    "    plst = list(params.items())\n",
    "    xgtrain = xgb.DMatrix(train_X, label=train_y)\n",
    "    xgval = xgb.DMatrix(val_X, label = val_y)\n",
    "    xgtest = xgb.DMatrix(test_X)\n",
    "    watchlist = [ (xgtrain,'train'), (xgval, 'test') ]\n",
    "    model = xgb.train(plst, xgtrain, num_rounds, watchlist, early_stopping_rounds=early_stopping_rounds, verbose_eval=20)\n",
    "\n",
    "    pred_val = model.predict(xgval, ntree_limit=model.best_ntree_limit)\n",
    "    pred_test = model.predict(xgtest, ntree_limit=model.best_ntree_limit)\n",
    "\n",
    "    return pred_val, pred_test\n",
    "\n",
    "def runLGB(train_X, train_y, val_X, val_y, test_X, eta=0.05, max_depth=5, min_child_weight=1, subsample=0.8, colsample=0.7, num_rounds=8000, early_stopping_rounds=50, seed_val=2017):\n",
    "    params = {}\n",
    "    params[\"objective\"] = \"regression\"\n",
    "    params['metric'] = \"l2_root\"\n",
    "    params[\"learning_rate\"] = eta\n",
    "    params[\"min_child_weight\"] = min_child_weight\n",
    "    params[\"bagging_fraction\"] = subsample\n",
    "    params[\"bagging_seed\"] = seed_val\n",
    "    params[\"feature_fraction\"] = colsample\n",
    "    params[\"verbosity\"] = 0\n",
    "    params[\"max_depth\"] = max_depth\n",
    "    params[\"nthread\"] = -1\n",
    "\n",
    "    lgtrain = lgb.Dataset(train_X, label=train_y)\n",
    "    lgval = lgb.Dataset(val_X, label = val_y)\n",
    "    model = lgb.train(params, lgtrain, num_rounds, valid_sets=lgval, early_stopping_rounds=early_stopping_rounds, verbose_eval=20)\n",
    "\n",
    "    pred_val = model.predict(val_X, num_iteration=model.best_iteration)\n",
    "    pred_test = model.predict(test_X, num_iteration=model.best_iteration)\n",
    "\n",
    "    return pred_val, pred_test, model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "721477f4-70fe-49aa-ab48-b53859daecac",
    "_uuid": "8a32febd905c631408e9deba932661db7ee07379"
   },
   "source": [
    "Now let us build the baseline model using K-fold cross validation and save the scores in a csv file so as to build ensembles / stacking models later.\n",
    "\n",
    "Please increase the number of rounds ('num_rounds') to a high value (1000) and then run the model in local. Using just 10 rounds here "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "705ffd18-1649-48d6-9a4d-535d293c54db",
    "_uuid": "c4c215237c7a8086f8f1b8ae4c23fe66bc8e86e3"
   },
   "outputs": [],
   "source": [
    "# Increase the num_rounds parameter to a higher value (1000) and run the model #\n",
    "kf = model_selection.KFold(n_splits=5, shuffle=True, random_state=2017)\n",
    "cv_scores = []\n",
    "pred_test_full = 0\n",
    "pred_val_full = np.zeros(train_df.shape[0])\n",
    "for dev_index, val_index in kf.split(train_X):\n",
    "    dev_X, val_X = train_X.ix[dev_index], train_X.ix[val_index]\n",
    "    dev_y, val_y = train_y[dev_index], train_y[val_index]\n",
    "    pred_val, pred_test, model = runLGB(dev_X, dev_y, val_X, val_y, test_X, num_rounds=5, max_depth=8, eta=0.3)\n",
    "    pred_val_full[val_index] = pred_val\n",
    "    pred_test_full += pred_test\n",
    "    cv_scores.append(np.sqrt(metrics.mean_squared_error(val_y, pred_val)))\n",
    "print(cv_scores)\n",
    "print(\"Mean CV score : \",np.mean(cv_scores))\n",
    "\n",
    "pred_test_full = pred_test_full / 5.\n",
    "pred_test_full = np.expm1(pred_test_full)\n",
    "pred_val_full = np.expm1(pred_val_full)\n",
    "\n",
    "# saving train predictions for ensemble #\n",
    "train_pred_df = pd.DataFrame({'id':train_id})\n",
    "train_pred_df['trip_duration'] = pred_val_full\n",
    "train_pred_df.to_csv(\"train_preds_lgb_baseline.csv\", index=False)\n",
    "\n",
    "# saving test predictions for ensemble #\n",
    "test_pred_df = pd.DataFrame({'id':test_id})\n",
    "test_pred_df['trip_duration'] = pred_test_full\n",
    "test_pred_df.to_csv(\"test_preds_lgb_baseline.csv\", index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "7151a0d7-9825-45e3-a5c5-1f1280294d14",
    "_uuid": "eae0e10201b25197b3dda281326bf53d4d2d8d4d",
    "collapsed": true
   },
   "source": [
    "**The mean cv score (when trained offline with 1000 rounds) is 0.4147 and the LB score is 0.41412 (around rank 400 which is not bad for a baseline model)**. \n",
    "\n",
    "This baseline model has helped us understand the following:\n",
    "1. Our overall setup is ready so that now we can make additional changes wherever needed\n",
    "2. The scores seem to be matching between cross validation and leaderboard and so we are probably good in that front\n",
    "\n",
    "So some of the next steps are as follows:\n",
    "1. Feature engineering to create more useful variables.\n",
    "2. Ascertain that the cross validation and LB scores are in sync.\n",
    "3. Parameter tuning and building varied models.\n",
    "4. Ensembling / Stacking\n",
    "\n",
    "**Feature Engineering:**\n",
    "\n",
    "Now that we have our base model and the overall setup ready, let us dive into creating more variables (This is where I generally spend most of my time during competitions). It is a good idea to look at the feature importances of the model which we have built to understand what type of features are generally more predictive. So I got the feature importances from the light gbm model and is as follows:\n",
    "\n",
    "| Feature Name | Feature Importance |\n",
    "|:----------------|------------------------:|\n",
    "| dropoff_latitude | 0.1761 |\n",
    "| pickup_latitude | 0.1729 | \n",
    "| pickup_longitude | 0.172 |\n",
    "| dropoff_longitude | 0.1581 |\n",
    "| pickup_hour | 0.0999 | \n",
    "| pickup_day | 0.0611 |\n",
    "| week_of_year | 0.0538 |\n",
    "| day_of_week | 0.0499 | \n",
    "| pickup_month | 0.0203 | \n",
    "| passenger_count | 0.0203 | \n",
    "| vendor_id | 0.0132 |\n",
    "| store_and_fwd_flag | 0.0021 |\n",
    "\n",
    "The important variables order seem to be the lat-lon co-ordinates followed by the time based variables. So some of my ideas to create new variables and the reasons are as follows\n",
    "1. Difference between pickup and dropoff latitude - will give an idea about the distance covered which could be predictive\n",
    "2. Difference between pickup and dropoff longitude - same reason as above\n",
    "3. Haversine distance between pickup and dropoff co-ordinates - to capture the actual distance travelled (commented out so as to use a faster function written by beluga)\n",
    "4. Pickup minute - since pickup hour is an important variable, the minute of pickup might well have been predictive\n",
    "5. Pickup day of year - same reason as above\n",
    "\n",
    "So let us create these variables first and re-run it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "8297ff05-10a4-4b9c-af0f-5a8b17338a77",
    "_uuid": "dff776b1bd0727785b4337f04b419418bfda0c87",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# difference between pickup and dropoff latitudes #\n",
    "train_df['lat_diff'] = train_df['pickup_latitude'] - train_df['dropoff_latitude']\n",
    "test_df['lat_diff'] = test_df['pickup_latitude'] - test_df['dropoff_latitude']\n",
    "\n",
    "# difference between pickup and dropoff longitudes #\n",
    "train_df['lon_diff'] = train_df['pickup_longitude'] - train_df['dropoff_longitude']\n",
    "test_df['lon_diff'] = test_df['pickup_longitude'] - test_df['dropoff_longitude']\n",
    "\n",
    "## compute the haversine distance ##\n",
    "#train_df['haversine_distance'] = train_df.apply(lambda row: haversine( (row['pickup_latitude'], row['pickup_longitude']), (row['dropoff_latitude'], row['dropoff_longitude']) ), axis=1)\n",
    "#test_df['haversine_distance'] = test_df.apply(lambda row: haversine( (row['pickup_latitude'], row['pickup_longitude']), (row['dropoff_latitude'], row['dropoff_longitude']) ), axis=1)\n",
    "\n",
    "# get the pickup minute of the trip #\n",
    "train_df['pickup_minute'] = train_df['pickup_datetime'].dt.minute\n",
    "test_df['pickup_minute'] = test_df['pickup_datetime'].dt.minute\n",
    "\n",
    "# get the absolute value of time #\n",
    "train_df['pickup_dayofyear'] = train_df['pickup_datetime'].dt.dayofyear\n",
    "test_df['pickup_dayofyear'] = test_df['pickup_datetime'].dt.dayofyear"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "df797b4b-b122-4df2-930f-78379cbd13c3",
    "_uuid": "4530ed7731b4e3e0ea8a1d9481f5fe121d3f7f93"
   },
   "source": [
    "Now let us re-run the model again with these new variables and check the score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "fde08d71-3d49-4bf4-a203-759948f58eea",
    "_uuid": "e16c980d0f2f5c714585bfc14093666f8771d03e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop off the variables which are not needed #\n",
    "cols_to_drop = ['id', 'pickup_datetime', 'pickup_date']\n",
    "train_id = train_df['id'].values\n",
    "test_id = test_df['id'].values\n",
    "train_y = train_df.log_trip_duration.values\n",
    "train_X = train_df.drop(cols_to_drop + ['dropoff_datetime', 'trip_duration', 'log_trip_duration'], axis=1)\n",
    "test_X = test_df.drop(cols_to_drop, axis=1)\n",
    "\n",
    "# Increase the num_rounds parameter to a higher value (1000) and run the model #\n",
    "kf = model_selection.KFold(n_splits=5, shuffle=True, random_state=2017)\n",
    "cv_scores = []\n",
    "pred_test_full = 0\n",
    "pred_val_full = np.zeros(train_df.shape[0])\n",
    "for dev_index, val_index in kf.split(train_X):\n",
    "    dev_X, val_X = train_X.ix[dev_index], train_X.ix[val_index]\n",
    "    dev_y, val_y = train_y[dev_index], train_y[val_index]\n",
    "    pred_val, pred_test, model = runLGB(dev_X, dev_y, val_X, val_y, test_X, num_rounds=5, max_depth=8, eta=0.3)\n",
    "    pred_val_full[val_index] = pred_val\n",
    "    pred_test_full += pred_test\n",
    "    cv_scores.append(np.sqrt(metrics.mean_squared_error(val_y, pred_val)))\n",
    "print(cv_scores)\n",
    "print(\"Mean CV score : \",np.mean(cv_scores))\n",
    "\n",
    "pred_test_full = pred_test_full / 5.\n",
    "pred_test_full = np.expm1(pred_test_full)\n",
    "pred_val_full = np.expm1(pred_val_full)\n",
    "\n",
    "# saving train predictions for ensemble #\n",
    "train_pred_df = pd.DataFrame({'id':train_id})\n",
    "train_pred_df['trip_duration'] = pred_val_full\n",
    "train_pred_df.to_csv(\"train_preds_lgb.csv\", index=False)\n",
    "\n",
    "# saving test predictions for ensemble #\n",
    "test_pred_df = pd.DataFrame({'id':test_id})\n",
    "test_pred_df['trip_duration'] = pred_test_full\n",
    "test_pred_df.to_csv(\"test_preds_lgb.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "607fdfb0-4995-453a-8269-44bec62b28af",
    "_uuid": "c4a30c2e4e97e75fe18294bd5684994f49fc7343"
   },
   "source": [
    "**The CV score of this new model is 0.3875 and the LB score is 0.38809. **\n",
    "\n",
    "So the scores are pretty much in sync with each other. This is really a great news since in many competitions, they will be far away (in which cases, directional improvement can be looked at).\n",
    "\n",
    "Our new feature importances are as follows:\n",
    "\n",
    "|Feature name | Feature Importance|\n",
    "|:----------------|------------------:|\n",
    "| pickup_longitude | 0.1060 |\n",
    "| dropoff_latitude | 0.1055| \n",
    "| haversine_distance | 0.1007 | \n",
    "| dropoff_longitude | 0.0990 | \n",
    "| pickup_latitude | 0.0983 |\n",
    "| pickup_hour | 0.0890 |\n",
    "| lon_diff | 0.0860 | \n",
    "| lat_diff | 0.0815 | \n",
    "| pickup_dayofyear | 0.0560 | \n",
    "| pickup_minute | 0.0459 |\n",
    "| pickup_day | 0.0433 | \n",
    "| day_of_week | 0.0358 |\n",
    "| week_of_year | 0.0246 |\n",
    "| passenger_count | 0.0110 |\n",
    "| vendor_id | 0.0080 |\n",
    "| pickup_month | 0.0078 |\n",
    "| store_and_fwd_flag | 0.0005 |\n",
    "\n",
    "As the next step, we can look into the forum posts / kernels  and see if there are any good feature ideas / implementations and try to add them as well into the features list.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "6e25fed8-173f-483a-8e67-ebd51a8293cd",
    "_uuid": "12356d75e3423876a327004c754ebcd678cbabe5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def haversine_array(lat1, lng1, lat2, lng2):\n",
    "    lat1, lng1, lat2, lng2 = map(np.radians, (lat1, lng1, lat2, lng2))\n",
    "    AVG_EARTH_RADIUS = 6371  # in km\n",
    "    lat = lat2 - lat1\n",
    "    lng = lng2 - lng1\n",
    "    d = np.sin(lat * 0.5) ** 2 + np.cos(lat1) * np.cos(lat2) * np.sin(lng * 0.5) ** 2\n",
    "    h = 2 * AVG_EARTH_RADIUS * np.arcsin(np.sqrt(d))\n",
    "    return h\n",
    "\n",
    "def bearing_array(lat1, lng1, lat2, lng2):\n",
    "    AVG_EARTH_RADIUS = 6371  # in km\n",
    "    lng_delta_rad = np.radians(lng2 - lng1)\n",
    "    lat1, lng1, lat2, lng2 = map(np.radians, (lat1, lng1, lat2, lng2))\n",
    "    y = np.sin(lng_delta_rad) * np.cos(lat2)\n",
    "    x = np.cos(lat1) * np.sin(lat2) - np.sin(lat1) * np.cos(lat2) * np.cos(lng_delta_rad)\n",
    "    return np.degrees(np.arctan2(y, x))\n",
    "\n",
    "train_df['haversine_distance'] = haversine_array(train_df['pickup_latitude'].values, \n",
    "                                                     train_df['pickup_longitude'].values, \n",
    "                                                     train_df['dropoff_latitude'].values, \n",
    "                                                     train_df['dropoff_longitude'].values)\n",
    "test_df['haversine_distance'] = haversine_array(test_df['pickup_latitude'].values, \n",
    "                                                    test_df['pickup_longitude'].values, \n",
    "                                                    test_df['dropoff_latitude'].values, \n",
    "                                                    test_df['dropoff_longitude'].values)\n",
    "\n",
    "train_df['direction'] = bearing_array(train_df['pickup_latitude'].values, \n",
    "                                          train_df['pickup_longitude'].values, \n",
    "                                          train_df['dropoff_latitude'].values, \n",
    "                                          train_df['dropoff_longitude'].values)\n",
    "test_df['direction'] = bearing_array(test_df['pickup_latitude'].values, \n",
    "                                         test_df['pickup_longitude'].values, \n",
    "                                         test_df['dropoff_latitude'].values, \n",
    "                                         test_df['dropoff_longitude'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "5d932952-bc8b-4253-90de-76615c6ab279",
    "_uuid": "8fb88926f42e3a52b4ca73b79344f2df48d3d806"
   },
   "source": [
    "Also we have this [wonderful osrm dataset](https://www.kaggle.com/oscarleo/new-york-city-taxi-with-osrm) published by oscarleo which gets the fastest and the second fastest routes for both train and test sets. Thank you oscarleo for this very helpful dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "9fed0b51-efdd-4816-8595-8c497d244d3a",
    "_uuid": "7a314c2e229b508a81693001e9dbdd9605733d57"
   },
   "outputs": [],
   "source": [
    "train_fr_part1 = pd.read_csv('../input/new-york-city-taxi-with-osrm/fastest_routes_train_part_1.csv', \n",
    "                             usecols=['id', 'total_distance', 'total_travel_time', 'number_of_steps'])\n",
    "train_fr_part2 = pd.read_csv('../input/new-york-city-taxi-with-osrm/fastest_routes_train_part_2.csv', \n",
    "                             usecols=['id', 'total_distance', 'total_travel_time', 'number_of_steps'])\n",
    "test_fr = pd.read_csv('../input/new-york-city-taxi-with-osrm/fastest_routes_test.csv', \n",
    "                             usecols=['id', 'total_distance', 'total_travel_time', 'number_of_steps'])\n",
    "train_fr = pd.concat((train_fr_part1, train_fr_part2))\n",
    "\n",
    "train_df = train_df.merge(train_fr, how='left', on='id')\n",
    "test_df = test_df.merge(test_fr, how='left', on='id')\n",
    "\n",
    "del train_fr_part1, train_fr_part2, train_fr, test_fr\n",
    "import gc; gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "bd85df18-adf4-4595-842b-687ceacd55f8",
    "_uuid": "10bddd9644ff5f187e672a25e2775f5e17ba1bae",
    "collapsed": true
   },
   "source": [
    "Why just the fastest route alone when the second fastest route is available? I tried adding the second fastest variables but it did not help much in the CV and so did not include them.\n",
    "\n",
    "Let us create few more variables by binning the latitudes and longitudes together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### some more new variables ###\n",
    "train_df['pickup_latitude_round3'] = np.round(train_df['pickup_latitude'],3)\n",
    "test_df['pickup_latitude_round3'] = np.round(test_df['pickup_latitude'],3)\n",
    "\n",
    "train_df['pickup_longitude_round3'] = np.round(train_df['pickup_longitude'],3)\n",
    "test_df['pickup_longitude_round3'] = np.round(test_df['pickup_longitude'],3)\n",
    "\n",
    "train_df['dropoff_latitude_round3'] = np.round(train_df['dropoff_latitude'],3)\n",
    "test_df['dropoff_latitude_round3'] = np.round(test_df['dropoff_latitude'],3)\n",
    "\n",
    "train_df['dropoff_longitude_round3'] = np.round(train_df['dropoff_longitude'],3)\n",
    "test_df['dropoff_longitude_round3'] = np.round(test_df['dropoff_longitude'],3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us run our models again now to check the scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop off the variables which are not needed #\n",
    "cols_to_drop = ['id', 'pickup_datetime', 'pickup_date']\n",
    "train_id = train_df['id'].values\n",
    "test_id = test_df['id'].values\n",
    "train_y = train_df.log_trip_duration.values\n",
    "train_X = train_df.drop(cols_to_drop + ['dropoff_datetime', 'trip_duration', 'log_trip_duration'], axis=1)\n",
    "test_X = test_df.drop(cols_to_drop, axis=1)\n",
    "\n",
    "# Increase the num_rounds parameter to a higher value (1000) and run the model #\n",
    "kf = model_selection.KFold(n_splits=5, shuffle=True, random_state=2017)\n",
    "cv_scores = []\n",
    "pred_test_full = 0\n",
    "pred_val_full = np.zeros(train_df.shape[0])\n",
    "for dev_index, val_index in kf.split(train_X):\n",
    "    dev_X, val_X = train_X.ix[dev_index], train_X.ix[val_index]\n",
    "    dev_y, val_y = train_y[dev_index], train_y[val_index]\n",
    "    pred_val, pred_test, model = runLGB(dev_X, dev_y, val_X, val_y, test_X, num_rounds=5, max_depth=8, eta=0.3)\n",
    "    pred_val_full[val_index] = pred_val\n",
    "    pred_test_full += pred_test\n",
    "    cv_scores.append(np.sqrt(metrics.mean_squared_error(val_y, pred_val)))\n",
    "print(cv_scores)\n",
    "print(\"Mean CV score : \",np.mean(cv_scores))\n",
    "\n",
    "pred_test_full = pred_test_full / 5.\n",
    "pred_test_full = np.expm1(pred_test_full)\n",
    "pred_val_full = np.expm1(pred_val_full)\n",
    "\n",
    "# saving train predictions for ensemble #\n",
    "train_pred_df = pd.DataFrame({'id':train_id})\n",
    "train_pred_df['trip_duration'] = pred_val_full\n",
    "train_pred_df.to_csv(\"train_preds_lgb.csv\", index=False)\n",
    "\n",
    "# saving test predictions for ensemble #\n",
    "test_pred_df = pd.DataFrame({'id':test_id})\n",
    "test_pred_df['trip_duration'] = pred_test_full\n",
    "test_pred_df.to_csv(\"test_preds_lgb.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The CV score of this version is 0.3784 and the LB score is 0.3799. **\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}