Exercise 3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Validation and cross-validation \n",
    "\n",
    "In this exercise you will implement a validation pipeline. \n",
    "\n",
    "At the end of Exercise 2, you tested your model against the training and test datasets. As you should observe, there's a gap between the results. By validating your model, not only should you be able to anticipate the test time performance, but also have a method to compare different models.\n",
    "\n",
    "Implement the basic validation method, i.e. a random split. Test it with your model from Exercise 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "m2, syp, laz, rok, garaz, dzielnica[4]\n[104, 2, 2, 1940, 1, 0, 1, 0, 0]\n(200, 9) (200,)\n"
     ]
    }
   ],
   "source": [
    "####################################################### \n",
    "# TODO: Implement the basic validation method,        # \n",
    "# compare MSLE on training, validation, and test sets #\n",
    "#######################################################\n",
    "import numpy  as np\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from mieszkania import open_dataset\n",
    "\n",
    "xs, ys = open_dataset('mieszkania.csv')\n",
    "\n",
    "print xs.shape, ys.shape\n",
    "txs, vxs, tys, vys = train_test_split(xs, ys, test_size=0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0843325391444\nw: [ 0.67035012  0.1629057   0.14264752  1.19804938  0.05109827  0.09546981\n  0.20954945  0.21935257  0.14597829]\n0.0843326686621\n[107 109 100 108  97  98 108  91  94  81  92  90  96 101  98  99 100 106\n  99  99 102  84  94 100  97  94 106 107 113  83 101 107 106 107 104 104\n 100  98  98  88 102 100 103  80 101  94 100 108  92  99 134  98  98  98\n 107 113 101 104  93 109 104 101  97 118  99 100  88 108 107  92  96 104\n  96 103 100  93  93 106  93  94 105 100  97 140  87  95  95  90  96  99\n  94 104  99  86 103  98  90  99  87 106 100  94 100  98 108 106  99 100\n 110 105  96 130 102 104  96  99  92 101  91  90]\nVal: 0.0878867698745\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEMFJREFUeJzt3X+MXlWdx/H3dzptmYKl3XUpSvkhMYi7YsDsLipu9gmN\n2EhA/1nDsqurm/UvDUazLiB/MP5jljUbNRv3DyNLKsJiZDdaDRsrKY9G1x8oLa20lBoEFewAUqyV\naWmZ7/5xn6Gz43Rm+vy4d3qe9yu5ee69c+49Z06mnzk998dEZiJJKstI0w2QJPWf4S5JBTLcJalA\nhrskFchwl6QCGe6SVKAFwz0ibo2IiYjYMWPfv0TE7ojYHhH/FRGrB9tMSdKJWMzI/TbgbbP2bQH+\nJDMvBvYCN/a7YZKk7i0Y7pn5HWD/rH33ZuZUZ/P7wPoBtE2S1KV+zLn/PfA/fTiPJKlPegr3iLgJ\nOJKZd/apPZKkPhjt9sCIeC/wduDyBcr58hpJ6kJmRrfHLnbkHp2l2ojYCHwUuDozDy90cGa6ZHLz\nzTc33oalstgX9oV9Mf/Sq8XcCnkn8L/ABRHx84h4H/BvwGnANyPigYj4955bIknqmwWnZTLz2jl2\n3zaAtkiS+sQnVGvUarWabsKSYV8cY18cY1/0T/RjbmfeCiJy0HVIUmkigqzhgqok6SRiuEtSgQx3\nSSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJek\nAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgRYM94i4\nNSImImLHjH1rI2JLROyJiG9ExOmDbaYk6UQsZuR+G/C2WftuAO7NzNcAW4Eb+90wSVL3Fgz3zPwO\nsH/W7ncAmzrrm4B39rldkqQedDvnfkZmTgBk5j7gjP41SZLUq35dUM0+nUeS1AejXR43ERHrMnMi\nIs4Enpqv8Pj4+EvrrVaLVqvVZbWSVKZ2u0273e7b+SJz4UF3RJwHfC0zL+ps3wI8m5m3RMT1wNrM\nvOE4x+Zi6pAkHRMRZGZ0ffxCwRsRdwIt4A+BCeBm4CvAl4GzgceBd2Xmc8c53nCXpBM08HDvleEu\nSSeu13D3CVVJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpUS7h7m7sk1auWcD9ypI5aJEnT\nDHdJKpDhLkkFMtwlqUCGuyQVqJZwf+GFOmqRJE1z5C5JBaol3A8frqMWSdI0p2UkqUCGuyQVyHCX\npAIZ7pJUIMNdkgrk3TKSVCBH7pJUIMNdkgpkuEtSgQx3SSqQF1QlqUCO3CWpQIa7JBWop3CPiBsj\n4qGI2BERd0TEirnKGe6SVK+uwz0izgXeD1ySma8HRoFr5ipruEtSvUZ7OPYA8AJwakRMAauAJ+cq\naLhLUr26Hrln5n7gX4GfA08Az2XmvXOV9W4ZSapX1yP3iDgf+DBwLvAb4O6IuDYz75xd9nvfG2d8\nvFpvtVq0Wq1uq5WkIrXbbdrtdt/OF5nZ3YER7wLempnv72y/G7g0Mz84q1y+5z3Jpk09t1WShkZE\nkJnR7fG93C2zB3hjRJwSEQFsAHbPVdA5d0mqVy9z7g8CXwB+DDwIBPC5ucoa7pJUr17uliEzPwl8\ncqFyXlCVpHr5hKokFchwl6QCGe6SVKBawv3QoTpqkSRN833uklQgR+6SVCDDXZIKZLhLUoEMd0kq\nUC3hPjkJXb6fTJLUhVrCfdkyOHKkjpokSVBTuJ9yilMzklSnWsJ9bMxwl6Q6OXKXpAIZ7pJUIMNd\nkgpkuEtSgQx3SSqQ4S5JBTLcJalAtYX75GQdNUmSwJG7JBXJcJekAhnuklQg3y0jSQVy5C5JBTLc\nJalAhrskFaincI+I0yPiyxGxOyIeiohL5ypnuEtSvUZ7PP4zwD2Z+VcRMQqsmquQ4S5J9eo63CNi\nNfAXmflegMw8ChyYq6zhLkn16mVa5lXAMxFxW0Q8EBGfi4ixuQr6+gFJqlcv0zKjwBuAD2TmjyLi\n08ANwM2zC9599zg/+QmMj0Or1aLVavVQrSSVp91u0263+3a+yMzuDoxYB3wvM8/vbL8FuD4zr5pV\nLrduTT7+cehjuyWpaBFBZka3x3c9LZOZE8AvIuKCzq4NwK65yjrnLkn16vVumeuAOyJiOfAo8L65\nChnuklSvnsI9Mx8E/myhcr5bRpLq5ROqklQgw12SCmS4S1KBDHdJKlAt4b5yJbzwAnR5S70k6QTV\nEu4RsGIFHD5cR22SpFrCHXy/jCTVqdZwd95dkurhyF2SClRbuJ92Gvzud3XVJknDrbZwP/VUw12S\n6lLryP3gwbpqk6Th5shdkgrkyF2SCmS4S1KBnJaRpAI5cpekAjlyl6QCOXKXpAIZ7pJUIKdlJKlA\njtwlqUCO3CWpQI7cJalAvvJXkgpU67SMI3dJqofTMpJUoNovqGbWVaMkDa+ewz0iRiLigYjYPF+5\n0VFYvtw/ki1JdejHyP1DwK7FFPR2SEmqR0/hHhHrgbcDn19MeefdJakevY7cPwV8FFjUTLrhLkn1\nGO32wIi4EpjIzO0R0QLieGXHx8cBeO45+Pa3W7zuda1uq5WkIrXbbdrtdt/OF9nl7SsR8Qngb4Gj\nwBjwMuC/M/M9s8rldB2XXw433QQbNvTUZkkqXkSQmccdNC+k62mZzPxYZp6TmecD1wBbZwf7bF5Q\nlaR61HafOzjnLkl16XrOfabM/BbwrYXKGe6SVI9aR+5Oy0hSPWoN99Wr4cCBOmuUpOFUa7ivWQP7\n99dZoyQNp1rDfe3a6l53SdJgOXKXpAI5cpekAtUe7o7cJWnwnJaRpAI5LSNJBar99QOTk3DkSJ21\nStLwqTXcR0bg9NPhN7+ps1ZJGj61hjt4UVWS6lB7uK9Z47y7JA2aI3dJKpDhLkkFclpGkgrkyF2S\nCuTIXZIK5MhdkgpkuEtSgZyWkaQCOXKXpALVHu5nnAFPPVV3rZI0XCIzB1tBRM6sY3Kympo5dAgi\nBlq1JJ20IoLM7Dolax+5j41Vi/PukjQ4tYc7wLp1sG9fEzVL0nBoJNzPPNNwl6RBaizcJyaaqFmS\nhkPX4R4R6yNia0Q8FBE7I+K6xR7ryF2SBmu0h2OPAh/JzO0RcRrw44jYkpkPL3Sg4S5Jg9X1yD0z\n92Xm9s76QWA3cNZijjXcJWmw+jLnHhHnARcDP1hMecNdkgar53DvTMncDXyoM4JfkLdCStJg9TLn\nTkSMUgX77Zn51eOVGx8ff2m91WpxwQUtw12SZmi327Tb7b6dr6fXD0TEF4BnMvMj85TJ2XUcOQKr\nVlWvIFi2rOvqJalYjb1+ICIuA/4GuDwitkXEAxGxcTHHLl9evR3y6ae7rV2SNJ+up2Uy87tA1+Pu\nV74SfvnL6uKqJKm/GnlCFeC88+Dxx5uqXZLK1li4n3uu4S5Jg9LoyP2xx5qqXZLK5shdkgrU6Mj9\nZz9rqnZJKlvtf2Zv2oED8IpXwG9/CyON/YqRpKXppPsze9NWr66WJ59sqgWSVK5Gx8wXXACPPNJk\nCySpTI2H+549TbZAksrUaLhfeCHs3t1kCySpTI2G++tfDzt3NtkCSSpTo+F+0UWwYwcM+IYdSRo6\njYb7unXVK3+9Y0aS+qvRcI+AN7wBfvSjJlshSeVp/PGhSy+FHyzqL69KkhbLcJekAjX2+oFp+/fD\nOefAr38NK1YMtCmSdNI4aV8/MG3tWnjNaxy9S1I/NR7uABs2wJYtTbdCksqxJML9qqtg8+amWyFJ\n5VgS4f6mN8G+ffDTnzbdEkkqw5II92XL4NprYdOmplsiSWVo/G6ZaTt3wsaN8OijsHLlQJskSUve\nSX+3zLSLLqoWR++S1LslM3IHuP9+uPrq6jXAa9YMtFmStKT1OnJfUuEO8IEPwFNPwV13VXPxkjSM\nigv3yUm48ko46yy49VafWpU0nIqZc582NgZf/zo8/3w1B3/HHXDwYNOtkqSTS08j94jYCHya6pfE\nrZl5yxxlTmjkPtM998BnPwv33Qfnnw+vfW31DviXv7xaXvYyWLVq/mVsDJYvr14vLEkni8amZSJi\nBHgE2AA8CdwPXJOZD88q13W4Tzt8GHbtgocfhmeegaefrj4PHqxG+JOT1edcy+QkHD1a3V45cznl\nlONvz/e1lSurqaLly39/GR2de//0sm1bmze/uTVvmWG5ztBut2m1Wk03Y0mwL46xL47pNdxHe6j7\nz4G9mfl4pyF3Ae8AHp73qC6sXAmXXFIt3Ziaqn5BzFwOHTr+9vG+dugQPPccHDmyuOXo0f+//eST\nbdasac17DFS/JEZHYWSkCvv5ln6XGRmplohqmV7v97777mvz3e+2ujrPtOn12Z8n29e+9KU2zz7b\n6umcvVhK5/jiF9tMTrYab0fT5zj77N7b0Eu4nwX8Ysb2L6kCf8kZGammZ8bGmm3H+Hi1zGdqqgr5\nF19ceJma6n+Zqanqb9pOf85c78e+qalj9T3//ImdZ2rqWD9N/2dw9ufJ+LVdu6qBQLfn7MVSO8fe\nvfDss823o+lzXHVV7+foJdw1ACMjw/GE7mJ+0Q0L++IY++KY667r7fhe5tzfCIxn5sbO9g1Azr6o\nGhGDvddSkgrV1AXVZcAeqguqvwJ+CPx1Zu7utjGSpP7oelomM1+MiA8CWzh2K6TBLklLwMCfUJUk\n1W9gT6hGxMaIeDgiHomI6wdVz1IREbdGxERE7Jixb21EbImIPRHxjYg4fcbXboyIvRGxOyKuaKbV\ngxER6yNia0Q8FBE7I+K6zv6h64+IWBkRP4iIbZ3++ERn/9D1xbSIGImIByJic2d7KPsiIh6LiAc7\nPxs/7OzrX19kZt8Xql8aPwXOBZYD24ELB1HXUlmAtwAXAztm7LsF+KfO+vXAP3fW/xjYRjUtdl6n\nr6Lp76GPfXEmcHFn/TSqazMXDnF/rOp8LgO+D1w2rH3R+R4/DHwR2NzZHsq+AB4F1s7a17e+GNTI\n/aUHnDLzCDD9gFOxMvM7wP5Zu98BTL+hfhPwzs761cBdmXk0Mx8D9rJEnxHoRmbuy8ztnfWDwG5g\nPcPbH893VldSDXz2M6R9ERHrgbcDn5+xeyj7Agh+f/akb30xqHCf6wGnswZU11J2RmZOQBV4wBmd\n/bP75wkK7Z+IOI/qfzTfB9YNY390piG2AfuAdmbuYkj7AvgU8FFg5sW+Ye2LBL4ZEfdHxD909vWt\nL3yIqV5DdfU6Ik4D7gY+lJkH53jmYSj6IzOngEsiYjXwjYho8fvfe/F9ERFXAhOZub3TB8dTfF90\nXJaZv4qIPwK2RMQe+vhzMaiR+xPAOTO213f2DZuJiFgHEBFnAk919j8BzHx7RHH9ExGjVMF+e2Z+\ntbN7aPsDIDMPAPcAf8pw9sVlwNUR8Sjwn8DlEXE7sG8I+4LM/FXn82ngK1TTLH37uRhUuN8PvDoi\nzo2IFcA1wOYB1bWURGeZthl4b2f974Cvzth/TUSsiIhXAa+megisJP8B7MrMz8zYN3T9EREvn77j\nISLGgLdSXRgbur7IzI9l5jmZeT5VJmzNzHcDX2PI+iIiVnX+Z0tEnApcAeyknz8XA7wSvJHqLom9\nwA1NX5ke9ALcSfXq48PAz4H3AWuBezv9sAVYM6P8jVRXvHcDVzTd/j73xWXAi1R3SW0DHuj8PPzB\nsPUHcFHn+98GPAj8Y2f/0PXFrH75S47dLTN0fQG8asa/j53TGdnPvvAhJkkq0JL7M3uSpN4Z7pJU\nIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFej/AJZcj4y/pibwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f366f94cbd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from rmse_regr import RMSERegr\n",
    "from mieszkania import msle\n",
    "\n",
    "lr = 1E-3\n",
    "steps = 500 #TODO\n",
    "\n",
    "regr = RMSERegr()\n",
    "\n",
    "zs = np.concatenate((txs[:,:-4], txs[:, -4:] * txs[:,:1]), axis=1)\n",
    "\n",
    "zs = np.log(zs + 1)\n",
    "w = regr.regression(zs, np.log(1+tys), [lr] * steps)\n",
    "my_prices=np.exp(np.sum(zs*w, axis=1))+1\n",
    "print msle(tys, my_prices)\n",
    "print (my_prices / tys * 100).astype(int)\n",
    "# print np.concatenate((prices.reshape((-1,1)), my_prices.reshape((-1,1)), \n",
    "                      # (prices / my_prices).reshape((-1,1))\n",
    "                      # ), axis=1)\n",
    "\n",
    "zs = np.concatenate((vxs[:,:-4], vxs[:, -4:] * vxs[:,:1]), axis=1)\n",
    "zs = np.log(zs + 1)\n",
    "my_prices=np.exp(np.sum(zs*w, axis=1))+1\n",
    "print 'Val:', msle(vys, my_prices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To make the random split validation reliable, a huge chunk of training data may be needed. To get over this problem, one may apply cross-validaiton.\n",
    "\n",
    "![alt-text](https://chrisjmccormick.files.wordpress.com/2013/07/10_fold_cv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's now implement the method. Make sure that:\n",
    "* number of partitions is a parameter,\n",
    "* the method is not limited to `mieszkania.csv`,\n",
    "* the method is not limited to one specific model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0863076454157\nw: [ 0.66879436  0.16266923  0.14064712  1.19868237  0.05371825  0.09350977\n  0.21074706  0.21520924  0.14932828]\nRMSE: 0.0852973680231\nw: [ 0.67140398  0.16492256  0.14191868  1.19598931  0.05748013  0.09512591\n  0.21005542  0.21640904  0.14981361]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0854388458323\nw: [ 0.67084673  0.16459234  0.14210722  1.19666444  0.05435321  0.09487059\n  0.21148854  0.21468758  0.14980002]\nRMSE: 0.0831254494889\nw: [ 0.67448438  0.16467328  0.14210954  1.19415021  0.05343904  0.09696398\n  0.21169802  0.21749896  0.14832342]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0854420812449\nw: [ 0.66845903  0.16194374  0.1402115   1.19916462  0.05495748  0.09391778\n  0.21048629  0.21557133  0.14848362]\nRMSE: 0.0860350526294\nw: [ 0.66978631  0.16373791  0.14083877  1.19793795  0.05454137  0.09428778\n  0.21026777  0.21567985  0.14955091]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0859948369684\nw: [ 0.67008546  0.16316857  0.14073658  1.19783449  0.05244546  0.09425246\n  0.21117639  0.21545231  0.14920431]\nRMSE: 0.0825706204846\nw: [ 0.66995904  0.16423324  0.14088649  1.1974534   0.05601193  0.09450968\n  0.21072952  0.21627304  0.1484468 ]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0840853793258\nw: [ 0.67031102  0.16362748  0.14145983  1.19735442  0.05202366  0.09417899\n  0.21111172  0.2156234   0.14939691]\nRMSE: 0.0856041776673\nw: [ 0.67087212  0.1639911   0.14034649  1.19724585  0.05573136  0.09561842\n  0.21044902  0.21583973  0.14896494]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0827091664831\nw: [ 0.66834178  0.16279575  0.1412571   1.19909322  0.05361364  0.09320069\n  0.21131716  0.21490215  0.14892179]\nRMSE: 0.0859102546396\nw: [ 0.66950413  0.16333116  0.13966408  1.19806089  0.05471489  0.09460288\n  0.21051028  0.21552777  0.14886319]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0844775390981\nw: [ 0.67317091  0.16438316  0.14154     1.19493426  0.05373774  0.09590764\n  0.21143924  0.21671477  0.14910925]\nRMSE: 0.0859916354085\nw: [ 0.67032865  0.16409969  0.14115838  1.19752094  0.05463209  0.09459781\n  0.21082325  0.21558373  0.14932385]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0817901929884\nw: [ 0.6691335   0.16353949  0.14083338  1.19849437  0.0552826   0.09380353\n  0.21079669  0.21520204  0.14933123]\nRMSE: 0.0856758198765\nw: [ 0.67022203  0.16366057  0.14117062  1.19731695  0.05479052  0.09450206\n  0.21013962  0.21555543  0.15002493]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0811891893653\nw: [ 0.66930142  0.16279408  0.14062863  1.19848622  0.05393584  0.0951164\n  0.21050935  0.21509846  0.14857721]\nRMSE: 0.0860356840216\nw: [ 0.66929913  0.16255172  0.14071082  1.19847033  0.05369223  0.09446675\n  0.2100929   0.21602045  0.14871903]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.0862801382697\nw: [ 0.66995933  0.16327693  0.1400483   1.19804182  0.05439927  0.09417299\n  0.21089747  0.21579624  0.14909263]\nRMSE: 0.0864907716031\nw: [ 0.6703908   0.16366637  0.14084205  1.19738709  0.05372425  0.09426045\n  0.21060911  0.21605796  0.14946328]\n0.0818350440954\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEACAYAAAC9Gb03AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE5FJREFUeJzt3X2QZXdd5/H3997bPY95IjgJySSBoFkLRCJKgoh6JbsS\ncY3+I2Z94MGS2j+kwqKFSfSPdLm17CKVlahYJRqzwJJViSjRxTKG7LUKBIlMhsBkEgZSJOOQTEjm\nKZnMTPft/vrHPcn09PY89L23f2dy7vtVdWvuPf0753vOr3s+/evfOffcyEwkSc3VqnsHJEmry6CX\npIYz6CWp4Qx6SWo4g16SGs6gl6SGO2nQR8StEbE7Iu5ftOx3ImJ7RGyNiL+MiDNXdzclScM6lRH9\nbcCbliy7C3hlZl4O7ABuHPeOSZLG46RBn5mfBfYuWXZ3Zi5UL78AbF6FfZMkjcE45uh/Gfi7MWxH\nkrQKRgr6iPgtYC4zbx/T/kiSxqwz7IoR8XbgzcAbT9LOm+lI0hAyM8axnVMd0Uf1GLyIuBp4L3BN\nZh452cqZ6SOTm266qfZ9OF0e9oV9YV+c+DFOp3J55e3APwGXRcSjEfEO4PeBjcA/RMSWiPjDse6V\nJGlsTjp1k5k/v8zi21ZhXyRJq8B3xhbU7Xbr3oXThn1xlH1xlH2xOmLcc0H/X4GIXO0aktQ0EUEW\nPhkrSXqBMuglqeEMeklqOINekhrOoJekhjPoJanhDHpJajiDXpIazqCXpIYz6CWp4Qx6SWo4g16S\nGs6gl6SGM+glqeEMeklqOINekhrOoJekhjPoJanhDHpJajiDXpIazqCXpIYz6CWp4Qx6SWo4g16S\nGs6gl6SGM+glqeFOGvQRcWtE7I6I+xctOyci7oqIhyLi7yPirNXdTUnSsE5lRH8b8KYly24A7s7M\nfwfcA9w47h2TJI3HSYM+Mz8L7F2y+KeBj1TPPwL8zJj3S5I0JsPO0W/KzN0Amfk4sGl8uyRJGqdx\nnYzNMW1HkjRmnSHX2x0R52Xm7og4H3jiRI1nZmaef97tdul2u0OWlaRm6vV69Hq9Vdl2ZJ58MB4R\nLwX+JjNfVb1+P7AnM98fEdcD52TmDcdZN0+lhiTpqIggM2Ms2zpZCEfE7UAXOBfYDdwE/DXwCeAi\n4BHgLZm57zjrG/SStEJFg37kAga9JK3YOIPed8ZKUsMZ9JLUcAa9JDWcQS9JDWfQS1LDGfSS1HAG\nvSQ1nEEvSQ1n0EtSwxn0ktRwBr0kNZxBL0kNZ9BLUsMZ9JLUcAa9JDWcQS9JDWfQS1LDGfSS1HAG\nvSQ1nEEvSQ1n0EtSwxn0ktRwBr0kNZxBL0kNZ9BLUsMZ9JLUcAa9JDWcQS9JDWfQS1LDjRT0EXFj\nRGyLiPsj4uMRMT2uHZMkjcfQQR8RlwDvBL4vM78X6ADXjmvHJEnjMcqI/gAwC2yIiA6wHvjWcg13\nPbprhDKSpFEMHfSZuRe4GXgU2AXsy8y7l2v7yMMPD1tGkjSizrArRsSlwHuAS4D9wB0R8fOZefvS\ntrf8/h9wV+8zAHS7Xbrd7rBlJamRer0evV5vVbYdmTncihFvAf5DZr6zev1LwJWZ+a4l7fLWW/6Y\nX77uV0beWUmaFBFBZsY4tjXKHP1DwOsiYm1EBHAVsH25hgef3j9CGUnSKEaZo/8y8FHgS8CXgQA+\nvFzbg08/M2wZSdKIhp6jB8jMDwAfOFm7w4cOjVJGkjSCIu+MnZ+dLVFGkrSMIkHfN+glqTZlRvTz\n/RJlJEnLKBL0adBLUm3KBH0ulCgjSVpGkaAP5kuUkSQto0zQx3DvvpUkja5I0LeGvM2CJGl0ZYI+\nnKOXpLoUCvoSVSRJyykzR99y6kaS6lJmRO9HkEtSbYpEcNurbiSpNoVG9Aa9JNXFqRtJargyUzeO\n6CWpNl5eKUkN5xy9JDVcoRG9QS9JdXGOXpIazqkbSWo4L6+UpIZzjl6SGs6pG0lqOO9eKUkN5weP\nSFLDOXUjSQ1n0EtSw40U9BFxVkR8IiK2R8S2iLhy2SItp24kqS6dEde/Bfh0Zv5sRHSA9cs18vJK\nSarP0EEfEWcCP5yZbwfIzD5wYNm2juglqTajTN28DHgyIm6LiC0R8eGIWLdsEefoJak2o0zddIDX\nAL+amf8SER8EbgBuWtrwjv/3dR7szwDQ7XbpdrsjlJWk5un1evR6vVXZdmQON9qOiPOAz2fmpdXr\nNwDXZ+ZPLWmXf/G+H+Nnb7xn5J2VpEkREWTmWD62aeipm8zcDeyMiMuqRVcBDyzX1jl6SarPqFfd\nXAd8PCKmgIeBdyzXyKCXpPqMFPSZ+WXgtSdr5y0QJKk+hW5qZtBLUl0MeklqOINekhqu0E3NDHpJ\nqkuhEf18iTKSpGWUCXqvupGk2jhHL0kN59SNJDVcmaBvG/SSVBeDXpIarlDQz5UoI0laRqGg75co\nI0lahlM3ktRwZYK+44hekupSJOjpOEcvSXVxRC9JDVdmRO/JWEmqTaGpmz79vidkJakOZYIe2PGV\nHaVKSZIWKRP082223ndfkVKSpGOVCfr+FE88urNIKUnSsYoEffY77HtqT4lSkqQlio3ojzz7TJFS\nkqRjlRnRz7fpzx0uUUqStESxqZtc8PJKSapDoRF9B9Kgl6Q6FAr6KdqRJUpJkpYoNkffCj8gXJLq\nMHLQR0QrIrZExJ3Ha7Mw36HddkQvSXUYx4j+3cADJ2qQ822nbiSpJiMFfURsBt4M/MmJ2i3Md+i0\nR6kkSRrWqCP63wXeC5xwuJ7zbdpt5+glqQ6dYVeMiJ8Edmfm1ojoAnG8th//9GM89dSzHJmZodvt\n0u12hy0rSY3U6/Xo9Xqrsu3IHG7uPCLeB/wi0AfWAWcAn8zMty5pl5+8+fU8/I0L+fUP/cWo+ytJ\nEyEiyMzjDqBXYuipm8z8zcy8ODMvBa4F7lka8s9ZmG971Y0k1aTIdfQL8206Br0k1WLoOfrFMvMf\ngX883tcXFlqejJWkmpQZ0S+0aXkdvSTVokzQ99t0HNFLUi2KBH1/vk27Y9BLUh3KBH2/RafdL1FK\nkrREkaCfn2/R7ng/ekmqQ5Ggn+u3abcNekmqQ6E5+hbtqbkSpSRJS5QZ0c8HbefoJakWhU7GBu2O\nQS9JdSg2om+1nbqRpDoUCfrZ+aDlHL0k1aLQ5ZVBq2PQS1IdCl11E4RBL0m1KBL02ZqiNXWkRClJ\n0hJFgj46awjn6CWpFkWCft2ZZxFTsyVKSZKWKBL0L7noImLaqRtJqkORoH/1FT8A07McPuyoXpJK\nKxL0V7z+SgDu+/yXSpSTJC1SJOjb7RbMTrP1Cwa9JJVWJOgBcm6ax3Y+WqqcJKlSLuhn13DomadL\nlZMkVYoGfc4fKlVOklQpFvTzs2uZCj9lSpJKKxb0C7NrmO4slConSaoUC/r+7FrWTBn0klRauamb\nuSmDXpJqUG5EP7eG6Wnn6CWptKGDPiI2R8Q9EbEtIr4SEdedqP3c3BTTUwa9JJXWGWHdPvBrmbk1\nIjYCX4qIuzLzwWUbz00xZdBLUnFDj+gz8/HM3Fo9fwbYDlx4vPZzcx2mp/rDlpMkDWksc/QR8VLg\ncuCfj9dmdrZNx6CXpOJGDvpq2uYO4N3VyH5Zs3NtpvzwEUkqbpQ5eiKiwyDkP5aZnzpeu5mZGT73\nT9s595wDnHtlj263O0pZSWqcXq9Hr9dblW1HZg6/csRHgScz89dO0CYzk//2n3+OV7/q6/zHd3mr\nYkk6mYggM2Mc2xrl8sofAn4BeGNE3BcRWyLi6uO1PzLbou3HCUpScUNP3WTm54D2qbY/stCivca7\nV0pSaeVuatZaQ3vdwVLlJEmVYkF/3steTmvDcS/KkSStkpFOxp5Sgepk7Lce28VDD1zMhvbnuaJ7\nxarWlKQXutPiZOxKXfCSC8lnN/B3n/irUiUlSRQMeoCFZzfy9N7HS5aUpIlXNOjnD21gTXuuZElJ\nmnhFg75/aAPrvCe9JBVVNOjnDq9j/Ro/ZUqSSioa9LOH17F2rSN6SSqpaNAfPrKG9Wu9g6UklVQ0\n6A8+u5b1670NgiSVVDTonz7YYe2Gp0uWlKSJVzToDzzbZnrjgZIlJWniFQ36I7GOqTP3liwpSROv\naNB/34/8CK2z97D3iX0ly0rSRCsa9Ne+9W0w3+GP/vuHSpaVpIlWNOhbrRb9feey57EdJctK0kQr\nGvQAh/Zs4twz/EhBSSqleNA/s/9szj3boJekUooH/b79GznrLK+ll6RSigf9U/un2HDOk6XLStLE\nKh70Bztnsvb8nRz4tm+ckqQSigf9zAf+AKbm+K+/emPp0pI0kYoH/dr163hm18vZdOb+0qUlaSIV\nD3qAb+8+j4tf4q0QJKmEWoL+a49t5MWXfZWndu6po7wkTZRagv6mm28jNz7LB6/7rTrKS9JEqSXo\n12/cyENfvpzXf/+DPHLvI3XsgiRNjJGCPiKujogHI+JrEXH9StZ95Y9eR7ziAb74wd/h9/79H3Fo\np588JUmrITJzuBUjWsDXgKuAbwH3Atdm5oNL2uXxavzP9/0Kl73ib5naczadLa/iyN5zOTI3zeH+\nNIdmOxxZCGbng8MLwZE+HF4IZudbHO4H0OGMdVO85MUbuPiC83jx5k286KJNfMcF38GGCzYwvWma\n9hltImKo41sNvV6Pbrdb926cFuyLo+yLo+yLoyKCzBxLgI0S9K8DbsrMn6he3wBkZr5/SbvjBj3A\n/Nxh/vTma3mmtZN16xZYM73Amqk+U1N92u1Fj84crfbc4N/OLNHuk/Mdcm4aZtfA3BT0OzDXJvod\not+G+Q4532ZhoU0uLP63xcJCh1xokQstFubbZLbIDBYIMiGJo6+BzMHyBTi6bCGgWp6L/120DTJY\nqA7//35+Gz/xg6/k+d6ovoVJQsJCJAGDbVdNju266qvV9365bj1mWQ5WeW5ZEsd+rVoW1Yvnvp5L\nd3DxJmNlPy959GifPzaAz3xxG1dd8coVbWvVnOiQVvLfbLj/Snzmi9t446K+ONWSQ5Ybk+d/eMfq\nnnu/yo+99nvGu9EXoP588Nsf/vOxBX1nhHUvBHYuev2vwBUr3Uh7ai3vvOGvV1w8+3PMHtjN/se/\nyc5vfoMnD36bPfv2sf/APg4c3M+hI8/S788ScwsE0CJpZdCJIFpBu5W0WtBuQysWaLWSiMGP7+Df\npFUF73PLWy2IalkLiKnBOkQSJNGqnlcPGLQnBvuwYeNBzjv/iWP+I5/4eR599XzAPrfdY3pjmQ1U\nvxAW78eizR2zHs8dw5L1T2ZJk1hBKn51x35e9fKdJ2+4IkOkzmnwB99Xv7Gf7/2uR+vejVMWK/xl\nvxLbHt7H5Zd9c/wbPg2+zyvRn10z1u2NEvS1is4Ua160mU0v2symV7yh7t05JQ8cmuEt18/UvRun\nhS37Z7jmPTN178ZpYcu+Ga75LzN178Zp4Ut7Z/gp+2LgN8b322nUqZuZzLy6en3cqZuR91KSJtDp\nMEffBh5icDL2MeCLwH/KzO3j2DFJ0ngMPXWTmfMR8S7gLgZT1rca8pJ0+hl6RC9JemFYtXfGjvJm\nqheiiNgcEfdExLaI+EpEXFctPyci7oqIhyLi7yPirEXr3BgROyJie0T8eH17vzoiohURWyLizur1\nRPZFRJwVEZ+ojm1bRFw5wX1xY9UH90fExyNielL6IiJujYjdEXH/omUrPvaIeE3Vf1+LiA+eUvHM\nHPuDwS+QrwOXAFPAVuC7V6PW6fIAzgcur55vZHD+4ruB9wO/US2/Hvgf1fNXAPcxmD57adVfUfdx\njLlP3gP8b+DO6vVE9gXwv4B3VM87wFmT2BdVHjwMTFev/xx426T0BfAG4HLg/kXLVnzswD8Dr62e\nfxp408lqr9aI/gpgR2Y+kplzwJ8BP71KtU4Lmfl4Zm6tnj8DbAc2Mzjuj1TNPgL8TPX8GuDPMrOf\nmd8EdjDE+xBOVxGxGXgz8CeLFk9cX0TEmcAPZ+ZtANUx7mcC+wI4AMwCGyKiA6wDdjEhfZGZnwWW\n3p99RcceEecDZ2TmvVW7jy5a57hWK+iXezPVhatU67QTES9l8Jv7C8B5mbkbBr8MgE1Vs6V9tItm\n9dHvAu/l2HcxTWJfvAx4MiJuq6axPhwR65nAvsjMvcDNwKMMjmt/Zt7NBPbFIptWeOwXMsjT55xS\nttZy98omi4iNwB3Au6uR/dKz3Y0/+x0RPwnsrv7COdF1wI3vCwZ/er8G+FBmvgY4CNzAZP5cXMpg\nOu8S4AIGI/tfYAL74gRW5dhXK+h3ARcver25WtZo1Z+jdwAfy8xPVYt3R8R51dfPB56olu8CLlq0\nepP66IeAayLiYeD/AG+MiI8Bj09gX/wrsDMz/6V6/ZcMgn8Sfy5+APhcZu7JzHngr4DXM5l98ZyV\nHvtQfbJaQX8v8J0RcUlETAPXAneuUq3TyZ8CD2TmLYuW3Qm8vXr+NuBTi5ZfW1118DLgOxm86ewF\nLzN/MzMvzsxLGXzv78nMXwL+hsnri93Azoi4rFp0FbCNCfy5YHCBwusiYm0Mbit7FfAAk9UXwbF/\n5a7o2Kvpnf0RcUXVh29dtM7xreIZ5qsZfGN3ADfUfcZ7tR8MRrHzDK4wug/YUvXBi4C7q764Czh7\n0To3Mjibvh348bqPYZX65Uc5etXNRPYF8GoGg5+twCcZXHUzqX3xXga/6O5ncPJxalL6AridwS3d\njzA4T/EO4JyVHjvw/cBXqmy95VRq+4YpSWo4T8ZKUsMZ9JLUcAa9JDWcQS9JDWfQS1LDGfSS1HAG\nvSQ1nEEvSQ33b2vmAYu1qQJ6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f366f4c37d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "####################################\n",
    "# TODO: Implement cross-validation # \n",
    "####################################\n",
    "def cross_val(part_n, xs, ys, evaluate):\n",
    "    perm = np.random.permutation(xs.shape[0])\n",
    "    xs = xs[perm]\n",
    "    ys = ys[perm]\n",
    "    pts = np.linspace(0, xs.shape[0], num=part_n+1, endpoint=True, dtype=int)\n",
    "    errs = []\n",
    "    for i in range(part_n):\n",
    "        a, b = pts[i], pts[i+1]\n",
    "        errs.append(evaluate(np.concatenate((xs[:a], xs[b:])),\n",
    "                             np.concatenate((ys[:a], ys[b:])),\n",
    "                             xs[a:b], ys[a:b]))\n",
    "    return np.average(errs)\n",
    "\n",
    "def evaluate(txs, tys, vxs, vys):\n",
    "    lr = 1E-3\n",
    "    steps = 1000\n",
    "    regr = RMSERegr()\n",
    "    zs = np.concatenate((txs[:,:-4], txs[:, -4:] * txs[:,:1]), axis=1)\n",
    "    zs = np.log(zs + 1)\n",
    "    w = regr.regression(zs, np.log(1+tys), [lr] * steps)\n",
    "\n",
    "    zs = np.concatenate((vxs[:,:-4], vxs[:, -4:] * vxs[:,:1]), axis=1)\n",
    "    zs = np.log(zs + 1)\n",
    "    my_prices=np.exp(np.sum(zs*w, axis=1))+1\n",
    "    err = msle(vys, my_prices)\n",
    "    return err\n",
    "\n",
    "print cross_val(20, xs, ys, evaluate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Recall that sometimes validation may be tricky, e.g. significant class imbalance, having a small number of subjects, geographically clustered instances...\n",
    "\n",
    "What could in theory go wrong here with random, unstratified partitions? Think about potential solutions and investigate the data in order to check whether these problems arise here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "##############################\n",
    "# TODO: Investigate the data #\n",
    "##############################"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2.0
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}