SVM classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e082252c",
   "metadata": {},
   "source": [
    "# SVM classifier from scratch\n",
    "This file contains code to train a support vector machine classifier from scratch. The only advanced tool that I have used is scipy.optimize, necessary to minimize the loss function numerically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c6177253",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from scipy.spatial import distance_matrix\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import train_test_split\n",
    "from scipy.optimize import minimize \n",
    "from scipy.optimize import Bounds\n",
    "from scipy.optimize import LinearConstraint\n",
    "import math\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a37aea3b",
   "metadata": {},
   "source": [
    "The SVM is defined by two classes. $\\texttt{SVM_binary}$ is a binary classifier, taking $y \\in \\{-1, 1\\}$. Then $\\texttt{multiclass_SVM}$ takes arbitrary number of classes $y \\in \\mathbf{N}$. $\\texttt{multiclass_SVM}$ works by calling $\\texttt{SVM_binary}$ $\\frac{c(c-1)}{2}$ times, where $c$ is a number of different classes. Then, so called 1 VS 1 classification is performed. Having $\\frac{c(c-1)}{2}$ classifiers trained, the new point is classified by each of them and final classification is simply a majority vote of all classifiers. \n",
    "\n",
    "The program can work with two kernels, gaussian and linear. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cda8ebe4",
   "metadata": {},
   "source": [
    "The classifier minimizes the loss in the form of Lagrangian dual\n",
    "$$ \\max_v \\sum_{i=1}^n v_i-\\frac{1}{2}\\sum_{i=1}^n\\sum_{j=1}^nv_iy_i\\kappa(x_i,x_j)v_jy_j$$\n",
    "$$ \\textrm{subject to} \\sum_{i=1}^nv_iy_i=0 \\textrm{  and  } 0 \\leq v \\leq C$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d340e0d0",
   "metadata": {},
   "source": [
    "Then, the classifier is given by \n",
    "$$\\textrm{sgn}(g(x))$$\n",
    "where \n",
    "$$g(x)=a_0+\\sum_{i=1}^nv_iy_i\\kappa(x_i,x)$$\n",
    "and $a_0$ is calculated as \n",
    "$$\\frac{1}{|J|}\\sum_{j \\in J}\\left(y_j - \\sum_{i=1}^ny_iv_i\\kappa(x_i,x_j)\\right)$$\n",
    "where $J=\\{v_i: v_i\\in (0,C)\\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d64e12",
   "metadata": {},
   "source": [
    "$\\texttt{K_function(self,x)}$ works for a classifier trained using $X=\\left[x_1,\\ldots,x_n\\right]$, takes a new argument $x$ to be classified and returns a vector of scalar products $\\kappa(x_1,x),\\ldots,\\kappa(x_n,x)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "197ed870",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SVM_binary: #assuming y is either -1 or 1\n",
    "    def __init__(self, kernel='linear', C=1, gamma='auto'): #C and gamma have the same function as in sklearn.SVC\n",
    "        self.kernel = kernel\n",
    "        self.C = C\n",
    "        self.gamma = gamma\n",
    "        \n",
    "    def K_function(self,x):\n",
    "        if(self.kernel=='linear'): return self.X@x\n",
    "        if(self.kernel=='rbf'): return np.exp(-self.rbf_gamma*np.power(distance_matrix(self.X,x.reshape((1,x.size)))[:,0],2))\n",
    "            \n",
    "    def fit(self,X,y):    \n",
    "        def gram_matrix(): #calculates gram matrix\n",
    "            if(self.kernel=='linear'): return np.array(X@X.T)\n",
    "            if(self.kernel=='rbf'):\n",
    "                if(self.gamma=='auto'): self.rbf_gamma=1/p\n",
    "                elif(self.gamma=='scale'): self.rbf_gamma=1/(p*X.var())\n",
    "                else: self.rbf_gamma=self.gamma\n",
    "                return np.array(np.exp(-self.rbf_gamma*np.power(distance_matrix(X,X),2)))\n",
    "\n",
    "        def optim_problem(v):  #loss function to be optimized\n",
    "            v_F=np.matrix(v)\n",
    "            return -float((np.matrix.sum(v_F))-v_F@K_hat@v_F.T/2)\n",
    "\n",
    "        n=len(y) #number of observarions\n",
    "        p=X.shape[1] #number of dimensions\n",
    "        y_F=y.reshape((n,1))\n",
    "        K=gram_matrix()\n",
    "        K_hat=y_F*y_F.T*K\n",
    "        bounds = Bounds(np.repeat(0,n), np.repeat(self.C,n)) #optimization constraints\n",
    "        linear_constraint = LinearConstraint(y_F.T,0, 0) #optimization constraints\n",
    "        \n",
    "        #return support vectors by minimizing loss function\n",
    "        self.V=minimize(optim_problem, np.repeat(0,n),method='trust-constr',constraints=linear_constraint,\n",
    "                 options={'verbose': 0}, bounds=bounds).x\n",
    "\n",
    "        #calculating a0\n",
    "        SV_indices=np.where(self.V>=0.00001)[0] #due to the numeric optimization, vectors smaller than a treshhold are treated as zeros\n",
    "        SVs=self.V[SV_indices]\n",
    "        J_indices=np.where((self.V>=0.00001) & (self.V<=self.C-0.00001))[0] #J are used to approximate a0 as an average\n",
    "        self.a0=0\n",
    "        for j in J_indices:\n",
    "            t_sum=0\n",
    "            for i in SV_indices:\n",
    "                t_sum+=self.V[i]*y[i]*K[i,j]\n",
    "            self.a0+=y[j]-t_sum\n",
    "        self.a0/=J_indices.size\n",
    "        \n",
    "        self.y=y\n",
    "        self.X=X\n",
    "\n",
    "    def predict(self,X):\n",
    "        def g_preclassifier(x): return self.a0+np.sum(self.y*self.V*self.K_function(x))\n",
    "        prediction=np.array([])\n",
    "        for i in range(X.shape[0]):\n",
    "            prediction=np.append(prediction,math.copysign(1,g_preclassifier(X[i,:])))\n",
    "        return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "299f2cd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "class multiclass_SVM: #arbitrary y\n",
    "    def __init__(self, kernel='linear', C=1, gamma='auto'):\n",
    "        self.kernel = kernel\n",
    "        self.C = C\n",
    "        self.gamma = gamma\n",
    "        \n",
    "    def fit(self,X,y):\n",
    "        self.classes=np.array(list(set(y))) #stores set of different y\n",
    "        self.set_of_clf=[] #stores all binary classifiers\n",
    "        self.class_pairs=np.zeros(shape=(0,2)) #stores a history of comparisons made\n",
    "        \n",
    "        #this loop compares all possible pairs of classes\n",
    "        for i in range(self.classes.size):\n",
    "            for j in range(i+1,self.classes.size):\n",
    "                index_plus=self.classes[i] #this class is now \"-1 class\"\n",
    "                index_minus=self.classes[j] #this class is now \"+1 class\"\n",
    "                \n",
    "                #new_X and new_Y are subsets of X and y used for binary comparison\n",
    "                new_X=np.vstack((X[np.where(y==index_plus)[0],:],X[np.where(y==index_minus)[0],:]))\n",
    "                new_y=np.concatenate((np.repeat(1,np.where(y==index_plus)[0].size),np.repeat(-1,np.where(y==index_minus)[0].size)))\n",
    "                \n",
    "                clf=SVM_binary(kernel=self.kernel,C=self.C,gamma=self.gamma)\n",
    "                clf.fit(new_X,new_y)\n",
    "                self.set_of_clf.append(clf)\n",
    "                self.class_pairs=np.vstack((self.class_pairs,np.array((self.classes[i],self.classes[j])))) \n",
    "                \n",
    "    def predict(self, X): #assuming classes of X are a subset of classes used previously for training\n",
    "        n=X.shape[0]\n",
    "        \n",
    "        #temp_pairs stores pairs of categories compared by each binary classifier\n",
    "        temp_pairs_matrix=np.zeros(shape=(0,2))\n",
    "        for i in range(self.classes.size):\n",
    "            for j in range(i+1,self.classes.size):\n",
    "                temp_pairs_matrix=np.vstack((temp_pairs_matrix,(i,j)))\n",
    "        \n",
    "        #matrix of predictions counts how many times each new x is classified to each category by all binary classifiers\n",
    "        matrix_of_predictions=np.zeros(shape=(n,len(self.classes)))\n",
    "        for i in range(len(self.set_of_clf)):\n",
    "            prediction=self.set_of_clf[i].predict(X)\n",
    "            matrix_of_predictions[np.where(prediction==-1)[0],int(temp_pairs_matrix[i,1])]+=1\n",
    "            matrix_of_predictions[np.where(prediction==1)[0],int(temp_pairs_matrix[i,0])]+=1\n",
    "        prediction=np.array([]) #final vector to be returned\n",
    "        for i in range(n):\n",
    "            index=np.where(matrix_of_predictions[i,:]==np.max(matrix_of_predictions[i,:]))[0][0] #for each x choosing class which has the most votes\n",
    "            prediction=np.append(prediction,self.classes[index])\n",
    "        return prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a70679f7",
   "metadata": {},
   "source": [
    "### Binary linear classification - hard margin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4505222c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD6CAYAAAC8sMwIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAATS0lEQVR4nO3df4wd1XnG8efFbNolNF4qVorwj5q2rhsKbl1dpVSuqgZHQFIg1Eot0l9pkWpFShpoEA2uJUBUCCpLJKWNWlmB9o+aggUOISQtEIiUthIo69gYqHGLUFPbpM2miZ00rOS1efvH3Y13787cvffOmZlzZr4fCZk7ez33jFiee+ac95wxdxcAIF3n1N0AAEAxBDkAJI4gB4DEEeQAkDiCHAASR5ADQOKCBLmZTZjZo2b2qpkdNrNfDnFeAMDyzg10nr+Q9E/u/kEze5uk8/q9+cILL/R169YF+mgAaIf9+/d/290ne48XDnIze4ekX5X0+5Lk7qckner3d9atW6epqamiHw0ArWJm38g6HmJo5SclTUv6WzM7YGafNbO3BzgvAGAAIYL8XEm/KOmv3X2TpB9Iuq33TWa23cymzGxqeno6wMcCAKQwQX5M0jF3f2Hu9aPqBvsi7r7b3Tvu3pmcXDLEAwAYUeEgd/f/lnTUzDbMHdoi6d+KnhcAMJhQVSt/JGnPXMXK65L+INB5AQDLCBLk7n5QUifEuQDE4/EDx7XrqSN648SMLpoY161XbdD1m1bV3Sz0CNUjB9Awjx84rh37XtLM7BlJ0vETM9qx7yVJIswjwxJ9AJl2PXXkhyE+b2b2jHY9daSmFiEPQQ4g0xsnZoY6jvoQ5AAyXTQxPtRx1IcgB5Dp1qs2aHxsxaJj42MrdOtVG3L+BurCZCeATPMTmlStxI8gB5Dr+k2rCO4EMLQCAIkjyAEgcQQ5ACSOIAeAxBHkAJA4ghwAEkeQA0DiCHIASBxBDgCJI8gBIHEEOQAkjiAHgMSxaRaC4hmPQPUIcgTDMx6BejC0gmB4xiNQD4IcwfCMR6AeBDmC4RmPQD0IcgTDMx6BejDZiWB4xmO7UbFUH4IcQfGMx3boDe33/OykHtt/nIqlmjC0AmAo82Wmx0/MyNUN7T3P/xcVSzUiyAEMJavM1HPeS8VSNQhyAEMZJpypWKoGQQ5gKHnhbD2vqViqDkEOYCh5Zaa/fflarZoYl0laNTGue7ZeFvVE5+MHjmvzvc/p4tu+qM33PqfHDxyvu0kjo2oFwFCaUGbatH2BCHIAQ0u9zLTfvkApXhdDKwBap2n7AhHkAFqnafsCBQtyM1thZgfM7MlQ5wSAMjRtX6CQY+Q3STos6R0BzwkUwv4fyNKECduFggS5ma2W9OuS7pb0iRDnBIpqWmUCwkp9wnahUEMrn5b0J5LeCnQ+oDCeWIS2KBzkZnaNpG+5+/5l3rfdzKbMbGp6erroxwLLalplApAnRI98s6TrzOw/JT0s6Qoz+/veN7n7bnfvuHtncnIywMcC/TWtMgHIUzjI3X2Hu69293WSbpD0nLv/TuGWAQU1rTIByMPKTjRW0yoTgDzmnreTcHk6nY5PTU1V/rkAkDIz2+/und7j9MgBoAJlrmkgyAGgZGWvaSDIAZSK1bXl77ZIkAMoDatru8pe08DuhwBKe1oOq2u7yl7TQJADiSgrbOd7zcdPzMh1ttcc4vysru0qe00DQytApBaOLa8cH9MPTp3W7JluuXDIIYoyx28vmhjX8YzQnu+JtmX8vOw1DQQ5EKHeseUTM7NL3hMqbMvsNd961YZF1yGd7Ym2bfy8zN0WCXIgIvM91KxebJYQYbtcr7mIfj3Rzfc+16jnZtaJIAci0dtDHUSIsO3Xaw4hryfK+Hk4THYCkcgaq+4nVNhev2mV7tl6mVZNjMskrZoY1z1bLyu9V8zulOHQIwcisVxPdOwc0/k/eq5OvDkbfLKsjqfllH0n0CYEORCJvLFqqdtLblpFB7tThkOQYyhtKRerQ14PtYphjrr0hvn8QqGmXm9ZCHIMrG3lYlVrYw+V36kwCHIMrOyNf9CsJ7sPgt+pMAhyDKwp5WKpDg+l2u5+mvI7VTeCHAMrc+FIVVK9lc9q982PHNTNjxzUBeeN6Y5rf26o9sfypdCE36kYUEeOgTXhYcap7sbXr8b8u2/O6tZHXxx4k6syN8kaVhN+p2JAkGNgdS0cCSnVW/nl2jd7xgf+Morpy6wJv1MxYGgFQ0l9Mi7VW/l+NebzBv0yiu3LLPXfqRjQI0erpHorn9XuXoN+GbE0vnkIcrRKqrfy8+2+4LyxzJ+PrbCBv4xS/TJLTVkPAsli7l7ayfN0Oh2fmpqq/HOBJnj8wHHd+cQrP9yjPOWqlabK2skyxCpdM9vv7p0lxwlyIF4Ebpo23/tc5pzGqolx/ettV4x83rwgZ7ITiFSqNe+ofkKZMXIgUjGVCWI4VU8oE+RApGIrEwyhygnAOlU9oczQChCpVGve87RpqKjqnSwJciBSTXuCTtt2OqxyoRNBDkSqafuTN3GoKBYEORCxJi1fb9pQUUyY7ARQiZATgG2ZNB0UPXIAlQg1VBTrpGmdi7cIcgCVCTFUFOOkad1fLgytAEhKjJOmeV8ut+x9sZLhn8JBbmZrzOwrZnbYzF4xs5tCNAwAssS4DW/el8gZ90qewhSiR35a0i3u/i5Jl0v6qJldEuC8ACqUygRijNvwDvIlUub2CoWD3N2/6e5fn/v370s6LKkZ9VJAS8T0HM/lxLin/CAP/pDKG/4JOtlpZuskbZL0QsbPtkvaLklr164N+bEACopxArGf2OrreytyzjHTmYwtwssa/gkW5GZ2vqTHJN3s7t/r/bm775a0W+ruRx7qcwEUF+MEYiwGLStc+OWS92CJqDfNMrMxdUN8j7vvC3FOANVh1WW2UcsKk9s0y8xM0gOSDrv7fcWbBKBqTdugK5QiQ05VDv+EqFrZLOl3JV1hZgfn/nl/gPMCqEiME4gxSGXIqXCP3N3/RZIFaAuAGsU2gRiDVIacWNkJADlirFnPwl4rAJAjlT3hCXI0Wp070qEZUhhyIsjRWHXvSAdUhTFyNFa/0jGgSQhyNFYqpWNAUQytoLFSKR1rC+YrykOPHI2VSulYG6S0u2KKCHI0FqsV48F8RbkYWkGjpVA61gbMV5SLHjmA0sX4eLYmIcgBlI75inIxtAKgdKksdU8VQQ6gEsxXlIehFfR3aK/0qUulOye6fx7aW3eLAPSgR458h/ZKX/i4NDtXWXDyaPe1JG3cVl+7ACxCjxz5nr3rbIjPm53pHgcQDYIc+U4eG+44gFoQ5Mi3cvVwxwHUgiBHvi23S2M9CzbGxrvHAUSDIEe+jduka++XVq6RZN0/r72fiU4gMlStoL+N2whuIHL0yAEgcQQ5ACSOIAeAxBHkAJA4ghwAEkeQA0DiCHIASBxBDgCJI8gBIHEEOQAkjiAHgMQR5ACQOIIcABIXJMjN7GozO2Jmr5nZbSHOCQAYTOEgN7MVkj4j6X2SLpH0ITO7pOh5AQCDCdEjf7ek19z9dXc/JelhSR8IcF4AwABCBPkqSUcXvD42dwwAUIEQQW4Zx3zJm8y2m9mUmU1NT08H+FgAgBQmyI9JWrPg9WpJb/S+yd13u3vH3TuTk5MBPhYAIIUJ8q9JWm9mF5vZ2yTdIOmJAOdFXQ7tlT51qXTnRPfPQ3vrbhGAPgo/fNndT5vZxyQ9JWmFpAfd/ZXCLUM9Du2VvvBxaXam+/rk0e5riYcwA5EKUkfu7l9y959x959y97tDnBM1efausyE+b3amexxAlFjZicVOHhvuOIDaEeRYbOXq4Y4DqB1BjsW23C6NjS8+NjbePQ4gSgQ5Ftu4Tbr2fmnlGknW/fPa+5noBCJWuGoFDbRxW/OD+9De7gTuyWPdYaMttzf/mtFYBDnahxJLNAxDK2gfSizRMAQ52ocSSzQMQY72ocQSDUOQo30osUTDpBfkbOiEoiixRMOkVbVCtQFCaUOJJVojrR451QYAsERaQU61AQAskVaQU20AAEukFeRUGwDAEmkFOdUGALBEWlUrEtUGANAjrR45AGAJghwAEkeQA0Di0glyluYDQKY0JjtZmg8AudLokbM0f3TcyQCNl0aPnKX5Zw3zrEnuZIBWSKNHntrS/LJ6wU9+Qtq3vRvI8rPBnHd+7mSAVkgjyFNamj/fCx40bIc579SD3XMu1C+YuZMBWiGNIE9haf58L3zfH5bTC372Li0J8Xl5wTzqnQzj6kBS0hgjl+Jemt87Fp2laC+439/PC+Ytty9t13J3MoyrA8lJo0ceu6yx6F5Fx/Nz/77lB/ModzKMqwPJSadHHrPletujjOf3Vqesv1J68aGekDWpc2P/YB72Tma5cfVhqmYAVIIeeQj9etujjOdnTZi++JD087+1uHe9dbd0zX1FW9/T3j7j6mVN5AIohB55CHlj0aNOyOYNb/zH09Ifv1ysrcvpN67eb9iFXjlQG3rkIYSqqpmvFjl5NPvnVZQN9rsWyhmBKNEjD6VoVc0glS9VLYDKu5aVq7O/ZGJdmAW0BD3yWCxX+RLDAqiUFmYBLVIoyM1sl5m9amaHzOxzZjYRqF3t07dOPJIFUCkszAJaqOjQyjOSdrj7aTP7c0k7JH2yeLMiUWWpXe6wxZryJziHEfPCLKClCgW5uz+94OXzkj5YrDk1Wxjc4xdIp/5POnOq+7OyVziOsgoTABR2jPxGSf8Y8HzV6q2RnvnO2RCfV+YKR4YtAIxo2R65mX1Z0jszfrTT3T8/956dkk5L2tPnPNslbZektWvXjtTYUg2yzF7KHssONQTDsAWAESwb5O7+3n4/N7MPS7pG0hZ3z9meT3L33ZJ2S1Kn08l9X20GrYXuLbVjk6lqsUUAsETRqpWr1Z3cvM7d3wzTpJoMUgudNWZd1SZTbC3LFgFAjqJj5H8l6cckPWNmB83sbwK0qR5ZNdLnjEnjP66+Y9ZVrHYkwLrYmRHIVLRq5adDNaR28wG9sGpFkma+2/8WvorVjssFWFuGGtgiAMjEys6FNm7r1mxv3S2dnulWrizXA65itWNugB1tV089tWe3AhUhyLMMcwtfRdlgXlDZinYNNbBFAJCJTbOyDHsLX3bZYN5iobxyyaYONfQOfzV9KAkYEEGeJbZd/vIC7Nm74mpnFai1B5ZIN8jLrCeOYbl81vVl7blSdzuHQQ04UIo0g7zsRTh138IPen11t3MYLJwCSmN9FmOWptPp+NTU1OgnyHuKTmw7BY4qhesbtnedwjUBkTOz/e7e6T2eZo+86fXEsV/fKL3r2K8JSFia5YdNryeO/fpGWWEZ+zUBCUszyMusJ45hT5O861t/Zf1tk0brXVMDDpQmzaGVsib5YpmQy7q+9VdKLz5Uf9uk0cozU5qYBRKT5mRnWWKekIupbb1feFK3d82DMIBS5U12pjm0UpaYJ+RiahtPMwKikubQSlliW9HZ24aY2sYKSyAa9MgXinlCLua2AagVQb5Q0SGDMiteGM4AkIOhlYWK7AVSRcULwxkAMtAjn1f0cWo8hgxATdIM8jKGMIoGcUxVJQBaJb0gL+tBxEWDmCXoAGqSXpCXNYRRNIipKgFQk/SCvKwhjKJBTFUJgJqkV7VS1sKYEHuBUFUCoAbpBXmZj2EjiAEkKL2hFYYwAGCR9HrkEj1nAFggvR45AGARghwAEkeQA0DiCHIASBxBDgCJq+WZnWY2LekblX/wYhdK+nbNbQiFa4kT1xKnlK/lJ9x9svdgLUEeAzObynqIaYq4ljhxLXFq0rXMY2gFABJHkANA4toc5LvrbkBAXEucuJY4NelaJLV4jBwAmqLNPXIAaITWBrmZ/ZmZHTKzg2b2tJldVHebRmVmu8zs1bnr+ZyZTdTdplGZ2W+a2Stm9paZJVlZYGZXm9kRM3vNzG6ruz1FmNmDZvYtM3u57rYUYWZrzOwrZnZ47vfrprrbFFJrg1zSLnff6O6/IOlJSSk/k+0ZSZe6+0ZJ/y5pR83tKeJlSVslfbXuhozCzFZI+oyk90m6RNKHzOySeltVyN9JurruRgRwWtIt7v4uSZdL+mji/10WaW2Qu/v3Frx8u6RkJwvc/Wl3Pz338nlJyT7x2d0Pu/uRuttRwLslvebur7v7KUkPS/pAzW0ambt/VdJ36m5HUe7+TXf/+ty/f1/SYUmr6m1VOGnuRx6Imd0t6fcknZT0npqbE8qNkh6puxEttkrSwmcRHpP0SzW1BRnMbJ2kTZJeqLkpwTQ6yM3sy5LemfGjne7+eXffKWmnme2Q9DFJd1TawCEsdy1z79mp7i3knirbNqxBriVhlnEs2bu9pjGz8yU9JunmnrvypDU6yN39vQO+9SFJX1TEQb7ctZjZhyVdI2mLR15TOsR/lxQdk7RmwevVkt6oqS1YwMzG1A3xPe6+r+72hNTaMXIzW7/g5XWSXq2rLUWZ2dWSPinpOnd/s+72tNzXJK03s4vN7G2SbpD0RM1taj0zM0kPSDrs7vfV3Z7QWrsgyMwek7RB0lvq7sT4EXc/Xm+rRmNmr0n6EUn/O3foeXf/SI1NGpmZ/Yakv5Q0KemEpIPuflWtjRqSmb1f0qclrZD0oLvfXW+LRmdm/yDp19TdMfB/JN3h7g/U2qgRmNmvSPpnSS+p+/+8JP2pu3+pvlaF09ogB4CmaO3QCgA0BUEOAIkjyAEgcQQ5ACSOIAeAxBHkAJA4ghwAEkeQA0Di/h9uTs3o39a3tQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#creating and plotting the data set\n",
    "x_1=np.random.multivariate_normal([1,3], [[1,0],[0,1]], size=30) \n",
    "x_2=np.random.multivariate_normal([-2,-2], [[1,0],[0,1]], size=30) \n",
    "plt.scatter(np.array(x_1[:,0]),np.array(x_1[:,1]))\n",
    "plt.scatter(np.array(x_2[:,0]),np.array(x_2[:,1]))\n",
    "plt.show()\n",
    "X=np.vstack((x_1,x_2))\n",
    "y=np.concatenate((np.repeat(-1,30),np.repeat(1,30)))\n",
    "X_train,X_test,y_train,y_test=train_test_split(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6c0fe1c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#training the classifier\n",
    "clf=SVM_binary(C=np.inf)\n",
    "clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "68064bad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1.  1.  1.  1. -1. -1. -1.  1.  1.  1. -1.  1.  1.  1.  1.]\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "#printing the result of classification\n",
    "print(clf.predict(X_test))\n",
    "#printing accuracy score\n",
    "print(accuracy_score(y_test, clf.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94e05b8e",
   "metadata": {},
   "source": [
    "### Binary linear classification - soft margin - different regularization parameters C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f95104b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAYhklEQVR4nO3de4xdV3XH8d/CnVAnQbbauI0ytnGkRubhWLE6CpXCH80DnBJITCpcqFRAVLKQiiCIpiRYCm4QilVLBJUiFasg/kkBVwlOQkAhDyTaqFDGSXAcnKCUksYTKKYwhtZTZeys/nHnxpM793Ee+zz2Pt+PFDlzZ+acfebOrLPP2mvvbe4uAEC8XtF0AwAA5RDIASByBHIAiByBHAAiRyAHgMj9RhMnPe+883zTpk1NnBoAonXo0KGfu/u6wdcbCeSbNm3S7OxsE6cGgGiZ2bPDXie1AgCRI5ADQOQI5AAQOQI5AESOQA4AkWukagUI5eBjc9p3/9N6fn5BF6xdrRu3b9aObdNNNwuoFYEc0Tr42JxuvusJLSyeliTNzS/o5ruekCSCOTqF1Aqite/+p18K4n0Li6e17/6nG2oR0AwCOaL1/PxCrteBVBHIEa0L1q7O9TqQKgI5onXj9s1aPbXqZa+tnlqlG7dvbqhFQDMY7ES0+gOaVK2g6wjkiNqObdMEbnQeqRUAiByBHAAiVzqQm9kGM/uWmR01syfN7EMhGgYAyCZEjvyUpI+4+6Nm9ipJh8zsAXf/QYBjAwAmKN0jd/efuPujS///a0lHJTH6BAA1CZojN7NNkrZJ+m7I4wIARgsWyM3sXEl3SrrB3X815PO7zGzWzGaPHz8e6rQA0HlBArmZTakXxO9w97uGfY2773f3GXefWbduxSbQAICCQlStmKTPSzrq7p8q3yQAQB4heuSXSfozSVeY2eNL/70lwHEBABmULj9093+RZAHaAgAogLVWUCu2ZkMR/N6MRyBHbdiaDUXwezMZa62gNmzNhiL4vZmMQI7asDUbiuD3ZjICOWrD1mwogt+byQjkqE3VW7MdfGxOl+19WBfedJ8u2/uwDj42F+S4aBZb+k3GYCdqU+XWbAyIpYst/SYzd6/9pDMzMz47O1v7eZGuy/Y+rLkhOdPptav1yE1XNNAiIDwzO+TuM4Ovk1pBEhgQQ5eRWkFURk0MuWDt6qE9cgbEhmOCTVrokXfd4QPS7VukPWt7/x4+0HSLRurnwefmF+Q6kwc/+NgcA2I5jPs5Ik4E8i47fEC694PSieckee/fez/Y2mA+bmLIjm3Tuu36izW9drVMvdz4bddfTC9zCCbYpIfUSpc9dKu0OJCOWFzovb51ZzNtGmNSHnzHtmkCdwaMJ6SHQN5lJ47le71hXcuDV5XH7trPsQtIrXTZmvX5Xm9Yl/LgVeaxu/Rz7AoCeZddeYs0NdALm1rde72FupQHrzKP3aWfY1eQWumyfh78oVt76ZQ163tBvIX58b6u5MGrzmN35efYFQTyrtu6M1jgpjY5nFB5bN6TbiC1giCoTQ4rRB4773vComPxIpAjCGqTwwqRx87znnAjjhupFQRBbXJ4ZfPYed6TSZOt0G5BeuRm9gUz+5mZHQlxPMSHxf/bJ897wo04bqFSK1+UdHWgYyFC1Ca3T573hBtx3IIEcnf/tqRfhDgW4tR0bTIDdSvleU+4Ecct2MYSZrZJ0tfcfcuIz++StEuSNm7c+PvPPvtskPMCg7sDSb0gxCSXfChVbL9RG0vUFsiXY4cghMTuQOgKdghCshioQ9cRyBE9BurQdaHKD78k6V8lbTazY2b25yGOC2QxaqDu8tesYwAUnRBkQpC7vyvEcYAi+gNyywfqLn/NOt15aO6lAdD+TMXlXw+kgpmdyKWtlQ2DsyAv2/swMxXRGQRyZDZY5tfmXi4DoOgSBjuRWUwLYzEAii6hR47MYurl3rh989BJQqFmKtaVYmprKgvtQiBHZjFt2jtsADRUEKwrxVT0PAT/7iGQI7Oqe7mhlV0GdlRADLnk67igW+Q8TYxjcONoHoEcmVXZy22bcQExVIpp2Dlu/Kfv66/vfVLzJxc1avGMceepe13xmAbAU0YgRy5d2bR3XEAMlWIado7FF12/PLk49vvGnafucQw2pGgHqlaAIcYFxFBLvhYJrpPOU3e1ThsGwFnCmB45IlZlbnZcrztUimnUOYaxpa+fdJ66xzGaHgCPKbVT5e8rgRxRqvoPeFJADJFiGnaOYfIsx1v3OEbTA+CxpHaq/n0lkCNKVf8B1xEQB8+xZvWU/veFU1o8fWaYs0hQrHMco+kB8DakdrKo+veVQI4ojfpDnZtf0MHH5oIF86oD0uA5Yizla3IAvOnUTlZV33AI5IjSuPxyW3OkWXSlKiiUplM7WVV9w6FqBVEaVjnS19b1XxBe0U2/6650qXpza3rkiMKwlMNt11+sG77y+NCvb1uOFNXJ+xTTRKVL1WMJBHK03qg/vH5PbNgj69qzp+puZi4x5sJT0VSlS5VpM1IraL1xf3g3bt+sqVW24nv+5/9OtXZiSP/GNDe/INeZG1Nb2zso9gk4sVS65EEgR+uN+8PbsW1a55y18sFy8UVvbZ48pnXdB7XxJpT3xpLiWvUEcrTepD+8EwvD1yZpaw8r5h5h225CRW4sVQ88NiFIIDezq83saTN7xsxuCnFMoG/SH15sPazY2rtc225CRW4sRStd2qz0YKeZrZL0WUlvknRM0vfM7B53/0HZYwPS5BH/WGqJ+2Jr73Jtm4BT9MaSWr1+iKqVSyU94+4/kiQz+7Kk6yQRyCegciG7cX94tU4TP3xAeuhW6cQxac166cpbpK07cx2i6WntZbTtJtS2G0tTQgTyaUnPLfv4mKQ3DH6Rme2StEuSNm7cGOC0cYtp1bYgAgTAcWrpYR0+IN37QWlxKXCceK73sVQomMf4PrftJtS2G0tTQgTylbVfWrm5ibvvl7RfkmZmZkZtftIZsazaFkTAANioh249cw19iwu912O6jpLadBNq242lKSEC+TFJG5Z9vF7S8wGOm7S2DRpVKpUAeOJYvtdRizbdWJoSomrle5IuMrMLzewsSe+UdE+A4yYt5sqF3FIJgGvW53sdqEnpQO7upyR9QNL9ko5KOuDuT5Y9bupqr2U9fEC6fYu0Z23v38MHqjnPMKkEwCtvkaYGbrRTq3uvjxH7TEi0X5C1Vtz965K+HuJYXVF7pUXoHHWewcsrb3n5+aVMAbB1+teXY9C2c4PaaIS51z/uODMz47Ozs7Wft7Nu39IL3oPWbJA+fERSzlLIwRuD1AvMb/vb0UGt4qqVtrps78NDy+PybN/WRpTONsPMDrn7zODrrH7YBRNy1Ll7jUUGL7fu7ETgHpTioDZPGe3DWitNqitvPSFHnXuacyqDlzVIcVC7beutgEDenH564sRzkvxM3rqKYD5hkC53rzGVwctQxtyQU1ygKcWnjNgRyJsyLj0R2tadvfz1mg2SrPfvsnx27l5jweqNiZqorCl7zgk35BQXaAr1lEE1TzjkyJtSJj1RZOBwTI469zTnAtUbEzUx+zPEOTOMF6Q2YSXEtHjy7GERyJuyZv2ISpIJ6YlJwadAkC9UChl68LKJ2Z8hztnB8YIQpbOdWqKiBgTyphStrZ6UkinYw2y819hEQAxxzqI35MiV/X0hzx4WOfKmTMhbjzQu+NSZdw+tiQHUEOesarwgcSlW8zSJQN6krTt7E3L2zPf+zfI4Py74xPyY30RADHHOojfkjkuxmqdJpFZiMy4l89Ct8T7mVzGAWtc5OzrZqQyWnw2LKfoxGjWgWWTqPIBoMEU/JaN6gE30alFYf72SufkFrTLTaXdN0zNFAQTy1PCYH4XBOurTS0/G1FOjCAY7Y9PkuuIdUceMw2F11H0Li6d1w1ceZ7YjMqNHHpNU9r5ssVAzDict85qlXpreObKiR94WWXraMdeJRyLEyn79m8Hc/IJcZwLy8t511nppVhVEFgTyNsi6EmLMdeJZNZw6CjHjMMvNYFgddd42AX0E8jbI2tNOffnYOpf2HSHEjMMsN4PlqyJK0iqz3G0C+gjkbZC1p536dPAWpI5CzDjMejPYsW1aj9x0hX689xr9+21v0af/5JIV5zb1UjMMfGIcAnkbZO1ppz4dvAWpoxDrhxe9GQz20k1Sf7resDw70FdqZqeZvUPSHkmvlXSpu2earpnczM6yGwszI7MnwybRsciyOfG4r0l102aUU9XMziOSrpf0uZLHiVeIkkBmZPYUXdq3hSYt8zqpzJFlXpFHqUDu7kclycYM1CQv1IYIXZuROe4pJrEb2rCe96SNFS5Yu3poj5yBTwxT24QgM9slaZckbdy4sa7TVq8Fed3oTHqKiTxwLzeq5z1qVme/xx1iOzV0x8RAbmYPSjp/yKd2u/vdWU/k7vsl7Zd6OfLMLWy7ju4QU0oT27o1ZFTPu79I1qB+j5tlXs/IMt7QdRMDubtfVUdDopVQXrc2kT/F5Akso3Lap921emrV2B5349vvtQCbNGdD+WFZqZcEViHnxKY6FrHKKsv0++VG5bT7ZY1lyhy7IMSSCV1QKkduZm+X9BlJ6yTdZ2aPu/v2IC2LSZa8btkSxZTkeIppW48s7+7v43Ld9Lgno3onm1I9cnf/qruvd/dXuvvvdjKIZ9GCqeetkuMpZmKPrOzaLDm/P3NgWTrujrtfr0Pn3qD3nvtvlfS82/S0UgU2ac6GZWzr0KHBvcwyVqeMDZxla/gLfH+mssCB45698BPtmfqc9vzp66Wt10xuV0Zte1qpAtU72ZAjr0Pkg3uFBVjJcGyPrOzaLAW+f9SqhSdfOHWmN1zTmjFdyB+HWDKhC+iR16GKEsW259wDbYIxtkd2d8kbZIEbbD+A7LnnSc0vLL70+i9PLp7pDdd04+5K/pixhMnokdch9KqFMeTcA/VKx/bIyi7rW/D7d2yb1jmvXNkHeqk3XNNyw+SP0Ucgr0PoEsUWLPc6UZZeacbUS3+51//Ye40euemKM72zsjfIEt8/tjdc03LDIZbcRRpIrdQl5NTzGHLuk9JJbVhsrMT3jx307A9oVpz6YvYn+kotY1tUcsvY1i2G5V4nLc0bwzWMMVgxIvV6wwzEoUqjlrEltRKjGHYKmpROastTRcHKGqop0CakVmIUy3Kv49JJbVhsrGR6h2oKtEWcPfLQO603vHN7IVt39lIQe+Z7QfyhW+NqfxueKmIYNE5FjH9jEYmvRx6oPrmy49Ut1va34amiLemd1MX6OxqR+AY7QwySLZ9MY6+QfMgi/5EMuuX+ebR9IlGdIh9wjQY/52DSGews24sanEwzLIjnOV7T8vw8YphIFNq4R/o2pHe6gCefysUXyMvOmhuWFy1zvKbl+Xl0LSc86cbFWvL1qGmma5fFF8jL9qKy9AJi6pXl+Xl0rWeU5ca1fND4w0cI4lXgyady8QXysr2oUb0AW1XseE3L8/PoWs+oazeutuLJp3LxDXaWNWnGYcq6du0MsiEx6Qx2ltXl3kGd196GumEe6dGQundu6l6PHNVrU8+fckvUrMp1eEb1yAnkCI+URnIOPjbHKosZXbb34aErY06vXa1Hbrqi1LErSa2Y2T4ze8rMDpvZV81sbZnjIREMMial38Ocm1+Q68zeoKlt9BxKEzs3lc2RPyBpi7tvlfRDSTeXb9IIbci5IpuuVcckrgt7g4bUxM5NpQK5u3/T3U8tffgdSdX8pVY1I5GbQzUYZExKV/YGDaWJnZtCVq28T9I3Rn3SzHaZ2ayZzR4/fjzfkauYkdjF6ep16XJlUILYGzSfJtaqnzjYaWYPSjp/yKd2u/vdS1+zW9KMpOs9w+hp7sHOPWslDTus9WbkFcGA3HBVVHlQORKvwwd08hu36DdP/lTP+2/rb07t1D0vvpHdkBoyarBz4jK27n7VhAO/R9JbJV2ZJYgXUsUmBAzIrVTFcqMsYRqvpffu7MUFyaT19nPtnfoH/dbUWbrkml0E8RYpW7VytaSPSrrW3U+GadIQVeRcGZBbqYoUVtcW6krJkPfubHtBe865kyDeMmVz5H8n6VWSHjCzx83s7wO0aaUqcq4MyK1UxVMKTz7x4r2LRqkdgtz990I1ZKJx+z8WPZ5E7na5KlJYbdibE8Xw3kWje2utLMcSpi9XxVMKTz7x4r2LRrcDOV6uihQWpYjx4r2LBmutIF2plz2mfn1YoXD5IRCl1MseU78+5EJqBWlKvewx9etDLgRypCn10rnUrw+5EMiRptQnfKV+fciFQF4Gqye2V+qlc6lfH3IhkBfF6ontVqR0LqYbM6WBWIbyw6JYPTEtbdpnFBihkq3eOo3BprRQBYKIEciLYrApLdyYETECeVEMNqWFGzMiRiAvisGmlWIaLBzEjRkRY4p+GaGX1o1Z7FPGWdYYESOQI4xxg4WxBENuzIgUqRWEwWBhe8Wc8kImBHKEwWBhOzFxrRMI5AiDwcJ2oj6+EwjkCIMqnnYi5dUJpQY7zewTkq6T9KKkn0l6r7s/H6JhiBCDhe3DBsqdULZHvs/dt7r7JZK+JonnaKBNSHl1QqlA7u6/WvbhOZLqX4ErFEb2kSJSXp1Quo7czD4p6d2STki6vHSLmhD7ZJbQ2NQ3LaS8kjexR25mD5rZkSH/XSdJ7r7b3TdIukPSB8YcZ5eZzZrZ7PHjx8NdQQiM7J9BuRoQnWDrkZvZqyXd5+5bJn1t69Yj37NWw7NCJu2Zr7ctTWOddaC1KlmP3MwuWvbhtZKeKnO8xoSezBJzvp1yNSA6ZatW9i6lWQ5LerOkDwVoU/1CjuzHnppghuZoMd+gkbSyVSt/7O5blkoQ3+buc6EaVquQI/ux59spVxsu9hs0ksbqh32hRvZjT02wnOtwKazuiGQRyENLYSYd5WorxX6DRtJYayU0UhNpYuwALUYgD42ZdGniBo0WI7VSBVIT6WHsAC1GIAey4gaNliK10hXUQAPJokfeBSwKBiSNHnkXxD5JCcBYBPIuoAYaSBqBvAuogQaSRiDvAmqggaQRyIuKqQqESUpA0qhaKSLGKhBqoIFk0SMvgioQAC1CIC+CKhAALUIgL4IqEAAtQiAvgioQAC1CIC+CKhAALULVSlFUgQBoCXrkABC5IIHczP7SzNzMzgtxPABAdqUDuZltkPQmSf9ZvjkAgLxC9Mhvl/RXkjzAsQAAOZUK5GZ2raQ5d/9+hq/dZWazZjZ7/PjxMqcFACwzsWrFzB6UdP6QT+2W9DFJb85yInffL2m/JM3MzNB7B4BAJgZyd79q2OtmdrGkCyV938wkab2kR83sUnf/adBWAgBGKlxH7u5PSPqd/sdm9mNJM+7+8wDtAgBkRB05AEQu2MxOd98U6lgAgOzokQNA5AjkABA5AvkwMe3HCaDzWP1wUIz7cQLoNHrkg9iPE0BkCOSD2I8TQGQI5IPYjxNAZAjkg9iPE0BkCOSD2I8TQGSoWhmG/TgBRIQeOQBEjkAOAJEjkANA5AjkABA5AjkARM7c698+08yOS3q2osOfJ6kruxRxrWniWtMU4lpf7e7rBl9sJJBXycxm3X2m6XbUgWtNE9eapiqvldQKAESOQA4AkUsxkO9vugE14lrTxLWmqbJrTS5HDgBdk2KPHAA6hUAOAJFLMpCb2SfM7LCZPW5m3zSzC5puU1XMbJ+ZPbV0vV81s7VNt6kqZvYOM3vSzF40s+RK1szsajN72syeMbObmm5PlczsC2b2MzM70nRbqmRmG8zsW2Z2dOl390NVnCfJQC5pn7tvdfdLJH1NUsq7QjwgaYu7b5X0Q0k3N9yeKh2RdL2kbzfdkNDMbJWkz0r6I0mvk/QuM3tds62q1BclXd10I2pwStJH3P21kv5A0l9U8b4mGcjd/VfLPjxHUrIjuu7+TXc/tfThdyQluyedux9196ebbkdFLpX0jLv/yN1fkPRlSdc13KbKuPu3Jf2i6XZUzd1/4u6PLv3/ryUdlTQd+jzJbixhZp+U9G5JJyRd3nBz6vI+SV9puhEoZFrSc8s+PibpDQ21BRUws02Stkn6buhjRxvIzexBSecP+dRud7/b3XdL2m1mN0v6gKSP19rAgCZd69LX7FbvMe6OOtsWWpZrTZQNeS3ZJ8muMbNzJd0p6YaBjEEQ0QZyd78q45f+o6T7FHEgn3StZvYeSW+VdKVHPjEgx/uammOSNiz7eL2k5xtqCwIysyn1gvgd7n5XFedIMkduZhct+/BaSU811ZaqmdnVkj4q6Vp3P9l0e1DY9yRdZGYXmtlZkt4p6Z6G24SSzMwkfV7SUXf/VGXnibwDN5SZ3Slps6QX1Vsu9/3uPtdsq6phZs9IeqWk/1566Tvu/v4Gm1QZM3u7pM9IWidpXtLj7r690UYFZGZvkfRpSaskfcHdP9lsi6pjZl+S9IfqLe36X5I+7u6fb7RRFTCzN0r6Z0lPqBePJOlj7v71oOdJMZADQJckmVoBgC4hkANA5AjkABA5AjkARI5ADgCRI5ADQOQI5AAQuf8HTOkex9Fet9gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_1=np.random.multivariate_normal([0,0], [[1,0],[0,1]], size=50) \n",
    "x_2=np.random.multivariate_normal([-1.5,-1.5], [[1,0],[0,1]], size=50) \n",
    "plt.scatter(np.array(x_1[:,0]),np.array(x_1[:,1]))\n",
    "plt.scatter(np.array(x_2[:,0]),np.array(x_2[:,1]))\n",
    "plt.show()\n",
    "X=np.vstack((x_1,x_2))\n",
    "y=np.concatenate((np.repeat(-1,50),np.repeat(1,50)))\n",
    "X_train,X_test,y_train,y_test=train_test_split(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ff1f4043",
   "metadata": {},
   "outputs": [],
   "source": [
    "#training classifiers with different regularization parameters\n",
    "clf0_001=SVM_binary(C=0.001)\n",
    "clf0_01=SVM_binary(C=0.01)\n",
    "clf0_1=SVM_binary(C=0.1)\n",
    "clf1=SVM_binary()\n",
    "clf0_001.fit(X_train,y_train)\n",
    "clf0_01.fit(X_train,y_train)\n",
    "clf0_1.fit(X_train,y_train)\n",
    "clf1.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8d3362cd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.72\n",
      "0.8\n",
      "0.84\n",
      "0.8\n"
     ]
    }
   ],
   "source": [
    "print(accuracy_score(y_test, clf0_001.predict(X_test)))\n",
    "print(accuracy_score(y_test, clf0_01.predict(X_test)))\n",
    "print(accuracy_score(y_test, clf0_1.predict(X_test)))\n",
    "print(accuracy_score(y_test, clf1.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3c90a36",
   "metadata": {},
   "source": [
    "### Multiclass radial basis kernel classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69a11b07",
   "metadata": {},
   "source": [
    "Let us make one more interesting dataset and visualize the decision boundaries that our SVM produces. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "7c7a1c84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA4VklEQVR4nO2df5Ac5Xnnv8/OrLQrKcxAGaFZSQQTy7CYbAqdDovYiBSKBY68J0EAw1VdkbIrlOt8FUm+A7S7/JAB/TCqsuByzh+k7ESpcxA/jCTGGxdyxAWBjewI4ciGlUAnO0HsGpHDMwnaXbE7+94fMz3b0/O+3W/PdE+/3fN8qlTa7e6ZeWd2+umnnx/fh4QQYBiGYZJJR9QLYBiGYcKDjTzDMEyCYSPPMAyTYNjIMwzDJBg28gzDMAkmHfUC7HzsYx8Tl1xySdTLYBiGiRWvvfbavwohLpTtM8rIX3LJJThy5EjUy2AYhokVRPTPqn0crmEYhkkwbOQZhmESDBt5hmGYBMNGnmEYJsGwkWcYhkkwbOSZtqOYz+Pt61djpPcKvH39ahTz+aiXxDChYVQJJcOETTGfx9j9D0BMTgIApkdHMXb/AwCATH9/lEtjmFBgT55pK87seqxq4C3E5CTO7HosmgUxTMiwkWciIaqQyfTYmK/tQcDhISZKOFzDtJwoQybpXA7To6PS7WHA4SEmatiTZ1pOMyGTZr3ihZs2grq6arZRVxcWbtoYyutxeIiJGvbkmZajDJmMjqKYzyPT349iPo8zux7D9NgY0rlc1Qg36xVbxzmfW/b4ILzwKMJDDGOHTJrxumLFCsECZcnn7etXS0MmQNmrzty0HsW9+2o8YOrqQkdXF0qFQt1j0j09WPbiwervsguEl1GWPebMrsfkoR3H67k+n+J9IpVCz47tHLJhAoGIXhNCrJDuYyPPtBqnh1xHKgWUSv6elAjpXA4LrlslvUDkHn5IaVBl66GuLvX6iNA78qZyKZ7vT3NdYdPIxZAxEzcjzzF5puVk+vuRe/gh9QF+DTwACIHp0VEU9jzlOwauipsjlZIen87lXGP1sueTLrmB2HxQlTrWhWh6dLT62Y3d/wBX/iQQNvJMKHgZo0x/P1LZrPzBCuNK2Wxd0rQOxZ2pWwxcua9UkiZpF1y3ytVA+om3+zlWZphHBwbx1sprfBt9Tgi3D2zkmcDR8RKL+TxKH35Y91jq7ET2tlulxjU3NIjcww8h3dMDEPlak1uJpGpfuqen5vWs3z986ZCrgfRTjunnWOkdwvR0OU/hwxsv5vPKXAEnhJMHG3kmcHS8xDO7HgOmp+sfPH8+cg8+KDWumf5+ZPr7sezFg+gdebO8XwO3EknAvazS/nrLXjyITH+/Z8WM9Pk6O4F0bTGb17pUz++GlzduXYBVhNUvwEQHG/k2JcwuTJ2yQdUxolgEAKlxdaIyztk7bpdeIFRYOQLdxyg9/8p26fNt24qe7dt8rUv3dZ24XQzc8gWqi47ud4U7e82E6+TbkLC7MHW6SoPoPPVT867zXLqPW7hpo7Qax24gVc/XzOcre10Zbp+hsqQTkF50dL8r3NlrLuzJtyFBJt1k3ptOV6nfzlMVOh5/0Pj1/MN6Xcpmy2EgG17duyrSPT3S9et+VziRay5cJ9+GjPReoaxCSff0aHWCVht9iGqey6r9Brw9bK7Tbh4/n6GyCY0IPY9+Q/o45XfF0SugexwTDtwMxdTg1nFqR9aso9Poo9MRyrQet4t77/ER6XbVd8X5N9Y9jgkHboZKMI0ku2ShEhm6t+VOuAzPTNxKRVXohtWCCr8xwcNGPsY02rVYF9udN095rNNg6xhwLsMzk0YMsW7+Iao8BeMNh2tiTBC3yMV8HqP33Osao9e5LbeIWo+FcYfzIMnELVzDJZQxRqce3eukPrPrMaWBByC9LVfF5NM9PWw0DMdPqSiTDNjIxxivWnNV7fL40aP48KVD5YuBi4GnbFZ6Ww4EU5vOtDd8V9Ea2MjHGK+mHFXtcuHJPVrPnxsalG5nb5BpFrfmKYCdiCBp2sgT0VIAfwNgEYAZAE8IIR4nogsAPAXgEgC/AnCbEOI3zb4eM4uXV61TJimFCNnbv8gnFhMaKgfkva3bMHP2LMTUFICK8R8cAsCds43SdOKViHIAckKIo0T0WwBeA7AewJ8A+EAIsYOINgM4Xwhxr9tzceI1OIr5PEbvvsffgyqDN9hzYsLGrWZfRiqbxScPvxriiuJNqIlXIcQYgLHKz/9ORCMAFgNYB+APKoftBvAPAFyNPBMcftvJuWmFaQVWHN6PgQcgHfvI6BFonTwRXQLgKgA/AXBR5QJgXQgWKh5zFxEdIaIj77//fpDLaUus5iivMkfn79y0woRNTV+HBJ0GPcY/gRl5IloA4HsANgoh/k33cUKIJ4QQK4QQKy688MKgltOWeJ1EgHoQBodnmLBx65a2voekmBZmbWc5Y/8EUl1DRJ0oG/jvCiGeq2x+j4hyQoixStz+TBCvxajxkhywD8Jgo860GmW3NFFNqHB0YLB2oEw6jdzQIMsZN0jTnjwREYBvAxgRQnzTtut5AHdWfr4TwP5mX4txx01ygD12Jmq8hq0AZWPtHK7Ss30bMv39GNu6jeWMGyCIcM1nAPwXANcT0c8q//4IwA4AnyOitwF8rvI74wO/t6ZuAlSt0lpnZhk+NYw1z65B3+4+rHl2DYZPDWvtSyq62jmyGQHFfB5CkXx1C09yeCeY6ppXAKimKq9u9vnblUZuTXUmFjHhMXxqGI8ffRy/PvtrnDfnPIxPj2NqplzvPXZ2DFt+vKV67JYfb8FkabJu39pL17Z62S2jmW5pV2+dCMV8vuGpVkmHBcoMpVHxMW4Vj4bhU8M1hltFbn75bmvsbH1oLTc/hwO3HAhlfXHHq65edl60k8Y9C5TFEB3xMRlJSKraPeJF8xdhw/INgXi4YT0vADx+9HFPAw8Avz7764b2tTsqnSaL6dHRslG3OTeNnkNJg/XkDUUnSZVELI947OwYBEQ1lNFszDqs57XQNdCL5i/CovmLlPsYOTqDbpxzFSiTkR6X9HPICRt5g7AniUrj476GNCcFmUc8WZrE40cfN/J5LXQMdFeqCxuWb8CG5RvQlao3WONT422RgG0EaygJSJX+q0VMTqID3PgHsJE3BueUJ1EoQAiBVDbbVk1LKo+42VCG6vFjZ8cCqXKRGe40pZGdmwWBkJufw5bf34K1l67F2kvXYsvvb0F2brbm+OJHxUDvLpKG3+9+qVjkxj9wTN4YpI1M09OgefPQ20bCTIvmL5ImJZsNZaieF5hNgjZT5WIdr4r5W/mAgZcHqvu6090onCvUPI91d5HkKptm8IrNO49NQo6qWdiTbwE6tbqcJCoj84itMEfQzytjsjSJ7T/Z7uu5rZr3gZcHAADbr92OA7ccqDHwsnyA6qLDCVg1CzdtrAtjymjHsIwKNvIhoztsu10TrXYsb9cZO+9KNy9cZYVIcvNz1fCJiuJHRe2QiU5CV5UP6CD56efnrqXdmqoy/f3IbdvqGptv17CMCq6TDxndWl1n4wbQXkOxderMM3MyGPj0QGChjDXPrlF607o166rnsD++b3cfBOTnWVeqq+Y9d6W6qrF7L9w+s9z8XKAloqYR5vkSx14Ttzp59uRDRjcMY1UPtGuSSKfOPOjEpFsISDdkopMoVnnmVjLWfneha+AB98/MuqN45PAjoXn6Ud5FhHW+6N55xwn25EOmnbrumsHN23USZGfotXuurUt++nkNHU9e5nH78dhV+PnMgnxdILz3FDVxPV/Zk48QXVGmdsdPHDrIxOTmqzc3lejVSRTL8gFBGMNGKo6C6g1Q5RkGXxmMdV4giQUQXEIZMs2IMrUTG5Zv0NJ+AYLtDPUqfQzq8VZ9fJD4+czsBHGRVD3HjJiJpdia11jCOBdAsJEPAVnixuRbPRNwGsvz5pyHqZkpjE+P1xwXRDml7LWbMUhhGHDd1wXKn5kqgSwjiIukW99B3Gr9ZUlcO3G/8+aYfMC0U5VMmIJfrXyNpPDI4Ufw1ImnXI+Rxc3dPmPVPq9qKALh2J3HgntzIeI2Eznd0xOLO2+3mDwb+YCJa+LGL6qT/IuXfRH3rbwvolW1N6okcAd1QAghvUi6JVABuCZXh08NY/CVQcyIGelrbvvstlhckJUyxkToHXmz9QtqAJYabiFJTNzIUJXvPXXiKbzwqxdQPFdkz7vFqOLkQgilV+0l3KbaZw9RyS72cYrNq6QS4hyHt8PVNQETp87VZuqc3ZJ3hXOFUOR8GXcakTB2q/PX6QGwKodk3btBqnyGiW4FXFxHCbKRD5i4lEw2q6+um7yLy4meBBrR/XG7MOheNNZeuhaqsG8cdHh0Gqvi3CTFRj5g4tK5qrpN3/zyZi2v3k+FSxxO9CTQSD2+24XBz0Uj7oNQZMPD7chUYsXkpPvsWUPgmLwLjWpYxEHe1M3w6kjurr10LV4/87pnNQcQnxM9Cfgt59Sp8/eqbho+NYyJ6Ym65w6j3DUq4pxr4+oaBUkvhXQT57LQae23l9idN+c8jE+PY2pmqro/Ca3ujBpVlVXQYnJRY3rVHMsaNIDO7VlcEzGAnr66Tphl7aVrceCWAzh25zG8cscrePgzDwfevs+Yi6rKal7nvET93eOSa5PB4RoFXrdnTk/fSsQA/seURYFOt2QjYZaouj+ZaAhrXGOUuIVp4yhPwkZegVftrJunH4c/PDBr6O//0f01IRagPJ80KfFUJjzCGtcYFV7OW1zObTscrlHgdXsW50SMncePPl5n4AFgwZwF7JEznoQ1rjEqvMK01RDt5b0Y+dSVGLm81/hQLXvyCqwr9tjWbRCFAgCgw2b0k9Ilp7qtLp4rtnglTBxxq86Jo+6Qm/NWV4xRKpX3GR6q5eoaF6TqdESAEKBsFjh7FmJq1guOY/WNztALhvFLXIeKuFXRAFAKmQEAUilgZiaSeD1X1zSI7NbNEjIShQKEEEhls0Y3PXmRtNttxgy8NHFMxS1M6xmKLZWM7IYNxMgT0XeI6AwR/cK27QIi+iERvV35//wgXissZOWQnn/U6WnQvHnKLrk4ENbUIqa9iWvVjVvHeiqT0X4ek7phAwnXENEqAB8C+BshxJWVbY8C+EAIsYOINgM4Xwhxr9vzRBWuUTU+oaurGo9XEiM50rbm2NPAwYeA4jsApQBRAjJLgdUPAH23Rb26xOEme2xJEMctZn985TXe9sBOC21D6FLDQohDRHSJY/M6AH9Q+Xk3gH8A4Grko0KVUU91dWGmq0s5MQaIX6K1Laga9NNAZgmwbA3wT38LTFVa70U5YYbiO0D+z8o/h23onWtK+MVl1ZJVUskLS4L49TOvY//J/dWQjo6URtT4MvAwxzaEGZO/SAgxBgCV/xfKDiKiu4joCBEdef/990NcjhpVWKZUKCBz0/pq0sVJXDre2opjT5cNd/EdAKL8/5HvzBp4J1MTwN6vlB/XyjXl/yzc14yYQ6cPKfdNlibxzFvPxCpm7ze+bpJtiDzxKoR4QgixQgix4sILL4xkDW5X3OLefVi4aSN6j4+gZ+ejxqtLtj0HH5IYdI+QpCh5G91jTwO7rgS2ZICvX1D+f9eVeoZatqapifL2hOIVe5dNk9J5XFT4ia+bZhvCNPLvEVEOACr/nwnxtZpCllG3sCdQvORIGQMonm7scW5Gt8YTR324x8vQq9bU6FpjgFfHq2zIiM7jokK3ydESLDPJNoRp5J8HcGfl5zsB7A/xtTxxExOzMuoqTOpibWaaU1uQWdL4Yy0j7kR6d1DBujhUPf1svYevWlMzazUcNwG8rlQXbv3krcrSXRO/47rx9QXXrQp5Jf4JqoTySQCvAriMiE4T0ZcB7ADwOSJ6G8DnKr9Hgs5Ul0x/vzL2bkoCpdlpTm3B6geAzu7abZ3dQOd878dSSr7dy+O2PHpVzF21ptUPeK8ppthLc4FZz90q0b1v5X1Y94l11e0d1IF1n1gHoKylZP+O3/+j+yP/jkvv9onqjivu3WdMfbxFW3S86mpBqzpcs7d/EbkHHwx8XX5py+5Ue1VKd6XVYuI3s1Uzbx+or1iRVbIAZcOr8sgttkjkHHZdqfbygdmSTCeZpcCmX9S/jzaorvFC1RHbQR0Ynx6vOz47N4uXb3+5lUusw6lOKcbHUZJU3EShMR96CaXp6IqJZfr7MX70KApP7pndKASKe/dh3vLlkcfZ4tpg0jBWLNwyzBMfzO4rvgMc+Xbt7/ZySJUBtWrlZWSWyrevfkB9gejsVl847HcAbmtqQ1QdsSoK5wohr8gbpwrlSO8V0uNMCu8CBlTXtAJVuEW2/cOX6ku/TOlei/scTd+4xcJleJVD9t1W9qxv/kt/4ZO+24D+/zl7EbDCOpmltdudJDjm3ixeU8nigB+7EiWJ9OSdt1ULrluF4t59dR2tsjpWkyWENyzfIL3FTazOTCPVJ1Y5JKD2nK3tfsInXp6409MPO+Ye8/BPB3UoyyhlZOboSwq0ioWbNko75U2pj7dInJGXif4X9+5D5qb1+PClQ55TXUyXEJ6bmls18tm5WWy+erOxHYJNk1niHgtXYVW8NGO0/dDIRaMZnGGsVnbuBoSbgU9TGtNiuub3gU8PtGJZvojLtKjEJV6bHbhr6gDvuEq3NoXTmPmCgC2FoFdkBqpEsD3RazhuRQQblm+o0bRZtWQVDp0+FBuNmyhoK6lh3XBLMZ/H8ZXXlCe8XN6Lt1Zeg2I+76pCFyVxlW5tippYOAHdF5T/gcrbVnxZXfaY5Hh4Apqr3CSu7cPhNyzfgP0n93PZcBMkLlyjE24p5vMYHRgEpmdvCUuFAsYGhwCYOctRlahKbGWNhVdY5eKVrY+HR40qjBWjC5vbRCk7bs4Ne/N6JM6T95rNCpRH+tkNvIWYmjKiisaJm9eSuMoat85RGU5v36p4iUlsuiES0lxl99gP3HJAarTbrmw4BBLnyXslQ4r5vKtkqAlVNE7cQjLjU+Po292XjFjl979WVoy0BMV0E4rtVoMuS/QuW1P+/bm7Yllto2LR/EXSu9jEOTchkrjEqxdvrbxG2qVmEUW3mhd9u/sgvJQUEfNE7LGnywZK9j5jlFCMBFmCurMb+L3/rN8RbOgFoS0LDhqgrRKvbhTzeVcDD5gpMKTrtcQ6EXvwISglgWOUUIwElZTxke/U6+l8/2ux0rZvh/GUbuKJQZAII6/7IenE200UGHJT9HMS21ilmyGPUUIxEpSfneOiOTUBvPbXsdO214ndm4Jfg60jntgssTfyfj4knXi7KRIGdpzeTHZuVnmskbFKnWSq0pBT7BKKLcfPRVAmpAbw3VIANGKwVaNHg7RBsTfyfj4k3a5VE5Ovljez/drtmJyWCzkZKXGgO/pOVjECAlZ8ydh4sTGoPjsZMekrMFFT3gtdW2T39mXl3kCwNij2Rt6P1oyupoQpEgYyZHXDQFkLxMhYpe7oO1kp5M1PAF/4ZsuWGltkn92KL8nLLP/Dnxhffqk7N8G0C4GOLXJ6+yqCtEGxL6H0qzVD8+ZBjNfrVVf3GygwZEcVcxdCmGfgAX/dme1WChkkss/u4pXyKhrndsPKL1UNUDt+uqPaPHXenPMwPj2OqZkpAKheCABEdh7o2KKxrdvqvH0nQdug2HvyOs1PgE2Txmngu7uRymaNkjBwI3Zyw204+s4YLGnlLYXy/5bhtm9f/QDwT39rVLWNypEpnCtUvfviR8WqgbeIurrMyxZ59eiEZYNi78nrKsHJ4mUAkD7/fOPq4t2IndywbOCGYeGBtsYtnBaRN69qgNIhyuoyL1vkmkxNpdD7Rji9ILE38oCe1ozJOvF2hk8Nu+p56Gp+GEOrZXgZd5yNUCop5wirbWSOjC5R39G62SJXW1NSVD0FQCKMvBvWABFVksOkJKuzu08VZ1x76VpzjboMjrVHS9Wwv4Ny1Y1NNsL+u50Iw2kyR2Z8ahzFjyTzd22YekfrZYOAcqd9WCRa1kA6mNuGCTrxdlQa29m5WXSnu+PhuTNmoaXJLzH03RcAn/+GMRfnRw4/gqdOPFWzLU1pLJizAMVzRWPPCy8bBABIp9GzfVtTdqitBnnbR/+ho0N5G5Tu6TFuiotbwskaZGxCFQETI7Tm5IqyUbcPSp/4wJhpUzIDDwB//Mk/xn0r74tgRfqocoEWlM0iNzQYqh2KfXWNnboaVFWciwjLXjxolIEH2kSjhmktOrH1zFJgzvz67QbIHQyfGpYaeAA4dPpQi1fjH2Ucngi9x0eQGxrEmV2PhaZbAyTMyHtdNasIEdoH2gxtoVHDtBav2LpV6WTotCk3ZyYO54Aq55fO5VqiWwMkLFzjp1LG+kABGOPR+0k4RV1F0E7ct+/nePIn76AkBFJEuOPTS/HI+t/Fvtffxc4XTuDdwgRSRCgJgcXZbtx9w2VYf9XiqJddRlbCasXgM0tnK52qiVkHEfczuBnyOJwDCzdtlM6MXrhpo6sMQpA2KVGJV9UQb6RSrrF5k+vkZfFI1tNuDftefxdDe3+Osx/Vf3c+8zsX4Oi/FDExVb+vuzOF7Tf/rjmGXkc/XqVJH/GULVUxgoU1+Nvkc8GeJ7TXzo9c3it/ABF6R9709Rptoyev6jjr2bEdILlgk2l18naGTw1j/8n9ddvXfWKdmV9qv6P7DOa+fT/Hpqd+JjXwAPCj//uB1MADwMRUCTtfOBHm8vyh6nx1HmPgGEWvEGYcBntn+vux7MWD6Hn0GwCA0XvuxVsrr1EeH3RZd6KMfKa/H7mHHyrXnDpahN1iY6aiEiMzMuGkUpv8/tdiZ/j3vf4uvnv4XzRmcakZLXhVtBhI321lLz+zpOz1H3wo8r+XU2a7g+pNVhwKEZzxd7fhRUEPLkpUTB5Qd5wtuG4VCk/ukW43lVgNMVZOJ/r27O+6M1sjZucLJ5oy8ADQk3VK/8YAZ8jGkL+Xvfmvb3ef9Bgjzwkb2kUhAD58KVgnLnRPnohuJKITRHSSiDaH/Xoyivk8inv3yfcZOAnKIlZiZLpVGAaU5XnRrBfe3ZnC3TdcFtBqWoiuLHSExOqcsOGrKCTgEHKoRp6IUgC+BeDzAK4AcAcRXRHma8pwu4qaOAnKQhaPNLV121cVhuFTiBrxwlOVnM/ibLdZSVc/GFpGaSdW54QNP2HhoEPIYYdrrgZwUghxCgCIaA+AdQD8pY6bxOvKaGryNVZiZNJSPQWGywzffcNlGHju58rEqp3F2W78aPP1LVhVC1AJlhn094rVOWFDVkrpdmyQhG3kFwOwf2tOA/i0/QAiugvAXQBw8cUXh7IIlZi/fb+pxEaMzKk2SR2KeaLmz2xdf9ViHPnnD6q18SrsYZl9r7+LLc+/gcJEWeP8/HmdeLD/U+Z59G7llDGRhXYaeivpavp5IhQVfnYomw28bydsIy97VzVnjRDiCQBPAOU6+TAW4XYVNX0SVKywq01KhbGin9lqNTCNFibQo2hc2vf6u/jea++6GngAmJvuwKanfoav599AcXwKM7Z9vxmfwt3P/hMAmGPovRKrMZGF1lVrNYViPo/RgUFgetr1OOrqQm5oMPDXD9vInwaw1Pb7EgBqlzokasT8R0erzVEmipQlBgMNxr7X360Jw7xbmMDAcz8HUGuId75wQitUY3ntvxmfku6fKgnsfOGEOUZeZ0BIDGShVeMBHz/6uJFG/syuxzwNfJi2KOzqmn8EsIyIPk5EcwDcDuD5kF8TQO1E9LevXw0AWPbiQfQeH0HvG79A7/GRamtxmOJAbc9HZ1Gtm//BvZHWXcuMt6xxKcgad6Pq5WOQWNUhVqXF0Mj5EWF6dBRndj0WP4EyIcQ0gP8G4AUAIwCeFkK8EeZrAvWNBzLhn1aJA/nBtOnzTfH9rwHP/Wm9fO3+r0Zm6FUG17k9yBp3o+rlVQnU7vNj1bAWtzJKz5xfJSwYlg0KvU5eCPF3QohPCiF+RwixNazXsXvuo5sHpMI/o3ffU/XY3cSBosCKM1qDiuPQrq3k2NPAke/I95U+AvZ+JRKDkunu1NruVeM+f05K6/U6U2RWvfzqB8qJVDupOcC5fzdqkLcXcSujXLhpI5DWi4yHYYMSIWugrSOP2aulqtomqnJKtzhj7Dj4EKQj5SxECVEYFFVxg3P7+qsWo7tTfmp0d3agM+V92pw/rxM7b/k9c+LxgFyfZs4CYMaRUzCsAcqJU+ogNz9ntGBfpr8fPdu3gbJZreNj1QzVKvy0DAPlqyVScm8sqnLKuMUZXfET422hQSkoEqSy7dtv7kNnR6317+wgbL+5D8UJ+fMQgF/tWItf7ViL1x9YE62Bd4rFWRpCz91V3n/zE2WxMns4zY5qwHfEWCHNgZcHAAC3XVZOEg+8PGB0iDPT34/LD7+Knp2P1okoOmGBMgluNfBKSiWpYmVU5ZRxizO64rd5pkWJP1V8XLZ9/VWLsfPW38PibDcI5aannbeWPXM/zxMJMrG4I9+Wh2RIEXpSbY8QWUjzqRNPxSrEWSOiKCEMG5QII6/yyt1I9/Qgc9P62cemUsjctD6yckq3OKM9IfvZJz+La/dca3ZydvUD5VivLi3qqLz7hsvQ3Vn7XXHTmVl/1WL8aPP1+OWOtfjR5uurnrnf52k5OnNdrTsoacMa1NsjRKXKasekEKezws9KqFrSw73HR9Cz81Gpam6QJEOF0iUGL4O6urDgulVl0TLrsaUSinv3Yd7y5ZEYelW7NoCaxg/7lChjm0D6biuXS6pCATW0rgPWMtJezVCtep7Q0L0zKp4ux+WlUgZL67dFjG7o0oQQp5UntMLIqkl0KtXcIEnEZCjlRCgZqRR6dmyfbYxyYNqkKK/JOEB5Os6BWw60aEWabMnCNflac2z9eEOmCXZdqRdTt8b/GTgRSobOuQCYcT6obFJY9iXxk6F8acLPzCDT36/MYJsmVqbjlZjgudShG4Ix0GOMPbJSyTpotgPZwIlQMlYt8T7PTSmlNMm+xD5cI9WKJwK6uoCJ+riklblWiZaZJla2aP4iT+/FyOSsjiqlgeJXiaBGUkLl0YtYSRkA6oloVJHIMkmR0iT7EntPXlo+KQRSc+e6Vs+o5sGaJlbmNePSFM+lDpmHuOLLsfAYE4E111V1p2T4HZSs+1t1xyogsP3a7ThwywEjDDxgln2JvSevuv0pFYvoefQb0inpgEO0TLLfFNZeuhavn3kdz7z1DGbEDAiErlQXJkoT6KCOmmoCU77gVWLiIcYaN+lgIDbywXZUKpPnzTmvpvDAjmniZJYdGdu6DaIyz7XDoz4+LGJv5N1ui7wy163IbDfL8Klh7D+5HzOiLGQrIDA1M4XOjk5MVToVja2y8TJATHPozGQ1UA3UC1X3d1dabSSNzEsBgC3KUCoUpBU2YRP7cI2f2yJV3arJyL7w02K6auAtTKoPBiBvyDFcEyV26M5ktUI3Wwrl/w028IDaYBfPFZGZk5HuMzEvZYo+VuyNfE0HmUtDgYmqkzr48VCM8mZiMBQ69nhJBzulDWJygXXr/h749EBsxMlMqbCJvZEHbB1kI29i2YsHpbdCplxV/eLHQzHKm0mIdrnRqMpUM0tifSfl1v1tqjiZLEqgqqRpdYVNIoy8Dsqr6uio0aEb2Rc+TWl0dtTK4xrnzbgZICYYZPXwVlI1xndSXoZ87aVrceCWAzh25zEjKmpUUYIF160yosIm9olXXdyGeatajk3ATe7A6In1MazqiB1uSVVLbdJJTO6kYjPAHuoowYcvHULu4Ycir+BLhKyBG9aAkOnR0XKTlMv7jVrSYPjUsNmG2y9cXRMdKmmDzNJy8pUJjJHeK5R2JXvH7cg9+GDoa3CTNUi0J+8UCXIz8EC0kgZxm0CvRZh18nwBcYfvpFqGW5Sg8OQeAGiJoVeR6Ji8cpiIYQNDgIRNhgqbGCcVm8JPtUyMNGnijqyM207h6WdauJp6Eu3JKz3zysAQ+wUgakmDZidDJS7U44ZbUjHORszt7kSn8clJQjuOTfuuWzH20bvvkR/gUwo9aBLtyStLmCq19Pba+sxN63Fm12ORNUo1MxkqUUPA3bA8WZXoVkySilJUdyfVsX1/GttqmWZwatg8cvgRI7/rmf5+9fCiBoYaBUmijbxbN6y9tn7hpo0o7t0XaaNUMxPo2yLUU2MEFcS5PFN1d1Id26cgzhc2D1Tj/kz9rmdvu9XX9laR6HCNrgiZW6NUq8qdVKWSOrehqpDO2NkxrHl2jTG3tU3hOdKOgGVrWracwBO/jRrrOF/YPNAZ92dhQre3lVwtPP1MOUSTSiF7262RJl2BNiihlFEtq6wYfuVUKSL0jrwZ+nqaRXdiTleqy4juwIbQmTQV9EQjlSF3xseDeG3daU52DJ3gpItXbL1vdx+E5nQxE6ZBRUniJ0P5QdadBiLpsaYNEFHhpTlvYcptbUPoeKxBxqjdKnjC6CbVmuZkI+bVMrJQzH2v3FczpD4zVy5G5sS4bm/DSHS4RoZqyIiTqKtt/CAL9ag8exNuaxtCZ9IUoB/2qHrp7wAgVO8Sui8APv8Nd0Mehi6Ps3uVOgChqMpY8WXgC99s/LUMQKWuWjhXAFAONVryHXbF1a5UF9Z9Yh0OnT5kVBjSGR0waTZF24Vr3LrTLCibRW5o0Jg/UiOoQjixvq2tMcwKdDo6jz0N7PuvgEOuuUpqDlD6SPFgKt9VhN1NKgsJWcQsTCMLywy8PKAViulOdePczDnMiBl0UAdu/eStuG/lfcrnjcLY1zVdouwkytRww4LDNTa0QjCyBqqY0Uy1jrH03eYe1tDt6PzBvWoDD5QNPCnK3qzYvEoYzAvdhiarmUm2jhiVTqrKe8+bc57W4ydKE9WBOTNiBvtP7sfwqeGWlw27zaIwXeG27Yy8V3caYNYfqFFMlWRtGlWVDaX0vduJD7yPESW1IW+0m9Rvp27fbUDFwNURk9JJVXkvEWnlkZxYeaVWlg17zaIwRTdeRVNGnohuJaI3iGiGiFY49g0Q0UkiOkFENzS3zOBwDhlRYcofqBlMk2QNBJVxEzPBhi8sw60y5I1MW2okYRtzyWa3KU92JyQzJ1Mnn+32nM12iPvBy1M3RTdeRbOe/C8A3AzgkH0jEV0B4HYAnwJwI4C/IFLd/7YeeyNUuqdHegxl9DL7TIsJwuh1X+C+PzVn1mMPcmxeIwnbZkJDBqDq2CYiDLw8AADYfu12vHLHK3j4Mw/X3Hm6jfprpkPcL16e+oLrVkn3q7a3mqaMvBBiRAhxQrJrHYA9QohzQohfAjgJ4OpmXissFm7aCKQlRUZnzxo7SKStCcLoff4bZUMuo/sCYN23wklqNnKBirnQmKq8d0bM1MXSnXeebqP+WplzcvPIi/k8PnzpkHSfanurCSsmvxiAvfzgdGVbHUR0FxEdIaIj77//fkjLUZPp70dqwYK67WJqShqXj+Mw8EQRhNHru61syO3PcfNfAluKwL2/DM+ANnqBitkgbidzU3OrPxPqQ6SqWLpbXqmVOSelIygExgaH1MOIDAn5epZQEtHfA5DdAw0JIfZXjvkHAP9DCHGk8vu3ALwqhPjfld+/DeDvhBDfc3utVnW8OlGWVTo6Xk0olWJiThvp4DtnJLhBIBy781gLVtUYx69aDjGh6NFIpaRKk60cQtTU0BAhxB828JqnASy1/b4EgEI7IHpU0gbO2zQTNG6YmJNQ+V8ZfrRnjBpC76CYz6sNPGCkdLmdsMI1zwO4nYjmEtHHASwD8NOQXqtp3NQq7ZheKsUwJqFb6WJ6/4ZXObVMutyku/umZA2I6CYAfw7gQgDDRPQzIcQNQog3iOhpAG8CmAbwVSFUPdrRo6tWqevxMwwDpbxGZk4G8zrnRd6pqourE9fRATE+jtF77kU6l0PPo98wxrhbtJ2sQTNwTJ5hZvGSFZDF5E1XQpVp0JzZ9ZhaqbajA5iZbViLyh64xeTZyPvEZCGisDFFK4SJHl0DHqfvTDGfx+jAIDA9PbsxnUb3f1yBicM/8dS8qj6khQlXCzbyTNPE0Stjgscy2iqV0zgL4B1feQ1EodD8E0Uwh4IFypimaYsRg4wrdlEwFbGVsgaCMfAwL0fHRt4FbnyapZVaIYyZ6JREmlwK2QpMKp20aLuhIbo4k6yW8pxFu8TlrdtzlfZ3u5/U7YBXiMbC9FJIL1LZLEo+vXnq7ATmz4coFo21BWzkFagan97bug0zk5NS45/p709UYtarYzHuJzXjjW7Xam5+zuikqg4XDQ1ibHAIYso2a6CjA5RK1W6rkO7picX5zUZegao2Vnalt8uOqrx/078IMtxuz5NwUjPeeIVo4p58dzplmVv+GB++dKjGSQPifefO1TUK3r5+tbo2VkG6p0feLBVBSVUQ9O3uk4ZpGtUZiVM5XdJo9LNXfQeA+F/oi/l8nedOnZ3IbdsaKyMOcHVNQ6ikDpSDRlKpxMkeBKnZ3epxbcwszXz2qr+1VSoZZwM/es+9dWEYMTWF97Zui2hV4cBGXoFzgpSlR6FsiCiV1KVTHR2xrMwJUrObSzCjo5nPPomzgq2iCtW57Df5ajock3ch099fd9umanG2kjBO2QMAQKkUy9i85aUFEWLxU4LJYZ1gaab8NcjvgCnIiiqSDBt5n8gMuVUbaxnw0c0DdfrScZUktgY0NItKrMoZDnBWc1ihBWst7URQFzvVZ09E6Nvd5/ncQX0HTMErfErZbGsW0iI4XOMTVRjHMt6Z/v4awSI7cY3NB4HubT+HdcoEmcPwM4KvHUi5zW9Op5EbGmzdYloAe/INIAvj2GFJ4np0b/u5s7aM28XO+sx0PX3nZ09EmBEzrs+dZOQuGAAi9GzfFru7bS/YyIeAW0inndG57dcN6yQdt4vd8KlhbP/JdhQ/Kla3e4W17J993+4+X68ZN9waEov5vKtGTdIMPMDhmlDwCukwapJYzSFj+NQw1jy7Bn27+7Dm2TU1oZLhU8MgRaluZm4GW368pcbAW0yWJjH4yqD0Oe0EWRprGlblzPToKCBEtSGxmM/PVtUoSOqdNjdDtYAkSR20giASjqZW6AyfGsaOn+5A4VyhZrvVOQpAKSPQlerC3NRcqYGXoepGTbJstLKJMZUC/dZvKb34uA//YT35COFpUq3HzYgB9XkB2bYwjJ2XDkxuftmTlIWrOqgD2z67DQMvDyg7UFXPKdN3N/Ui2CwjvVdoD/ew07Pz0Vifj2zkI0TlWcRV6iAOrHl2jXK26LnSuRojm6Y0iAhTM7Odj15ebaMGUrUuC0I5RKMy4juu3aGlBul8zkYkKOJKo3IkcT8XWdYgQnSkDli3vha3eLUOqgRi8aNinRc9LaZrDDwwW2kiW4estHHzy5tx7Z5rPdfpldhcNH+Ra1x8y4+3YNWSVdJySOsCIXvOdkImR+JGOxREsCcfMl6ePIdzagkiXuzlMevSleqqW4dbTDxNaSyYswDFc0Wph++2Lp2YPDArCiYLOSU1zu6XYj4vbUgEyprxNG9e4vJjHK6JEC8jrhvOKebzGNu6rZo4SmWzuGhoMBFfUDsqQ+hndqjqQtGV7qpLeKrooI66WnK/OI2sKiafmZPBwKcHao7b/PJm6XO6hV+SGmdvhHZzntyMPNfJh4z1hVJV1+iGc5xT5EuFAsYGh2peIwkE0QylarwC6r1dVUzea0iGDs4GI92GsLWXrlXG3t3CL0mTH2gGr/OunWBPPmJ0PHmvZFJcJtToEIQn74bM2wXqDa/KyGbnZjE5Pal9EWhGe5/DL7VwKbIa9uQNRqc71kvzJu4TqOxsWL5BatyCaoZSebuybbJ1bL66HEZxdpyqaDTxmUT1x2Zwm7kc9+982LAnbwBeHopuWVgSSsEAc2LLXuuw7z9vznkYnx73VYrJ6MOlyO5w4jWG2A1/KpNBqVj0bvIgQu/Im61ZIFOHKRenJKJscuLvPAAO18QCu1GnTAY4e7Y6mqxUKIA6OyHSaWBiQvkcSdXeiAuc+AwPVnZtnKaaoYhoJxEdJ6JjRLSXiLK2fQNEdJKIThDRDU2vNME4RZVEoSCdPZk+/3z1jFkg8U0dTHthbxIsjY+DOjtr9rdDI1MQNNvx+kMAVwoh+gC8BWAAAIjoCgC3A/gUgBsB/AURpZp8rcSiO47MitnLoGyWE1BMYpA6PkIglc2ysqtPmjLyQogDQgirePswgCWVn9cB2COEOCeE+CWAkwCubua1kozuxCgrKets26auLuSGBlkegYkVbt9XqeMzPQ2aNw+9I29i2YsH2cBrEmRM/ksAnqr8vBhlo29xurKNkaCKN9pxzpF1VuMAkJaY/ea55zDx038st3gTAV1d5bh+KgWUSomqsWfig1dJpE6TIKOHp5Enor8HICv2HRJC7K8cMwRgGsB3rYdJjpeWhhDRXQDuAoCLL75YY8nJQ1Yrj3QaqQULUCoW68oqZeMH375+dZ3nIyYnMfGq7VorxGzitqLrwfXGTBTIPHX7sHtOtAaHp5EXQvyh234iuhPAFwCsFrP1mKcBLLUdtgSA1FUVQjwB4AmgXEKpsebEEUQLdjMejv3kYpiwKebzyjtX63vMIzSDo6lwDRHdCOBeANcJIcZtu54H8LdE9E0APQCWAfhpM6+VdLyGg3uhE/Jxg2+DmVagO4KPtWeCo9mY/P8CMBfADyszKQ8LIb4ihHiDiJ4G8CbKYZyvCiHqdT+ZwJCGfHzAt8FMK3CrJHN66s06PkyZpoy8EOITLvu2AtjazPMz+sg8n87fvrg2Jq+Ab4OZVuF2x8glkeHAk6ESRKa/H8tePFgtMbvkr/4K2TtuL1fSqEilGj65uGSzvQji7626Y0z39LCBDwnWrmkT3IYoAP5jn7LnA5I7zKRdqcptjI6WS3Bt9qKR70+7DfNoFaxd0+ZYJ6qYnKyrjx8/ehSFJ/dUj9UtqVTFVkuFApdkJoRiPo+xwaFZiQ2HQygmJzG2dRswOaktAcwJ1dbDnnzC8fLgR++5V6ru5yXhqlQF1Hw8Yz5vrbwGpcq4Sb/w37+1sCffxrg1nZR/kRtqr5JKr5JNLsmMD6p5Bo0aeID//ibBideE49Ye7nYiepVUyjR0/DyeCZ5GEqPW/GBLCGx6dBSjA4Naj6WurrJgmAT++5sDG/mEo6xmyOVcT0SvkspMfz9yDz8Ekp3kRJgeHa0amjCrcLjCp4xTtdGKjXt9HmNbt9UMiAcATE9jbOs2+d+2gqUCedHQoFQwj0tyzYGNfMJRqVYu3LRR7o0TIXvH7VqJsEx/Py4//Cp6dj6KdE/P7I5KCMjyCscGh3wbHx0aNWxBvK5pFxbPsJwCoQjJiEIBuaFBIO2I6KbT6Nn5aFUF0rrYp3t6WALYUDjx2ga4zZD1mi/rB91ZtEAwiTmvuZ9BvjeLVpUA+l17o+PxRi7vVe7rPT4SymfIBA/PeGVaglfFTQ0BzOZ0e72enY+GYowbvbD4MZaNXEh0B10711EqFCDGx+sel8pm8cnDr3p+HowZuBl5DtcwgeEn2dZsYq6YzwMd6q/v2NZtDYUvvHBLZKvCR2Nf/7qvsJIq9DK6eUD5GLewnIVsfTIDT52duGho0O1jYGIEG3kmMKQx/nQ68NmcVSXDklrzThVrbra0T3lx6ujAe4oLS+HpZ+RG++57MPKpKzFyeW9NbF+5xlJJeXHQiY3rjJlMZbPIbdvKIZkEwXXyTGC4Ta0KMq6rOxNXRjN3EMV8HiWJ5wsAKJXUdeUuFyPZ8JZUJqN8LuvicGbXY3Wfo5dqo84FjubNYwOfMNjIM4GiMjRBGg5tbzydrikPbOYOQqXVEyRichLvbd2G0ocfeh7byEQvnZkD3MSUPDhcw8QOXW88tWBBIKV9xXweo5sHQjXwFqVCob5uXYHfHINXAxvATUxJhD15JnYs3LQRo3ff43lcqVhEb5MVIjrxfyUO1cYw8ON514TTJB49NzElE/bkmdiR6e937ca0CMIrbSb+79fAU1eX1vuy4/c9VmcOHB+ZbWLjJqZEw548E0tyQ4OuMfKgvFI3T5k6OyHSaWBiwvfz0rx55fJFh/QzAGmNfOam9Sju3RfoYGser9cesJFnYomzkocyGXSgHKIJsjNTmaxMpaqlhipJ3lQ2ixmb1jqg35Alq0aat3w5d58yvuGOV4ZxQaf7NOipWwzjF9aTZ5gG0Zlk5HUMG3UmStiTZxiGiTmsXcMwDNOmsJFnGIZJMGzkGYZhEgwbeYZhmATDRp5hGCbBGFVdQ0TvA/jnqNcRMB8D8K9RL6IF8PtMFvw+48VvCyEulO0wysgnESI6oiptShL8PpMFv8/kwOEahmGYBMNGnmEYJsGwkQ+fJ6JeQIvg95ks+H0mBI7JMwzDJBj25BmGYRIMG3mGYZgEw0Y+JIhoJxEdJ6JjRLSXiLK2fQNEdJKIThDRDREusymI6FYieoOIZohohWNfIt6jBRHdWHkvJ4loc9TrCQoi+g4RnSGiX9i2XUBEPySityv/nx/lGoOAiJYS0f8hopHKd3ZDZXvi3qsTNvLh8UMAVwoh+gC8BWAAAIjoCgC3A/gUgBsB/AURpSJbZXP8AsDNAA7ZNybsPaKy9m8B+DyAKwDcUXmPSeCvUf4b2dkM4KAQYhmAg5Xf4840gP8uhOgFsBLAVyt/wyS+1xrYyIeEEOKAEGK68uthAEsqP68DsEcIcU4I8UsAJwFcHcUam0UIMSKEOCHZlZj3WOFqACeFEKeEEB8B2IPye4w9QohDAD5wbF4HYHfl590A1rdyTWEghBgTQhyt/PzvAEYALEYC36sTNvKt4UsAflD5eTGAd2z7Tle2JYmkvcekvR8vLhJCjAFl4whgYcTrCRQiugTAVQB+goS/V4DH/zUFEf09gEWSXUNCiP2VY4ZQvlX8rvUwyfHG1rHqvEfZwyTbjH2PGiTt/bQtRLQAwPcAbBRC/BuR7E+bLNjIN4EQ4g/d9hPRnQC+AGC1mG1IOA1gqe2wJQBGw1lh83i9RwWxeo8aJO39ePEeEeWEEGNElANwJuoFBQERdaJs4L8rhHiusjmR79UOh2tCgohuBHAvgP8khBi37XoewO1ENJeIPg5gGYCfRrHGEEnae/xHAMuI6ONENAflpPLzEa8pTJ4HcGfl5zsBqO7YYgOVXfZvAxgRQnzTtitx79UJd7yGBBGdBDAXwP+rbDoshPhKZd8QynH6aZRvG38gfxazIaKbAPw5gAsBFAD8TAhxQ2VfIt6jBRH9EYDHAKQAfEcIsTXaFQUDET0J4A9Qltx9D8CDAPYBeBrAxQD+BcCtQghncjZWENFnAbwM4OcAZiqbB1GOyyfqvTphI88wDJNgOFzDMAyTYNjIMwzDJBg28gzDMAmGjTzDMEyCYSPPMAyTYNjIMwzDJBg28gzDMAnm/wMafOWjgkJt3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_1=np.random.multivariate_normal([0,0], [[1,0],[0,1]], size=20) \n",
    "x_2=np.random.multivariate_normal([0,0], [[1,0],[0,1]], size=60) \n",
    "x_3=np.random.multivariate_normal([0,0], [[1,0],[0,1]], size=100)\n",
    "x_4=np.random.multivariate_normal([0,0], [[1,0],[0,1]], size=200)\n",
    "x_2=8*x_2/np.matrix((np.linalg.norm(x_2,axis=1))).T+1*x_2\n",
    "x_3=16*x_3/np.matrix((np.linalg.norm(x_3,axis=1))).T+1*x_3\n",
    "x_4=24*x_4/np.matrix((np.linalg.norm(x_4,axis=1))).T+1*x_4\n",
    "\n",
    "plt.scatter(np.array(x_1[:,0]),np.array(x_1[:,1]))\n",
    "plt.scatter(np.array(x_2[:,0]),np.array(x_2[:,1]))\n",
    "plt.scatter(np.array(x_3[:,0]),np.array(x_3[:,1]))\n",
    "plt.scatter(np.array(x_4[:,0]),np.array(x_4[:,1]))\n",
    "plt.show()\n",
    "\n",
    "X=np.vstack((x_1,x_2,x_3,x_4))\n",
    "y=np.concatenate((np.repeat(1,20),np.repeat(6,60),np.repeat(2,100),np.repeat(4,200)))\n",
    "X_train,X_test,y_train,y_test=train_test_split(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdfb4c14",
   "metadata": {},
   "source": [
    "First, we will use 'auto' gamma value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "4ab7eb35",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "clf=multiclass_SVM(kernel='rbf',C=1)\n",
    "clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "645d9775",
   "metadata": {},
   "source": [
    "On the below plot, we will show a color of SVM's predictions for all integer pairs of points in the range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "ccf33f52",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAArAklEQVR4nO2df5BdZZnnv0/f7nR3IOnmR4dtIJgIwSJBCrFFqFiYWSj5oREdBzcyNcusbAHKrGLtVIHLriyrlDpWKdYuDlDqyNQSIjXoECSI4JihTAWkUQYTohB+KCxdpAkmISTp9O1+9o9zO7n33Pc99z3nfc8573vu86nq6tvvPfec9zz93vee873f53mJmSEIgiBUk56yOyAIgiDkh0zygiAIFUYmeUEQhAojk7wgCEKFkUleEAShwvSW3YFmjj32WF6yZEnZ3RAEQQiKp5566g1mHlE959Ukv2TJEoyPj5fdDUEQhKAgoj/onhO5RhAEocLIJC8IglBhZJIXBEGoMDLJC4IgVBiZ5AVBECqMtbuGiAYAPAagv7G/f2Lmm4joaAA/BLAEwMsAPsnMf7I9nordDzyAHd+6FfWJCfSOjmLRF64DgK5vG1q9WmIjcZDYBBibodWr4QqyrUJJRATgCGbeS0R9AH4J4PMA/hzAm8z8NSK6AcBRzHx90r7GxsY4rYVy9wMPYOJ/fAl84MDhxt5eEBF4erpr22hgAEMf/xh2//ifuzo2EgeJTWixoYEBjH75f6Wa6InoKWYeUz7nstQwEc1HNMl/BsA/AljFzBNENApgIzO/K+n1WSb55//9+ai/9lrWLlebWg2YmSm7F+UjcdAjsdFTYmx6jz8ey/7l58bbJ03yTjR5IqoR0dMAdgB4hJmfAHAcM08AQOP3Is1rryKicSIan5ycTH3s+sRE9o5XHXnzRkgc9Ehs9JQYG5fzmpNJnplnmPlMACcCOJuITk/x2juZeYyZx0ZGlFm5ifSOjqZ+TddQq5XdAz+QOOiR2OgpMTYu5zWn7hpm3gVgI4CLALzekGnQ+L3D5bHmOPKD5+Wx20rQt3RJ2V3wAomDHomNnjJj43Jes57kiWiEiIYbjwcBXADgdwDWA7iisdkVAO63PZaKvf/6WB67rQTTL71cdhe8QOKgR2Kjp8zYuJzXXBQoGwVwFxHVEH1o3MvMPyGizQDuJaIrAfwRwGUOjtWGaPIJiN4aIXHQI7HRUxFN3nqSZ+ZnALxH0b4TwPm2++9EbWgIM7t25X2YMCECZKF2iUMSEhs9JcamNjTkbF/BZ7zOlt0Bn5E3b4TEQY/ERk+JsXE5rwU/yfPu3WV3QRAEwSku57XgJ3mXtzWVg6jsHviBxEGPxEZPibERuaYJkWsSkFvxCImDHomNHpFr/EDkGkEQqobINU2IXJOA3IpHSBz0SGz0iFzjByLXJCC34hESBz0SGz0i1/iByDWCIFQNkWuakAJlCUjxqQiJgx6JjR4pUOYHUqBMjxSfipA46JHY6JECZZ4gBcr0SPGpCImDHomNnqoUKAt+kpcCZQlI8akIiYMeiY2eihQoC36SFwtlAmKPi5A46JHY6BELpR+IhTIBscdFSBz0SGz0iIXSD8RCKQhC1RALZRMi1yQgt+IREgc9Ehs9Itf4gcg1CciteITEQY/ERo/INX4gco0gCFVD5JomJOM1AclmjJA46JHY6JGMVz+QjFc9ks0YIXHQI7HRIxmvniAZr3okmzFC4qBHYqNHMl49QTJeE5BsxgiJgx6JjR7JePUDsVAmIPa4CImDHomNHrFQ+oFYKBMQe1yExEGPxEaPWCgjiGgxEf2CiLYR0VYi+nyj/WgieoSInm/8Psq+u+2IhVIQhKrhm4WyDuC/MvNpAM4BcC0RLQdwA4CfM/MyAD9v/O0ckWsSkFvxCImDHomNHpFrIph5gpl/3Xj8FoBtAE4AcCmAuxqb3QXgY7bHUiFyTQJyKx4hcdAjsdEjck07RLQEwHsAPAHgOGaeAKIPAgCLNK+5iojGiWh8cnIy9TFFrhEEoWr4JtcAAIjoSAD3AbiOmfeYvo6Z72TmMWYeGxkZSX1ckWsSkFvxCImDHomNHpFrDkNEfYgm+LuZ+UeN5teJaLTx/CiAHS6OFUfkmgTkVjyixDjEj3yQgOlYtnypY1jGiB6RayKIiAB8D8A2Zv5m01PrAVzReHwFgPttj6VC5BrBZxjA5MLoTTu5EPj71YTvfJha2uRaWojjcl7rdbCPlQD+CsBviejpRtt/A/A1APcS0ZUA/gjgMgfHaqN3dBT1117LY9fhU6tJRiNQahx2LgSuvbb9bbZpxeHHt91Wx4ixwOkYGSN6SoyNVwXKmPmXzEzMfAYzn9n42cDMO5n5fGZe1vj9posOx5ECZXpCLD7FAGZil7bTConDtG2qD9h+3jsx1Zft9XW0Sy6m/T7QC6xd1fk6fe0qwlTsc8D0uLaEOEaKoioFylxcyZeKFCjTE2LxKQLwdj8wNQ84Zk90JTw3UV6+kTO1bT79Dzh3AWV6ff80sHB/9n5vWtG5XG20zUym49oS4hgpiqoUKCP26IuXsbExHh8fT/Wabactly+PKsYsgDVf9OP6Y91X68a3uy77nea4QgUhwmnbnk2xOT3FzGOq54IfR2KhTCBQe9zOhW73RxZfbe4dzGfbIveVSKBjpBDEQukHYqFMIIA7nHgPTXXsdMewiEOal7oMt2Zf8Wbr8R/AGCmNilgo/bgntkAslGHDAN5YmF7HLoojD6Tb9rbb6k7ORXfceLyOLcuVI+SKbxbKUqkNDWFm166yu+EnRN5fqe0dVFsMXUKgzFfzOxfC2N5IOLztyB7g6g0MYCbTRK87btySaW2/DGCMlEaJsRG5pgmRaxII4c1bQBdt5Jq1qwgHYp9BujEXF5kG6pFTx9VxVVKWartUWbWaMeJ9pm4RiFzjByLXhE0aOaQMVPbGNBLJMRmvslXHVck/uu0AZO4zoJbRbPcpmCNyTROS8ZpAANmMrp00K7e2T3ibT+/DuVumO06Yprw1aO5h3zvQrtMDZv7+TStqLZmxOnTbGWXVasaI95m6RSAZr34gGa96isjYs8lQde2kWbl1BtdsYIzsiQb2yB7gmocYX9n8DlzzUGv71RsYK7d2fgOr9jl/v3kW7BFTaHntZx9gfPZB7thm2j9TVFm1umzgEDJ1i0AyXj1BMl71FJGxZ5uh6tJJc/lGRn+9ta1/GjjlsReB2Hw5p5d3ulJW7dP0TUMAarEZry/6LrZjm2n/TNHJOrpsYN8zdYtAMl5zQDJew6SsDNW4NHPsnnQVHWcRva6T7lxWSoxPmb+mSKauIxxmvIY1ghSIhTKBgixgrnV1E1ZuncHVGxgDjavskT0JMoEmDnHL42ceYFAP0DdjsM8CKCLr1cZeqmJviu8rvEcslH5QeRuXDSkGqIllTtWWR4aqCZdvPDzBz6HthSYO8e3n8eEJvuM+i8Bze6lmh9VBLJR+IBZKN5hY5orQ1QG1QyZ+3G6w77nMoC0K3y2xoSAWyiZErkkgxe2mLvO0kzXPNSoZxlpGCTSr02UGrf4YItdoEbnGD0SuSSDNAPVkDlTJMNYySoATPOA2g1aHyDUJVESuCX6SF7nGDb7cZmfNEO0WfI+PL+ModESuaUIyXhNIkbFXhkMGaNffc7ndDyDz1xSbDFpVmy4bOOs+KyXXVCTjNXif/MTNN2PXPety6lHY9J1yMqa3v9BxOwbw07OAf7iw2M/8uWzS5mSjOoAa3LpaTOPgOwxglloTrKYJQE+rnGXaNtUHvPKBZVj8y+fRP+1mn3n8/8qizHEz/Kk1GL3pJuPtK70ylGS86jHN2CMAY9tz7YoSXTap6wmiKuuY6jJo499XmLbNZQM3T/C2+8zj/1cWVcl4DV6uqU9MlN0Ff0lxq2mj9ZpYHku1QVZEqskFiY2eEmPjcl4LfpIXC2UCKS2UWTC1PJaaTRqohbIQJDZ6KmKhDH6SFwtlAgVYKHWWx3jBLVVbYbf1MonpSZkV3fw/O0ho+eDWtc0iUF24IhZKJ5M8EX0fwEcA7GDm0xttRwP4IYAlAF4G8Elm/pOL4zUjFko3qLIrAY8kF6F0smZFyxhJj8t5zYm7hojOA7AXwD82TfJ/B+BNZv4aEd0A4Chmvj5pP1ncNc+dc67INTpS3G5mvUqLv85LCNVK0nFJijEyqVlIpBPBLi5SplwzPIxTH99svH3u7hpmfgzAm7HmSwHc1Xh8F4CPuThWHJFrEkgxQE2KdXlXwEtJ6zlTbRbDJ+8D1WIjhWZBPVx8m2+fNoZrvNoUokuzTq5XiFzTkeOYeQIAmHmCiBapNiKiqwBcBQAnnXRS6oOIXCPE6Z1fR31fDb3zZ7DojLcwtGQ/5h87iB3PLGhpB1B4W32f3wXG5pjz5Pdw9PsXZyBzzRzbdXK7Ee/kGgAgoiUAftIk1+xi5uGm5//EzEcl7UPkGrfM9hB6Zj27csyZ3vl1LPvojlirP3rNcz86DjMHPZroNZJEXIY70AvccYm7KpjfvbXuf2asyDUdeZ2IRhsdGAUQf+c5IYjbvgJQ3V7/7Ey03SarasKHEcPOUgjVZg9dPSe+tkS8+8w1rLXvvDiab3FQURG5Js9Jfj2AKxqPrwBwfx4HEbkmghF9MTbb+H3HJYTvX1jDHZdQS/vfryZ858Otbf7p6mp650fLRPfOr+P49+/G6Nm7WtpG37cbQ0v8vjzk6SDNhADcFkeTQmbJeFegjIjuAbAKwLFE9CqAmwB8DcC9RHQlgD8CuMzFseJIgbKInQrnQw/1YNOKzjXhQ3A/9M6fUcgwMJvUqQZwOdmLu19u/S6gNm/WL7kmRREuVXG0rPLNzoXwfsxVpUCZK3fNp5h5lJn7mPlEZv4eM+9k5vOZeVnjd9x944QjP3heHrsNCgYwfkp7+9IFS41eP36K/3fPR45aXPods8xdR1Kw++VBTDw5hPq+qKJLfV8vZg6meMtRXKJi526dvqVLjLZjAEdMRRNzDw4vYrJya7ZJcO0qwlTsErMOv8ahaWzywOW8Fu69YwMpUKYvMPbSWy8ZvX5su/+Szd6Jgewv3vm8u46kYMczC8Az8beYeaSpdzYmUe1qk6hq8+zU2zRF7OLF0Wx0+k0rarg9JiXuG/RrHEqBMk+QAmURKr10ls0mAN8XogBgZz3USDVxKcU3uyRP92DZJ9rHd7NEtW2d5W29pRxxzJ7sEs6mFbUW2XDdV+v6jctACpT5gRQoi1AVGDNdvzOEhR5659u84dotlHNSytyVdn1fL157YghEBJ4lJ2224oPJVXr0YWLxNra0Cbpch9Y7nb4iBcqCl2vCsP8VgGIsGq/f6ZMQCkCVtaq2RmbbH6CRUrjn0CTtpM1SfDCxWy464632bN40/1DLScyl1dK7zNiKWCiDv5IXC2WEjSXNRzubKmvVBJUEM7Rkf1t7CJmnJnbLubj4dG5ZJRzJjD2MdxbKMhG5JkK1Rmuoco3OLtkJlQQz8eQQ9r3Rj90vDbS0e3j70obpl6pDS/a3fAimyqrNQZKwkXDiOn2pmbEi1/hBN8o1psWjwpBr3EkzKgmGZ3qw64X5Vi6XssiaHZvqdTlMYk6zZcscmxWRa4Kf5LtRrlFlt9rUFClbrnGVtaqVKfy/aFeSNTvWx6zarA6ussdmWYhc00Q3ZryqsltV9FCPkY2yTFdDVmkGSJFNWmLmog1ZHUWpHDcFxUYlJ5q+rjTHjWS8+kG3ZbzqsltVeJfxSu6kmTTZpGVmLtqQNcs3jeOmiNikGbNxyszGloxXT+i2jFdddqsK3zJe4xmcNtJMmmzSMjMXbcia5Tu0ZD9G37fbKDO2iNikGbNxyszGloxXT+jGjFdTi5pvGa+6DE4TrCyQAUo1QPQdw/PrF2WyksYdN9rM2IJik3WMlZqNXZGM1+Cv5F1ajUJhzqLWqVAUGV4DqbJl8yBrnRWVNJMK8t9Jo6P5nCeeHMLul7P9s7T6fkGxyTrGsmr5Tihx3IiFsolutFCaWtR8s1BmtQTaFvoq0wpnR+s58kwPdjyzINOeVDo91VRr0OZExsOUmgVbEQtl8HJNN1ooVdjc1hZlU+Ppnjb5AajOuqhFoJJwgPYYxmUdVWbsojPewmuPD6MI1TvrGNu0ooZTX63jQ785vN5sUZ9LZSIWyiYk4zUilIzXOamlyEJfZWYu5kGnGE48Gd3qqyb6eNvrvxnGzFT+fc4qu6zcOoM/e+ZwmeMaF+i2kYxXP+hGuSbcjNdyCn1VaYI3iWEaWWc2h9iYjk8TLt/IGIhVIC5MKRe5xg+6Ua5hAG8sROZl2FZubS0CtaBLswqrjKmswwfdT5m247OZENY6yAORa5roRrlm76BZxqtKrlm5dQZXbzh8dTSyJ9isf3MqJteYYiLrUC+BHa/VYTo+TfclBcrsELkmRIxVmPYNS739betPUc4Ou+PEX32QejBNrVem/o1DM1nH9QQf7djTfaU+djXkmuAn+W6Ua2zcMGXf/jZnYYYCA3h9cBizjd/fOus/4JtnfbKlLVwnvntcurWkQJk9wcs13VqgzARVgTKfipFFmnEBQ9Cy0NTk4DD++sL/3ta+cfF7Dz3+wcNfwXH7d2U+RmnkUIRr70D2dV/jSIEye4K/kpcCZXpUBcp8KkZmvXSdIaaFphhAnVrfEgdqffjB8os7vvYHyy/GgVpfS1sd/n/f4boIFwM4YsosI9sEKVBmT/CTvBQo06MqUOZTMbI0hbRsMC00RQDe7u1vkWG+feZftFyx69i4+L349pl/0fLat+fN917GcV2Ei3DY0z6HzaIhUqDMntzvlYnoIgDfBlAD8F1m/prL/XdrgTITVAXKfCtGZlxIy4YUt9wLpvdjzYe/nOkwGxe/t+UD4cF//ttM+ykUKVCmRwqUdYaIagBuA3AxgOUAPkVEy10eoxsLlJkWe1IVKPO9GFkeV/JpCk1NDg47O+xb8+Y721dueF6grKjxqkQKlBlxNoDtzPwiMx8EsA7ApS4P4J91rQAsLJS+FyPL+rpENFa4tsxMQ/3d9rheUVQfsx5GLJTW5D3JnwDglaa/X220HYKIriKicSIan5ycTH0AsVAW99o0hLA+adwaaaq/m7JguqwsHv/IOu7EQmlP3pq86n6n5eORme8EcCcAjI2Npf7o7MaMV1MLpSrjtagMQhu5RrlOqwVMBFJclemska54a958DB3cl9v+nVBQVqeNXCMZr3bkfdn0KoDFTX+fCMCpqb3qco1NsSeRayL29/S22RudSzMqSpVrYsem9trxIdSTF7nGnrwn+ScBLCOipUQ0D8AaAOtdHqDqcg0DmFwY/dMnFwJ3XJI9sQToTrlmYGa6zd7oWppRUbZc02xNPf79uzF69q42WyvPFDORiVyTjmDkGmauE9HfAHgYkYXy+8y81eUxqp7xunNh9mJPZWa8apebM3id6yzYN484qs3eWASTg8OlZcHGs4vniOcq7NhyFOp78+9P1nrykvFqT+7fcjHzBmY+lZlPZuZbXO+/yhmvabJbVZSZ8XrkaLZLsKyvS+KNd7/P+T5NeOK409zHmuIyDCtlmHh2sY4j372480aW2IzjtasIU7HP/KIyiSXj1ROqnPGaJrtVRZkZr3snBgp9XRLH/vZJ5/s04f2vb3Me63gm8fHv36WUYeJX7Dr2/vaVzhtZYjOON62o4fZLqEWy3DdYzBiWjFdPqHrG6zF7shd7KjPjNeu6rHms53r023/Cqleewl8/+xBG9u/C5ODwoS9d82xblINUo8skNp3U49T3MoqYMm3G3aYVNWxacfjvdV8tqIJpRTJeg5/kq26hJBzWJOeKPQEzRhO9ykJZlMbpk4VyqjYPn3/6nzAwMw0AOG7/Lnzh1z8EgdDHM7m15SEpuM4IrvWjkDVeXWauFmarFAulH1TdQhm/xkpT7ElloVy7inAg9tGeRwx9slAOzE4fmuDnmMezhybkvNryuD52HZ881nhVEuJCIhWxUAZ/JV91C6UK21tfoHWN12NzuLL3yUKpSoQKFZv47H55sJA1XlXYWCG7cU3iYCyURVB1uUZFmgJlqqv5uMb53Vvrzm9/fZJrqrTGa1Zr6u6XBzHx5BB4JvqQyHONVxVZLZSlrklcEbkm+Em+6nKNEpsCZRb7S4NPck24E3zrl6JprJFxdjyz4NAEf2jvM/motfGvctNkaccpdU3iisg1wWvy3SjXuM4CzCOr0Ce5JmSyWiPj5OFa0uEyS7vsNYnLQuSaJqqe8arCZo1X3f5cO26ob7axhuth/ddkgsoj47XMzEUbdFmrJsT1d60MlkNsbLK0VfuSjFc7gr9sqnLGqwrbNV5V5JEFy/WexmRNh/Tf3S93/jIhj4zXMjMXbcgaizn9vTn+MwfVb/U81ni1ydKOI2u82hP8JF/ljFcVtmu8qsglC5Zb98gzPdjxzIKOL8sj47XMzEUbssZCpb/r/sN5rPFqk6UdR9Z4tSd4uabqGa8qbNZ4tdmfLSpdOC4r5KIdByjVAOY6ulUMc4iNy/Eka7zaE/yVfDeu8Zpm0RATylr3VSUr5EKJa3XaYGKXtI5hDrFxnd1aGrLGqx9U3ULpfNEQk4PkRNwemUZWUBPvuOZEvLNQmi3oYWKXtI5hHrEJMbtVeexqWCiDl2uqbqFkAG8sRKYCZaYUuZBIs+PGhTTTO7+er9STE839npvM49moJm4kH8/Z5XiSRUPsCX6Sr3rG697B7HY0XcZrnCJtanNyQvTb7kopbjGMPkAUsfIs49V0QQ8TrDOEc4hNkMXIVFQk41XkGt+xGGOmck1RRcvaZYTs0oxKzlh0xlug2mzbdvbrmHaWV/T/qOwLephgnSEsck3CsUWu8YKqyzVHHsheT96UooqW2RKXOOJXvnN/x2WP1x4fhq0Rr5O8kiSbdOp3GuJOGh8zhEWusUfkmiaqLte4rievI1607Lbb6uVlGiqozZs1yv4cWrK/fR3T3w6h/nb2Sd5EXtFJRTZZq3FURcasL3VzkCSyFiNTIXKNPf5dBqSk6nKN63ryphQn4ZhhI0ssOmNPm4yjlFwsXC46qcilNGPvRlKQYhKLb3mQgOnYtYZNMTKjgxaJyDV+UHW5RkURCSK+STg2ssTQO/YBzG0yDtAu7ajaTOQVnVRkI83EKdtJo3J6AWgZI67lRJFr7Al+ku/GAmV7B9p1eqD9zbb59D6cu2U685vQJwknax11AADVlDIOoHa0ZJ2YdcdwRdnF23SFx5rHiGukQJk9wcs13Vig7IipaOD3IPr92QcYn32QW9queYjxlc3vwDUPtbZfvYGxcmu2gVtmsSirwmXHLHPXkRIps3ib68JjpqxdRZiKfa7UUcw4lAJlntCNBcpqsRHex0BfbN7unwZOeexF9LcubZpK049TZrEoq8JlO59315ESKbN4m+vCY6ZsWlHD7ZdQS336fYPFjEMpUAaAiC4D8D8BnAbgbGYeb3ruiwCuBDAD4HPM/LDNsXR0Y4EyYzS3msfsMZN74m1lavL1fbVM9ekBAGx3y61aGxVwp+ebHrfs4m2+LOCxoCi3TUUKlNkKfFsA/DmAO5obiWg5gDUAVgA4HsCjRHQqs+W7TUHVLZRWaCxgcVvmZx5gUM/huwFdW9k5o83ZshNPRhYzs0mUkLX3Ktvia08MgYjAs5TYlq6PnY+by38ghU3QpTXSFFnj1R4ruYaZtzHz7xVPXQpgHTNPMfNLALYDONvmWDqqbqG0QjNA47e68xRyj6qt3FqO2erTN7bOfFSlbZF7Dk3mSW3p+mhw3Dz+A5oxYlMYzyWyxqs9eblrTgDweNPfrzba2iCiqwBcBQAnnXRS6gN1o4VSiFBJOIBaMskqudhKJKZ9dH1cW1wXxlu5daajHOibRFgmLuc14g6fVkT0KIB/p3jqRma+v7HNRgB/O6fJE9FtADYz8/9t/P09ABuY+b6kY42NjfH4+HjSJm08d865Itfo8Kwwl3sYLdd1NNsimQBRQtLQ0gPY/dJA65WxYltlW/wYOfQxn+MaohkjewaB/3ydm2vAuOQCRIlUzXKgrq2gKKgpU64ZHsapj2823p6InmLmMdVzHf+LzHxBir7N8SqAxU1/nwggFzO7yDUJVHqCB9re/tzTdso804NdL8xXlJ5v31bZZj3FdO5jPsc1RDdGHA4dleQyL6rO0bGtVImwInJNXhbK9QDWEFE/ES0FsAzAr/I4kMg1Qkeq/lmXAy4zTX1x5YSENxmvRPRxAP8bwAiAB4noaWa+kJm3EtG9AJ5FlLtwbR7OGqA7M16NKTFjzyskDno0sVFlVZtq8nH9vdQiYzZIxivAzD9m5hOZuZ+Zj2PmC5ueu4WZT2bmdzHzQ/ZdVdNtGa9pKDNjzyckDnr2vXsppvpa26YJGJxGpkzplVtncM2G1izr+fvDvJmSjFdP6LaM1zSUmbHnExIHPQdeegm3X9yaUbp/oN0+a5opfflGRn9Mf4+WGA8PyXj1BMl4TUAkigiJg5bh3TPYtKK3pcjYuq/WlduaZEpXyvIoGa9+IBmvCVTeQmmIxEHLm0Pt19i6yo8mmdKVirJkvPqBWCgTCGBii/cwl/9nAHGwwWQxD90CH3d/sH1/ugVjTDKlQ5RltFTEQhn8lbxYKMMmnllZqdv9gjBdzEPVpnLM+LZgTDfijYXSB0SuSSAAmSK+EEUuC5MEEAcb0izmEW/TrQMcXzDmu7fWw7RB2iByjR+IXJOAZxObSdGrXNaW9SwOKmwkF5vCYcbrAPsfQveIXOMHIteEg0nRq26VClxLLrbEE5oWdOlaq2Uhck0TItck4JlMsXdQLSvEcS4VeBYHFTaSiw0quabUGu4+IXKNH4hck4BvE1vW7tieRolxMJFhyqrVDqjlmlJruPuEyDV+IHJNOGQteuWyWFbRmMoweUguWZGCYuUjck0TUqAsAc8Kc2UteqVLzjGmxDikkWHKoId6cO6W6WoUFHONFCjzAylQpsenwlwM4IipbEWv1q4iTMXmyWmF7KFqm+oDtp/3TmURLpPX27SVKcOY8levLME1D1WjoJhrqlKgLPgreSlQpsenwlwEoBabOeaKXnW6qlU5btK4Tzaf/gecu4Ayv96mzScZRsW5619A/3RrW/CTgiOqUqCs4/J/RZJl+b9tpy337wtGwZhZRBNiSBNjCJiuqfpf1nP4t/NVhAinbXs2xeYWy//5jlgoEwjAOhgvenX1hmgNOJcTvS6rs6qoLJCqYmJXb2AcnEcYONg9sUmFWCj9QCyUCXg+wQPt1jzTuuVp6KYJHtCvqaqqEd8vE7wesVD6gVgoq4dY+PSYyDBpMoT9/lq4exELZRMi1yQQgFyjYu+g2/1VRa4xlWFSnWmgY6QQKiLXBD/Ji1yTQKhvXsfdrsIED+hlGNjUdA91jBRBReSa4DV5kWuqR8gZrnkiMlb3IHJNE5LxmoBnGa+m7Fxotp2pTXDz6X1tWZ1Fed1N+2jSlksmaqBjpBAqkvEavE9+4uabseuedTn1KGz6TjkZ09tfKLsbqWAAPz0L+IcLk68/Vm6dwTUbGP1N8sU0AehpdZFM9QGvfGAZFv/y+ZakH9W2qrYDvcAdl2Sb6E37aNpWB1CD2y9LQxwjRVFmbIY/tQajN91kvH2STz54uUYyXvX4lPFqCgEY2955u8s3tk6eANCnsAn2TwOnPPZiW1analtVm42l07SPpm29cO+GCXGMFEVVMl6t5Boi+gaA1QAOAngBwH9i5l2N574I4EpEXwt9jpkftuuqmvrERB67rQaB3oYfs6e9kBmQ3SZoGweT/lj3sSwCHSOFUGJsXM5rVnINEX0IwL8wc52Ivg4AzHw9ES0HcA+AswEcD+BRAKcyc2LUssg1z51zrlgodQRqj2O0XrEeJLTYBFXbJGIZB5P+WPexLAIdI4VQpoVyeBinPr7ZePvc5Bpm/hkzz92QPg7gxMbjSwGsY+YpZn4JwHZEE75zxEKZQKBv3vjEqMrWLNImaNIf6z6WRaBjpBDEQtnGpwE81Hh8AoBXmp57tdHWBhFdRUTjRDQ+OTmZ+qBioRQEoWq4nNc6TvJE9CgRbVH8XNq0zY2Ivvy/e65JsSvlxyIz38nMY8w8NjIykvoEXGaGVQ4K4loyfyQOeiQ2ekqMTaEZr8x8QdLzRHQFgI8AOJ8PC/yvAljctNmJAHIxs4tck4DcikdIHPRIbPSIXAMQ0UUArgfwUWbe1/TUegBriKifiJYCWAbgVzbH0iFyjSAIVaNQuaYD/wfAAgCPENHTRHQ7ADDzVgD3AngWwE8BXNvJWZMVl5lhlaMmi28AkDgkIbHRU2JsvFnjlZlPYebFzHxm4+eapuduYeaTmfldzPxQ0n5skDVe9fi0xmuZSBz0SGz0VGWNV8l4rTCSzRghcdAjsdFTlYzX4Cd5yXhNQLIZIyQOeiQ2eiqS8Rr8JC8WygTEHhchcdAjsdFTEQtl8JO8WCgTEHtchMRBj8RGj1go/UAslIIgVA2fLJSlI3JNAnIrHiFx0COx0SNyjR+IXJOA3IpHSBz0SGz0iFzjByLXCIJQNUSuaULkmgTkVjxC4qBHYqNH5Bo/ELkmAbkVj5A46JHY6BG5xg9ErhEEoWqIXNOEFChLQIpPRUgc9Ehs9EiBMj+QAmV6pPhUhMRBj8RGjxQo8wQpUKZHik9FSBz0SGz0SIEyT5ACZQlI8akIiYMeiY0eKVDmB2KhTEDscRESBz0SGz1iofQDsVAmIPa4CImDHomNHrFQ+oFYKAVBqBpioWxC5JoE5FY8QuKgR2KjR+QaPxC5JgG5FY+QOOiR2OgRucYPRK4RBKFqiFzThGS8JiDZjBESBz0SGz2S8eoHkvGqR7IZIyQOeiQ2eiTj1RMk41WPZDNGSBz0SGz0SMYrACL6MhE9Q0RPE9HPiOj4pue+SETbiej3RHShfVfVSMZrApLNGCFx0COx0SMZrwCAbzDzGcx8JoCfAPgSABDRcgBrAKwAcBGA7xBRLgKXWCgTEHtchMRBj8RGj1goAWbe0/TnEQDmPEeXAljHzFPM/BKA7QDOtjmWDrFQJiD2uAiJgx6JjZ6KWCh7bXdARLcA+I8AdgP4s0bzCQAeb9rs1Uab6vVXAbgKAE466aTUxxcLpSAIVaNQCyURPUpEWxQ/lwIAM9/IzIsB3A3gb+Zepuq3av/MfCczjzHz2MjISOoTEAtlAmKPi5A46JHY6OkWCyUzX8DMpyt+7o9tuhbAJxqPXwWwuOm5EwG85qbLrSz6wnWggYHWxt5eUF9fV7fRwACGP3lZ18dG4qBvk9jo28qMDQ0MYNEXroMrrOQaIlrGzM83/vwogN81Hq8HsJaIvgngeADLAPzK5lg6hlavBgDs+NatqE9MoHd09FCAur1taPVqzD/rLG/6I3Hwr01i42ds5uY1FxBbfLlARPcBeBei7wn+AOAaZv5/jeduBPBpAHUA1zHzQ532NzY2xuPj45n7IwiC0I0Q0VPMPKZ6zupKnpk/kfDcLQBusdm/IAiCYEfwGa+CIAiCHpnkBUEQKoxM8oIgCBVGJnlBEIQKY+WucQ0RTSJy6WTlWABvOOpOmVTlPAA5Fx+pynkAci5zvIOZldmkXk3ythDRuM5GFBJVOQ9AzsVHqnIegJyLCSLXCIIgVBiZ5AVBECpM1Sb5O8vugCOqch6AnIuPVOU8ADmXjlRKkxcEQRBaqdqVvCAIgtCETPKCIAgVJvhJ3ofFxF1BRN8got81zufHRDTc9Fxo53IZEW0lolkiGos9F9q5XNTo63YiuqHs/qSBiL5PRDuIaEtT29FE9AgRPd/4fVSZfTSFiBYT0S+IaFtjbH2+0R7U+RDRABH9ioj+rXEeNzfa8zkPZg76B8DCpsefA3B74/FyAP8GoB/AUgAvAKiV3d8O5/IhAL2Nx18H8PWAz+U0RGWoNwIYa2oP6lwA1Bp9fCeAeY2+Ly+7Xyn6fx6AswBsaWr7OwA3NB7fMDfOfP8BMArgrMbjBQCea4ynoM4H0cp5RzYe9wF4AsA5eZ1H8Ffy7MFi4q5g5p8xc73x5+OIVtQCwjyXbcz8e8VToZ3L2QC2M/OLzHwQwDpE5xAEzPwYgDdjzZcCuKvx+C4AHyuyT1lh5glm/nXj8VsAtiFaOzqo8+GIvY0/+xo/jJzOI/hJHogWEyeiVwD8JYAvNZpPAPBK02baxcQ95dMA5hZaCf1cmgntXELrrwnHMfMEEE2cABaV3J/UENESAO9BdBUc3PkQUY2IngawA8AjzJzbeQQxyee9mHiRdDqXxjY3IlpR6+65JsWugjgX1csUbaWfSwKh9bfyENGRAO5DtOLcnk7b+wgzzzDzmYju1s8motPzOpbVylBFwcwXGG66FsCDAG5CgYuJp6HTuRDRFQA+AuB8bohzCPRcNHh5LgmE1l8TXieiUWaeIKJRRFeTQUBEfYgm+LuZ+UeN5mDPh5l3EdFGABchp/MI4ko+CSJa1vRnfDHxNUTUT0RLkeNi4q4goosAXA/go8y8r+mp4M4lgdDO5UkAy4hoKRHNA7AG0TmEzHoAVzQeXwHg/hL7YgwREYDvAdjGzN9seiqo8yGikTnnHBENArgA0byVz3mU/U2zg2+q7wOwBcAzAB4AcELTczcickb8HsDFZffV4Fy2I9J/n2783B7wuXwc0VXwFIDXATwc8LlcgsjJ8QKAG8vuT8q+3wNgAsB04/9xJYBjAPwcwPON30eX3U/Dc/kAIqnsmab3yCWhnQ+AMwD8pnEeWwB8qdGey3lIWQNBEIQKE7xcIwiCIOiRSV4QBKHCyCQvCIJQYWSSFwRBqDAyyQuCIFQYmeQFQRAqjEzygiAIFeb/A+LqH3wjMsUPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_x=np.array(range(-30,31))\n",
    "pairs=np.array(np.meshgrid(visualize_x, visualize_x)).T.reshape(-1,2)\n",
    "color=clf.predict(pairs)\n",
    "plt.scatter(pairs[color==1,0],pairs[color==1,1])\n",
    "plt.scatter(pairs[color==6,0],pairs[color==6,1])\n",
    "plt.scatter(pairs[color==2,0],pairs[color==2,1])\n",
    "plt.scatter(pairs[color==4,0],pairs[color==4,1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e5aa7ff",
   "metadata": {},
   "source": [
    "Notice how the decision boundaries are logical, given the nature of the training set, but SVM does not see the underlying pattern in the data. Let see if 'scale' gamma can change it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "a0a07729",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf2=multiclass_SVM(kernel='rbf',C=1,gamma='scale')\n",
    "clf2.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "fd4f87fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmeElEQVR4nO2dfZBc1Xnmn3d6Rhp9oNEnMEJSJBvhAnkpzHZRuGAxu1CYQEC2U1CELS+1xiVIyIJdu2vkJbHZ9VLGuMox2bAFFHYtqQUrJNhBBFjzkciUVGA8sIQgyxgBjtFqjL4YyUJImp5+94/bI/X0nNNze+7H+ejnVzU1Ped29znn7XvP3Pv0+7xXVBWEEELipMf1AAghhBQHF3lCCIkYLvKEEBIxXOQJISRiuMgTQkjE9LoeQDOLFy/WlStXuh4GIYQExcsvv7xHVZeYtnm1yK9cuRJDQ0Ouh0EIIUEhIv9s20a5hhBCIoaLPCGERAwXeUIIiRgu8oQQEjFc5AkhJGIyZ9eISD+A5wHMbLzf36jq10VkIYC/ArASwK8AXK2q72ftz8T+xx/Hrj/7LmrDw+gdHMSJX/4SAHR928AVVzA2jANjE2BsBq64AnkhWatQiogAmKOqB0WkD8BmALcA+ByAfap6p4isB7BAVW9t917ValU7TaHc//jjGP7Tr0EPHz7e2NsLEYGOjnZtm/T3Y+Czn8H+H/1tV8eGcWBsQouN9Pdj8Bv/raOFXkReVtWqcVuepYZFZDaSRf4PAfwlgAtVdVhEBgFsUtWPtXv9dBb5N//NRajt3DndIcdNpQKMjbkehXsYBzuMjR2HselduhSr//651M9vt8jnosmLSEVEXgWwC8AzqvpTACep6jAANH6faHntOhEZEpGh3bt3d9x3bXh4+gOPHR68CYyDHcbGjsPY5Lmu5bLIq+qYqp4FYBmAc0Tk4x289n5VrapqdckSoyu3Lb2Dgx2/pmuoVFyPwA8YBzuMjR2HsclzXcs1u0ZVRwBsAnApgPcaMg0av3fl2dc4cz91QRFvGwV9q1a6HoIXMA52GBs7LmOT57qWeZEXkSUiMr/xeBaAiwH8AsBGANc1nnYdgMey9mXi4E+eL+Jto2D0nV+5HoIXMA52GBs7LmOT57qWR4GyQQAPikgFyT+NR1T170TkBQCPiMj1AH4N4Koc+poENfk2UG9NYBzsMDZ2ItHkMy/yqvoagE8Y2vcCuCjr+09FZWAAYyMjRXcTJiIAb9TOOLSDsbHjMDaVgYHc3it4x2vd9QB8hgdvAuNgh7Gx4zA2ea5rwS/yun+/6yEQQkiu5LmuBb/I53lZEx0irkfgB4yDHcbGjsPYUK5pgnJNG3gpnsA42GFs7FCu8QPKNYSQ2KBc0wTlmjbwUjyBcbDD2NihXOMHlGvawEvxBMbBDmNjh3KNH1CuIYTEBuWaJligrA0sPpXAONhhbOywQJkfsECZHRafSmAc7DA2dligzBNYoMwOi08lMA52GBs7sRQoC36RZ4GyNrD4VALjYIexsRNJgbLgF3mmULaB6XEJjIMdxsYOUyj9gCmUbWB6XALjYIexscMUSj9gCiUhJDaYQtkE5Zo28FI8gXGww9jYoVzjB5Rr2sBL8QTGwQ5jY4dyjR9QriGExAblmiboeG0D3YwJjIMdxsYOHa9+QMerHboZExgHO4yNHTpePYGOVzt0MyYwDnYYGzt0vHoCHa9toJsxgXGww9jYoePVD5hC2QamxyUwDnYYGztMofQDplC2gelxCYyDHcbGDlMoE0RkuYj8g4hsE5GtInJLo32hiDwjIm82fi/IPtzJMIWSEBIbvqVQ1gD8R1U9HcC5AG4SkTMArAfwnKquBvBc4+/coVzTBl6KAwDqPYyDFe4jdijXJKjqsKq+0nj8WwDbAJwCYC2ABxtPexDAZ7L2ZYJyTRsiuhRvnclRAUYrU7cd7gWePiv5PZ3Xu2orbb+OaB/JnUjkmt6pn5IeEVkJ4BMAfgrgJFUdBpJ/BCJyouU16wCsA4AVK1Z03Cflmu5AAeyZByw6AOydBzx8YXKWde0mnbJty5oK3lg2luq5vrQtPlBCUIm35Lmuieb030pE5gL4CYA7VPWHIjKiqvObtr+vqm11+Wq1qkNDQx31+8tzP4mxkZFpjLgLEInmTG33POCmm6Z3TiIQ6KRrAb+5554alpSx0Ee0j+SOw9hU5s/HaS++kPr5IvKyqlZN23LJrhGRPgCPAnhIVX/YaH5PRAYb2wcB7Mqjr1Yo17QhgIM3jQxzuPf42e70+vA/Dq08fKFMW2Lq6JgIYB9xBuWaBBERAN8DsE1Vv9O0aSOA6wDc2fj9WNa+TFCuCZu0MsyWNd1VYyWZ7/QkJko94ZPnupaHJn8egM8D+CcRebXR9l+QLO6PiMj1AH4N4Koc+ppE7+Agajt3FvHW4VOpeO9o3GuRYbasya+PHulBXcO75tuypmKMw1RtHUk9AewjznAYG68KlKnqZlUVVT1TVc9q/DypqntV9SJVXd34vS+PAbfCAmV2XBVYUgBjLerKaAEyTFpWnbCq8D584uELBUda/m+a4n+kDzj0L7orNp0QS4GyXLNrXMACZXZcFVgSAB/MBI7M8EOGeee37xTeh090IvV8/ifvYLbb4XpLLAXKgl/kWaCsDQ4vw+ceBr745WJlmLSEKNVkJa3U8x821soZUIiwQJkf0PHaBoeOvb3znHU9CQFdnTb2DTA2Vuh49YPuO0frgJJSwFp7KUtrT0uIKZRl8fCnJqdq8phqEEkKZfCLPFMo3aNIzEr1xu/7Luu+lMdQ2bymB/ddJhM+P3/+PXcvvqVQOqUyMEDHq42SHHu2NEhfCNHxWhYCmaTfl+a29R2XjlfKNcfhpWUbMu6gZbhRy4ALvB1TbNK6baM/9iKRa/w9/UoJ5ZrioBu1O0mbgklnbXFQrmmCjtc2ZHTsleFGLYNQHa9lYItNmhTM6GUdOl79gI5XO2kdeyaHaggyTFq6zfHaCVliY3LW1jBZ5gsVOl49gY5XO2kdezaHaiwyTLc5XjshS2xMss7MUWDeh/mNzyV0vHoCHa9t6OBS0+ZQjQFKNXayxqZV1tnwzYgctHS8+gEdr23owLHnk0M1b+h4tZN3bKLaj+h49QOeo7XBkgLmu0M1b5hCaSfv2JjSL4M9RiNJoQx+kWcKZefQoUqKYsuaCh20OcAUyiboeLWjIhDD2YjvDtW8oePVThGxicZBS8erHwR7KVgCR3p10qVz7NKMCS7wdsqITbASDuUaP6BcY2fGKCZdOlOaIWVDCadzKNc0QcernZGBCraskeAcqnlDx6udsmITpIRDx6sf0PFqZ9/ZH3E9hNKptFxi99cVV/UtRX99YntvvY4+9aNN6u5kAVdu4BDcsnS8egIdr3YWvvI2cG53XRjPHatjtip+01vBybUx3PL+CC4/tBOfmD0Tdy+YP6EdgBdtH/YIRuBGQnPlBg7BLRuL41XU4ZcLrVSrVR0aGuroNdtOP8PpFyQ+UwdwzVeD/z/eEaKK1371ruthdMSZK5dDHRpvfGHDN2vhSwt5IYLTt/28g6fLy6paNW0LPqZ0vNrpxvt3nlwzaah+x8E85nLwyQ18cJbrEbRAx6sf8Ou0BJOL9aFPORlKMaTQtvvr9WNySMuLixtXDtzy/gj66xP3ZNP8irhi9Sq91KOhAGAKZTMi8n0R2SUirze1LRSRZ0TkzcbvBXn01QpTKBO6wcU6OFqDqGJwtIb/vmcfvrF774S22/fsw+UfHHI9zI65/INDuH3PvinnFztzD7segT/4mEL5vwD8BYC/bGpbD+A5Vb1TRNY3/r41p/6OQcdrwsFZk12sMTk9B2tjeHrH5FTZdIu6wL/TxIlc/sEh41ya2y5ZthTDffl+x+LTPrJ3HvxKq6Tj9Tiq+jyAfS3NawE82Hj8IIDP5NFXK5RrGhj2RV8O3rZkkmFSd5Lhtf5gknWy38fXn9h454yNRK4pMvXiJFUdBgBVHRaRE01PEpF1ANYBwIoVKzruhHJNQsiXuoOjtSnTDkOUYfJmPAbNsRnujUeSM6VVdut9ZH2Ua6aNqt4P4H4gSaHs9PWUaxJMdbx9uhS3kU2GSYv/ck1aWmWdrBKOb/tIqzP2ge/W3OXOU66ZkvdEZBAAGr93FdFJN8o1aevB+3TwAph0wGSXYVJ3XEIfbsiamePdPtKKy+FFItcUuchvBHBd4/F1AB4ropNulGtCzqSJIRvGJ2LPzAlZhsyCd3KNiPwAwIUAFovIDgBfB3AngEdE5HoAvwZwVR59tdKNBcrS1oP3rTCXTZopHKkA6s5wVDRZMnN820dacZpxwwJlx1HVP1DVQVXtU9Vlqvo9Vd2rqhep6urG79bsm1zotgJlCmDo1HTPdVV8CjAVCitLmjGwaLWbfj3igkPmKyaX+0gaXBYyi6VAWfCO124rUCYAqtvTPddV8SkgKRTmjTSz9003/XrE87NnG9td7iNp2LKmgntbatEfmlVOoYpYCpQ5z67JSm142PUQSmdRystXl5fhByo92OxLobCIpZq0/MaSaumzVDNOa8bNhm+W9B2DI6kGyHddC/5MvhsLlJnSJU24LD410GracYo/RbhcYSuC5lOBsrSUVsiMBcr8wKelpAjSpkuaX+suBcyv6s9eDcYJplTL/np9soM2BMr6OJlC6Qexp1CGmi55oBL8rhUVplTL2/fsw+HwTuS7Iq3SuxRKl8TueDUVHkuLSzejyxrpk4nH8ZoFU6rl3QsXBFcaobS0Sjpe/SDAi83OyLCPlbbA+5QuaYQLvI0iip4VTWmFzCjX+EHsck0ol6bepEuSjrj84AeTZBzf2bKmgvta0ioDVJ3aQrmmidgdr2kzaUyU5WZ05mRNS+SO10xIJfeiZ2XQmlZ5zz21/CUcOl79IGbHayfuVhNluRltbkpvSOl4VQA1nXhIHNEKjmrvlG01lTBFIUNsvP88DQydmr8oR8erJ8TseO3E3WqiLDejzU3pDSkdrwJgv87Gjvpi1FWwo74Y/3n0Bvyn0XVTth3AnDAlA0NsvP88DVS35y/Z0PHqCbE7XtO6W02U5Wa0uSm9oQOpZoF8gH955P5J7RuPnt+27e2Z105vbK4xxMb7z9NAluPECh2vfhC74zWLJl+Wm9GvdMnJaAdx2KmLptXHTl08rde5Z3JsfP88TRTigqXj1Q9iSqHM4m41v18BKrH36ZKTOYQZOKQzJrSZdPVDOgN31a6eVh931a5O1UfdO+F+8oBCTKss5AsRplD6QUwplKG4W0NLl5ylR7F+9ItT6urrR7+IjfXJskwaNtbPT9VHCLq9yR3rO6GkGqeFKZRNxOR4zeJuNVGE43V+ve53uqSBYSzCxvr5U+rqWUnTx+YZN2OZ7Mmtz+yY3cCtaZX/asUpGKn4d8IxzsFZyP9esHS8+kFMck3el5xFyDXeX7W3jO+QzsBdo9OTYIrAJOu4lXDSde775065xk7wi3xMck0Il5y+Fx5TYJIE89g0JZgiMMk6IUg4vn/uIRw7nUC5ponY5Jo8KUKu8atO/GRGMBfnH/3zCW2+lSdrlXVembkOC3HQ0WjSRWegXqdcUyKUa5rwe8npEMo1mTGNz/MhO44p5Rr7e1Ku8QLKNeXi+2X7AvnA9RA6ZoG4OotPj++fewjHTifkua75/cmlIM9CPq7JYnwy0SP5f7y+G2VMZqaKQ1NLGpwaqSSdBOP75573sQMgKVDmCBYoayKWAmVZi5GZKKJAmS/Fq0zFxGxmpo8s8bsWy3P1s9xJSimLt/nyudt4+ELBkZZvGGvIpuKwQJknxFKgLGsxMhNFFCjzpXiVqZiYzcz09m6/F6iLel51l2GTsnibL5+7jS1rKri3pcb8oVnZipaxQFlKRORSAHcDqAB4QFXvzPP9YypQlneRpSIKlPlUvMpWTKyVMc+/NVzq0hyVsnibT5+7jdYa8xu+mdGpywJlUyMiFQD3APhdAGcA+AMROSPPPmIqUFZECmXe+JRCOYI5qZ7ntyLvurhZuuj49LmnJfPxxAJlqTgHwHZVfVtVjwLYAGBtnh2Et+u1gSmUHZF2LB4N2YhbF2wkKZQmso6ZKZSpOAXAu01/72i0HUNE1onIkIgM7d69u+MOmEJZLj6l0oWYLmliY/18/PXYBahpD1STL5R9u/rw6XNPSwjHk42QUihN++qEf4+qer+qVlW1umTJko47oFxjh3JNgm8LZitX9mzGVZXn0St1iAC9UmaMKddYoVyTih0Aljf9vQxAriUMw9v12kC5piNikWu+0vsIZsvRCW3lrS+Ua+yvp1yThp8BWC0iq0RkBoBrAGzMswPKNeXi02V7LHKN0+yalPj0uaclhOPJRjAFylS1JiJ/DODHSFIov6+qW/Pso3dwELWdYdU3t1GE4zXvNEqfnI9pb9VXEfE6jXKnLnZXY14qqdIoT66NYbjP73qG520dw7WbFIsOJMdS5qJllYqzNMqgHK+q+qSqnqaqH1XVO/J+fzpe7cTueH2uflaq59Lx2oZIHK/nbR3DjU8qlhxIFrUlB4DZH9LxCtDx6g10vHaGIHGKpoGO1zZE4ni9dpNiZov3qRd0vAIR1JOn49VO7I7XpbIHm2fcjKWyBzt1Me6qXW0sa+CzVAPQ8ZoHeR87AOh49QWmUNqJPYVSACzr2YMeSX7f2fcAruzZbHyez4xgrsPe40ihzPvYAcAUSl/we9frEKZQdkTrMThbjuIrvY9Mep5HQzbCm4bkAG8aYiV4uYYplOXieypdWgnHJVf2bMZXeh85NsaFvGlIZkI4djohJMdr4VCusRO7XGPCJOGsrUyWcFxxZc9m3Nn3wIQxuoVyjRXKNX7g967XIZRrMpNWwnGFW3erCco19veMQ64JfpGnXFMuvl+2mxjEXtdDOEYI7lYTvn/uIRw7nRCM47UMYnK8HuwH7rmndsyx9/CFgi1rpp+6VoTjdd5YHZcsW4rf9FZwcm0Mt7w/gss/8DsPfZcsnqSDj98msLUti36fpo8RzMVCeKTBWxyvT8yZjbsXzD/2OQ/U6xhxeM/Tqdg7LzFA5UokjtfgF/m5n7oAIz/Y4HoYmVEAc44A8xpnJEsOADc8qQDGpr3QrzphFd468FZuYwSAg5Ue7G/oC8N9vbh98UIA8Hqhl0Ufxbf2PIBZDZlkmezBt/vug0AwQ2rH2u7sewAYxbQW+nGtffYUfdRUoPAorXPRamDPLyY0PTFnNm5fvBCHe5Kz9+G+Xkh5Be6nxdCpwKWv5BvXvlUrMbo93+MnLXS8NhGT47XSchz11xIn33QpwvE61iIgH+7pwd0L5ufeT56ctO9nxxb4cWbK2LHFd5ws+r1Jazf10SvqzwIPGB2vdy+Yf2yBH0d7vBr1JKrb8//HScerJ8TkeDWRxclXhOPVhO9uyLSuTsCcgglMlnVa204JVGs3xcb7z9MAHa92gl/kKwMDGBsZcT2MwsiSGiaQQjJsWvE9vS45x0sXh/EUTMAuuZjavM8+sTI5Nr7r7yYyV5w0IeIswybPFMrgF3nfl5fMZNjHyljggQDS6zqIQ2s640yZfDZnanObBpmFybHx//M0wBRKK8Ev8jGlUJoIITVsf6UnuIwbcpzWTJr9nqdLApNrx58QwHHSCUyhbIJyjZ2y5BoAx24o4WfGTXq5ptt4Yu4c3L5owYRMGt9P5c/bOoYbnlT0N77TXnKgoE83ErnG/3/ZU0C5pt1LS9pBvc+48XvRcokpk8Z37enaTccX+HEKGTHlGj/oBrkmT4NUWQz3VijhBMBvApBmWikkk8YzWKCsiZgKlJkQYMItzW54UnHe1nSpXUUUKOuE4b5eqMgxCeeJOa7uLuT3malLBjw3OZkopBiZCRYo84PY5ZrW3awTg1Rpco0JrySc8BaysggyMmUNOhK5JvhFPna5xkSol6shmmxi54DnTlYTIWScZYVyTRN5FvIJhb3z0j2vR/z6eMeLm525cjkuWba0PPlG+M8FSFIlW+N/coCXwmn3/8w4NITlua75tQpMgzwL+YSAIinGlIZVJ6wqdCydcrDS40anX7S6+D48Z7zo2MT4L8LvzPNrH0nD0KnlKDZ9q1aW0IsZFihrIpYCZWkRJMWY0lBEgbIsOCtuZijC1W2YUiUP9wheOvRrRyOaPkUUIzPBAmUAROQqALcDOB3AOao61LTtqwCuBzAG4GZV/XGWvmzEXqDMxKID6dIqyypQloVSUi07KFAWC60u1mHL9yEh7COt7tbFZX0nxQJlAIDXAXwOwH3NjSJyBoBrAKwBsBTAsyJymmr+R1vsjlcT42mVQPu682U6XrNQvFu2uxyvpnrwtkwR3/eR0tytJuh4BVR1m6q+Ydi0FsAGVT2iqu8A2A7gnCx92fD/PCR/0qZV+nzwHqMUCSeAOORIJy5W3/eR0tytJiJJoSzK8XoKgBeb/t7RaJuEiKwDsA4AVqxY0XFH3ZhCaSLUtEoTdMvaaZVhbnl/BABSSTMh4Eya8YxSC5SJyLMATjZsuk1VH7O9zNBm/LeoqvcDuB8AqtVqx/86u1GuMWFyAfp+Kd6OfCWcOOQakwzzJ4sXQkQw2nRLxk7OQH3aR5xKMyYikWumXORV9eJpvO8OAMub/l4GoJC7bXejXGPEsC/6cvB2jEXCmf4iH2gcWjDJMLVWWQboyI7v0z7iVJoxEYlcU1QK5UYA14jITBFZBWA1gJeK6IhyTULsLkC6ZeOPQUySY1a8qScvIp8F8D8ALAHwhIi8qqqfVtWtIvIIgJ8DqAG4qYjMGiBxhtV2FnKREBQH+yenVb7w8b4gUuTSMO6WbadFW7V7qXifRplGay/itnw90uPNPrJ33vGsMS+oVJylUXrjeFXVH6nqMlWdqaonqeqnm7bdoaofVdWPqepT2YdqptscryYUwJwjE6tV3viU4vPvrnQ8svxodcv+yeKF+NMli9I5aD13vJrcqKb57S+gTIVPruiynKxpoePVE7rN8WpCAFRajo6Zo8AnN77lZDxF0OqWrfX0HPuycRxr+qXnjleb1t46Py2gmJhPruiynKxpoePVE7rR8ZqW+fvHcN5WmZCSFspNR6aLLf0yjRziqs1lyqNLqcb7dMlIHK+iHt3PsVqt6tDQ0NRPbOKX536SKZQWDvcBUEzIWDjcC9x3WcQLveqE7JL+eh1rDx7CY3NnTzhb7q3XJ6QeumxrHXOZuEqhbE2XBBKpxqczeacplPPn47QXX0j9fBF5WVWrpm3ByzV+fGXkJzNrMiklrZObjgSJQcL563lzU8khrtpc3oHIVQqld+mSJiJJoQxermEKpR2x7KRpC5zFAk8E3OO9NOMZvGlIE7Hf4zUTljPELPeNDRHX97r1mTJiMy7NNO9zQcB7vPoBz9LaYK08OJHYJRyfXJ2+UUZsgpBmTFCu8QPKNfnQbRIOKQ5KM9mhXNME5Zo2dHC5GbOEQ7nGTt6xCVaaMUG5xg8o17Sho2qEE4lJwqFcYyfv2AQrzZiIRK4JfpGnXFMcLBhFOoX7TD54U6DMB1igrA0ZCyyZip6FqNP7VITLN7LGplV/PzgLmPdhjgN0CQuU+QELlNnJUmDJVPQsVJ3epyJcvpElNudtHcONLfr77A/9KjKWBRYo8wQWKLOTpcCSqehZqDq9T0W4fCNLbK7dpJjZor/3ImANvgUWKPMEFihrQwGXmiGmWlKqsdNJbLouNTKSAmXBn8kzhbINBaSAhZhqyRRKO2ljE1VqZFqYQukHPEdrQwEpYCGmWjKF0k7a2ESVGpmWSFIog5drmELpHpOEA6Cr6tiHTKsMY/r8opdmPIMplE1UBgZYT95GSfWwxyUcNH7/4eMK6QH6xo633fCkAhhzstC7qpkeAudvrWNdU1132+fXldFzWU+ecs1xKNe0oaQdtPWyfYYeXyDGcSnrcIG3c+1PJsswps8vemnGRCRyTfCLPOWacKAb0j8W7uc/QB+hXNMEHa9tcOjYM2Fy0ALFa/d0vCaYtPeRgQoW7vdnH/EKOl79gI5XOy4de62YHLR/9Ljij57QwlMy6Xg1u1NvfErRv4qxsUHHqyfQ8WrHpWOvFZODtq8k7Z6OV7M7deYoMPufGBsbdLwCEJFvA7gCwFEAbwH496o60tj2VQDXAxgDcLOq/jjbUM3Q8doGj6SaTsg7JTMmqSZNumNHKZCB7iOlEInjVTTDN8gicgmAv1fVmoh8CwBU9VYROQPADwCcA2ApgGcBnKaqbaNWrVZ1aGioozH88txPMoXShsMUsCwoJmZzHBVMSOkDgMO9wH2XpVvoY0mhHHedNmfDmGJjamuN6TEC3UdKwWUK5fz5OO3FF1I/X0ReVtWqaVsmuUZVn1bV8V3uRQDLGo/XAtigqkdU9R0A25Es+LkTzzlaAQR68OadkhnDAg+YXaem2HSUAhnoPlIKkaRQ5pld8wUAf9V4fAqSRX+cHY22SYjIOgDrAGDFihUdd8oUyu6lE1lnujKHT210nXYPea5rU8o1IvIsgJMNm25T1ccaz7kNQBXA51RVReQeAC+o6v9ubP8egCdV9dF2fVGuyZnIL8XTyjqbzhRc+Nr0ZA6f2qySSxYi30cyEYlcM+WZvKpePMWbXwfg9wBcpMf/Y+wAsLzpacsAFJLMTrmmDZEfvCZZBwZZ55JXgZ761M/1va0Q12nk+0gmIpFrMmnyInIpgFsBXKmqh5o2bQRwjYjMFJFVAFYDeClLXzYo15Cp6KlzISNhkee6ljVP/i8AnADgGRF5VUTuBQBV3QrgEQA/B/B/ANw0VWbNdMnTGRYdFVZ9BMA4tIOxseMwNt44XlX1VFVdrqpnNX5ubNp2h6p+VFU/pqpPZR+qGTpe7fjkeHUJ42CHsbFDx6sn0PFqxyfHq0sYBzuMjZ1YHK/BL/J0vLaBbsYExsEOY2MnEsdr8Is87/HaBof3qPQKxsEOY2OH93j1A6ZQtoHpcQmMgx3Gxg5TKP2AKZSEkNjwKYXSOZRr2sBL8QTGwQ5jY4dyjR9QrmkDL8UTGAc7jI0dyjV+QLmGEBIblGuaoFzTBl6KJzAOdhgbO5Rr/IByTRt4KZ7AONhhbOxQrvEDyjWEkNigXNMEC5S1gcWnEhgHO4yNHRYo8wMWKLPD4lMJjIMdxsYOC5R5AguU2WHxqQTGwQ5jY4cFyjyBBcrawOJTCYyDHcbGDguU+QFTKNvA9LgExsEOY2OHKZR+wBTKNjA9LoFxsMPY2GEKpR8whZIQEhtMoWyCck0beCmewDjYYWzsUK7xA8o1beCleALjYIexsUO5xg8o1xBCYoNyTRN0vLaBbsYExsEOY2OHjlc/oOPVDt2MCYyDHcbGDh2vnkDHqx26GRMYBzuMjR06XgGIyDdE5DUReVVEnhaRpU3bvioi20XkDRH5dPahmqHjtQ10MyYwDnYYGzt0vAIAvq2qZ6rqWQD+DsDXAEBEzgBwDYA1AC4F8D9FpBCBiymUbWB6XALjYIexscMUSkBVDzT9OQfAeM7RWgAbVPWIqr4DYDuAc7L0ZYMplG1gelwC42CHsbETSQplb9Y3EJE7APw7APsB/OtG8ykAXmx62o5Gm+n16wCsA4AVK1Z03D9TKAkhsVFqCqWIPCsirxt+1gKAqt6mqssBPATgj8dfZhq36f1V9X5VrapqdcmSJR1PgCmUbWB6XALjYIexsdMtKZSqerGqftzw81jLUx8G8PuNxzsALG/atgzAznyGPJETv/wlSH//xMbeXkhfX1e3SX8/5l99VdfHhnGwtzE29jaXsZH+fpz45S8hLzLJNSKyWlXfbPx5JYBfNB5vBPCwiHwHwFIAqwG8lKUvGwNXXAEA2PVn30VteBi9g4PHAtTtbQNXXIHZZ5/tzXgYB//aGBs/YzO+ruWBaIYvF0TkUQAfQ/I9wT8DuFFV/19j220AvgCgBuBLqvrUVO9XrVZ1aGho2uMhhJBuREReVtWqaVumM3lV/f022+4AcEeW9yeEEJKN4B2vhBBC7HCRJ4SQiOEiTwghEcNFnhBCIiZTdk3eiMhuJFk602UxgD05DcclscwD4Fx8JJZ5AJzLOL+jqkY3qVeLfFZEZMiWRhQSscwD4Fx8JJZ5AJxLGijXEEJIxHCRJ4SQiIltkb/f9QByIpZ5AJyLj8QyD4BzmZKoNHlCCCETie1MnhBCSBNc5AkhJGKCX+R9uJl4XojIt0XkF435/EhE5jdtC20uV4nIVhGpi0i1ZVtoc7m0MdbtIrLe9Xg6QUS+LyK7ROT1praFIvKMiLzZ+L3A5RjTIiLLReQfRGRbY9+6pdEe1HxEpF9EXhKRf2zM47822ouZh6oG/QNgXtPjmwHc23h8BoB/BDATwCoAbwGouB7vFHO5BEBv4/G3AHwr4LmcjqQM9SYA1ab2oOYCoNIY40cAzGiM/QzX4+pg/BcAOBvA601tdwFY33i8fnw/8/0HwCCAsxuPTwDwy8b+FNR8kNw5b27jcR+AnwI4t6h5BH8mrx7cTDwvVPVpVa01/nwRyR21gDDnsk1V3zBsCm0u5wDYrqpvq+pRABuQzCEIVPV5APtamtcCeLDx+EEAnylzTNNFVYdV9ZXG498C2Ibk3tFBzUcTDjb+7Gv8KAqaR/CLPJDcTFxE3gXwbwF8rdF8CoB3m55mvZm4p3wBwPiNVkKfSzOhzSW08abhJFUdBpKFE8CJjsfTMSKyEsAnkJwFBzcfEamIyKsAdgF4RlULm0cQi3zRNxMvk6nm0njObUjuqPXQeJPhrYKYi+llhjbnc2lDaOONHhGZC+BRJHecOzDV831EVcdU9SwkV+vniMjHi+or052hykJVL0751IcBPAHg6yjxZuKdMNVcROQ6AL8H4CJtiHMIdC4WvJxLG0IbbxreE5FBVR0WkUEkZ5NBICJ9SBb4h1T1h43mYOejqiMisgnApShoHkGcybdDRFY3/dl6M/FrRGSmiKxCgTcTzwsRuRTArQCuVNVDTZuCm0sbQpvLzwCsFpFVIjIDwDVI5hAyGwFc13h8HYDHHI4lNSIiAL4HYJuqfqdpU1DzEZEl45lzIjILwMVI1q1i5uH6m+Ycvql+FMDrAF4D8DiAU5q23YYkM+INAL/reqwp5rIdif77auPn3oDn8lkkZ8FHALwH4McBz+UyJJkcbwG4zfV4Ohz7DwAMAxhtfB7XA1gE4DkAbzZ+L3Q9zpRzOR+JVPZa0zFyWWjzAXAmgP/bmMfrAL7WaC9kHixrQAghERO8XEMIIcQOF3lCCIkYLvKEEBIxXOQJISRiuMgTQkjEcJEnhJCI4SJPCCER8/8BO3tm09vKn80AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_x=np.array(range(-30,31))\n",
    "pairs=np.array(np.meshgrid(visualize_x, visualize_x)).T.reshape(-1,2)\n",
    "color=clf2.predict(pairs)\n",
    "plt.scatter(pairs[color==1,0],pairs[color==1,1])\n",
    "plt.scatter(pairs[color==6,0],pairs[color==6,1])\n",
    "plt.scatter(pairs[color==2,0],pairs[color==2,1])\n",
    "plt.scatter(pairs[color==4,0],pairs[color==4,1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95c274bf",
   "metadata": {},
   "source": [
    "This are exaclty the decision boundaries we were looking for!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}