L07-SupportVectorMachine-AnaJaved.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "optional-paste",
   "metadata": {},
   "source": [
    "## Lesson 7 - Age of Abalone\n",
    "### Author: Ana Javed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pending-trunk",
   "metadata": {},
   "source": [
    "### Workplace Scenario\n",
    "\n",
    "Kennedy's oceanographic institute client pulled into port the other day with a ton (literally) of collected samples and corresponding data to process. Some of these data tasks are being distributed to others to work on; you've got the abalone (marine snails) data to classify and determine the age from physical characteristics. \n",
    "\n",
    "##### Background\n",
    "\n",
    "Age of abalone is determined by cutting the shell through the cone, staining it, and counting the number of rings through a microscope. Other measurements, which are easier to obtain, could be used to predict the age. According to the data provider, original data examples with missing values were removed (the majority having the predicted value missing), and the ranges of the continuous values have been scaled (by dividing by 200) for use with machine learning algorithms such as SVMs.\n",
    "\n",
    "The target field is “Rings”. Since the output is continuous the solution can be handled by a Support Vector Regression or it can be changed to a binary Support Vector Classification by assigning examples that are younger than 11 years old to class: ‘0’ and those that are older (class: ‘1’).\n",
    "\n",
    "### Instructions\n",
    "\n",
    "Using the Abalone csv file (location: https://library.startlearninglabs.uw.edu/DATASCI420/2019/Datasets/Abalone.csv) , create a new notebook to build an experiment using support vector machine classifier and regression. Perform each of the following tasks and answer the questions:\n",
    "\n",
    "- Convert the continuous output value from continuous to binary (0,1) and build an SVC\n",
    "- Using your best guess for hyperparameters and kernel, what is the percentage of correctly classified results?\n",
    "- Test different kernels and hyperparameters or consider using sklearn.model_selection.SearchGridCV. Which kernel performed best with what settings?\n",
    "- Show recall, precision and f-measure for the best model\n",
    "- Using the original data, with rings as a continuous variable, create an SVR model\n",
    "- Report on the predicted variance and the mean squared error\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "attached-model",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Importing Necessary Libraries & Packages \n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "import datetime as dt\n",
    "import csv\n",
    "import sklearn \n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Declaring inline visualizations \n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "color-field",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex</th>\n",
       "      <th>Length</th>\n",
       "      <th>Diameter</th>\n",
       "      <th>Height</th>\n",
       "      <th>Whole Weight</th>\n",
       "      <th>Shucked Weight</th>\n",
       "      <th>Viscera Weight</th>\n",
       "      <th>Shell Weight</th>\n",
       "      <th>Rings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M</td>\n",
       "      <td>0.455</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.5140</td>\n",
       "      <td>0.2245</td>\n",
       "      <td>0.1010</td>\n",
       "      <td>0.150</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0.350</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.090</td>\n",
       "      <td>0.2255</td>\n",
       "      <td>0.0995</td>\n",
       "      <td>0.0485</td>\n",
       "      <td>0.070</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.420</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.6770</td>\n",
       "      <td>0.2565</td>\n",
       "      <td>0.1415</td>\n",
       "      <td>0.210</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>0.440</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.5160</td>\n",
       "      <td>0.2155</td>\n",
       "      <td>0.1140</td>\n",
       "      <td>0.155</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I</td>\n",
       "      <td>0.330</td>\n",
       "      <td>0.255</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.0895</td>\n",
       "      <td>0.0395</td>\n",
       "      <td>0.055</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Sex  Length  Diameter  Height  Whole Weight  Shucked Weight  Viscera Weight  \\\n",
       "0   M   0.455     0.365   0.095        0.5140          0.2245          0.1010   \n",
       "1   M   0.350     0.265   0.090        0.2255          0.0995          0.0485   \n",
       "2   F   0.530     0.420   0.135        0.6770          0.2565          0.1415   \n",
       "3   M   0.440     0.365   0.125        0.5160          0.2155          0.1140   \n",
       "4   I   0.330     0.255   0.080        0.2050          0.0895          0.0395   \n",
       "\n",
       "   Shell Weight  Rings  \n",
       "0         0.150     15  \n",
       "1         0.070      7  \n",
       "2         0.210      9  \n",
       "3         0.155     10  \n",
       "4         0.055      7  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ## Reading data file into Dataframe \n",
    "url = 'https://library.startlearninglabs.uw.edu/DATASCI420/2019/Datasets/Abalone.csv'\n",
    "df = pd.read_csv(url, sep=\",\")\n",
    "\n",
    "## First 5 Rows from Dataframe\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "turned-stress",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4177, 9)\n",
      "Sex                object\n",
      "Length            float64\n",
      "Diameter          float64\n",
      "Height            float64\n",
      "Whole Weight      float64\n",
      "Shucked Weight    float64\n",
      "Viscera Weight    float64\n",
      "Shell Weight      float64\n",
      "Rings               int64\n",
      "dtype: object\n",
      "            Length     Diameter       Height  Whole Weight  Shucked Weight  \\\n",
      "count  4177.000000  4177.000000  4177.000000   4177.000000     4177.000000   \n",
      "mean      0.523992     0.407881     0.139516      0.828742        0.359367   \n",
      "std       0.120093     0.099240     0.041827      0.490389        0.221963   \n",
      "min       0.075000     0.055000     0.000000      0.002000        0.001000   \n",
      "25%       0.450000     0.350000     0.115000      0.441500        0.186000   \n",
      "50%       0.545000     0.425000     0.140000      0.799500        0.336000   \n",
      "75%       0.615000     0.480000     0.165000      1.153000        0.502000   \n",
      "max       0.815000     0.650000     1.130000      2.825500        1.488000   \n",
      "\n",
      "       Viscera Weight  Shell Weight        Rings  \n",
      "count     4177.000000   4177.000000  4177.000000  \n",
      "mean         0.180594      0.238831     9.933684  \n",
      "std          0.109614      0.139203     3.224169  \n",
      "min          0.000500      0.001500     1.000000  \n",
      "25%          0.093500      0.130000     8.000000  \n",
      "50%          0.171000      0.234000     9.000000  \n",
      "75%          0.253000      0.329000    11.000000  \n",
      "max          0.760000      1.005000    29.000000  \n"
     ]
    }
   ],
   "source": [
    "## Conducting Exploratory Data Analysis: \n",
    "print(df.shape)  # (4177, 9)\n",
    "print(df.dtypes) \n",
    "print(df.describe()) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prostate-diameter",
   "metadata": {},
   "source": [
    "### 1. Convert the continuous output value from continuous to binary (0,1) and build an SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "threatened-central",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "After:\n",
      "0    2730\n",
      "1    1447\n",
      "Name: ring_class, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Assigning examples that are younger than 11 years old to class: ‘0’ and those that are older (class: ‘1’).\n",
    "\n",
    "## Creating a copy of the Rings Column: \n",
    "df.loc[:, \"ring_class\"] = df.loc[:, \"Rings\"]\n",
    "\n",
    "## For Loop to create a binary value in \"ring_class\" column\n",
    "for each in df.loc[:, \"ring_class\"].unique():\n",
    "    if int(each) >= 11:\n",
    "        df.loc[df.loc[:, \"ring_class\"] == each, \"ring_class\"] = '1' # Older\n",
    "    elif int(each) < 11.0: \n",
    "        df.loc[df.loc[:, \"ring_class\"] == each, \"ring_class\"] = '0' # Younger\n",
    "    continue\n",
    "        \n",
    "print(\"\\nAfter:\")\n",
    "print(df.loc[:, \"ring_class\"].value_counts())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "executive-canon",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Making \"Sex\" One-hot encoded columns \n",
    "df_expanded = pd.get_dummies(df, columns = [\"Sex\"]) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "accurate-paraguay",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Length</th>\n",
       "      <th>Diameter</th>\n",
       "      <th>Height</th>\n",
       "      <th>Whole Weight</th>\n",
       "      <th>Shucked Weight</th>\n",
       "      <th>Viscera Weight</th>\n",
       "      <th>Shell Weight</th>\n",
       "      <th>Rings</th>\n",
       "      <th>ring_class</th>\n",
       "      <th>Sex_F</th>\n",
       "      <th>Sex_I</th>\n",
       "      <th>Sex_M</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.455</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.5140</td>\n",
       "      <td>0.2245</td>\n",
       "      <td>0.1010</td>\n",
       "      <td>0.150</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.350</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.090</td>\n",
       "      <td>0.2255</td>\n",
       "      <td>0.0995</td>\n",
       "      <td>0.0485</td>\n",
       "      <td>0.070</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.530</td>\n",
       "      <td>0.420</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.6770</td>\n",
       "      <td>0.2565</td>\n",
       "      <td>0.1415</td>\n",
       "      <td>0.210</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.440</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.5160</td>\n",
       "      <td>0.2155</td>\n",
       "      <td>0.1140</td>\n",
       "      <td>0.155</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.330</td>\n",
       "      <td>0.255</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.0895</td>\n",
       "      <td>0.0395</td>\n",
       "      <td>0.055</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Length  Diameter  Height  Whole Weight  Shucked Weight  Viscera Weight  \\\n",
       "0   0.455     0.365   0.095        0.5140          0.2245          0.1010   \n",
       "1   0.350     0.265   0.090        0.2255          0.0995          0.0485   \n",
       "2   0.530     0.420   0.135        0.6770          0.2565          0.1415   \n",
       "3   0.440     0.365   0.125        0.5160          0.2155          0.1140   \n",
       "4   0.330     0.255   0.080        0.2050          0.0895          0.0395   \n",
       "\n",
       "   Shell Weight  Rings ring_class  Sex_F  Sex_I  Sex_M  \n",
       "0         0.150     15          1      0      0      1  \n",
       "1         0.070      7          0      0      0      1  \n",
       "2         0.210      9          0      1      0      0  \n",
       "3         0.155     10          0      0      0      1  \n",
       "4         0.055      7          0      0      1      0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Final Dataframe \n",
    "df_expanded.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wicked-ecuador",
   "metadata": {},
   "source": [
    "### 2. Using your best guess for hyperparameters and kernel, what is the percentage of correctly classified results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "sticky-reconstruction",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Separating out Target Variable & Test/Train Sets \n",
    "\n",
    "col_names_list = list(df_expanded.columns)\n",
    "col_names_list.remove(\"ring_class\")\n",
    "col_names_list.remove(\"Rings\")\n",
    "\n",
    "X = df_expanded.loc[:, col_names_list]\n",
    "Y = df_expanded.loc[:, \"ring_class\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, Y, \n",
    "                    test_size = 0.25, random_state = 99)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "wooden-equity",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rbf Kernel\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.90      0.76      0.83       807\n",
      "           1       0.47      0.72      0.57       238\n",
      "\n",
      "    accuracy                           0.75      1045\n",
      "   macro avg       0.69      0.74      0.70      1045\n",
      "weighted avg       0.80      0.75      0.77      1045\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cost = .9 # penalty parameter \n",
    "gamma = 5 # defines the influence of input vectors on the margins\n",
    "\n",
    "from sklearn import svm, metrics\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# Test rbf \n",
    "clf = svm.SVC(gamma=gamma, kernel='linear', C=cost).fit(X_train, y_train)\n",
    "clf.predict(X_test)\n",
    "print(\"rbf Kernel\")\n",
    "print(classification_report(clf.predict(X_test), y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pointed-question",
   "metadata": {},
   "source": [
    "The percentage of correctly classified results i 0.75 or 75%. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sonic-apparatus",
   "metadata": {},
   "source": [
    "### 3. Test different kernels and hyperparameters or consider using sklearn.model_selection.SearchGridCV. Which kernel performed best with what settings?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fiscal-essay",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['0', '1', '0', ..., '0', '1', '0'], dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import model_selection\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "parameters = {'kernel':('linear', 'poly'), 'C':[.7, .8, .9, 1], 'gamma':[5, 10]}\n",
    "svc = svm.SVC()\n",
    "clf = GridSearchCV(svc, parameters)\n",
    "clf.fit(X_train, y_train)\n",
    "clf.predict(X_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "suited-graphics",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters identified: \n",
      "{'C': 0.7, 'gamma': 10, 'kernel': 'poly'}\n",
      "/n\n",
      "    mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
      "0        0.185732      0.008743         0.038187        0.000794     0.7   \n",
      "1        2.404163      0.212576         0.036837        0.001945     0.7   \n",
      "2        0.182182      0.003174         0.037997        0.000442     0.7   \n",
      "3       20.037556      2.785862         0.036111        0.001944     0.7   \n",
      "4        0.182693      0.003560         0.037234        0.000570     0.8   \n",
      "5        2.337878      0.206666         0.036232        0.001848     0.8   \n",
      "6        0.181574      0.005814         0.038001        0.001256     0.8   \n",
      "7       24.088684      3.467405         0.035141        0.000614     0.8   \n",
      "8        0.183876      0.006564         0.038716        0.001069     0.9   \n",
      "9        2.832718      0.410373         0.036512        0.002961     0.9   \n",
      "10       0.182851      0.004464         0.038089        0.001102     0.9   \n",
      "11      28.427846      2.840517         0.039416        0.004376     0.9   \n",
      "12       0.217551      0.004438         0.042451        0.002224       1   \n",
      "13       3.182565      0.467558         0.036257        0.001246       1   \n",
      "14       0.183440      0.003584         0.037560        0.000829       1   \n",
      "\n",
      "   param_gamma param_kernel                                       params  \\\n",
      "0            5       linear   {'C': 0.7, 'gamma': 5, 'kernel': 'linear'}   \n",
      "1            5         poly     {'C': 0.7, 'gamma': 5, 'kernel': 'poly'}   \n",
      "2           10       linear  {'C': 0.7, 'gamma': 10, 'kernel': 'linear'}   \n",
      "3           10         poly    {'C': 0.7, 'gamma': 10, 'kernel': 'poly'}   \n",
      "4            5       linear   {'C': 0.8, 'gamma': 5, 'kernel': 'linear'}   \n",
      "5            5         poly     {'C': 0.8, 'gamma': 5, 'kernel': 'poly'}   \n",
      "6           10       linear  {'C': 0.8, 'gamma': 10, 'kernel': 'linear'}   \n",
      "7           10         poly    {'C': 0.8, 'gamma': 10, 'kernel': 'poly'}   \n",
      "8            5       linear   {'C': 0.9, 'gamma': 5, 'kernel': 'linear'}   \n",
      "9            5         poly     {'C': 0.9, 'gamma': 5, 'kernel': 'poly'}   \n",
      "10          10       linear  {'C': 0.9, 'gamma': 10, 'kernel': 'linear'}   \n",
      "11          10         poly    {'C': 0.9, 'gamma': 10, 'kernel': 'poly'}   \n",
      "12           5       linear     {'C': 1, 'gamma': 5, 'kernel': 'linear'}   \n",
      "13           5         poly       {'C': 1, 'gamma': 5, 'kernel': 'poly'}   \n",
      "14          10       linear    {'C': 1, 'gamma': 10, 'kernel': 'linear'}   \n",
      "\n",
      "    split0_test_score  split1_test_score  split2_test_score  \\\n",
      "0            0.763955           0.770335           0.777955   \n",
      "1            0.765550           0.773525           0.809904   \n",
      "2            0.763955           0.770335           0.777955   \n",
      "3            0.762360           0.784689           0.814696   \n",
      "4            0.768740           0.770335           0.779553   \n",
      "5            0.770335           0.775120           0.809904   \n",
      "6            0.768740           0.770335           0.779553   \n",
      "7            0.762360           0.784689           0.814696   \n",
      "8            0.768740           0.768740           0.782748   \n",
      "9            0.765550           0.775120           0.809904   \n",
      "10           0.768740           0.768740           0.782748   \n",
      "11           0.762360           0.783094           0.813099   \n",
      "12           0.770335           0.768740           0.785942   \n",
      "13           0.762360           0.773525           0.809904   \n",
      "14           0.770335           0.768740           0.785942   \n",
      "\n",
      "    split3_test_score  split4_test_score  mean_test_score  std_test_score  \\\n",
      "0            0.784345           0.784345         0.776187        0.008001   \n",
      "1            0.787540           0.806709         0.788646        0.017558   \n",
      "2            0.784345           0.784345         0.776187        0.008001   \n",
      "3            0.793930           0.803514         0.791838        0.017795   \n",
      "4            0.785942           0.792332         0.779380        0.009010   \n",
      "5            0.790735           0.808307         0.790880        0.016347   \n",
      "6            0.785942           0.792332         0.779380        0.009010   \n",
      "7            0.792332           0.801917         0.791199        0.017563   \n",
      "8            0.785942           0.792332         0.779700        0.009466   \n",
      "9            0.793930           0.808307         0.790562        0.017688   \n",
      "10           0.785942           0.792332         0.779700        0.009466   \n",
      "11           0.790735           0.800319         0.789922        0.017042   \n",
      "12           0.789137           0.793930         0.781617        0.010198   \n",
      "13           0.797125           0.809904         0.790564        0.019375   \n",
      "14           0.789137           0.793930         0.781617        0.010198   \n",
      "\n",
      "    rank_test_score  \n",
      "0                15  \n",
      "1                 8  \n",
      "2                15  \n",
      "3                 1  \n",
      "4                13  \n",
      "5                 3  \n",
      "6                13  \n",
      "7                 2  \n",
      "8                11  \n",
      "9                 5  \n",
      "10               11  \n",
      "11                6  \n",
      "12                9  \n",
      "13                4  \n",
      "14                9  \n"
     ]
    }
   ],
   "source": [
    "## The Best Parameters: \n",
    "print(\"Best parameters identified: \")\n",
    "print(clf.best_params_)\n",
    "\n",
    "print(\"/n\")\n",
    "\n",
    "## Some Results of the GridSearchCV Output \n",
    "results = pd.DataFrame(clf.cv_results_)\n",
    "print(results.head(15))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secret-scholar",
   "metadata": {},
   "source": [
    "### 4. Show recall, precision and f-measure for the best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "legal-absolute",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Poly Kernel\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.79      0.84       770\n",
      "           1       0.55      0.73      0.63       275\n",
      "\n",
      "    accuracy                           0.77      1045\n",
      "   macro avg       0.72      0.76      0.73      1045\n",
      "weighted avg       0.80      0.77      0.78      1045\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## Updating Model to use Parameters from the GridSearchCV\n",
    "cost = 0.7 # penalty parameter \n",
    "gamma = 10 # defines the influence of input vectors on the margins\n",
    "\n",
    "from sklearn import svm, metrics\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# SVC Classifier with \"Best\" Parameters\n",
    "clf = svm.SVC(gamma=gamma, kernel='poly', C=cost).fit(X_train, y_train)\n",
    "clf.predict(X_test)\n",
    "print(\"Poly Kernel\")\n",
    "print(classification_report(clf.predict(X_test), y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "welsh-surgeon",
   "metadata": {},
   "source": [
    "Once using the polynomial kernal and the specific cost (0.7) and gamma (10), the accuracy increased by 2% to 77%. While this improved the accuracy metric, the overall number could still be improved proved."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "weird-straight",
   "metadata": {},
   "source": [
    "### 5. Using the original data, with rings as a continuous variable, create an SVR model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "compliant-makeup",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_expanded.loc[:, col_names_list]\n",
    "Y = df_expanded.loc[:, \"Rings\"]  # Only selecting \"Rings\" Continous Column for target \n",
    "\n",
    "\n",
    "## Standardizing Data Columns that are Not Binary:\n",
    "for each in ['Length', 'Diameter', 'Height', 'Whole Weight', 'Shucked Weight', \n",
    "            'Viscera Weight', 'Shell Weight']:\n",
    "    column_df = pd.DataFrame(df_expanded.loc[:, each])\n",
    "    standardization_scale = StandardScaler().fit(column_df)\n",
    "    column_df = pd.DataFrame(standardization_scale.transform(column_df))\n",
    "    X.loc[:, each] = column_df[0]\n",
    "\n",
    "\n",
    "## Splitting Testing /Training Data: \n",
    "X_train, X_test, y_train, y_test = train_test_split(X, Y, \n",
    "                    test_size = 0.25, random_state = 99)\n",
    "\n",
    "## SVR \n",
    "regr = svm.SVR()\n",
    "regr.fit(X_train, y_train) ## Training Model\n",
    "y_predict = regr.predict(X_test) ## Testing Model \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thrown-bronze",
   "metadata": {},
   "source": [
    "### 6. Report on the predicted variance and the mean squared error\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "pending-basket",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE is : 454.63868941135115%\n",
      "R^2:  0.5619097059750076\n",
      "Predicted Variance:  4.3959934413831805\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import * \n",
    "rmse = mean_squared_error(y_test, y_predict) * 100\n",
    "print(\"RMSE is : {}%\".format(rmse))\n",
    "\n",
    "rsqd = regr.score(X_train, y_train)\n",
    "print(\"R^2: \", rsqd)\n",
    "print(\"Predicted Variance: \", (1-rsqd) * np.var(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "younger-prior",
   "metadata": {},
   "source": [
    "#### Summary\n",
    "\n",
    "In this lesson, a few different Support Vector Machine techniques were applied for predicting the age of the Abalone. First a support vector machine classifier was used, with random hyperparameters chosen. This resulted in an accuracy rate of 75%. Cost was 0.9, Gamma =5, and the kernel chosen was linear. Next, to improve the results, GridSearchCV was utilize to identify the best combination of parameters for the support vector machine classifier. This resulted in using Cost =  0.7, gamma = 10, and a polynomial kernel, which achieved a 77% accuracy rate. \n",
    "\n",
    "Lastly, support vector machine regression model was generated and applied to the dataset. This resulted in a RMSE of 454 and R-squared of 0.56, which could certainly be improved perhaps with parameters. Parameters were passed to this model, however it was computationally expensive and did not result after 1-hr of processing - thus parameters were left off. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "virgin-concentration",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}