Chapter 7 Support Vector Machines and Kernel Methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 6: Support Vector Machines and Kernel Methods\n",
    "\n",
    "This notebook provides practical \"recipes\" for Support Vector Machines (SVMs). Each recipe includes explanations, code examples, visualizations, best practices, and common pitfalls when applicable.\n",
    "\n",
    "## Introduction to SVMs\n",
    "SVMs are powerful machine learning models used for both classification and regression tasks. SVMs are particularly effective in situations where the number of features is large compared to the number of samples, or when the data is high-dimensional. The core idea behind SVMs is to find the hyperplane that maximizes the margin between classes, which helps in achieving better generalization performance. In this section, we will explore the basics of SVMs, their role in classification and regression, and how they work.\n",
    "\n",
    "### Getting ready\n",
    "Before implementing SVMs, let’s ensure we have the necessary Python libraries installed and the dataset loaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries \n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC, SVR\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "import pandas as pd\n",
    "\n",
    "# Load the dataset\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "feature_names = iris.feature_names\n",
    "df = pd.DataFrame(X, columns=feature_names)\n",
    "df['target'] = y\n",
    "\n",
    "# Split the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2024)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to do it\n",
    "Now let's implement SVM for classification and regression tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model\n",
    "svm_classifier = SVC(kernel='linear')\n",
    "\n",
    "# Train the model\n",
    "svm_classifier.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = svm_classifier.predict(X_test)\n",
    "\n",
    "# Evaluate the model\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "report = classification_report(y_test, y_pred, output_dict=True)\n",
    "report_df = pd.DataFrame(report).transpose()\n",
    "\n",
    "# Stylize the DataFrame\n",
    "styled_df = (report_df\n",
    "    .style\n",
    "    .background_gradient(cmap='Blues', subset=['precision', 'recall', 'f1-score'])\n",
    "    .format({\n",
    "        'precision': '{:.3f}',\n",
    "        'recall': '{:.3f}', \n",
    "        'f1-score': '{:.3f}',\n",
    "        'support': '{:.0f}'\n",
    "    })\n",
    ")\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.2f}\")\n",
    "styled_df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's apply SVM to regression and evaluate our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model\n",
    "svm_regressor = SVR(kernel='rbf')\n",
    "\n",
    "# Train the model\n",
    "svm_regressor.fit(X_train[:, :2], y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred_reg = svm_regressor.predict(X_test[:, :2])\n",
    "\n",
    "# Evaluate the model\n",
    "# Note: For regression, we typically use metrics like MSE or R-squared\n",
    "from sklearn.metrics import mean_squared_error\n",
    "mse = mean_squared_error(y_test, y_pred_reg)\n",
    "print(f\"MSE: {mse:.2f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How it works\n",
    "SVMs work by finding the hyperplane that maximizes the margin between classes. The margin is the distance between the hyperplane and the nearest data points of each class, known as support vectors. SVMs can handle non-linearly separable data by using kernel functions, which map the data into higher-dimensional spaces where it becomes linearly separable.\n",
    "\n",
    "* Kernel Functions: Common kernels include linear, polynomial, and radial basis function (RBF). The choice of kernel depends on the complexity of the data and the problem at hand.\n",
    "\n",
    "* Soft Margin: SVMs can tolerate some misclassifications by introducing slack variables, allowing for a \"soft margin\" that improves robustness against outliers.\n",
    "\n",
    "Visualizing the decision boundaries of SVMs can provide insights into how they classify data. For linear SVMs, the decision boundary is a line that maximizes the margin between classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the libraries\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.inspection import DecisionBoundaryDisplay\n",
    "\n",
    "# Create a mesh grid\n",
    "x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.01),\n",
    "                     np.arange(y_min, y_max, 0.01))\n",
    "\n",
    "# Adjust the input to match the expected number of features\n",
    "# Use all four features for prediction, but only visualize the first two\n",
    "grid_points = np.c_[xx.ravel(), yy.ravel(), np.zeros((xx.ravel().shape[0], 2))]\n",
    "\n",
    "# Plot decision boundaries using DecisionBoundaryDisplay\n",
    "fig, ax = plt.subplots()\n",
    "disp = DecisionBoundaryDisplay.from_estimator(\n",
    "    svm_regressor,\n",
    "    grid_points[:, :2],\n",
    "    response_method=\"predict\",\n",
    "    plot_method=\"contourf\",\n",
    "    cmap=plt.cm.coolwarm,\n",
    "    alpha=0.8,\n",
    "    ax=ax\n",
    ")\n",
    "\n",
    "# Plot the data points\n",
    "ax.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', marker='o', cmap=plt.cm.coolwarm)\n",
    "ax.set_title(\"SVM Decision Boundary\")\n",
    "ax.set_xlabel(\"Feature 1\")\n",
    "ax.set_ylabel(\"Feature 2\")\n",
    "ax.set_xlim(xx.min(), xx.max())\n",
    "ax.set_ylim(yy.min(), yy.max())\n",
    "ax.set_xticks(())\n",
    "ax.set_yticks(())\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kernel Functions and Their Applications\n",
    "Kernel functions are the critical element that give SVMs their power, allowing them to handle non-linear data effectively. By mapping the original data into a higher-dimensional space, kernel functions enable SVMs to find linear decision boundaries in this new space, which correspond to non-linear boundaries in the original space\n",
    "\n",
    "### Getting ready\n",
    "Before implementing kernel functions with SVMs, let’s ensure we have the necessary Python libraries installed and the dataset loaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the libraries\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Load the dataset\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "feature_names = iris.feature_names\n",
    "df = pd.DataFrame(X, columns=feature_names)\n",
    "df['target'] = y\n",
    "\n",
    "# Split the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2024)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to do it\n",
    "Now let's implement SVMs using different kernel functions. We already used the linear kernel in the first example so now let’s use the Polynomial Kernel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model\n",
    "svm_poly = SVC(kernel='poly', degree=3)\n",
    "\n",
    "# Train the model\n",
    "svm_poly.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred_poly = svm_poly.predict(X_test)\n",
    "\n",
    "# Evaluate the model\n",
    "accuracy_poly = accuracy_score(y_test, y_pred_poly)\n",
    "report_poly = classification_report(y_test, y_pred_poly, output_dict=True)\n",
    "report_df_poly = pd.DataFrame(report_poly).transpose()\n",
    "\n",
    "# Stylize the DataFrame\n",
    "styled_df_poly = (report_df_poly\n",
    "    .style\n",
    "    .background_gradient(cmap='Blues', subset=['precision', 'recall', 'f1-score'])\n",
    "    .format({\n",
    "        'precision': '{:.3f}',\n",
    "        'recall': '{:.3f}', \n",
    "        'f1-score': '{:.3f}',\n",
    "        'support': '{:.0f}'\n",
    "    })\n",
    ")\n",
    "\n",
    "print(f\"Polynomial Kernel Accuracy: {accuracy_poly:.2f}\")\n",
    "styled_df_poly\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now with the RBF Kernel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model\n",
    "svm_rbf = SVC(kernel='rbf')\n",
    "\n",
    "# Train the model\n",
    "svm_rbf.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred_rbf = svm_rbf.predict(X_test)\n",
    "\n",
    "# Evaluate the model\n",
    "accuracy_rbf = accuracy_score(y_test, y_pred_rbf)\n",
    "report_rbf = classification_report(y_test, y_pred_rbf, output_dict=True)\n",
    "report_df_rbf = pd.DataFrame(report_rbf).transpose()\n",
    "\n",
    "# Stylize the DataFrame\n",
    "styled_df_rbf = (report_df_rbf\n",
    "    .style\n",
    "    .background_gradient(cmap='Blues', subset=['precision', 'recall', 'f1-score'])\n",
    "    .format({\n",
    "        'precision': '{:.3f}',\n",
    "        'recall': '{:.3f}', \n",
    "        'f1-score': '{:.3f}',\n",
    "        'support': '{:.0f}'\n",
    "    })\n",
    ")\n",
    "\n",
    "print(f\"RBF Kernel Accuracy: {accuracy_rbf:.2f}\")\n",
    "styled_df_rbf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualizing the decision boundaries of SVMs with different kernels can provide insights into how they classify data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a mesh grid using only the first two features for visualization\n",
    "x_min, x_max = X_train[:, 0].min() - 1, X_train[:, 0].max() + 1\n",
    "y_min, y_max = X_train[:, 1].min() - 1, X_train[:, 1].max() + 1\n",
    "\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.01),\n",
    "                     np.arange(y_min, y_max, 0.01))\n",
    "\n",
    "# Use only the first two features for prediction\n",
    "X_train_2d = X_train[:, :2]\n",
    "\n",
    "# Train new SVM models using only the first two features\n",
    "svm_linear_2d = SVC(kernel='linear').fit(X_train_2d, y_train)\n",
    "svm_poly_2d = SVC(kernel='poly').fit(X_train_2d, y_train)\n",
    "svm_rbf_2d = SVC(kernel='rbf').fit(X_train_2d, y_train)\n",
    "\n",
    "# Predict class probabilities across the grid for each kernel using only the first two features\n",
    "Z_linear = svm_linear_2d.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "Z_poly = svm_poly_2d.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "Z_rbf = svm_rbf_2d.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "\n",
    "Z_linear = Z_linear.reshape(xx.shape)\n",
    "Z_poly = Z_poly.reshape(xx.shape)\n",
    "Z_rbf = Z_rbf.reshape(xx.shape)\n",
    "\n",
    "# Plot decision boundaries\n",
    "fig, axes = plt.subplots(1, 3, figsize=(18, 6))\n",
    "\n",
    "axes[0].contourf(xx, yy, Z_linear, alpha=0.8)\n",
    "axes[0].scatter(X_train_2d[:, 0], X_train_2d[:, 1], c=y_train, edgecolors='k', marker='o')\n",
    "axes[0].set_title(\"Linear Kernel\")\n",
    "\n",
    "axes[1].contourf(xx, yy, Z_poly, alpha=0.8)\n",
    "axes[1].scatter(X_train_2d[:, 0], X_train_2d[:, 1], c=y_train, edgecolors='k', marker='o')\n",
    "axes[1].set_title(\"Polynomial Kernel\")\n",
    "\n",
    "axes[2].contourf(xx, yy, Z_rbf, alpha=0.8)\n",
    "axes[2].scatter(X_train_2d[:, 0], X_train_2d[:, 1], c=y_train, edgecolors='k', marker='o')\n",
    "axes[2].set_title(\"RBF Kernel\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning SVM Parameters\n",
    "As we’ve seen with all other ML models up to this point, hyperparameter tuning and optimization is a key step in improving the performance of SVMs as well. By adjusting parameters such as the regularization parameter (C) and kernel parameters, we can significantly improve the accuracy and robustness of SVM models. In this section, we will discover how to use classic grid search and cross-validation techniques to optimize SVM models using scikit-learn.\n",
    "\n",
    "### Getting ready\n",
    "Before tuning SVM parameters, let’s ensure we have the necessary Python libraries installed and the dataset loaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the libraries\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "import pandas as pd\n",
    "\n",
    "# Load the dataset\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "feature_names = iris.feature_names\n",
    "df = pd.DataFrame(X, columns=feature_names)\n",
    "df['target'] = y\n",
    "\n",
    "# Split the dataset\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to do it\n",
    "Now let's implement SVM parameter tuning using grid search and cross-validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the parameter grid\n",
    "param_grid = {\n",
    "    'C': [0.1, 1, 10],\n",
    "    'kernel': ['linear', 'rbf', 'poly'],\n",
    "    'degree': [2, 3, 4]\n",
    "}\n",
    "\n",
    "# Perform grid search with cross-validation\n",
    "grid_search = GridSearchCV(SVC(), param_grid, cv=5)\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# Get the best hyperparameters and score\n",
    "best_params = grid_search.best_params_\n",
    "best_score = grid_search.best_score_\n",
    "print(f\"Best Parameters: {best_params}\")\n",
    "print(f\"Best Cross-Validation Score: {best_score:.2f}\")\n",
    "\n",
    "# Train a new model with the best hyperparameters\n",
    "best_model = SVC(**best_params)\n",
    "best_model.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions with the best model\n",
    "y_pred = best_model.predict(X_test)\n",
    "\n",
    "# Evaluate the best model\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "report_df = pd.DataFrame(classification_report(y_test, y_pred, output_dict=True)).transpose()\n",
    "\n",
    "# Stylize the DataFrame\n",
    "styled_report_df = (report_df\n",
    "    .style\n",
    "    .background_gradient(cmap='Blues', subset=['precision', 'recall', 'f1-score'])\n",
    "    .format({\n",
    "        'precision': '{:.3f}',\n",
    "        'recall': '{:.3f}', \n",
    "        'f1-score': '{:.3f}',\n",
    "        'support': '{:.0f}'\n",
    "    })\n",
    ")\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.2f}\")\n",
    "styled_report_df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How it works\n",
    "\n",
    "Grid search is a systematic approach to hyperparameter tuning that involves evaluating all possible combinations of parameters specified in a grid. Cross-validation is used to assess the performance of each combination, ensuring that the model generalizes well to unseen data.\n",
    "\n",
    "* **Regularization Parameter (C)**: Controls the trade-off between margin and misclassification error. A high C value means the model is less tolerant of misclassifications.\n",
    "\n",
    "* **Kernel Parameters**: Different kernels (e.g., linear, RBF, polynomial) and their parameters (e.g., degree for polynomial, gamma for RBF) affect how the data is mapped into higher-dimensional spaces.\n",
    "\n",
    "Visualizing how different hyperparameters affect model performance can provide insights into the tuning process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform a grid search with a range of C values for a specific kernel\n",
    "param_grid_c = {\n",
    "    'C': np.logspace(-4, 4, 10),\n",
    "    'kernel': ['rbf']\n",
    "}\n",
    "grid_search_c = GridSearchCV(SVC(), param_grid_c, cv=5)\n",
    "grid_search_c.fit(X_train, y_train)\n",
    "\n",
    "# Plot the cross-validation scores against C values\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(grid_search_c.param_grid['C'], grid_search_c.cv_results_['mean_test_score'])\n",
    "plt.xscale('log')\n",
    "plt.xlabel('C (Regularization Parameter)')\n",
    "plt.ylabel('Cross-Validation Score')\n",
    "plt.title('Effect of C on Model Performance')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVMs in High-Dimensional Spaces\n",
    "SVMs are particularly effective in handling high-dimensional data, where the number of features is large compared to the number of samples. For example, maybe you are measuring a rare event but have several powerful instruments to gather hundreds of data points about them once they do occur. We call this a “wide” dataset. In this section, we will look more closely at how SVMs can be applied to high-dimensional data, both synthetically generated and from real-world datasets.\n",
    "\n",
    "### Getting ready\n",
    "Before applying SVMs to high-dimensional data, let’s ensure we have the necessary Python libraries installed and the dataset loaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the libraries\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Create a synthetic high-dimensional dataset\n",
    "X, y = make_classification(n_samples=1000, n_features=1000, n_informative=50, n_redundant=0, random_state=2024)\n",
    "\n",
    "# Split the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to do it...\n",
    "Now let's implement SVMs for high-dimensional data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model\n",
    "svm_model = SVC(kernel='linear')\n",
    "\n",
    "# Train the model\n",
    "svm_model.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluate the model\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "report_df = pd.DataFrame(classification_report(y_test, y_pred, output_dict=True)).transpose()\n",
    "\n",
    "# Stylize the DataFrame\n",
    "styled_report_df = (report_df\n",
    "    .style\n",
    "    .background_gradient(cmap='Blues', subset=['precision', 'recall', 'f1-score'])\n",
    "    .format({\n",
    "        'precision': '{:.3f}',\n",
    "        'recall': '{:.3f}', \n",
    "        'f1-score': '{:.3f}',\n",
    "        'support': '{:.0f}'\n",
    "    })\n",
    ")\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.2f}\")\n",
    "styled_report_df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How It Works...\n",
    "\n",
    "SVMs handle high-dimensional data efficiently by leveraging kernel functions. The choice of kernel depends on the nature of the data and the complexity of the decision boundary.\n",
    "* **Linear Kernel**: Suitable for linearly separable data or when the number of features is large.\n",
    "* **Non-Linear Kernels (e.g., RBF, Polynomial)**: Useful for non-linearly separable data, allowing SVMs to find complex decision boundaries.\n",
    "\n",
    "Visualizing high-dimensional data directly is challenging, but we can use dimensionality reduction techniques like PCA to project the data onto a lower-dimensional space for visualization. To visualize the data using PCA:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PCA from sklearn.decomposition\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Apply PCA to reduce dimensionality\n",
    "pca = PCA(n_components=2)\n",
    "X_pca = pca.fit_transform(X)\n",
    "\n",
    "# Plot the reduced data\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, edgecolors='k', marker='o')\n",
    "plt.title(\"High-Dimensional Data in 2D Space Using PCA\")\n",
    "plt.xlabel(\"Principal Component 1\")\n",
    "plt.ylabel(\"Principal Component 2\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating SVM Models\n",
    "Evaluating the performance of Support Vector Machine (SVM) models is crucial for understanding how well they generalize to new data. In this section, we will explore key metrics for evaluating SVM models, including accuracy, precision, recall, and ROC curves. By applying these metrics, we can assess the strengths and weaknesses of our SVM models and make informed decisions about their deployment.\n",
    "\n",
    "### Getting Ready\n",
    "Before evaluating SVM models, let’s ensure we have the necessary Python libraries installed and the dataset loaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the libraries\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.metrics import classification_report, roc_curve, auc\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Load the dataset\n",
    "data = load_breast_cancer()\n",
    "X = data.data\n",
    "y = data.target\n",
    "feature_names = data.feature_names\n",
    "df = pd.DataFrame(X, columns=feature_names)\n",
    "df['target'] = y\n",
    "\n",
    "# Split the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2024)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to Do It...\n",
    "Now let's implement SVM and evaluate its performance using various metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model\n",
    "svm_model = SVC(kernel='linear', probability=True)\n",
    "\n",
    "# Train the model\n",
    "svm_model.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = svm_model.predict(X_test)\n",
    "y_prob = svm_model.predict_proba(X_test)[:, 1]\n",
    "\n",
    "# Evaluate the model\n",
    "report = classification_report(y_test, y_pred, output_dict=True)\n",
    "report_df = pd.DataFrame(report).transpose()\n",
    "\n",
    "# Stylize the DataFrame\n",
    "styled_report_df = (report_df\n",
    "    .style\n",
    "    .background_gradient(cmap='Blues', subset=['precision', 'recall', 'f1-score'])\n",
    "    .format({\n",
    "        'precision': '{:.3f}',\n",
    "        'recall': '{:.3f}', \n",
    "        'f1-score': '{:.3f}',\n",
    "        'support': '{:.0f}'\n",
    "    })\n",
    ")\n",
    "\n",
    "print(f\"Accuracy: {accuracy:.2f}\")\n",
    "styled_report_df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the ROC curve\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_prob)\n",
    "roc_auc = auc(fpr, tpr)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (area = {roc_auc:.2f})')\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver Operating Characteristic (ROC)')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How It Works...\n",
    "\n",
    "Evaluating SVM models involves assessing their performance using various metrics:\n",
    "•\tAccuracy: Measures the proportion of correctly classified instances.\n",
    "•\tPrecision: Measures the proportion of true positives among all predicted positive instances.\n",
    "•\tRecall: Measures the proportion of true positives among all actual positive instances.\n",
    "•\tF1 Score: The harmonic mean of precision and recall, providing a balanced measure of both.\n",
    "•\tROC Curve: Plots the true positive rate against the false positive rate at different thresholds, providing a comprehensive view of the model's performance across all possible thresholds.\n",
    "•\tAUC (Area Under the Curve): Quantifies the overall ability of the model to distinguish between positive and negative classes.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practical Exercises with SVMs\n",
    "\n",
    "In this final section, we will engage in practical exercises that involve building, tuning, and evaluating SVM models on various datasets. These exercises are designed to reinforce the concepts learned throughout the chapter and demonstrate how to effectively apply SVMs in different scenarios. By the end of this section, we will have hands-on experience that can be leveraged in our own ML projects.\n",
    "\n",
    "### Exercise 1: Building a Simple SVM Classifier\n",
    "In this exercise, we will build a simple SVM classifier using the Iris dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Load the dataset\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Split the data\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Create and train a SVM classifier\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Make predictions\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Evaluate performance\n",
    "YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2: Tuning SVM Parameters with Grid Search\n",
    "In this exercise, we will tune SVM parameters using grid search on the Breast Cancer dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the libraries\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Load the dataset\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Split the data\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Define a hyperparameter for grid search\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Perform grid search with cross-validation\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Train a new model with the best hyperparameters\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Make predictions with the best model\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Evaluate the best model\n",
    "YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3: Visualizing SVM Decision Boundaries\n",
    "In this exercise, we will visualize the decision boundaries of an SVM model trained on a synthetic dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries for visualization and dataset creation\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Create a synthetic dataset for binary classification\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Split the data\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Create and train an SVM model\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Create a mesh grid for plotting decision boundaries\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Predict class probabilities across the grid\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Plot decision boundaries\n",
    "YOUR CODE HERE\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}