Chapter 4 Building Models with Distance Metrics and Nearest Neighbors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 4: Building Models with Distance Metrics and Nearest Neighbors\n",
    "\n",
    "This notebook provides practical \"recipes\" for using k-nearest neighbors (KNN) and distance metrics for clustering, classification, and regression tasks in scikit-learn. Each recipe includes explanations, code examples, visualizations, best practices, and common pitfalls. We'll cover implementation, tuning, and evaluation of KNN models.\n",
    "\n",
    "## Understanding k-nearest neighbors (KNN)\n",
    "\n",
    "### Getting ready\n",
    "KNN is a simple yet powerful classification algorithm that:\n",
    "- Makes predictions based on the \"k\" closest training examples\n",
    "- Is non-parametric (no assumptions about data distribution)\n",
    "- Can be used for both classification and regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Load the iris dataset\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "\n",
    "# Split the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=2024\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to do it\n",
    "To implement a basic KNN model using scikit-learn, follow these steps. As you can see, this follows the same design paradigm as the other models we've seen in this book so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Create and train a basic KNN model\n",
    "knn = KNeighborsClassifier(n_neighbors=3)\n",
    "knn.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = knn.predict(X_test)\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How it works\n",
    "The KNN algorithm works by:\n",
    "1. Storing all training examples\n",
    "2. For each new point:\n",
    "   - Calculating distances to all training examples\n",
    "   - Finding the k nearest neighbors\n",
    "   - Taking a majority vote (for classification) or average (for regression)\n",
    "\n",
    "## Distance Metrics Overview\n",
    "\n",
    "### Getting ready\n",
    "We’ll create two new datasets for illustrating the differences between distance metrics using scikit-learn’s built-in make_circles() function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.datasets import make_circles\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Create two synthetic datasets that highlight metric differences\n",
    "n_samples = 300\n",
    "\n",
    "# Circular dataset with high noise (Euclidean distance should work better)\n",
    "X_circles, y_circles = make_circles(n_samples=n_samples, noise=0.3, factor=0.3)\n",
    "X_circles_train, X_circles_test, y_circles_train, y_circles_test = train_test_split(\n",
    "    X_circles, y_circles, test_size=0.2, random_state=2024\n",
    ")\n",
    "\n",
    "# Checkerboard pattern dataset (Manhattan distance should work better)\n",
    "x = np.linspace(0, 4, int(np.sqrt(n_samples)))\n",
    "y = np.linspace(0, 4, int(np.sqrt(n_samples)))\n",
    "xx, yy = np.meshgrid(x, y)\n",
    "X_moons = np.column_stack((xx.ravel(), yy.ravel()))\n",
    "y_moons = np.mod(np.floor(xx.ravel()) + np.floor(yy.ravel()), 2)\n",
    "X_moons_train, X_moons_test, y_moons_train, y_moons_test = train_test_split(\n",
    "    X_moons, y_moons, test_size=0.2, random_state=2024\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to do it\n",
    "\n",
    "To compare various distance metrics, we’ll cycle through each one and build two KNN classifiers with it on our datasets, then visualize the dataset along with the performance metric, accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compare different distance metrics on both datasets\n",
    "metrics = ['euclidean', 'manhattan', 'minkowski']\n",
    "results = {'circles': {}, 'moons': {}}\n",
    "\n",
    "for metric in metrics:\n",
    "    # Test on \"circles\" dataset\n",
    "    knn_circles = KNeighborsClassifier(n_neighbors=3, metric=metric)\n",
    "    knn_circles.fit(X_circles_train, y_circles_train)\n",
    "    y_pred_circles = knn_circles.predict(X_circles_test)\n",
    "    results['circles'][metric] = accuracy_score(y_circles_test, y_pred_circles)\n",
    "    \n",
    "    # Test on \"moons\" dataset\n",
    "    knn_moons = KNeighborsClassifier(n_neighbors=3, metric=metric)\n",
    "    knn_moons.fit(X_moons_train, y_moons_train)\n",
    "    y_pred_moons = knn_moons.predict(X_moons_test)\n",
    "    results['moons'][metric] = accuracy_score(y_moons_test, y_pred_moons)\n",
    "\n",
    "# Set up plot specs\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))\n",
    "\n",
    "# Plot circles dataset\n",
    "scatter1_train = ax1.scatter(X_circles_train[:, 0], X_circles_train[:, 1], c=y_circles_train, cmap='viridis', alpha=0.6, label='Training Points')\n",
    "scatter1_test = ax1.scatter(X_circles_test[:, 0], X_circles_test[:, 1], color='red', marker='x', s=50, label='Test Points')\n",
    "ax1.set_title('Noisy Circles Dataset')\n",
    "\n",
    "# Create dummy lines for legend\n",
    "lines1 = [plt.Line2D([0], [0], color='white') for _ in metrics]\n",
    "legend_text1 = [\n",
    "    f\"Euclidean: {results['circles']['euclidean']:.3f}\",\n",
    "    f\"Manhattan: {results['circles']['manhattan']:.3f}\",\n",
    "    f\"Minkowski: {results['circles']['minkowski']:.3f}\"\n",
    "]\n",
    "ax1.legend(lines1 + [scatter1_train, scatter1_test], legend_text1 + ['Training Points', 'Test Points'], \n",
    "          title='Accuracy by Metric', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
    "\n",
    "# Plot moons dataset\n",
    "scatter2_train = ax2.scatter(X_moons_train[:, 0], X_moons_train[:, 1], c=y_moons_train, cmap='viridis', alpha=0.6, label='Training Points')\n",
    "scatter2_test = ax2.scatter(X_moons_test[:, 0], X_moons_test[:, 1], color='red', marker='x', s=50, label='Test Points')\n",
    "ax2.set_title('Checkerboard Dataset')\n",
    "\n",
    "# Create dummy lines for legend\n",
    "lines2 = [plt.Line2D([0], [0], color='white') for _ in metrics]\n",
    "legend_text2 = [\n",
    "    f\"Euclidean: {results['moons']['euclidean']:.3f}\",\n",
    "    f\"Manhattan: {results['moons']['manhattan']:.3f}\",\n",
    "    f\"Minkowski: {results['moons']['minkowski']:.3f}\"\n",
    "]\n",
    "ax2.legend(lines2 + [scatter2_train, scatter2_test], legend_text2 + ['Training Points', 'Test Points'],\n",
    "          title='Accuracy by Metric', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning in KNN\n",
    "\n",
    "Hyperparameter tuning is a critical step in optimizing the performance of machine learning models, including KNN\n",
    "\n",
    "### Getting ready\n",
    "\n",
    "We will use our original dataset, the Iris dataset, and tune the number of neighbors, whether or not to weight our training data by distance from our test data, and the choice of distance metric.\n",
    "\n",
    "### How to do it\n",
    "\n",
    "To perform hyperparameter tuning for KNN, we will use the grid search approach. Grid search is a hyperparameter technique that tests an exhaustive number of combinations of model hyperparameters – all combinations are tested with the resulting combination returned to us as the optimal set. Additionally, cross-validation is used to add an element of randomization that aids in making sure our model is robust when used in the real world."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# Create KNN classifier\n",
    "knn = KNeighborsClassifier()\n",
    "\n",
    "# Define parameter grid\n",
    "param_grid = {\n",
    "    'n_neighbors': [3, 5, 7, 9, 11],\n",
    "    'weights': ['uniform', 'distance'], \n",
    "    'metric': ['euclidean', 'manhattan']\n",
    "}\n",
    "\n",
    "# Create grid search\n",
    "grid_search = GridSearchCV(\n",
    "    knn, param_grid, cv=5, scoring='accuracy'\n",
    ")\n",
    "\n",
    "# Fit grid search\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# Print best parameters and score\n",
    "print(\"Best parameters:\", grid_search.best_params_)\n",
    "print(\"Best cross-validation score:\", grid_search.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating KNN Performance\n",
    "\n",
    "Evaluating the performance of KNN models is essential for understanding how well the model makes predictions and where it may need improvement. This section will cover various techniques for assessing KNN performance, including confusion matrices, precision, recall, and F1 scores. \n",
    "\n",
    "### Getting ready\n",
    "Again, we will use our toy dataset from before, the Iris dataset, and import a few new functions to help us evaluate our model. We'll evaluate our model using:\n",
    "- Cross-validation scores\n",
    "- Learning curves\n",
    "- Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "from sklearn.model_selection import learning_curve, cross_val_score\n",
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "import seaborn as sns\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to do it...\n",
    "\n",
    "To evaluate the performance of your KNN model, we’ll use the three different approaches mentioned above. If you refer to the visualizations located in the output of this code block, you'll get some insights into the performance of your model. For example, the learning curves show that the model doesn't improve much after ~50 training examples, and the confusion matrix shows that the model is having trouble distinguishing between the first and second classes.\n",
    "\n",
    "The classification report provides a detailed breakdown of the model's performance, including precision, recall, and F1 scores for each class. The fact that the setosa class has a high precision, recall, and F1 score indicates that the model is doing well at predicting this class, but since it's predicting with 100 percent accuracy it's probably an indication of overfitting.\n",
    "\n",
    "(NOTE: The \"accuracy\" measure is shown in the table three times, but it's the same value for each class since accuracy is an overall measure of the model's performance across classes.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get cross-validation scores\n",
    "cv_scores = cross_val_score(grid_search.best_estimator_, X_train, y_train, cv=5)\n",
    "print(\"Cross-validation scores:\", cv_scores)\n",
    "print(\"Mean CV score:\", cv_scores.mean())\n",
    "print(\"Standard deviation:\", cv_scores.std())\n",
    "\n",
    "# Generate learning curves\n",
    "train_sizes, train_scores, test_scores = learning_curve(\n",
    "    grid_search.best_estimator_, X_train, y_train,\n",
    "    train_sizes=np.linspace(0.1, 1.0, 10),\n",
    "    cv=5\n",
    ")\n",
    "\n",
    "# Plot learning curves\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(train_sizes, np.mean(train_scores, axis=1), label='Training score')\n",
    "plt.plot(train_sizes, np.mean(test_scores, axis=1), label='Cross-validation score')\n",
    "plt.xlabel('Training examples')\n",
    "plt.ylabel('Score (Accuracy)')\n",
    "plt.legend(loc='best')\n",
    "plt.title('Learning Curves')\n",
    "plt.show()\n",
    "\n",
    "# Make predictions on test set\n",
    "y_pred = grid_search.best_estimator_.predict(X_test)\n",
    "\n",
    "# Get class names from iris dataset\n",
    "class_names = iris.target_names\n",
    "\n",
    "# Generate and plot confusion matrix\n",
    "cm = confusion_matrix(y_test, y_pred)\n",
    "plt.figure(figsize=(8, 6))\n",
    "sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', \n",
    "            xticklabels=class_names, \n",
    "            yticklabels=class_names,\n",
    "            cbar_kws={'label': 'Number of Samples'})\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('True')\n",
    "plt.title('Confusion Matrix')\n",
    "plt.show()\n",
    "\n",
    "# Get classification report as a dict\n",
    "report_dict = classification_report(y_test, y_pred, \n",
    "                                  target_names=class_names,\n",
    "                                  output_dict=True)\n",
    "\n",
    "# Create a DataFrame for better visualization\n",
    "report_df = pd.DataFrame(report_dict).transpose()\n",
    "\n",
    "# Style the DataFrame\n",
    "styled_df = (report_df\n",
    "    .style\n",
    "    .background_gradient(cmap='Blues', subset=['precision', 'recall', 'f1-score'])\n",
    "    .format({\n",
    "        'precision': '{:.3f}',\n",
    "        'recall': '{:.3f}',\n",
    "        'f1-score': '{:.3f}',\n",
    "        'support': '{:.0f}'\n",
    "    })\n",
    ")\n",
    "\n",
    "print(\"\\nClassification Report:\")\n",
    "display(styled_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practical Exercises with KNN Models\n",
    "\n",
    "### Exercise 1: Building a KNN Classifier\n",
    "Implement a KNN classifier for a binary classification problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Load the Dataset\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Preprocess the Data\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Create and Train the KNN Classifier\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Make Predictions\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Evaluate Performance\n",
    "YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2: Tuning Hyperparameters with Grid Search\n",
    "Compare the performance of different KNN models using grid search\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Load a different dataset\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Preprocess the data\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Set up grid search: \n",
    "YOUR CODE HERE\n",
    "\n",
    "# Fit grid search\n",
    "YOUR CODE HERE\n",
    "# Evaluate best model\n",
    "YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3 Evaluating a KNN Classifier\n",
    "Evaluate the performance of a KNN classifier using the techniques we've covered in this chapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Load the Dataset\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Preprocess the Data\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Create and Train the KNN Classifier\n",
    "YOUR CODE HERE\n",
    "\n",
    "# Make Predictions\n",
    "YOUR CODE HERE\n",
    "# Evaluate Performance\n",
    "YOUR CODE HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}