03-02_Decision_Trees.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "title: \"ClassifyAnything Part 3-2: Decision Trees\"\n",
    "author: \"Joseph Chao-Chung Kuo (ckuo@ukaachen.de)\"\n",
    "date: today\n",
    "output:\n",
    "  general:\n",
    "    output_stdout: true\n",
    "    output_error: false\n",
    "    output: true\n",
    "format:\n",
    "  html:\n",
    "    toc: true\n",
    "    toc-depth: 3\n",
    "    html-math-method: katex\n",
    "    code-fold: true\n",
    "    code-tools: true\n",
    "    number_sections: true\n",
    "    embed-resources: true\n",
    "    page-layout: full\n",
    "execute:\n",
    "    echo: true\n",
    "    warning: false\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Supervised learning for classification_\n",
    "\n",
    "Decision Trees are interpretable algorithms used for supervised classification. They create a tree-like model by recursively splitting the data based on feature values. Each internal node represents a feature test, and the leaf nodes represent the predicted class labels. Decision Trees are easy to interpret and handle both numerical and categorical data. Overfitting can be a concern, but techniques like pruning and ensemble methods (e.g., Random Forests) help address this issue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.1 Import data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.tree import DecisionTreeClassifier, plot_tree\n",
    "import pickle\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "with open(\"outputs/03_Variables.pkl\", 'rb') as file:\n",
    "    (X_train, X_test, y_train, y_test, kfold) = pickle.load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.2 Train the model with cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparameter grid for grid search\n",
    "param_grid = {\n",
    "    'criterion': ['gini', 'entropy'],\n",
    "    'max_depth': [1, 5, 10, 20, 30],\n",
    "    'min_samples_split': [2, 5, 10],\n",
    "    'min_samples_leaf': [1, 2, 4]\n",
    "}\n",
    "\n",
    "# Create a DecisionTreeClassifier\n",
    "dt_classifier = DecisionTreeClassifier()\n",
    "# Create a GridSearchCV object with cross-validation\n",
    "grid_search = GridSearchCV(dt_classifier, param_grid, cv=kfold)\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# Get the best hyperparameters and the best model\n",
    "best_params = grid_search.best_params_\n",
    "best_model = grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the best hyperparameters\n",
    "for p in best_params:\n",
    "    print(\":\\t\".join([p,str(best_params[p])]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.3 Visualize fine-tuning of hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access the grid search results\n",
    "results = pd.DataFrame(grid_search.cv_results_)\n",
    "\n",
    "# Create a pivot table to reshape the data for the heatmap\n",
    "pivot_table = results.pivot_table(index='param_criterion', \n",
    "                                  columns='param_max_depth', \n",
    "                                  values='mean_test_score')\n",
    "\n",
    "# Create the heatmap using seaborn\n",
    "plt.figure()\n",
    "sns.heatmap(pivot_table, annot=True, cmap='viridis', fmt=\".3f\")\n",
    "plt.title('Grid Search Results: Criterion vs Max Depth')\n",
    "plt.xlabel('Max Depth')\n",
    "plt.ylabel('Criterion')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pivot table to reshape the data for the heatmap\n",
    "pivot_table = results.pivot_table(index='param_min_samples_leaf', \n",
    "                                  columns='param_min_samples_split', \n",
    "                                  values='mean_test_score')\n",
    "\n",
    "# Create the heatmap using seaborn\n",
    "plt.figure()\n",
    "sns.heatmap(pivot_table, annot=True, cmap='viridis', fmt=\".3f\")\n",
    "plt.title('Grid Search Results: Min Sample Leaf vs Min Sample Split')\n",
    "plt.xlabel('Min Sample Split')\n",
    "plt.ylabel('Min Sample Leaf')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.4 Visualize decision trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the Decision Tree\n",
    "plt.figure(figsize=(15, 10))\n",
    "plot_tree(best_model, \n",
    "          feature_names=X_train.columns.to_list(),\n",
    "          class_names=np.unique(y_train.to_list()).tolist())\n",
    "# Use the line below to verify the label names\n",
    "# np.unique(y_train.to_list(), return_counts=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2.5 Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "# Save the final trained model to a file\n",
    "joblib.dump(best_model, 'outputs/03-2_Decision_Tree_final_model.joblib')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}