tutorials/tutorial_00_imputer_plugins.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "molecular-moscow",
   "metadata": {},
   "source": [
    "# Imputation Plugins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "corrected-basis",
   "metadata": {},
   "source": [
    "Missing data is a crucial issue when applying machine learning algorithms to real-world datasets.\n",
    "\n",
    "**HyperImpute** provides a set of default imputation plugins and can be extended with any number of other plugins."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "auburn-hygiene",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wanted-point",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import warnings\n",
    "import time\n",
    "from tqdm import tqdm\n",
    "from math import sqrt\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "\n",
    "from sklearn.experimental import enable_iterative_imputer\n",
    "from sklearn.impute import IterativeImputer\n",
    "\n",
    "from hyperimpute.plugins.utils.metrics import RMSE\n",
    "from hyperimpute.plugins.utils.simulate import simulate_nan\n",
    "\n",
    "import xgboost as xgb\n",
    "\n",
    "from IPython.display import HTML, display\n",
    "import tabulate\n",
    "\n",
    "if not sys.warnoptions:\n",
    "    warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "devoted-console",
   "metadata": {},
   "source": [
    "### Loading the Imputation plugins\n",
    "\n",
    "Make sure that you have installed HyperImpute in your workspace.\n",
    "\n",
    "You can do that by running `pip install .` in the root of the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coated-innocent",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperimpute.plugins.imputers import Imputers, ImputerPlugin\n",
    "\n",
    "imputers = Imputers()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "refined-booth",
   "metadata": {},
   "source": [
    "### List the existing plugins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "incident-familiar",
   "metadata": {},
   "outputs": [],
   "source": [
    "imputers.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "czech-karma",
   "metadata": {},
   "source": [
    "### Adding a new Imputation plugin\n",
    "\n",
    "By default, HyperImpute automatically loads the imputation plugins with the pattern `hyperimpute/plugins/imputers/plugin_*`. \n",
    "\n",
    "Alternatively, you can call `Imputers().add(<name>, <ImputerPlugin derived class>)` at runtime.\n",
    "\n",
    "Next, we show two examples of custom Imputation plugins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sophisticated-ground",
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_ice_plugin = \"custom_ice\"\n",
    "\n",
    "\n",
    "class NewPlugin(ImputerPlugin):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        lr = LinearRegression()\n",
    "        self._model = IterativeImputer(\n",
    "            estimator=lr, max_iter=500, tol=1e-10, imputation_order=\"roman\"\n",
    "        )\n",
    "\n",
    "    @staticmethod\n",
    "    def name():\n",
    "        return custom_ice_plugin\n",
    "\n",
    "    @staticmethod\n",
    "    def hyperparameter_space():\n",
    "        return []\n",
    "\n",
    "    def _fit(self, *args, **kwargs) -> \"NewPlugin\":\n",
    "        self._model.fit(*args, **kwargs)\n",
    "        return self\n",
    "\n",
    "    def _transform(self, *args, **kwargs):\n",
    "        return self._model.transform(*args, **kwargs)\n",
    "\n",
    "    def save(self) -> bytes:\n",
    "        raise NotImplemented(\"placeholder\")\n",
    "\n",
    "    @classmethod\n",
    "    def load(cls, buff: bytes) -> \"NewPlugin\":\n",
    "        raise NotImplemented(\"placeholder\")\n",
    "\n",
    "\n",
    "imputers.add(custom_ice_plugin, NewPlugin)\n",
    "\n",
    "assert imputers.get(custom_ice_plugin) is not None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sufficient-broad",
   "metadata": {},
   "source": [
    "### List the existing plugins\n",
    "\n",
    "Now we should see the new plugins loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "loved-hepatitis",
   "metadata": {},
   "outputs": [],
   "source": [
    "imputers.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dominican-tulsa",
   "metadata": {},
   "source": [
    "### Testing the performance\n",
    "\n",
    "We simulate some testing datasets using 3 amputation strategies:\n",
    "- **Missing Completely At Random** (MCAR) if the probability of being missing is the same for all observations\n",
    "- **Missing At Random** (MAR) if the probability of being missing only depends on observed values.\n",
    "- **Missing Not At Random** (MNAR) if the unavailability of the data depends on both observed and unobserved data such as its value itself."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "advance-expert",
   "metadata": {},
   "source": [
    "#### Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "preceding-vermont",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import pandas as pd\n",
    "\n",
    "preproc = MinMaxScaler()\n",
    "\n",
    "\n",
    "def dataset():\n",
    "    X, y = load_breast_cancer(return_X_y=True)\n",
    "    X = pd.DataFrame(preproc.fit_transform(X, y))\n",
    "    y = pd.Series(y)\n",
    "\n",
    "    return train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "\n",
    "def ampute(x, mechanism, p_miss):\n",
    "    x_simulated = simulate_nan(np.asarray(x), p_miss, mechanism)\n",
    "\n",
    "    mask = x_simulated[\"mask\"]\n",
    "    x_miss = x_simulated[\"X_incomp\"]\n",
    "\n",
    "    return pd.DataFrame(x), pd.DataFrame(x_miss), pd.DataFrame(mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "chief-jimmy",
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets = {}\n",
    "headers = [\"Plugin\"]\n",
    "\n",
    "pct = 0.3\n",
    "\n",
    "mechanisms = [\"MAR\", \"MNAR\", \"MCAR\"]\n",
    "percentages = [pct]\n",
    "\n",
    "plugins = [\"mean\"]  # imputers.list()  # default plugins\n",
    "\n",
    "X_train, X_test, y_train, y_test = dataset()\n",
    "\n",
    "for ampute_mechanism in mechanisms:\n",
    "    for p_miss in percentages:\n",
    "        if ampute_mechanism not in datasets:\n",
    "            datasets[ampute_mechanism] = {}\n",
    "\n",
    "        headers.append(ampute_mechanism + \"-\" + str(p_miss))\n",
    "        datasets[ampute_mechanism][p_miss] = ampute(X_train, ampute_mechanism, p_miss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opened-finding",
   "metadata": {},
   "source": [
    "#### Evaluation\n",
    "\n",
    "We compare the methods in terms of root mean squared error (RMSE) to the initial dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "recovered-tonight",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []\n",
    "duration = []\n",
    "\n",
    "for plugin in tqdm(plugins):\n",
    "    plugin_results = [plugin]\n",
    "    plugin_duration = [plugin]\n",
    "\n",
    "    for ampute_mechanism in mechanisms:\n",
    "        for p_miss in percentages:\n",
    "            ctx = imputers.get(plugin)\n",
    "            x, x_miss, mask = datasets[ampute_mechanism][p_miss]\n",
    "\n",
    "            start = time.time() * 1000\n",
    "            x_imp = ctx.fit_transform(x_miss)\n",
    "\n",
    "            plugin_duration.append(round(time.time() * 1000 - start, 4))\n",
    "            plugin_results.append(RMSE(x_imp.values, x.values, mask.values))\n",
    "\n",
    "    results.append(plugin_results)\n",
    "    duration.append(plugin_duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "narrative-amount",
   "metadata": {},
   "source": [
    "### Reconstruction error(RMSE)\n",
    "\n",
    "__Interpretation__ : The following table shows the reconstruction error -  the __Root Mean Square Error(RMSE)__ for each method applied on the original full dataset and the imputed dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "normal-decline",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(HTML(tabulate.tabulate(results, headers=headers, tablefmt=\"html\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "overhead-favor",
   "metadata": {},
   "source": [
    "### XGBoost test score after imputation\n",
    "\n",
    "__Interpretation__ The following table shows different metrics on the test set for an XGBoost classifier, after imputing the dataset with each method.\n",
    "Metrics:\n",
    " - accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "nonprofit-content",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "\n",
    "def get_metrics(X_train, y_train, X_test, y_test):\n",
    "    xgb_clf = xgb.XGBClassifier(verbosity=0)\n",
    "    xgb_clf = xgb_clf.fit(X_train, y_train)\n",
    "\n",
    "    y_pred = xgb_clf.predict(X_test)\n",
    "\n",
    "    score = xgb_clf.score(X_test, y_test)\n",
    "\n",
    "    fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred)\n",
    "    auroc = metrics.auc(fpr, tpr)\n",
    "\n",
    "    prec, recall, thresholds = metrics.precision_recall_curve(y_test, y_pred)\n",
    "    aurpc = metrics.auc(recall, prec)\n",
    "\n",
    "    return score, auroc, aurpc\n",
    "\n",
    "\n",
    "metrics_headers = [\"Plugin\", \"Accuracy\", \"AUROC\", \"AURPC\"]\n",
    "xgboost_test_score = []\n",
    "\n",
    "\n",
    "x, x_miss, mask = datasets[\"MAR\"][pct]\n",
    "\n",
    "xgboost_test_score.append(\n",
    "    [\"original dataset\", *get_metrics(X_train, y_train, X_test, y_test)]\n",
    ")\n",
    "\n",
    "for plugin in plugins:\n",
    "    X_train_imp = imputers.get(plugin).fit_transform(x_miss.copy())\n",
    "\n",
    "    score, auroc, aurpc = get_metrics(X_train_imp, y_train, X_test, y_test)\n",
    "\n",
    "    xgboost_test_score.append([plugin, score, auroc, aurpc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77086d08",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    HTML(\n",
    "        tabulate.tabulate(xgboost_test_score, headers=metrics_headers, tablefmt=\"html\")\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "numeric-million",
   "metadata": {},
   "source": [
    "### Duration(ms) results\n",
    "\n",
    "__Info__ : Here we measure the duration of imputing the dataset with each method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "viral-suggestion",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(HTML(tabulate.tabulate(duration, headers=headers, tablefmt=\"html\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "earlier-description",
   "metadata": {},
   "source": [
    "## Debugging\n",
    "\n",
    "HyperImpute supports **debug** logging. __WARNING__: Don't use it for release builds. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "constitutional-toyota",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperimpute import logger\n",
    "\n",
    "imputers = Imputers()\n",
    "\n",
    "logger.add(sink=sys.stderr, level=\"DEBUG\")\n",
    "\n",
    "x, x_miss, mask = datasets[\"MAR\"][pct]\n",
    "\n",
    "x_imp = imputers.get(\"EM\").fit_transform(x)\n",
    "\n",
    "imputers.get(\"softimpute\").fit_transform(x_miss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "037f0841",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}