MTH_IDS_IoTJ.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles\n",
    "This is the code for the paper entitled \"[**MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles**](https://arxiv.org/pdf/2105.13289.pdf)\" accepted in IEEE Internet of Things Journal.  \n",
    "Authors: Li Yang (liyanghart@gmail.com), Abdallah Moubayed, and Abdallah Shami  \n",
    "Organization: The Optimized Computing and Communications (OC2) Lab, ECE Department, Western University\n",
    "\n",
    "L. Yang, A. Moubayed, and A. Shami, “MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles,” IEEE Internet of Things Journal, 2021."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score,precision_recall_fscore_support\n",
    "from sklearn.metrics import f1_score,roc_auc_score\n",
    "from sklearn.ensemble import RandomForestClassifier,ExtraTreesClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "import xgboost as xgb\n",
    "from xgboost import plot_importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the CICIDS2017 dataset\n",
    "The CICIDS2017 dataset is publicly available at: https://www.unb.ca/cic/datasets/ids-2017.html  \n",
    "Due to the large size of this dataset, the sampled subsets of CICIDS2017 is used. The subsets are in the \"data\" folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Read dataset\n",
    "df = pd.read_csv('./data/CICIDS2017.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Fwd Packets</th>\n",
       "      <th>Total Backward Packets</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Total Length of Bwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Min</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Fwd Packet Length Std</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>...</th>\n",
       "      <th>min_seg_size_forward</th>\n",
       "      <th>Active Mean</th>\n",
       "      <th>Active Std</th>\n",
       "      <th>Active Max</th>\n",
       "      <th>Active Min</th>\n",
       "      <th>Idle Mean</th>\n",
       "      <th>Idle Std</th>\n",
       "      <th>Idle Max</th>\n",
       "      <th>Idle Min</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>109</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830738</th>\n",
       "      <td>32215</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>112</td>\n",
       "      <td>152</td>\n",
       "      <td>28</td>\n",
       "      <td>28</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>76</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830739</th>\n",
       "      <td>324</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>84</td>\n",
       "      <td>362</td>\n",
       "      <td>42</td>\n",
       "      <td>42</td>\n",
       "      <td>42.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>181</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830740</th>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>6</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>15.5</td>\n",
       "      <td>21.92031</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>32</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830741</th>\n",
       "      <td>1048635</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>192</td>\n",
       "      <td>256</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>128</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830742</th>\n",
       "      <td>94939</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>188</td>\n",
       "      <td>226</td>\n",
       "      <td>47</td>\n",
       "      <td>47</td>\n",
       "      <td>47.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>113</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2830743 rows × 78 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Flow Duration  Total Fwd Packets  Total Backward Packets  \\\n",
       "0                    3                  2                       0   \n",
       "1                  109                  1                       1   \n",
       "2                   52                  1                       1   \n",
       "3                   34                  1                       1   \n",
       "4                    3                  2                       0   \n",
       "...                ...                ...                     ...   \n",
       "2830738          32215                  4                       2   \n",
       "2830739            324                  2                       2   \n",
       "2830740             82                  2                       1   \n",
       "2830741        1048635                  6                       2   \n",
       "2830742          94939                  4                       2   \n",
       "\n",
       "         Total Length of Fwd Packets  Total Length of Bwd Packets  \\\n",
       "0                                 12                            0   \n",
       "1                                  6                            6   \n",
       "2                                  6                            6   \n",
       "3                                  6                            6   \n",
       "4                                 12                            0   \n",
       "...                              ...                          ...   \n",
       "2830738                          112                          152   \n",
       "2830739                           84                          362   \n",
       "2830740                           31                            6   \n",
       "2830741                          192                          256   \n",
       "2830742                          188                          226   \n",
       "\n",
       "         Fwd Packet Length Max  Fwd Packet Length Min  Fwd Packet Length Mean  \\\n",
       "0                            6                      6                     6.0   \n",
       "1                            6                      6                     6.0   \n",
       "2                            6                      6                     6.0   \n",
       "3                            6                      6                     6.0   \n",
       "4                            6                      6                     6.0   \n",
       "...                        ...                    ...                     ...   \n",
       "2830738                     28                     28                    28.0   \n",
       "2830739                     42                     42                    42.0   \n",
       "2830740                     31                      0                    15.5   \n",
       "2830741                     32                     32                    32.0   \n",
       "2830742                     47                     47                    47.0   \n",
       "\n",
       "         Fwd Packet Length Std  Bwd Packet Length Max  ...  \\\n",
       "0                      0.00000                      0  ...   \n",
       "1                      0.00000                      6  ...   \n",
       "2                      0.00000                      6  ...   \n",
       "3                      0.00000                      6  ...   \n",
       "4                      0.00000                      0  ...   \n",
       "...                        ...                    ...  ...   \n",
       "2830738                0.00000                     76  ...   \n",
       "2830739                0.00000                    181  ...   \n",
       "2830740               21.92031                      6  ...   \n",
       "2830741                0.00000                    128  ...   \n",
       "2830742                0.00000                    113  ...   \n",
       "\n",
       "         min_seg_size_forward  Active Mean  Active Std  Active Max  \\\n",
       "0                          20          0.0         0.0           0   \n",
       "1                          20          0.0         0.0           0   \n",
       "2                          20          0.0         0.0           0   \n",
       "3                          20          0.0         0.0           0   \n",
       "4                          20          0.0         0.0           0   \n",
       "...                       ...          ...         ...         ...   \n",
       "2830738                    20          0.0         0.0           0   \n",
       "2830739                    20          0.0         0.0           0   \n",
       "2830740                    32          0.0         0.0           0   \n",
       "2830741                    20          0.0         0.0           0   \n",
       "2830742                    20          0.0         0.0           0   \n",
       "\n",
       "         Active Min  Idle Mean  Idle Std  Idle Max  Idle Min   Label  \n",
       "0                 0        0.0       0.0         0         0  BENIGN  \n",
       "1                 0        0.0       0.0         0         0  BENIGN  \n",
       "2                 0        0.0       0.0         0         0  BENIGN  \n",
       "3                 0        0.0       0.0         0         0  BENIGN  \n",
       "4                 0        0.0       0.0         0         0  BENIGN  \n",
       "...             ...        ...       ...       ...       ...     ...  \n",
       "2830738           0        0.0       0.0         0         0  BENIGN  \n",
       "2830739           0        0.0       0.0         0         0  BENIGN  \n",
       "2830740           0        0.0       0.0         0         0  BENIGN  \n",
       "2830741           0        0.0       0.0         0         0  BENIGN  \n",
       "2830742           0        0.0       0.0         0         0  BENIGN  \n",
       "\n",
       "[2830743 rows x 78 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BENIGN          2273097\n",
       "DoS              380699\n",
       "PortScan         158930\n",
       "BruteForce        13835\n",
       "WebAttack          2180\n",
       "Bot                1966\n",
       "Infiltration         36\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing (normalization and padding values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\41364\\AppData\\Roaming\\Python\\Python35\\site-packages\\pandas\\compat\\_optional.py:106: UserWarning: Pandas requires version '2.6.2' or newer of 'numexpr' (version '2.6.1' currently installed).\n",
      "  warnings.warn(msg, UserWarning)\n"
     ]
    }
   ],
   "source": [
    "# Z-score normalization\n",
    "features = df.dtypes[df.dtypes != 'object'].index\n",
    "df[features] = df[features].apply(\n",
    "    lambda x: (x - x.mean()) / (x.std()))\n",
    "# Fill empty values by 0\n",
    "df = df.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data sampling\n",
    "Due to the space limit of GitHub files and the large size of network traffic data, we sample a small-sized subset for model learning using **k-means cluster sampling**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "labelencoder = LabelEncoder()\n",
    "df.iloc[:, -1] = labelencoder.fit_transform(df.iloc[:, -1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2273097\n",
       "3     380699\n",
       "5     158930\n",
       "2      13835\n",
       "6       2180\n",
       "1       1966\n",
       "4         36\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# retain the minority class instances and sample the majority class instances\n",
    "df_minor = df[(df['Label']==6)|(df['Label']==1)|(df['Label']==4)]\n",
    "df_major = df.drop(df_minor.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = df_major.drop(['Label'],axis=1) \n",
    "y = df_major.iloc[:, -1].values.reshape(-1,1)\n",
    "y=np.ravel(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# use k-means to cluster the data samples and select a proportion of data from each cluster\n",
    "from sklearn.cluster import MiniBatchKMeans\n",
    "kmeans = MiniBatchKMeans(n_clusters=1000, random_state=0).fit(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "klabel=kmeans.labels_\n",
    "df_major['klabel']=klabel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "318    22146\n",
       "2      20340\n",
       "258    20225\n",
       "308    18461\n",
       "432    18154\n",
       "       ...  \n",
       "366       70\n",
       "92        21\n",
       "596       14\n",
       "756       10\n",
       "295        3\n",
       "Name: klabel, Length: 997, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_major['klabel'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cols = list(df_major)\n",
    "cols.insert(78, cols.pop(cols.index('Label')))\n",
    "df_major = df_major.loc[:, cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Fwd Packets</th>\n",
       "      <th>Total Backward Packets</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Total Length of Bwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Min</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Fwd Packet Length Std</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>...</th>\n",
       "      <th>Active Mean</th>\n",
       "      <th>Active Std</th>\n",
       "      <th>Active Max</th>\n",
       "      <th>Active Min</th>\n",
       "      <th>Idle Mean</th>\n",
       "      <th>Idle Std</th>\n",
       "      <th>Idle Max</th>\n",
       "      <th>Idle Min</th>\n",
       "      <th>klabel</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.439347</td>\n",
       "      <td>-0.009819</td>\n",
       "      <td>-0.010421</td>\n",
       "      <td>-0.053765</td>\n",
       "      <td>-0.007142</td>\n",
       "      <td>-0.281099</td>\n",
       "      <td>-0.210703</td>\n",
       "      <td>-0.280518</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.447423</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>391</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.439344</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.054365</td>\n",
       "      <td>-0.007139</td>\n",
       "      <td>-0.281099</td>\n",
       "      <td>-0.210703</td>\n",
       "      <td>-0.280518</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.444340</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>498</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.439345</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.054365</td>\n",
       "      <td>-0.007139</td>\n",
       "      <td>-0.281099</td>\n",
       "      <td>-0.210703</td>\n",
       "      <td>-0.280518</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.444340</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>499</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.439346</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.054365</td>\n",
       "      <td>-0.007139</td>\n",
       "      <td>-0.281099</td>\n",
       "      <td>-0.210703</td>\n",
       "      <td>-0.280518</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.444340</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>787</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.439347</td>\n",
       "      <td>-0.009819</td>\n",
       "      <td>-0.010421</td>\n",
       "      <td>-0.053765</td>\n",
       "      <td>-0.007142</td>\n",
       "      <td>-0.281099</td>\n",
       "      <td>-0.210703</td>\n",
       "      <td>-0.280518</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.447423</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>391</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830738</th>\n",
       "      <td>-0.438390</td>\n",
       "      <td>-0.007151</td>\n",
       "      <td>-0.008416</td>\n",
       "      <td>-0.043758</td>\n",
       "      <td>-0.007075</td>\n",
       "      <td>-0.250424</td>\n",
       "      <td>0.153902</td>\n",
       "      <td>-0.162296</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.408376</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>813</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830739</th>\n",
       "      <td>-0.439337</td>\n",
       "      <td>-0.009819</td>\n",
       "      <td>-0.008416</td>\n",
       "      <td>-0.046560</td>\n",
       "      <td>-0.006982</td>\n",
       "      <td>-0.230903</td>\n",
       "      <td>0.385923</td>\n",
       "      <td>-0.087065</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.354429</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>916</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830740</th>\n",
       "      <td>-0.439344</td>\n",
       "      <td>-0.009819</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.051863</td>\n",
       "      <td>-0.007139</td>\n",
       "      <td>-0.246240</td>\n",
       "      <td>-0.310140</td>\n",
       "      <td>-0.229468</td>\n",
       "      <td>-0.167112</td>\n",
       "      <td>-0.444340</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>267</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830741</th>\n",
       "      <td>-0.408187</td>\n",
       "      <td>-0.004484</td>\n",
       "      <td>-0.008416</td>\n",
       "      <td>-0.035753</td>\n",
       "      <td>-0.007029</td>\n",
       "      <td>-0.244846</td>\n",
       "      <td>0.220194</td>\n",
       "      <td>-0.140802</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.381659</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>634</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830742</th>\n",
       "      <td>-0.436526</td>\n",
       "      <td>-0.007151</td>\n",
       "      <td>-0.008416</td>\n",
       "      <td>-0.036153</td>\n",
       "      <td>-0.007042</td>\n",
       "      <td>-0.223931</td>\n",
       "      <td>0.468788</td>\n",
       "      <td>-0.060196</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.389366</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>978</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2826561 rows × 79 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Flow Duration  Total Fwd Packets  Total Backward Packets  \\\n",
       "0            -0.439347          -0.009819               -0.010421   \n",
       "1            -0.439344          -0.011153               -0.009418   \n",
       "2            -0.439345          -0.011153               -0.009418   \n",
       "3            -0.439346          -0.011153               -0.009418   \n",
       "4            -0.439347          -0.009819               -0.010421   \n",
       "...                ...                ...                     ...   \n",
       "2830738      -0.438390          -0.007151               -0.008416   \n",
       "2830739      -0.439337          -0.009819               -0.008416   \n",
       "2830740      -0.439344          -0.009819               -0.009418   \n",
       "2830741      -0.408187          -0.004484               -0.008416   \n",
       "2830742      -0.436526          -0.007151               -0.008416   \n",
       "\n",
       "         Total Length of Fwd Packets  Total Length of Bwd Packets  \\\n",
       "0                          -0.053765                    -0.007142   \n",
       "1                          -0.054365                    -0.007139   \n",
       "2                          -0.054365                    -0.007139   \n",
       "3                          -0.054365                    -0.007139   \n",
       "4                          -0.053765                    -0.007142   \n",
       "...                              ...                          ...   \n",
       "2830738                    -0.043758                    -0.007075   \n",
       "2830739                    -0.046560                    -0.006982   \n",
       "2830740                    -0.051863                    -0.007139   \n",
       "2830741                    -0.035753                    -0.007029   \n",
       "2830742                    -0.036153                    -0.007042   \n",
       "\n",
       "         Fwd Packet Length Max  Fwd Packet Length Min  Fwd Packet Length Mean  \\\n",
       "0                    -0.281099              -0.210703               -0.280518   \n",
       "1                    -0.281099              -0.210703               -0.280518   \n",
       "2                    -0.281099              -0.210703               -0.280518   \n",
       "3                    -0.281099              -0.210703               -0.280518   \n",
       "4                    -0.281099              -0.210703               -0.280518   \n",
       "...                        ...                    ...                     ...   \n",
       "2830738              -0.250424               0.153902               -0.162296   \n",
       "2830739              -0.230903               0.385923               -0.087065   \n",
       "2830740              -0.246240              -0.310140               -0.229468   \n",
       "2830741              -0.244846               0.220194               -0.140802   \n",
       "2830742              -0.223931               0.468788               -0.060196   \n",
       "\n",
       "         Fwd Packet Length Std  Bwd Packet Length Max  ...  Active Mean  \\\n",
       "0                    -0.245069              -0.447423  ...    -0.125734   \n",
       "1                    -0.245069              -0.444340  ...    -0.125734   \n",
       "2                    -0.245069              -0.444340  ...    -0.125734   \n",
       "3                    -0.245069              -0.444340  ...    -0.125734   \n",
       "4                    -0.245069              -0.447423  ...    -0.125734   \n",
       "...                        ...                    ...  ...          ...   \n",
       "2830738              -0.245069              -0.408376  ...    -0.125734   \n",
       "2830739              -0.245069              -0.354429  ...    -0.125734   \n",
       "2830740              -0.167112              -0.444340  ...    -0.125734   \n",
       "2830741              -0.245069              -0.381659  ...    -0.125734   \n",
       "2830742              -0.245069              -0.389366  ...    -0.125734   \n",
       "\n",
       "         Active Std  Active Max  Active Min  Idle Mean  Idle Std  Idle Max  \\\n",
       "0         -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "1         -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "2         -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "3         -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "4         -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "...             ...         ...         ...        ...       ...       ...   \n",
       "2830738   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "2830739   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "2830740   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "2830741   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "2830742   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "\n",
       "         Idle Min  klabel  Label  \n",
       "0       -0.338993     391      0  \n",
       "1       -0.338993     498      0  \n",
       "2       -0.338993     499      0  \n",
       "3       -0.338993     787      0  \n",
       "4       -0.338993     391      0  \n",
       "...           ...     ...    ...  \n",
       "2830738 -0.338993     813      0  \n",
       "2830739 -0.338993     916      0  \n",
       "2830740 -0.338993     267      0  \n",
       "2830741 -0.338993     634      0  \n",
       "2830742 -0.338993     978      0  \n",
       "\n",
       "[2826561 rows x 79 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_major"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def typicalSampling(group):\n",
    "    name = group.name\n",
    "    frac = 0.008\n",
    "    return group.sample(frac=frac)\n",
    "\n",
    "result = df_major.groupby(\n",
    "    'klabel', group_keys=False\n",
    ").apply(typicalSampling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18185\n",
       "3     3029\n",
       "5     1280\n",
       "2      118\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result['Label'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Fwd Packets</th>\n",
       "      <th>Total Backward Packets</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Total Length of Bwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Min</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Fwd Packet Length Std</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>...</th>\n",
       "      <th>Active Mean</th>\n",
       "      <th>Active Std</th>\n",
       "      <th>Active Max</th>\n",
       "      <th>Active Min</th>\n",
       "      <th>Idle Mean</th>\n",
       "      <th>Idle Std</th>\n",
       "      <th>Idle Max</th>\n",
       "      <th>Idle Min</th>\n",
       "      <th>klabel</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6980</th>\n",
       "      <td>-0.437857</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.054965</td>\n",
       "      <td>-0.007142</td>\n",
       "      <td>-0.289465</td>\n",
       "      <td>-0.310140</td>\n",
       "      <td>-0.312760</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.447423</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1506627</th>\n",
       "      <td>-0.438252</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.054965</td>\n",
       "      <td>-0.007142</td>\n",
       "      <td>-0.289465</td>\n",
       "      <td>-0.310140</td>\n",
       "      <td>-0.312760</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.447423</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1377524</th>\n",
       "      <td>-0.438860</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.054965</td>\n",
       "      <td>-0.007142</td>\n",
       "      <td>-0.289465</td>\n",
       "      <td>-0.310140</td>\n",
       "      <td>-0.312760</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.447423</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2056871</th>\n",
       "      <td>-0.435684</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.054965</td>\n",
       "      <td>-0.007142</td>\n",
       "      <td>-0.289465</td>\n",
       "      <td>-0.310140</td>\n",
       "      <td>-0.312760</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.447423</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005567</th>\n",
       "      <td>-0.437738</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.054965</td>\n",
       "      <td>-0.007142</td>\n",
       "      <td>-0.289465</td>\n",
       "      <td>-0.310140</td>\n",
       "      <td>-0.312760</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.447423</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1031173</th>\n",
       "      <td>-0.438439</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.050963</td>\n",
       "      <td>-0.007110</td>\n",
       "      <td>-0.233691</td>\n",
       "      <td>0.352777</td>\n",
       "      <td>-0.097812</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.410431</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1608048</th>\n",
       "      <td>-0.438422</td>\n",
       "      <td>-0.009819</td>\n",
       "      <td>-0.008416</td>\n",
       "      <td>-0.047160</td>\n",
       "      <td>-0.007083</td>\n",
       "      <td>-0.235086</td>\n",
       "      <td>0.336204</td>\n",
       "      <td>-0.103186</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.413000</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>817023</th>\n",
       "      <td>-0.437935</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.050863</td>\n",
       "      <td>-0.007108</td>\n",
       "      <td>-0.232297</td>\n",
       "      <td>0.369350</td>\n",
       "      <td>-0.092438</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.408376</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>559006</th>\n",
       "      <td>-0.437946</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.050763</td>\n",
       "      <td>-0.007111</td>\n",
       "      <td>-0.230903</td>\n",
       "      <td>0.385923</td>\n",
       "      <td>-0.087065</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.411459</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>985052</th>\n",
       "      <td>-0.437554</td>\n",
       "      <td>-0.011153</td>\n",
       "      <td>-0.009418</td>\n",
       "      <td>-0.050963</td>\n",
       "      <td>-0.007110</td>\n",
       "      <td>-0.233691</td>\n",
       "      <td>0.352777</td>\n",
       "      <td>-0.097812</td>\n",
       "      <td>-0.245069</td>\n",
       "      <td>-0.410431</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.125734</td>\n",
       "      <td>-0.104565</td>\n",
       "      <td>-0.149326</td>\n",
       "      <td>-0.101016</td>\n",
       "      <td>-0.351926</td>\n",
       "      <td>-0.10946</td>\n",
       "      <td>-0.356868</td>\n",
       "      <td>-0.338993</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>22612 rows × 79 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Flow Duration  Total Fwd Packets  Total Backward Packets  \\\n",
       "6980         -0.437857          -0.011153               -0.009418   \n",
       "1506627      -0.438252          -0.011153               -0.009418   \n",
       "1377524      -0.438860          -0.011153               -0.009418   \n",
       "2056871      -0.435684          -0.011153               -0.009418   \n",
       "2005567      -0.437738          -0.011153               -0.009418   \n",
       "...                ...                ...                     ...   \n",
       "1031173      -0.438439          -0.011153               -0.009418   \n",
       "1608048      -0.438422          -0.009819               -0.008416   \n",
       "817023       -0.437935          -0.011153               -0.009418   \n",
       "559006       -0.437946          -0.011153               -0.009418   \n",
       "985052       -0.437554          -0.011153               -0.009418   \n",
       "\n",
       "         Total Length of Fwd Packets  Total Length of Bwd Packets  \\\n",
       "6980                       -0.054965                    -0.007142   \n",
       "1506627                    -0.054965                    -0.007142   \n",
       "1377524                    -0.054965                    -0.007142   \n",
       "2056871                    -0.054965                    -0.007142   \n",
       "2005567                    -0.054965                    -0.007142   \n",
       "...                              ...                          ...   \n",
       "1031173                    -0.050963                    -0.007110   \n",
       "1608048                    -0.047160                    -0.007083   \n",
       "817023                     -0.050863                    -0.007108   \n",
       "559006                     -0.050763                    -0.007111   \n",
       "985052                     -0.050963                    -0.007110   \n",
       "\n",
       "         Fwd Packet Length Max  Fwd Packet Length Min  Fwd Packet Length Mean  \\\n",
       "6980                 -0.289465              -0.310140               -0.312760   \n",
       "1506627              -0.289465              -0.310140               -0.312760   \n",
       "1377524              -0.289465              -0.310140               -0.312760   \n",
       "2056871              -0.289465              -0.310140               -0.312760   \n",
       "2005567              -0.289465              -0.310140               -0.312760   \n",
       "...                        ...                    ...                     ...   \n",
       "1031173              -0.233691               0.352777               -0.097812   \n",
       "1608048              -0.235086               0.336204               -0.103186   \n",
       "817023               -0.232297               0.369350               -0.092438   \n",
       "559006               -0.230903               0.385923               -0.087065   \n",
       "985052               -0.233691               0.352777               -0.097812   \n",
       "\n",
       "         Fwd Packet Length Std  Bwd Packet Length Max  ...  Active Mean  \\\n",
       "6980                 -0.245069              -0.447423  ...    -0.125734   \n",
       "1506627              -0.245069              -0.447423  ...    -0.125734   \n",
       "1377524              -0.245069              -0.447423  ...    -0.125734   \n",
       "2056871              -0.245069              -0.447423  ...    -0.125734   \n",
       "2005567              -0.245069              -0.447423  ...    -0.125734   \n",
       "...                        ...                    ...  ...          ...   \n",
       "1031173              -0.245069              -0.410431  ...    -0.125734   \n",
       "1608048              -0.245069              -0.413000  ...    -0.125734   \n",
       "817023               -0.245069              -0.408376  ...    -0.125734   \n",
       "559006               -0.245069              -0.411459  ...    -0.125734   \n",
       "985052               -0.245069              -0.410431  ...    -0.125734   \n",
       "\n",
       "         Active Std  Active Max  Active Min  Idle Mean  Idle Std  Idle Max  \\\n",
       "6980      -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "1506627   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "1377524   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "2056871   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "2005567   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "...             ...         ...         ...        ...       ...       ...   \n",
       "1031173   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "1608048   -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "817023    -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "559006    -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "985052    -0.104565   -0.149326   -0.101016  -0.351926  -0.10946 -0.356868   \n",
       "\n",
       "         Idle Min  klabel  Label  \n",
       "6980    -0.338993       0      0  \n",
       "1506627 -0.338993       0      0  \n",
       "1377524 -0.338993       0      0  \n",
       "2056871 -0.338993       0      0  \n",
       "2005567 -0.338993       0      0  \n",
       "...           ...     ...    ...  \n",
       "1031173 -0.338993     999      0  \n",
       "1608048 -0.338993     999      0  \n",
       "817023  -0.338993     999      0  \n",
       "559006  -0.338993     999      0  \n",
       "985052  -0.338993     999      0  \n",
       "\n",
       "[22612 rows x 79 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "result = result.drop(['klabel'],axis=1)\n",
    "result = result.append(df_minor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "result.to_csv('./data/CICIDS2017_sample_km.csv',index=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### split train set and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df=pd.read_csv('./data/CICIDS2017_sample_km.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = df.drop(['Label'],axis=1).values\n",
    "y = df.iloc[:, -1].values.reshape(-1,1)\n",
    "y=np.ravel(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, random_state = 0,stratify = y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature selection by information gain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import mutual_info_classif\n",
    "importances = mutual_info_classif(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# calculate the sum of importance scores\n",
    "f_list = sorted(zip(map(lambda x: round(x, 4), importances), features), reverse=True)\n",
    "Sum = 0\n",
    "fs = []\n",
    "for i in range(0, len(f_list)):\n",
    "    Sum = Sum + f_list[i][0]\n",
    "    fs.append(f_list[i][1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# select the important features from top to bottom until the accumulated importance reaches 90%\n",
    "f_list2 = sorted(zip(map(lambda x: round(x, 4), importances/Sum), features), reverse=True)\n",
    "Sum2 = 0\n",
    "fs = []\n",
    "for i in range(0, len(f_list2)):\n",
    "    Sum2 = Sum2 + f_list2[i][0]\n",
    "    fs.append(f_list2[i][1])\n",
    "    if Sum2>=0.9:\n",
    "        break        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_fs = df[fs].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26794, 44)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_fs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature selection by Fast Correlation Based Filter (FCBF)\n",
    "\n",
    "The module is imported from the GitHub repo: https://github.com/SantiagoEG/FCBF_module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from FCBF_module import FCBF, FCBFK, FCBFiP, get_i\n",
    "fcbf = FCBFK(k = 20)\n",
    "#fcbf.fit(X_fs, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_fss = fcbf.fit_transform(X_fs,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26794, 20)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_fss.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Re-split train & test sets after feature selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_fss,y, train_size = 0.8, test_size = 0.2, random_state = 0,stratify = y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(21435, 20)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    14548\n",
       "3     2423\n",
       "6     1744\n",
       "1     1573\n",
       "5     1024\n",
       "2       94\n",
       "4       29\n",
       "dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SMOTE to solve class-imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\41364\\AppData\\Roaming\\Python\\Python35\\site-packages\\sklearn\\externals\\six.py:31: DeprecationWarning: The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/).\n",
      "  \"(https://pypi.org/project/six/).\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from imblearn.over_sampling import SMOTE\n",
    "smote=SMOTE(n_jobs=-1,sampling_strategy={2:1000,4:1000})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, y_train = smote.fit_sample(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    14548\n",
       "3     2423\n",
       "6     1744\n",
       "1     1573\n",
       "5     1024\n",
       "4     1000\n",
       "2     1000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine learning model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training four base learners: decision tree, random forest, extra trees, XGBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of XGBoost: 0.9917910447761195\n",
      "Precision of XGBoost: 0.991821481887011\n",
      "Recall of XGBoost: 0.9917910447761195\n",
      "F1-score of XGBoost: 0.9917663641435618\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      0.99      0.99      3645\n",
      "           1       0.99      0.98      0.99       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       0.99      0.99      0.99       609\n",
      "           4       1.00      0.71      0.83         7\n",
      "           5       0.99      1.00      0.99       251\n",
      "           6       0.97      0.99      0.98       436\n",
      "\n",
      "    accuracy                           0.99      5360\n",
      "   macro avg       0.99      0.95      0.97      5360\n",
      "weighted avg       0.99      0.99      0.99      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPX1//HXmSQgm6BCWZJUUFBx\nBWV1QUA2EYS6ICqKFkUtfIXa4r4UFbWtWqX1Z4uCghugguybICKtQBAQDfsmBMIiIAq0kuX8/pib\nMGAmyQz33plMztPHfTD5zL3zvncih8/dPldUFWOMMb8UiPUKGGNMvLICaYwxYViBNMaYMKxAGmNM\nGFYgjTEmDCuQxhgThhVIY4wJwwqkMcaEYQXSGGPCSI71ChRLxG7zMSYWVCWaxXK+3xTx39mUmmdE\nleWHuC6QOXs2xiw7pdaZJKfUi1l+bs6Ocpufm7MDwPJjnG/ivEAaY8qY/LxYr4GrrEAaY9yj+bFe\nA1dZgTTGuCffCqQxxhRJE6wHaZf5GGPck58f+VQCETlJRJaIyNcikikiQ532t0Vks4iscKYmTruI\nyHAR2SAiK0Xk4pDP6isi652pb0nZ1oM0xrjHmx7kz0B7VT0oIinAQhGZ4bw3RFU/Om7+q4FGztQS\neB1oKSKnAk8BzQAFvhKRyaq6P1yw9SCNMe7Jz4t8KoEGHXR+THGm4q637AGMcZZbBNQQkbpAZ2CO\nqu5ziuIcoEtx2VYgjTHu0fyIJxHpLyJLQ6b+x3+siCSJyApgN8Eit9h5a5izG/03EanotKUC20IW\nz3LawrWHZbvYxhj3RHEWW1VHACNKmCcPaCIiNYCJInI+8AiwE6jgLP8Q8DRQ1J05Wkx7WNaDNMa4\nRjU/4imyz9cfgPlAF1XNdnajfwbeAlo4s2UB6SGLpQE7imkPywqkMcY93pzFruX0HBGRSkAHYI1z\nXBEREaAn8K2zyGTgdudsdivggKpmA7OATiJyioicAnRy2sKyXWxjjHu8OYtdFxgtIkkEO3XjVXWq\niMwTkVoEd51XAPc6808HugIbgMPAnQCquk9EngEynPmeVtV9xQVbgTTGuMeDe7FVdSXQtIj29mHm\nV2BAmPdGAaNKm12mCuTPPx+h74AhHMnJIS83j47tLmfgXbehqgwfMZrZny0kEAhw02+uoc+NPZg6\nax4j3/sQgMqVKvHEHwdyTqMzyN61h0efeZHv9+0nIMINPa7mtl49XVnHihUrMn/ex1SoWJHk5CQm\nTJjG0KdfcuWzS6tzp7a8/PLTJAUCjHrrA/7y19d8zd+wbhE/HTxIXl4+ubm5tGrd1df8WG5/LH7/\nb4x4iWu6dmD3nu9p0vQqAP78/ONc060jR44cYdOm7+h31wMcOPCjp+sBJNy92BIstvHp+LHlVJX/\n/vd/VK5ciZzcXG6/7488POgeNn23jSXLVjLssQcIBALs3f8Dp51Sg+XfrOKM09OpfnI1vvgyg/83\n6j0+eOMV9ny/jz1793Hu2Q05dOgwvfrdz/Dnn+DMBqcXZp3IcGdVqlTm0KHDJCcns2D+RH7/wFMs\nXrIsos+IdrixQCDA6swv6NL1ZrKysln05XT63PY7Vq9e70s+BAtky9ZXs3dv2OtvS8yG6Ib7cmP7\nT3S4sRP9/Ueaf8XlLTl48BBvvfVqYYHs2KEN8z77N3l5eTz/3KMAPPLoc6XPj3I8yJ9XfxZxQanY\nuF3cjgfp2UkaETlHRB5ybvl51Xnd+AQ/k8qVKwGQm5tLbm4uIsK4idO4785bCASCm3PaKTUAaHrB\nuVQ/uRoAF553Drt2fw9ArZqncu7ZDYHg/8xnnJ7Orj17T2TVjnHo0GEAUlKSSU5Jwc9/hFo0b8rG\njVvYvHkrOTk5jB8/iWu7d/YtP9biYfv9/v1/sXAx+/b/cEzbnE8XkJcX3N1dtHgZqal1PV2HQh6c\npIklTwqkiDwEjCV48HQJwYOiAnwgIg+fyGfn5eVxfd8BtOl2M62bN+XC885h2/ZsZsz9nF6/vZ97\n//AE323b/ovlJkydxeWtmv2ifXv2Llav38iF5519Iqt1jEAgwNKM2WRvX8ncuQtYkrHctc8uSb3U\nOmzLOnrlQtb2bOrVq+NbPgR7+jOmf8DiRTO4q9+tvmbHw/bH8vdflDvv6M3MWZ/5ExbFheLxzKse\nZD+guaq+oKrvOtMLBK9T6nciH5yUlMTHo19j7sR3+GbVOtZv2sKRnBwqVqjA+FHDub57F5547m/H\nLLPkq6+ZMHU2D/zut8e0Hz78X37/2LM8dP89VK1S5URW6xj5+fk0a96J0xs0o3mzppznYvEtSfCK\nh2P5fRilTduetGjZhW7d+3DffXdwxeUtfcuOh+2P5e//eI88fD+5ubm8//4EfwKtB1kq+UBRB1Dq\nOu+FFXrb0ZtjPgg738nVqtL84gtZuGgpdWrVpGPbywHocOWlrNu4uXC+tRs28+QLr/D3F56kRvWT\nC9tzcnMZ/NizXNOpHR3bXhbRxpXWgQM/8vmC/9C5U1tPPr8o27OySU87+tWnpdYlO3uXb/lAYd6e\nPXuZNGkGzZs38S07Hra/QCx+/6Fuu+1GrunagdtuH+hbpmpexFM886pADgbmisgMERnhTDOBucCg\n4hZU1RGq2kxVm911+83HvLdv/w/8+FPwnvX//fwzizKW0+D0dNq3ac3ir1YAkLH8G05PD95emb1z\nN4MffYbnnxxC/V+nhWbw5POvcMbp6fTtfZ17Ww3UrHkq1Z1CfNJJJ3FV+ytYu9a/Z+tkLF1Bw4YN\nqF8/nZSUFHr16sGUqbN9y69cuRJVq1YpfN2xw5VkZq71LT/W2x/r33+Bzp3aMuSPv6PndXfw3//+\nz7/gBNvF9uQyH1WdKSJnEdylTiV4/DELyNAT+Cdjz979PPbsi+Tl56P5Suf2V9D2spZcfOF5PDT0\nL7wz7hMqVzqJoQ8PBuD1t97nwI8/8eyLwcs8kpKSGD9qOMtXZjJl5lwanVmf6/sGL5cadE9f2lza\nImx2adWtW5tRI18hKSlAIBDgo4+mMG36pyf8uaWVl5fHoMGPM33a+yQFArw9ehyrVq3zLb927Vp8\n9OFIAJKTkxg79hNmzZ7vW36stz8Wv/9333mNK9u0pmbNU9myaSlDn36Rhx4cSMWKFZk5YywAixcv\nY8DAEzr8XzpxvsscqTJ1mY+f7KmG9lTDcp0f5WU+//vqk4j/zp50Sc+4vcynTF0oboyJc/ZUQ2OM\nCSPOjylGygqkMcY9CXYM0gqkMcY91oM0xpgwrAdpjDFhWIE0xpiixfudMZGyAmmMcY/1II0xJowE\nO0ljD+0yxpgwrAdpjHGP7WIbY0wYCbaLbQXSGOMe60EaY0wY1oP0T0qtM2OaXzDslOVbfnnMj4r1\nII0xJgwrkP6J9YCxjWs1j1n+6j0ZMd/+cj1grOVHx3axjTEmDOtBGmNMGAnWg7Q7aYwx7vHgudgi\ncpKILBGRr0UkU0SGOu0NRGSxiKwXkXEiUsFpr+j8vMF5v37IZz3itK8Vkc4lZVuBNMa4x5vHvv4M\ntFfVi4AmQBcRaQX8GfibqjYC9gP9nPn7AftVtSHwN2c+RORcoDdwHtAF+H8iklRcsBVIY4x7POhB\natBB58cUZ1KgPfCR0z4a6Om87uH8jPP+VSIiTvtYVf1ZVTcDGwg+mjosK5DGGPd4UCABRCRJRFYA\nu4E5wEbgB1XNdWbJAlKd16nANgDn/QPAaaHtRSxTJCuQxhj3qEY8iUh/EVkaMvX/5cdqnqo2AdII\n9voaF5Xu/FnUc7a1mPaw7Cy2McY9UVzmo6ojgBGlnPcHEZkPtAJqiEiy00tMAwou4MwC0oEsEUkG\nqgP7QtoLhC5TJOtBGmPc481Z7FoiUsN5XQnoAKwGPgNucGbrC0xyXk92fsZ5f56qqtPe2znL3QBo\nBCwpLtt6kMYY93hzHWRdYLRzxjkAjFfVqSKyChgrIs8Cy4GRzvwjgXdEZAPBnmNvAFXNFJHxwCog\nFxigJTxExwqkMSauqepKoGkR7Zso4iy0qv4PuDHMZw0DhpU22wqkMcY9dquhMcaEocWeFC5zEvIk\nzaD77+brFfNYsXwu777zGhUrVnQ9o0LFCoyb+RYTP3uPKQvGMvDBuwFodUVzPv50DBPmvcu7U0bw\n6wZpAPS86Rr+vWoWE+a9y4R573LDrT1cX6cCnTu1JfPbBaxZtZAHhwzwLMfy4ys7HvK9ug4yVhKu\nQNarV4eBA35Ly1ZdadL0KpKSkripl/vF6MjPR7jz+t/xm3a38pv2t3J5u9ZcdMn5PPWXhxhy35Nc\n174P0ybM4t7f/7ZwmRmT5nBd+z5c174PH703qZhPj14gEGD4q8Po1r0PF1zUjptu6knjxo08ybL8\n+MmOh3zACmRZkJycTKVKJ5GUlETlSpXIzt7pSc7hQ/8N5qUkk5KSjKqiqlStVgWAqtWqsnvnHk+y\nw2nRvCkbN25h8+at5OTkMH78JK7tXuI9+ZZfxrPjIR/w6l7smIlJgRSRO7367B07dvLy3/7J5o1L\nyNq6nAM//sicTxd4khUIBJgw710WrprFfz5fwsplmTzx+2H86/1X+GzFFK698WreGD6mcP5O3drz\nyfz3eGXk89Sp9ytP1qleah22ZR299jVrezb16tXxJMvy4yc7HvIBNF8jnuJZrHqQQ8O9EXrbUX7+\noYg/uEaN6lzbvTMNz2pF+ukXU6VKZW655boTWtlw8vPzua59H9pd1I0Lmp5Lo3POoO+9N3PPLYNp\n16Q7E8dO5eGnBwMwf/ZCrrqkBz3b3sqXCzJ4/u9/8mSdgvfkH0t9PHBenvPL87YXsl3s0hGRlWGm\nb4Da4ZZT1RGq2kxVmwUCVSLOveqqK9i8ZSvff7+P3NxcJn4yg9atmp3IppTopx8PsuQ/y7jiqks5\n+7xGrFyWCcCMT+bQpPkFAPyw/wA5R3IA+PCdTzjvonM8WZftWdmkpx0dqj8ttS7Z2bs8ybL8+MmO\nh3zAdrEjUBu4HehexLTXq9BtW7fTsuXFVKp0EgDt213OmjXrXc855bQaVDu5KgAVT6pI6zYt2LRu\nC9WqVaX+Gb8G4NIrW7Jp/RYAav3qtMJl23dpw6Z1m11fJ4CMpSto2LAB9eunk5KSQq9ePZgydbYn\nWZYfP9nxkA9AvkY+xTEvr4OcClRV1RXHv+HcbO6JJRnLmTBhGhlLZpGbm8uKFZm88eZ7rufUql2T\n5//+FElJAQISYObkT5k/ZyFP/uE5Xh31Avmq/PjDjzw2+BkA+tx9E+07tyE3L48D+w/wyP1Pu75O\nAHl5eQwa/DjTp71PUiDA26PHsWrVOk+yLD9+suMhH4j7XeZIie/HKCKQXCE1ZitnTzW0pxqW63zV\nooYGK9HhV++N+O9s5UH/jCrLD3YnjTHGPXHc4YqGFUhjjHsSbBfbCqQxxj1xftIlUlYgjTHuifPL\ndiJlBdIY454E60Em5L3YxhjjButBGmNco3aSxhhjwkiwXWwrkMYY99hJGmOMCcN6kMYYE4YdgzTG\nmDCsB2mMMWHYMUhjjAnDepD+KRj2KVZW78mIaX6st9/yy3d+NOw6SB+V1/EQC/JTYpifY+NBlvv8\nqFgP0hhjwrACaYwxYSTYSRobrMIY4x4PHtolIuki8pmIrBaRTBEZ5LT/SUS2i8gKZ+oasswjIrJB\nRNaKSOeQ9i5O2wYRebikbOtBGmNco97sYucCf1DVZSJSDfhKROY47/1NVV8MnVlEzgV6A+cB9YBP\nReQs5+3XgI5AFpAhIpNVdVW4YCuQxhj3eFAgVTUbyHZe/yQiq4HUYhbpAYxV1Z+BzSKyAWjhvLdB\nVTcBiMhYZ96wBdJ2sY0x7snPj3gSkf4isjRk6h/u40WkPtAUWOw0DRSRlSIySkROcdpSgW0hi2U5\nbeHaw7ICaYyJKVUdoarNQqYRRc0nIlWBj4HBqvoj8DpwJtCEYA/zpYJZi4oppj0s28U2xrjHo8t8\nRCSFYHF8T1UnAKjqrpD33wCmOj9mAekhi6cBBRd3hmsvkvUgjTHu8eYstgAjgdWq+nJIe92Q2X4D\nfOu8ngz0FpGKItIAaAQsATKARiLSQEQqEDyRM7m4bOtBGmNco+pJD/Iy4DbgGxFZ4bQ9CtwsIk0I\n7iZvAe5x1iFTRMYTPPmSCwxQ1TwAERkIzAKSgFGqmllcsBVIY4x7vDmLvZCijx9OL2aZYcCwItqn\nF7fc8axAGmPcY7caGmNM0Ty6UDxmrEAaY9yTYAUyIc9id+7UlsxvF7Bm1UIeHDIg4fPfGPES27O+\nZvnyuYVtF154Ll8smMzyZZ8yceLbVKtW1fP1KFDevv94yY6HfPKjmOJYwhXIQCDA8FeH0a17Hy64\nqB033dSTxo0bJXT+6DHj6dbt1mPa/vXPv/LoY8/R9OIOTPpkBn/4w32erkOB8vj9x0N2PORDcBc7\n0imeeVYgReQcEbnKufo9tL2LV5kALZo3ZePGLWzevJWcnBzGj5/Etd07l7xgGc5fuHAx+/b/cEzb\nWWedyRdfLALg07lf8JvfdC1qUdeVx+8/HrLjIR/w5DrIWPKkQIrI/cAk4P+Ab0WkR8jbz3mRWaBe\nah22ZR29OD5rezb16tXxMjKu8gtkZq6le/dOANxwfTfS0/wZnTrW2x/L/PK87YVsF7tU7gYuUdWe\nQFvgiYIx3Cj6eqZCoTeu5+cfijg4eNH9sTy6eDUu8wvc3f8B7rv3DhYvmkHValU4ciTHl9xYb38s\n88vzthfmJdgutldnsZNU9SCAqm4RkbbARyJyOiUUSOdG9REAyRVSI/72tmdlH9NbSkutS3b2rmKW\ncFes8wusXbuRrtfcAkCjRmfQ9eqrfMmN9fbHMr88b3uhOO8RRsqrHuRO5xYgAJxi2Q2oCVzgUSYA\nGUtX0LBhA+rXTyclJYVevXowZepsLyPjKr9ArVqnAcFexaOPDGLEiHd8yY319scyvzxvewHrQZbO\n7QTvgSykqrnA7SLyL48yAcjLy2PQ4MeZPu19kgIB3h49jlWr1nkZGfP8d955jSvbtKZmzVPZvGkp\nTz/9IlWrVuHe++4A4JNPpvP26HGerkOB8vj9x0N2POQDCdeDlFgcHyutaHax3WKPfbXHvpbrfNVi\nD4WFs7f7lRH/nT1tyudRZfkh4a6DNMYYt9ithsYY9yTYLrYVSGOMaxLssdhWII0xLrICaYwxRbMe\npDHGhGEF0hhjwrACaYwx4UR3+WTcsgJpjHGN9SCNMSYMzbcepDHGFMl6kMYYE0aUt3DHLSuQxhjX\nWA/SGGPCsGOQPioY9qm85ueU8+23/NjmRyOOR0+MSlwXyFiPx1je89um+vOYhuPN3x58vne5Ho8x\nDvKjkWg9SBsP0hjjGs2XiKeSiEi6iHwmIqtFJLPgAYAicqqIzBGR9c6fpzjtIiLDRWSDiKwUkYtD\nPquvM/96EelbUrYVSGNMvMsF/qCqjYFWwAARORd4GJirqo2Auc7PAFcDjZypP/A6BAsq8BTQEmgB\nPFVQVMOxAmmMcY1q5FPJn6nZqrrMef0TsBpIBXoAo53ZRgM9ndc9gDEatAioISJ1gc7AHFXdp6r7\ngTlAl+Ky4/oYpDGmbPH6GKSI1AeaAouB2qqaDcEiKiK/cmZLBbaFLJbltIVrD8t6kMYY16hKxJOI\n9BeRpSFT/6I+W0SqAh8Dg1X1x2JWo6gqrcW0h1VigRSR2iIyUkRmOD+fKyL9SlrOGFP+aH4Uk+oI\nVW0WMo04/nNFJIVgcXxPVSc4zbucXWecP3c77VlAesjiacCOYtrDKk0P8m1gFlBwzcE6YHApljPG\nlDP5KhFPJRERAUYCq1X15ZC3JgMFZ6L7ApNC2m93zma3Ag44u+KzgE4icopzcqaT0xZWaY5B1lTV\n8SLyCICq5opIXimWM8aUMx7di30ZcBvwjYiscNoeBV4Axjt7tFuBG533pgNdgQ3AYeDO4LrpPhF5\nBshw5ntaVfcVF1yaAnlIRE7D2VcvqMil3DBjTDnixUkaVV1I0ccPAX5xN4OqKjAgzGeNAkaVNrs0\nBfIBgl3WM0Xk30At4IbSBhhjyo9yd6uhqi4TkSuBswlW8bWqmuP5mhljypxEu9WwxAIpIrcf13Sx\niKCqYzxaJ2NMGVWaky5lSWl2sZuHvD6J4D7/MsAKpDHmGOVuwFxV/b/Qn0WkOvCOZ2tkjCmzEu0Y\nZDR30hwmeBN43NqwbhHLl33K0ozZLPpyuu/5nTu1JfPbBaxZtZAHhxR5Mi0h8queXIWh/3qSMfNH\nMfqzkZx7cWOq1ajGi+//mXe/eJsX3/8zVatXPWaZsy86m7nfzeLKa67wbL1i+f2Xl999OF5cBxlL\npbmTZoqITHamqcBajl6QGbc6dLyRZs070ap1V19zA4EAw18dRrfufbjgonbcdFNPGjf2798TP/MH\nDh3AkvkZ3N72t/TrdA9bN2zllgG9Wfbv5fS54g6W/Xs5twzofcy63fPoXWR8vtST9SnIiNX3X55+\n9+FEc6thPCtND/JF4CVneh5oo6oPF78IiEgLEWnuvD5XRB4QEX+rVQy0aN6UjRu3sHnzVnJychg/\nfhLXdu+ccPmVq1bmopYXMO2DGQDk5uRy8MdDXNbpUmZ+OBuAmR/O5vLOlxUuc92dPVkw/Qt++P4H\n19enQCy///Lyuy+OF6P5xFKxBVJEkoAnVPVzZ/q3qmaV9KEi8hQwHHhdRJ4H/gFUBR4WkcfcWPHi\nqCozpn/A4kUzuKvfrV7HHaNeah22ZR29vTNrezb16tVJuPx6v67LD/sO8PDLQ3hj5j8Z8tcHOKnS\nSZxa8xT27Q7enLBv9z5OOa0GADXrnMblV1/G5Hemur4ux6xXDL//8vK7L06i7WIXe5JGVfNE5LCI\nVFfVSO6euQFoAlQEdgJpqvqjiPyV4DBFw8It6Izk0R9AkqoTCFSJIDaoTdueZGfvolat05g5Yyxr\n127gi4WLI/6caARvGz2W+vjPpF/5SclJnHV+I4Y/8Q9WL1/DwKG/O2Z3+ngD//Q7Rjz3Jvn53j72\nLpbff3n53Rcn3neZI1Way3z+R/AeyDnAoYJGVb2/mGVyVTUPOCwiGwuGJlLV/4pIsX9DnJE8RgAk\nV0iN6rebnb0LgD179jJp0gyaN2/iW4HcnpVNetrRZ4mkpdYtXJ9Eyt+TvYc92XtYvXwNAJ9PW8At\nA25m3/f7OfVXp7Jv9z5O/dWp7N8b3J0++8KzePK14M5D9VOr07J9C/Jy81g46z+urlcsv//y8rsv\nT0pzDHIa8ASwAPjKmUo6yn5ERCo7ry8paHQuEfK0C1G5ciWqVq1S+LpjhyvJzFzrZeQxMpauoGHD\nBtSvn05KSgq9evVgytTZCZe/b89+du/YQ/oZaQBccvnFfLf+O/4z50u63NgJgC43duLfs4MF8OZL\nb6N36z70bt2Hz6ct4JXHhrteHCG23395+d0Xp1ztYjtqqOqroQ0FD80pRhtV/RlA9ZhHiadwdHgi\nT9SuXYuPPhwJQHJyEmPHfsKs2fO9jDxGXl4egwY/zvRp75MUCPD26HGsWrUuIfOHP/EPHv/7IyRX\nSCH7u2xe+MNfCUiAp/75OF17d2HX9t386d5nPMkOJ5bff3n63YcT5+dcIiYlHaMQkWWqevFxbctV\ntamna0b0u9huiIfHrsY63x77Wo7zozyY+J+610f8d/bS7I/jthsZtgcpIjcDtwANRGRyyFvVgL1e\nr5gxpuwpTydp/gNkAzUJXgNZ4CdgpZcrZYwpm7y9RsF/YQukqn4HfAe0Lu4DRORLVS12HmNM+aBh\nx7Utm9x47OtJLnyGMSYB5CfYWRo3CmSCfSXGmGjlWw/SGGOKlmi72KUZzWeg84jEsLO4uD7GmDIs\nP4opnpXmTpo6QIaIjBeRLvLLGz5v82C9jDFlkCIRT/GsxAKpqo8THCB3JHAHsF5EnhORM533v/V0\nDY0xZUZ57EEWPGd2pzPlAqcAH4nIXzxcN2NMGZNoBbI0TzW8n+D9098DbwJDVDVHRALAeuBBb1fR\nGFNWxPsuc6RKcxa7JnCdc+F4IVXNF5Fu3qyWMaYsSrDHYpfqqYZPFvPeandXxxhTliXadZDRPNXQ\nGGPKhbi+ULxg2CfLj42CYcdiJdbbX97zo5Fot9XFdYGM9XiIll+Ox0O0/Kh4cVZaREYB3YDdqnq+\n0/Yn4G5gjzPbo6o63XnvEaAfkAfcr6qznPYuwKtAEvCmqr5QUnZcF0hjTNmSX8SDw1zwNsEno445\nrv1vqvpiaIOInAv0Bs4D6gGfishZztuvAR2BLII3v0xW1VXFBVuBNMa4xotdbFVdICL1Szl7D2Cs\n88iXzSKyAWjhvLdBVTcBiMhYZ95iC6SdpDHGuCaaC8VFpL+ILA2Z+pcybqCIrBSRUSHjRaQC20Lm\nyXLawrUXywqkMcY1+RL5pKojVLVZyDSiFFGvA2cCTQg++aDgqQdF7eNrMe3Fsl1sY4xr/LoOUlUL\nH/gtIm8AU50fs4D0kFnTgIKzTuHaw7IepDHGNRrFFA0RqRvy42+AgkFzJgO9RaSiiDQgONDOEiAD\naCQiDUSkAsETOaEPIyyS9SCNMa7x4lZDEfkAaAvUFJEs4CmgrYg0IVhjtwD3AKhqpoiMJ3jyJRcY\noKp5zucMBGYRvMxnlKpmlpRtBdIY4xovroNU1ZuLaB5ZzPzDgGFFtE8HpkeSbQXSGOMau5PGGGPC\nKHej+RhjTGnF+wC4kbICaYxxjRVIY4wJQxNsFzvhroNMS6vHp7M/5JuV8/l6xTz+b2A/39ehc6e2\nZH67gDWrFvLgkAGW77MN6xaxfNmnLM2YzaIvIzppecJive2xzi93z6Qpa3Jzcxny4FCWr/iWqlWr\nsGTxTD6du4DVq9f7kh8IBBj+6jC6dL2ZrKxsFn05nSlTZ1u+T/kFOnS8kb179/uaGettj3V+Ikq4\nHuTOnbtZviJ4Uf3Bg4dYs2Y9qfXq+JbfonlTNm7cwubNW8nJyWH8+Elc272z5ZcDsd72WOdD4vUg\nfSuQInL8WG6eO/30NJpcdD6Llyz3LbNeah22ZR29xTNrezb1fCzQ5T0fQFWZMf0DFi+awV39bvUt\nN9bbHut88O9WQ794sostIscv5P07AAAS9ElEQVTf4yhAOxGpAaCq13qRG6pKlcqMH/cGD/zxKX76\n6aDXcYWkiAFDg48Vt3y/tGnbk+zsXdSqdRozZ4xl7doNfLFwsee5sd72WOeDXQdZWmkE74V8k6ND\nDTXj6JBEYTljwfUHkKTqBAJVIg5PTk7mw3Fv8MEHE/nkkxkRL38itmdlk552dKj8tNS6ZGfvKmYJ\ny3dbQd6ePXuZNGkGzZs38aVAxnrbY50P8b/LHCmvdrGbAV8BjwEHVHU+8F9V/VxVPy9uwdCx4aIp\njgBvjHiJ1Ws28MqrpRlWzl0ZS1fQsGED6tdPJyUlhV69ejBl6mzL90nlypWoWrVK4euOHa4kM3Ot\nL9mx3vZY50PiHYP0pAepqvnA30TkQ+fPXV5lHe+yS5tzW58bWPnNKpZmBP/neOKJF5gxc54f8eTl\n5TFo8ONMn/Y+SYEAb48ex6pV63zJtnyoXbsWH30YHMcgOTmJsWM/Ydbs+b5kx3rbY50P8X9MMVLi\nxzEKEbkGuExVH41kueQKqTH7vsvzUwVjnR8vT/Ur1/ka3SXffzm9T8R/Zx/87t24PXLpS69OVacB\n0/zIMsbETrzvMkcq4S4UN8bETqLtYluBNMa4Jj/BSqQVSGOMa2wX2xhjwkis/qMVSGOMi6wHaYwx\nYdithsYYE4adpDHGmDASqzwm4HiQxhjjFutBGmNcYydpjDEmDDsGaYwxYSRWebQCaYxxke1i+6hg\n2CfLt3zLLxu82MUWkVFAN2C3qp7vtJ0KjAPqA1uAXqq6X4LPnXgV6AocBu5Q1WXOMn2Bx52PfVZV\nR5eUbWexjTGu8eihXW8DXY5rexiYq6qNgLnOzwBXA42cqT/wOhQW1KeAlkAL4CkROaWk4LjuQZbX\nAWPLe35cDBgLpNY4Nyb5239YBcR++6PhxS62qi4QkfrHNfcA2jqvRwPzgYec9jEaHAl8kYjUEJG6\nzrxzVHUfgIjMIVh0PyguO64LpDGmbFH/TtPUVtVsAFXNFpFfOe2pwLaQ+bKctnDtxbJdbGOMa6J5\naJeI9BeRpSFT/xNYhaLuBtdi2otlPUhjjGuiOUmjqiOASB9BuktE6jq9x7rAbqc9C0gPmS8N2OG0\ntz2ufX5JIdaDNMa4xqOTNEWZDPR1XvcFJoW03y5BrQg+djobmAV0EpFTnJMznZy2YlkP0hjjGo8u\n8/mAYO+vpohkETwb/QIwXkT6AVuBG53ZpxO8xGcDwct87gRQ1X0i8gyQ4cz3dMEJm+JYgTTGuMaj\ns9g3h3nrqiLmVWBAmM8ZBYyKJNsKpDHGNT6exfaFFUhjjGvsVkNjjAkj0XqQdhbbGGPCsB6kMcY1\ntottjDFh5Gti7WJbgTTGuCaxymMCHoN8Y8RL7Mj6mhXL58ZsHTp3akvmtwtYs2ohDw4p8pIsyy/D\n+fVS6/Dh5LeYv2gy8/4ziX739AHggYd+x9LMecxe8DGzF3xM+45XFC4z8Pd3sfCrGSxYMpUr21/m\n+joViPV3n49GPMWzhCuQY8aM55put8YsPxAIMPzVYXTr3ocLLmrHTTf1pHHjRpafQPm5ubkMffwv\ntG11Ld073cwdd91Mo7PPBOCN18fQqc31dGpzPfPmfAFAo7PPpMd1XWnf+lpuveEennvxcQIB9//q\nxfq7h+BZ7Ej/i2cJVyC/WLiYfft/iFl+i+ZN2bhxC5s3byUnJ4fx4ydxbffOlp9A+bt3fc+3K1cD\ncOjgYdav20Sdur8KO3/nru2YNGE6R47ksG3rdrZs2kbTSy5wdZ0g9t89RDeaTzzzpUCKyOUi8oCI\ndPIjL5bqpdZhW9bRAUeztmdTr14dy0/Q/LT0epx/YWOWf7USgDvvvoU5Cyfw0t+foXr1kwGoU7c2\nO7bvLFwme8dO6tSt7fq6xPq7B9vFLhURWRLy+m7gH0A1gsOcPxx2wQQQfCTGsdTHM3uW719+5SqV\neWPMKzz1yAsc/OkQY0aN49KmXeh0xfXs3rWHJ58d4us6xfq7B9vFLq2UkNf9gY6qOpTgEEPFHiAM\nHTwzP/+QR6vnne1Z2aSnHR0qPy21LtnZuyw/wfKTk5N5Y/QrTPxwGjOmfgrA93v2kp+fj6ry3uiP\naOLsRmfv2Em91KM9ubr16rBr5+4iP/dExPq7B9vFLvXnOuOunQaIqu4BUNVDQG5xC6rqCFVtpqrN\nAoEqHq2edzKWrqBhwwbUr59OSkoKvXr1YMrU2ZafYPkv/f1pNqzbxIj/d/TBeL+qXbPw9dXdOrB2\n9XoAZs/4jB7XdaVChRTSf51KgzN/zfKvvnF9nWL93UOwxxrpFM+8ug6yOvAVwWHOVUTqqOpOEalK\n0UOfu+bdd17jyjatqVnzVLZsWsrQp1/krbfHehl5jLy8PAYNfpzp094nKRDg7dHjWLVqneUnUH7z\nVhdzQ+8erMpcy+wFHwPwwjOv0PP6rpx7wTmoKllbd/DQ7/8EwLo1G5nyyUw+WzSZvNw8HhvyLPn5\n7vedYv3dgzfjQcaS+Hx8qDLBh+1sLs38yRVSY/Ztl+enCsY6355qGAdPNVSNqiPT/dfdIv47O2Xr\nVE87TSfC1ztpVPUwUKriaIwpe+L9pEuk7FZDY4xrEm0X2wqkMcY18X7SJVJWII0xron3y3YiZQXS\nGOOaRDsGmXD3YhtjjFusB2mMcY2dpDHGmDDsJI0xxoRhPUhjjAkj0U7SWIE0xrjGHtpljDFhJFZ5\ntAJpjHFRoh2DtOsgjTGu8eqRCyKyRUS+EZEVIrLUaTtVROaIyHrnz1OcdhGR4SKyQURWisjF0W5P\nXPcgC4adsnzLj4WCYcdiJdbbHw2PL/Npp6rfh/z8MDBXVV9wHuXyMPAQcDXQyJlaAq87f0bMepDG\nGNf4/NCuHkDBkO6jgZ4h7WM0aBFQQ0TqRhMQ1z3I8jpgbHnPj5cBc2Odf2HtVjHJX7lrUdTLRnOZ\nj4j0J/jsqgIjVHXELz4aZouIAv9y3q+tqtkAqpotIgXP3k0FtoUsm+W0ZUe6bnFdII0xZUs0u9hO\nsTu+IB7vMlXd4RTBOSKypph5ixqhPKquqhVIY4xrvDqLrao7nD93i8hEoAWwS0TqOr3HukDBoyKz\ngPSQxdOAqA7o2jFIY4xrvHiqoYhUEZFqBa8JPj76W2Ay0NeZrS8wyXk9GbjdOZvdCjhQsCseKetB\nGmNc41EPsjYwUUQgWLPeV9WZIpIBjBeRfsBW4EZn/ulAV2ADcBi4M9pgK5DGGNd4cS+2qm4CLiqi\nfS9wVRHtCgxwI9t2sY0xJgzrQRpjXGODVRhjTBg23JkxxoRhPUhjjAnDepDGGBOG9SCNMSYM60Ea\nY0wYidaDTMjrIDesW8TyZZ+yNGM2i76c7nt+505tyfx2AWtWLeTBIa5cr2r5ZSTfz+xAIMC4OaP5\n+zsvAvCnlx/lw7lj+GjeO7z05jAqVa4EwCWtmjBu9tssy/qCjt3aebpOGsV/8SwhCyRAh4430qx5\nJ1q17uprbiAQYPirw+jWvQ8XXNSOm27qSePGjSy/HOT7nX3r3b3YvH5L4c9/ffIVbrzqdm5ofxvZ\nWbu4+bc3AJC9fSePD3qGGRPneLYuBVTzI57imScFUkRaisjJzutKIjJURKaIyJ9FpLoXmfGiRfOm\nbNy4hc2bt5KTk8P48ZO4tntnyy8H+X5m165bizYdLmPCe5ML2w4dPFz4+qRKFQt7Zzu27WT96o3k\n53tfjHweMNdzXvUgRxG8SRzgVaA68Gen7S2PMgupKjOmf8DiRTO4q9+tXscdo15qHbZlHR1ZKWt7\nNvXq1bH8cpDvZ/aDzwzm5Wf+Qf5xPbCnX3mMz76ZRv2Gp/PByA89yS6OF6P5xJJXBTKgqrnO62aq\nOlhVF6rqUOCM4hYUkf4islRElubnH4oqvE3bnrRo2YVu3ftw3313cMXlUT2OIirOiCPH8PN/AsuP\nXb5f2W06Xsa+7/ezeuXaX7z35OBhXHVRdzav30LnHh1czy6J9SBL51sRKRhi6GsRaQYgImcBOcUt\nqKojVLWZqjYLBKpEFZ6dvQuAPXv2MmnSDJo3bxLV50Rje1Y26WlHh+pPS61buD6Wn9j5fmU3aX4h\nbTtdwYyMCfzln8/Q4rJLeO4fTxW+n5+fz8xJc+lwjbcnZIpiPcjSuQu4UkQ2AucCX4rIJuAN5z3P\nVK5ciapVqxS+7tjhSjIzf/kvrVcylq6gYcMG1K+fTkpKCr169WDK1NmWXw7y/coe/tzrdLy4B1c3\nv44H732CJf/+ikcHDiW9flrhPG07Xc6WDd+5nl2SfNWIp3jmyXWQqnoAuMMZBfgMJydLVT3/p7x2\n7Vp89OFIAJKTkxg79hNmzZ7vdWyhvLw8Bg1+nOnT3icpEODt0eNYtWqd5ZeD/FhmiwjPDn+CqtWq\nIAJrMzfw7EN/AeC8Jo15ZdQLnFyjGld2vJz7htzFdVd6c2w+3i/biZTEcxc3uUJqzFauPD9VMNb5\n8fJUwVjnx/SphqpFPfiqRLWrnxPx39ldB9ZEleWHhL0O0hhjTpTdamiMcU28n5WOlBVIY4xr4vmQ\nXTSsQBpjXBPvZ6UjZQXSGOMa60EaY0wYdgzSGGPCsB6kMcaEYccgjTEmjES7k8YKpDHGNdaDNMaY\nMBLtGKTdamiMcY1Xz6QRkS4islZENojIwx5vRiHrQRpjXONFD1JEkoDXgI5AFpAhIpNVdZXrYcex\nHqQxxjUeDZjbAtigqptU9QgwFujh6YY44roHWTDsk+VbfnnMX7lrUUzzo+HREchUYFvIz1mAL89R\niesCGe2YdAVEpL+qjnBrdcpKtuVbfqzyc49sj/jvrIj0B/qHNI04bt2L+kxfzgYl+i52/5JnSchs\ny7f8WOeXWuhzqJzp+MKeBaSH/JwG+NK9T/QCaYwp+zKARiLSQEQqAL2BySUs44r43sU2xpR7qpor\nIgOBWUASMEpVM/3ITvQCGbNjQDHOtnzLj3W+q1R1OjDd79y4fmiXMcbEkh2DNMaYMKxAGmNMGAlZ\nIGN136aTPUpEdovIt37mhuSni8hnIrJaRDJFZJDP+SeJyBIR+drJH+pnvrMOSSKyXESm+p3t5G8R\nkW9EZIWILPU5u4aIfCQia5z/B1r7mZ9oEu4YpHPf5jpC7tsEbvbjvk0nvw1wEBijquf7kXlcfl2g\nrqouE5FqwFdATx+3X4AqqnpQRFKAhcAgVfXtthAReQBoBpysqt38yg3J3wI0U9XvY5A9GvhCVd90\nLomprKo/+L0eiSIRe5Axu28TQFUXAPv8yisiP1tVlzmvfwJWE7xVy698VdWDzo8pzuTbv8IikgZc\nA7zpV2a8EJGTgTbASABVPWLF8cQkYoEs6r5N3wpEPBGR+kBTYLHPuUkisgLYDcxRVT/zXwEeBPJ9\nzDyeArNF5CvnNjq/nAHsAd5yDjG8KSJVfMxPOIlYIGN232Y8EZGqwMfAYFX90c9sVc1T1SYEbwlr\nISK+HGoQkW7AblX9yo+8YlymqhcDVwMDnMMufkgGLgZeV9WmwCHA12PwiSYRC2TM7tuMF86xv4+B\n91R1QqzWw9m9mw908SnyMuBa5xjgWKC9iLzrU3YhVd3h/LkbmEjwsI8fsoCskB77RwQLpolSIhbI\nmN23GQ+ckyQjgdWq+nIM8muJSA3ndSWgA7DGj2xVfURV01S1PsHf+zxV7eNHdgERqeKcHMPZve0E\n+HJFg6ruBLaJyNlO01WALyfnElXC3WoYy/s2AUTkA6AtUFNEsoCnVHWkX/kEe1G3Ad84xwEBHnVu\n1fJDXWC0czVBABivqjG53CZGagMTg/9OkQy8r6ozfcz/P+A9p3OwCbjTx+yEk3CX+RhjjFsScRfb\nGGNcYQXSGGPCsAJpjDFhWIE0xpgwrEAaY0wYViBNXBGR+rEaCcmY41mBNL5wros0pkyxAmmKJCLP\nhI4lKSLDROT+IuZrKyILRGSiiKwSkX+KSMB576CIPC0ii4HWInKJiHzuDOIwyxmaDaf9axH5Ehjg\n1zYaUxIrkCackUBfAKfg9QbeCzNvC+APwAXAmcB1TnsV4FtVbUlwRKG/Azeo6iXAKGCYM99bwP2q\naoO7mriScLcaGneo6hYR2SsiTQnePrdcVfeGmX2Jqm6CwlstLyc4UEIewUEzAM4GzgfmOLfhJQHZ\nIlIdqKGqnzvzvUNwFBxjYs4KpCnOm8AdQB2CPb5wjr9fteDn/6lqnvNagMzje4nOwBZ2v6uJS7aL\nbYozkeBQZc0JDv4RTgtn9KQAcBPBxywcby1Qq+AZKSKSIiLnOUOiHRCRy535bnVv9Y05MdaDNGGp\n6hER+Qz4IaQnWJQvgRcIHoNcQLCwFvVZNwDDnd3qZIKjf2cSHHFmlIgcpvhCbIyvbDQfE5bTI1wG\n3Kiq68PM0xb4YywejmWM12wX2xRJRM4FNgBzwxVHYxKd9SBNqYjIBQTPMIf62bmEx5iEZAXSGGPC\nsF1sY4wJwwqkMcaEYQXSGGPCsAJpjDFhWIE0xpgw/j9Fyj7q1laZnAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xg = xgb.XGBClassifier(n_estimators = 10)\n",
    "xg.fit(X_train,y_train)\n",
    "xg_score=xg.score(X_test,y_test)\n",
    "y_predict=xg.predict(X_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of XGBoost: '+ str(xg_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of XGBoost: '+(str(precision)))\n",
    "print('Recall of XGBoost: '+(str(recall)))\n",
    "print('F1-score of XGBoost: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter optimization (HPO) of XGBoost using Bayesian optimization with tree-based Parzen estimator (BO-TPE)\n",
    "Based on the GitHub repo for HPO: https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████| 20/20 [00:30<00:00,  1.53s/trial, best loss: -0.9957089552238806]\n",
      "XGBoost: Hyperopt estimated optimum {'learning_rate': 0.7340229699980686, 'n_estimators': 70.0, 'max_depth': 14.0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'learning_rate':  abs(float(params['learning_rate'])),\n",
    "\n",
    "    }\n",
    "    clf = xgb.XGBClassifier( **params)\n",
    "    clf.fit(X_train, y_train)\n",
    "    y_pred = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test, y_pred)\n",
    "\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 10, 100, 5),\n",
    "    'max_depth': hp.quniform('max_depth', 4, 100, 1),\n",
    "    'learning_rate': hp.normal('learning_rate', 0.01, 0.9),\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"XGBoost: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of XGBoost: 0.9957089552238806\n",
      "Precision of XGBoost: 0.9956893766598436\n",
      "Recall of XGBoost: 0.9957089552238806\n",
      "F1-score of XGBoost: 0.9956902750637269\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3645\n",
      "           1       0.99      1.00      1.00       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       1.00      1.00      1.00       609\n",
      "           4       0.83      0.71      0.77         7\n",
      "           5       0.99      1.00      0.99       251\n",
      "           6       0.99      0.99      0.99       436\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       0.97      0.96      0.96      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8FVXawPHfc5OgNEEEKUlWUHCt\nKyhFRRFRAyKIuypiRRdFd1FhdcGylkXFl9214rquINVCsSAdaSKi0kGkdyEQioIosGLK8/5xJ+GC\nmSQ3zJ253DxfP/PJzbkz88zcax7OmTPnjKgqxhhjfi0U9AEYY0y8sgRpjDEuLEEaY4wLS5DGGOPC\nEqQxxriwBGmMMS4sQRpjjAtLkMYY48ISpDHGuEgO+gCKJGLDfIwJgqqUZrPs7zZE/TebUv3UUsXy\nQ1wnyOxd6wOLnVLjNJJT6gQWPyd7W5mNn5O9DcDiBxzfxHmCNMYcY/Jygz4CT1mCNMZ4R/OCPgJP\nWYI0xngnzxKkMcYUShOsBmm3+RhjvJOXF/1SDBE5XkTmicjXIrJcRHo75UNEZKOILHGWhk65iEg/\nEVknIktF5PyIfXUWkbXO0rm42FaDNMZ4JzY1yINAK1XdJyIpwGwRmeS811NVPzhi/auBBs7SDHgD\naCYi1YCngcaAAgtFZKyq7nELbDVIY4x38nKjX4qhYfucX1Ocpaj7LTsAw5zt5gBVRaQ20BqYqqq7\nnaQ4FWhTVGxLkMYY72he9EsJiEiSiCwBdhJOcnOdt/o4zeiXReQ4pywV2BKxeaZT5lbuyhKkMcY7\npbgGKSJdRWRBxNL1yN2qaq6qNgTSgKYicg7wGHAG0ASoBjzirF7YyBwtotyVJUhjjGdU80qxaH9V\nbRyx9Hffv/4AzATaqGqW04w+CAwGmjqrZQLpEZulAduKKHdlCdIY453Y9GLXEJGqzuvywJXAKue6\nIiIiwHXAMmeTscAdTm/2hcBeVc0CPgEyROREETkRyHDKXFkvtjHGO7Hpxa4NDBWRJMKVulGqOl5E\nZohIDcJN5yXAfc76E4G2wDrgAHAXgKruFpFngfnOes+o6u6iAluCNMZ4JwZjsVV1KdCokPJWLusr\n0M3lvUHAoJLGPqaa2AcP/kKnu7vzh85/psOt9/Lvt94GQFV59c0hXNPpbtrf0pV33h8DwIzPv+L3\nd/yJ6zt3o+MfH2TR18sK9jVm4lTa3tSFtjd1YczEqZ4d43HHHcdXX4xn4YKpfL1kBk8/9bBn+y6p\n1hktWb5sFqtWzKZXz0L/P4mZtLQ6TJvyPt8sncnXS2bwwP1dfI0/oP+LbMv8miWLp/saN1/Q33+V\nKicwckR/ln3zGd8sncmFzS7wNX6serGDckzVIMuVS2FQv75UqFCe7Jwc7vjTX7n0wsZs+HYL23d+\nx7j3+hMKhfh+zw8AXHhBQy6/5EJEhNXrNvLXJ59n3PAB7P3xJ94Y/B4jB/YD4KYuD9LykgupckLl\noz7GgwcPcmVGR/bvP0BycjKzZo5m8uRPmTtv0VHvuyRCoRD9Xu1Dm7Y3k5mZxZyvJjJu/BRWrlzr\nS/ycnBx69urN4iXLqFSpIvPmTmba9Fm+xR82bBT/+c9gBg9+1Zd4Rwr6+3/5pWf45JNPualTV1JS\nUqhQobwvcRNVzGqQInKGiDziDPl51Xl95lHus+ALz8nJIScnBxFh5OgJ/OmuWwiFwqdz0olVAahQ\noTzh67fwv59/Buf1F3MXclGTRlQ5oTJVTqjMRU0a8cXchUdzaIfZv/8AACkpySSnpBCu8fujaZNG\nrF+/iY0bN5Odnc2oUWO4tn1r3+Jv376TxUvCNfV9+/azatVaUuvU8i3+57Pnstv5BzIoQX3/lStX\n4tJLmjFo8HAAsrOz2bv3R19iF4hBJ02QYpIgReQRYAThi6fzCF8UFWC4iDx6NPvOzc3l+s7daNHu\nZi5q0ojfnX0GW7ZmMWn6Z3T844Pc9/CTfLtla8H60z77gvY338Of//oUzz7+FwB27PqOWifXKFin\nZo3q7Nj13dEc1mFCoRAL5k8ha+tSpk+fxbz5iz3bd3HqpNZiS+ahOxcyt2ZRx8cEFemUU9JoeN45\nzJ3n3/nHg6C+/1NPPYXvvvuegW+9zPx5n/Dmf//lfw0ywZrYsapBdgGaqGpfVX3HWfoSvk/pqC5K\nJSUl8eHQ15k++m2+WbGGtRs28Ut2NseVK8eoQf24vn0bnnz+5YL1r7ysOeOGD6Bf36f494BhABT2\nD3p+TdMLeXl5NG6SwSn1GtOkcSPOPvu3nu27OIWdh5812HwVK1Zg1MgBPPTXp/npp33Fb5BAgvr+\nk5OSaNToXN58cxhNmrZm//4DPNLrfl9iF7AaZInkAYXNF1/bec9V5F31bw0b7rreCZUr0eT83zF7\nzgJq1ajOVS0vAeDKyy5mzfqNv1q/ccNz2bI1iz0/7KXWydXZvnNXwXs7dn3HydVPKtGJRWPv3h/5\nbNaXtM5o6fm+3WzNzCI97dBHn5Zam6ysHb7FB0hOTub9kQMYPnw0H388qfgNEpTf33/m1iwyM7MK\naqwffTSBRg3P9SV2PtXcqJd4FqsE2QOYLiKTRKS/s0wGpgPdi9ow8q76u++4+bD3du/5gR+d2sjP\nBw8yZ/5i6p2STqsWFzF34RIA5i/+hlPSw8MrN2duK6g9rVi9juzsHKpWOYHmzS7gy3mL2PvjT+z9\n8Se+nLeI5h719lWvXo0qVU4A4Pjjj+eKVpeyerV/z9aZv2AJ9evXo27ddFJSUujYsQPjxk/xLT6E\ne5JXrlrHK6+6DohIWEF+/zt27CIzcxunn34aAK1aXcLKlWt8iV0gwZrYMenFVtXJInI64SZ1KuHr\nj5nAfD2KfzJ2fb+Hvz33Arl5eWie0rrVpbRs3ozzf3c2j/T+J2+P/JgK5Y+n96M9AJg6czZjJ00n\nOTmZ448rxwvPPIqIUOWEytx75810ujucq++76xZPerABateuyaCBr5CUFCIUCvHBB+OYMHGaJ/su\nidzcXLr3eIKJE94jKRRiyNCRrFjh3x9J84ubcPttN7D0mxUsmB9OzE8+2ZdJk2f4Ev+dt1/nshYX\nUb16NTZtWEDvZ15g8JARvsSG4L//7n95kmFDX6NcuRQ2btxMl7sf8i02EPdN5mhJENenSqo0j5D0\nij3V0J5qWKbjl/Kxrz8v/Djqv9njL7jOHvtqjCkD7KmGxhjjIs6vKUbLEqQxxjsJdg3SEqQxxjtW\ngzTGGBdWgzTGGBeWII0xpnDxPjImWpYgjTHesRqkMca4SLBOmmNqRnFjjPGT1SCNMd6xJrYxxrhI\nsCa2JUhjjHesBmmMMS6sBumflBqnBRo/f9opi2/xy2L8UrEapDHGuLAE6Z+gJ4w9s0aTwOKv3DU/\n8PMv0xPGWvzSsSa2Mca4sBqkMca4sBqkMca4SLAapA01NMZ4JwaPfRWR40Vknoh8LSLLRaS3U15P\nROaKyFoRGSki5Zzy45zf1znv143Y12NO+WoRaV1cbEuQxhjv5OVFvxTvINBKVc8DGgJtRORC4B/A\ny6raANgDdHHW7wLsUdX6wMvOeojIWUAn4GygDfAfEUkqKrAlSGOMd2KQIDVsn/NrirMo0Ar4wCkf\nClznvO7g/I7z/hUiIk75CFU9qKobgXVA06JiW4I0xnhHNepFRLqKyIKIpeuRuxWRJBFZAuwEpgLr\ngR9UNcdZJRNIdV6nAlvCh6M5wF7gpMjyQrYplHXSGGO8U4pOGlXtD/QvZp1coKGIVAVGA2cWtprz\nU1zecyt3ZQnSGOOdGPdiq+oPIjITuBCoKiLJTi0xDci/wz0TSAcyRSQZqALsjijPF7lNoayJbYzx\nTmx6sWs4NUdEpDxwJbAS+BS4wVmtMzDGeT3W+R3n/Rmqqk55J6eXux7QAJhXVGyrQRpj4l1tYKjT\n4xwCRqnqeBFZAYwQkeeAxcBAZ/2BwNsiso5wzbETgKouF5FRwAogB+imxTxlzBKkMcY7MWhiq+pS\noFEh5RsopBdaVX8GbnTZVx+gT0ljW4I0xnhHi+zzOOYk5DXI1hktWb5sFqtWzKZXz24xiVHuuHKM\nnDyY0Z++y7hZI7i/1z0ANLukMR9OG8bYz4bzf689TVJS+D7UVm1a8PHMd/loxju8P2Uo5zc7LybH\nBf6cv8WPv9jxED9GN4oHRjSOM35yudSoDy4UCrFy+ee0aXszmZlZzPlqIrfd/mdWrlwb1X5KMt1Z\nhYrlObD/fyQnJ/HOuAH0feplXur/PH+8vhubNmzmgUe6sm3Ldj58b2zBugCnn1Wflwc8zzXNO7ru\nu7TTnXl5/kHFP5rpvoKM7+VnH3h81cJuiSnW/wb+Neq/2fJdXihVLD8kXA2yaZNGrF+/iY0bN5Od\nnc2oUWO4tn2xQy5LJT/hJackk5KSTF5uHr/88gubNmwG4MuZ88hod/lh6wJUqFCeWP3D5Of5W/z4\niR0P8YGY9GIHKZAEKSJ3xWrfdVJrsSXz0K1NmVuzqFOnVkxihUIhPprxDrNXfMKXn81j6aLlpCQn\nc/Z54XtYM9q3oladmgXrX9m2JRO+GMUb777EEz2ei8kx+Xn+Fj9+YsdDfADN06iXeBZUDbK32xuR\nw47y8vZHvePwkMvDxaq2lpeXxx9a3cbl57Xj3EZn0eCMU3n43id49Nm/MHLyYA7sO0BO7qG7CKZN\nnMk1zTvyQOdePPjovTE5Jj/P3+LHT+x4iA8k3DXImPVii8hSt7eAmi7vHTbsqDTXILdmZpGeduja\nTVpqbbKydkS7m6j89OM+5n25iEtaXcTg/7zL7deGh5Je3LIZp5z2m1+tv2DOYtJPSaNqtSr8sHuv\np8cSxPlb/OBjx0N8IO6bzNGKZQ2yJnAH0L6Q5ftYBZ2/YAn169ejbt10UlJS6NixA+PGT/E8zokn\nVaXyCZUAOO7447ioRVM2rv2WatVPBCClXAp3338HI4d+BMBv6qUVbHvWub8lpVyy58kR/Dt/ix9f\nseMhPgB5Gv0Sx2J5H+R4oJKqLjnyDWcsZUzk5ubSvccTTJzwHkmhEEOGjmTFijWex6lRs7pzG0+I\nkISYPHYaM6fO5q9PP0DLqy4hFAoxYsiHzJ29AICMdq3ocGNbsnNyOPjzQR7q+jfPjwn8O3+LH1+x\n4yE+EPdN5mgl3G0+XrGnGtpTDct0/FLe5nPg1fui/put0P2/cXubj42kMcZ4J44rXKVhCdIY450E\na2JbgjTGeCfOO12iZQnSGOOdBLvNxxKkMcY7CVaDTLix2MYY4xWrQRpjPKPWSWOMMS4SrIltCdIY\n4x3rpDHGGBdWgzTGGBd2DdIYY1xYDdIYY1zYNUhjjHFhNUj/5E/7FJSVu+YHGj/o87f4ZTt+adh9\nkD4qq/Mh5sdPCTB+ts0HWebjl4rVII0xxoUlSGOMcZFgnTQ2WYUxxjsxeGiXiKSLyKcislJElotI\nd6f87yKyVUSWOEvbiG0eE5F1IrJaRFpHlLdxytaJyKPFxbYapDHGMxqbJnYO8LCqLhKRysBCEZnq\nvPeyqr4QubKInAV0As4G6gDTROR05+3XgauATGC+iIxV1RVugS1BGmO8E4MEqapZQJbz+icRWQmk\nFrFJB2CEqh4ENorIOqCp8946Vd0AICIjnHVdE6Q1sY0x3snLi36JgojUBRoBc52i+0VkqYgMEpET\nnbJUYEvEZplOmVu5K0uQxphAiUhXEVkQsXR1Wa8S8CHQQ1V/BN4ATgMaEq5hvpi/aiGbaxHlrqyJ\nbYzxTima2KraH+hf1DoikkI4Ob6rqh852+2IeH8AMN75NRNIj9g8Dci/udOtvFBWgzTGeCc2vdgC\nDARWqupLEeW1I1b7PbDMeT0W6CQix4lIPaABMA+YDzQQkXoiUo5wR87YomJbDdIY4xnVmPRiNwdu\nB74RkSVO2ePAzSLSkHAzeRNwr3MMy0VkFOHOlxygm6rmAojI/cAnQBIwSFWXFxXYEqQxxjux6cWe\nTeHXDycWsU0foE8h5ROL2u5IliCNMd6xoYbGGFO4GN0oHhhLkMYY7yRYgkzIXuzWGS1ZvmwWq1bM\nplfPbgkff0D/F9ma+TWLF08vKPvd787i81ljWbxoGqNHD6Fy5UoxP458Ze3zj5fY8RCfvFIscSzh\nEmQoFKLfq31o1/42zj3vcm666TrOPLNBQscfOmwU7drdeljZm//9F4//7XkanX8lYz6exMMP/ymm\nx5CvLH7+8RA7HuJDuIkd7RLPYpYgReQMEbnCufs9srxNrGICNG3SiPXrN7Fx42ays7MZNWoM17Zv\nXfyGx3D82bPnsnvPD4eVnX76aXz++RwApk3/nN//vm1hm3quLH7+8RA7HuIDMbkPMkgxSZAi8iAw\nBngAWCYiHSLefj4WMfPVSa3FlsxDN8dnbs2iTp1asQwZV/HzLV++mvbtMwC44fp2pKf5Mzt10Ocf\nZPyyfO4FrIldIvcAF6jqdUBL4Mn8Odwo/H6mApHjMvPy9kcdOHzT/eFidPNqXMbPd0/Xh/jTfXcy\nd84kKlWuyC+/ZPsSN+jzDzJ+WT73gngJ1sSOVS92kqruA1DVTSLSEvhARE6hmAQZOS4zuVxq1J/e\n1sysw2pLaam1ycraUcQW3go6fr7Vq9fT9ppbAGjQ4FTaXn2FL3GDPv8g45flcy8Q5zXCaMWqBrnd\nGQIEgJMs2wHVgXNjFBOA+QuWUL9+PerWTSclJYWOHTswbvyUWIaMq/j5atQ4CQjXKh5/rDv9+7/t\nS9ygzz/I+GX53PNZDbJk7iA8BrKAquYAd4jImzGKCUBubi7dezzBxAnvkRQKMWToSFasWBPLkIHH\nf/vt17msxUVUr16NjRsW8MwzL1CpUkXu+9OdAHz88USGDB0Z02PIVxY//3iIHQ/xgYSrQUoQ18dK\nqjRNbK/YY1/tsa9lOr5qkZfC3Hzf/rKo/2ZPGvdZqWL5IeHugzTGGK/YUENjjHcSrIltCdIY45kE\neyy2JUhjjIcsQRpjTOGsBmmMMS4sQRpjjAtLkMYY46Z0t0/GLUuQxhjPWA3SGGNcaJ7VII0xplBW\ngzTGGBelHMIdtyxBGmM8YzVIY4xxYdcgfZQ/7VNZjZ9dxs/f4gcbvzTiePbEUonrBBn0fIxlPX7L\nVH8e03CkmVvDz/cu0/MxxkH80rAapDHGuEi0BGkT5hpj4pqIpIvIpyKyUkSW5z8hVUSqichUEVnr\n/DzRKRcR6Sci60RkqYicH7Gvzs76a0Wkc3GxLUEaYzyjGv1SAjnAw6p6JnAh0E1EzgIeBaaragNg\nuvM7wNVAA2fpCrwB4YQKPA00A5oCT+cnVTeWII0xntE8iXopdp+qWaq6yHn9E7ASSAU6AEOd1YYC\n1zmvOwDDNGwOUFVEagOtgamqultV9wBTgTZFxbYEaYzxjKpEvYhIVxFZELF0ddu/iNQFGgFzgZqq\nmhWOq1nAyc5qqcCWiM0ynTK3clfFdtKISE3geaCOql7tVG0vUtWBxW1rjClbSnOjuKr2B/oXt56I\nVAI+BHqo6o8irrXPwt7QIspdlaQGOQT4BMi/52AN0KME2xljypg8laiXkhCRFMLJ8V1V/cgp3uE0\nnXF+7nTKM4H0iM3TgG1FlLsqSYKsrqqjcJ42oao5QG4JtjPGlDGlaWIXR8JVxYHASlV9KeKtsUB+\nT3RnYExE+R1Ob/aFwF6nCf4JkCEiJzqdMxlOmauS3Ae5X0ROwqmK5gcswXbGmDImRvdBNgduB74R\nkSVO2eNAX2CUiHQBNgM3Ou9NBNoC64ADwF0AqrpbRJ4F5jvrPaOqu4sKXJIE+RDhjHyaiHwB1ABu\nKOGJGWPKkFgMNVTV2RR+/RDgV8O9VFWBbi77GgQMKmnsYhOkqi4SkcuA3zoHuVpVs0sawBhTdiTa\nSJqS9GLfcUTR+SKCqg6L0TEZY45RJe10OVaUpIndJOL18YSrtIsAS5DGmMOUuQlzVfWByN9FpArw\ndsyOyBhzzEq06c5KM5LmAOExjnFpQP8X2Zb5NUsWTw/sGFpntGT5slmsWjGbXj0LvVacEPErnVCR\n3m8+xbCZgxj66UDOOv9MKletzAvv/YN3Ph/CC+/9g0pVKgFQsXJFnh/8LG9NeZPB09+iTcfWMTuu\nID//svLdu4nVfZBBKTZBisg4ERnrLOOB1Ry63yjuDBs2imva3RpY/FAoRL9X+9Cu/W2ce97l3HTT\ndZx5pn//nvgZ//7e3Zg3cz53tPwjXTLuZfO6zdzSrROLvljMbZfeyaIvFnNLt04AXNf5Wjat/Za7\nM+6lx40P8+en7iU5xfvZ9oL8/MvSd+8mFvdBBqkkNcgXgBed5f+AFqr6aNGbgIg0FZEmzuuzROQh\nEWl7VEdbAp/PnsvuPT/EOoyrpk0asX79JjZu3Ex2djajRo3h2vaxqy0FFb9CpQqc1+xcJgyfBEBO\ndg77ftxP84yLmfz+FAAmvz+FS1o3B8JNrwoVKwBQvmJ5fvrhJ3JzvB9vEOTnX1a++6LEaDafwBSZ\nIEUkCXhSVT9zli9UNbO4nYrI00A/4A0R+T/g30Al4FER+ZsXBx6v6qTWYkvmodFLmVuzqFOnVsLF\nr/Ob2vywey+PvtSTAZP/S89/PcTx5Y+nWvUT2b0zfO/t7p27OfGkqgCMHvIxpzT4DR8uHMngaQN4\n7an/oDH46wjy8y8r331RylQTW1VzgQNOx0w0biB893sLwjdsXqeqzxCebuimojaMnNkjL29/lGGD\nV9gA+lgkgqDjJyUncfo5DRjz9jjuaXMf/zvwc0FzujBNWzZm3fL1XH/BTdzd+l66P3c/FSpV8Py4\ngvz8y8p3X5Sy2MT+mfAQn4HOLL39RKRfMdvkqGquqh4A1qvqjwCq+j+cMd1uVLW/qjZW1cahUMUS\nnUQ82ZqZRXraoWeJpKXWJitrR8LF35W1i11Zu1i5eBUAn02YRYNzG7D7uz1UO7kaANVOrsae78OX\nO9p0bMOsSZ+Hj3HTNrK2bOc39dML3/lRCPLzLyvffVlSkgQ5AXgSmAUsdJYFxWzzi4jkVw8uyC90\naqIJ9uTcw81fsIT69etRt246KSkpdOzYgXHjpyRc/N279rBz2y7ST00D4IJLzufbtd/y5dSvaHNj\nBgBtbszgiylfArBz604uuCQ88/2J1auSflo6Wd9meX5cQX7+ZeW7L0qiNbFL0o1YVVVfjSzIfyZE\nEVqo6kEA1cNmiEvh0OwbMfHO269zWYuLqF69Gps2LKD3My8weMiIWIY8TG5uLt17PMHECe+RFAox\nZOhIVqxYk5Dx+z35b5547TGSy6WQ9W0WfR/+FyEJ8fR/n6Btpzbs2LqTv9/3LADDXn2HR1/qyaBp\nAxCg//MD2LvnR8+PKcjPvyx9927ivM8lalLcNQoRWaSq5x9RtlhVG8X0yIDkcqmBfd7x8NjVoOPb\nY1/LcPxSXhz8svb1Uf/NXpz1YdxWI11rkCJyM3ALUE9Exka8VRn4PtYHZow59sR7p0u0impifwlk\nAdUJ3wOZ7ydgaSwPyhhzbEq0DgbXBKmq3wLfAhcVtQMR+UpVi1zHGFM2qOu0jccmL8Z6He/BPowx\nCSAvwXppvEiQCfaRGGNKK89qkMYYU7hEa2KXZDaf+50ngLmu4uHxGGOOYXmlWOJZSUbS1ALmi8go\nEWkjvx7weXsMjssYcwxSJOolnhWbIFX1CcIT5A4E7gTWisjzInKa8/6ymB6hMeaYURZrkPmPUdzu\nLDnAicAHIvLPGB6bMeYYk2gJsiRPNXyQ8Pjp74C3gJ6qmi0iIWAt0Cu2h2iMOVbEe5M5WiXpxa4O\n/MG5cbyAquaJSLvYHJYx5liUYI/FLtFTDZ8q4r2V3h6OMeZYlmj3QZbmqYbGGFMmxPWN4vnTPln8\nYORPOxaUoM+/rMcvjUQbVhfXCTLo+RAtfhmeD9Hil0oseqVFZBDQDtipquc4ZX8H7gF2Oas9rqoT\nnfceA7oAucCDqvqJU94GeBVIAt5S1b7FxY7rBGmMObbkFfLgMA8MIfxk1GFHlL+sqi9EFojIWUAn\n4GygDjBNRE533n4duArIJDz4ZayqrigqsCVIY4xnYtHEVtVZIlK3hKt3AEY4j3zZKCLrgKbOe+tU\ndQOAiIxw1i0yQVonjTHGMz7fKH6/iCwVkUER80WkAlsi1sl0ytzKi2QJ0hjjmTyJfhGRriKyIGLp\nWoJQbwCnAQ0JP/kg/6kHhbXxtYjyIlkT2xjjmdLcB6mq/YH+UW5T8MBvERkAjHd+zQQiH7ieBuT3\nOrmVu7IapDHGM1qKpTREpHbEr78H8ifNGQt0EpHjRKQe4Yl25gHzgQYiUk9EyhHuyIl8GGGhrAZp\njPFMLIYaishwoCVQXUQygaeBliLSkHCO3QTcC6Cqy0VkFOHOlxygm6rmOvu5H/iE8G0+g1R1eXGx\nLUEaYzwTi/sgVfXmQooHFrF+H6BPIeUTgYnRxLYEaYzxjI2kMcYYF2VuNh9jjCmpeJ8AN1qWII0x\nnrEEaYwxLjTBmtgJdx9kWlodpk15n2+WzuTrJTN44P4uvh9D64yWLF82i1UrZtOrZzeL77N1a+aw\neNE0Fsyfwpyvouq0PGpBn3vQ8cvcM2mONTk5OfTs1ZvFS5ZRqVJF5s2dzLTps1i5cq0v8UOhEP1e\n7UObtjeTmZnFnK8mMm78FIvvU/x8V151I99/v8fXmEGfe9DxE1HC1SC3b9/J4iXhm+r37dvPqlVr\nSa1Ty7f4TZs0Yv36TWzcuJns7GxGjRrDte1bW/wyIOhzDzo+JF4N0rcEKSJHzuUWc6eckkbD885h\n7rzFvsWsk1qLLZmHhnhmbs2ijo8JuqzHB1BVJk0cztw5k7i7y62+xQ363IOOD/4NNfRLTJrYInLk\nGEcBLheRqgCqem0s4kaqWLECo0YO4KG/Ps1PP+2LdbgCUsiEoeHHilt8v7RoeR1ZWTuoUeMkJk8a\nwerV6/h89tyYxw363IOOD3YfZEmlER4L+RaHphpqzKEpiVw5Ux11BZCkKoRCFaMOnpyczPsjBzB8\n+Gg+/nhS1Nsfja2ZWaSnHZoqPy21NllZO4rYwuJ7LT/erl3fM2bMJJo0aehLggz63IOOD/HfZI5W\nrJrYjYGFwN+Avao6E/ifqn67ovrdAAASdUlEQVSmqp8VtaGq9lfVxqrauDTJEWBA/xdZuWodr7wa\n1QxKnpi/YAn169ejbt10UlJS6NixA+PGT7H4PqlQoTyVKlUseH3VlZexfPlqX2IHfe5Bx4fEuwYZ\nkxqkquYBL4vI+87PHbGKdaTmFzfh9ttuYOk3K1gwP/w/x5NP9mXS5Bl+hCc3N5fuPZ5g4oT3SAqF\nGDJ0JCtWrPEltsWHmjVr8MH74XkMkpOTGDHiYz6ZMtOX2EGfe9DxIf6vKUZL/LhGISLXAM1V9fFo\ntksulxrY512WnyoYdPx4eapfmY6vpbvl+5+n3Bb132yvb9+J2yuXvtTqVHUCMMGPWMaY4MR7kzla\nCXejuDEmOInWxLYEaYzxTF6CpUhLkMYYz1gT2xhjXCRW/dESpDHGQ1aDNMYYFzbU0BhjXFgnjTHG\nuEis9JiA80EaY4xXrAZpjPGMddIYY4wLuwZpjDEuEis9WoI0xnjImtg+yp/2yeJbfIt/bEi0Jrb1\nYhtjPBOLh3aJyCAR2SkiyyLKqonIVBFZ6/w80SkXEeknIutEZKmInB+xTWdn/bUi0rkk5xPXNciy\nOmFsWY8fFxPGAqlVzwok/tYfVgDBn39pxKiJPQT4NxD5ZNRHgemq2ldEHnV+fwS4GmjgLM2AN4Bm\nIlINeJrw42AUWCgiY1W1yIenWw3SGOMZLcV/xe5TdRaw+4jiDsBQ5/VQ4LqI8mEaNgeoKiK1gdbA\nVFXd7STFqUCb4mJbgjTGeKY0D+0Ska4isiBi6VqCUDVVNQvA+XmyU54KbIlYL9MpcysvUlw3sY0x\nx5bSdNKoan/Aq0eQFjZdhhZRXiSrQRpjPBOLThoXO5ymM87PnU55JpAesV4asK2I8iJZgjTGeCYP\njXoppbFAfk90Z2BMRPkdTm/2hcBepwn+CZAhIic6Pd4ZTlmRrIltjPFMLHqxRWQ40BKoLiKZhHuj\n+wKjRKQLsBm40Vl9ItAWWAccAO4CUNXdIvIsMN9Z7xlVPbLj51csQRpjPFOSXumo96l6s8tbVxSy\nrgLdXPYzCBgUTWxLkMYYz9hQQ2OMcRGLGmSQrJPGGGNcWA3SGOMZa2IbY4yLPE2sJrYlSGOMZxIr\nPSbgNcgB/V9kW+bXLFk8PbBjaJ3RkuXLZrFqxWx69Sz0jgOLfwzHr5Nai/fHDmbmnLHM+HIMXe69\nDYCHHvkzC5bPYMqsD5ky60NaXXVpwTb3/+VuZi+cxKx547msVXPPjylf0J+9jzeK+yLhEuSwYaO4\npt2tgcUPhUL0e7UP7drfxrnnXc5NN13HmWc2sPgJFD8nJ4feT/yTlhdeS/uMm7nz7ptp8NvTABjw\nxjAyWlxPRovrmTH1cwAa/PY0OvyhLa0uupZbb7iX5194glDI+z+9oD97iM1sPkFKuAT5+ey57N7z\nQ2DxmzZpxPr1m9i4cTPZ2dmMGjWGa9u3tvgJFH/nju9YtnQlAPv3HWDtmg3Uqn2y6/qt217OmI8m\n8ssv2WzZvJVNG7bQ6IJzPT0mCP6zh9LN5hPPfEmQInKJiDwkIhl+xAtSndRabMk8NAY+c2sWderU\nsvgJGj8tvQ7n/O5MFi9cCsBd99zC1Nkf8eJrz1KlygkA1Kpdk21btxdsk7VtO7Vq1/T8WIL+7MGa\n2CUiIvMiXt9DeDbgysDTzuy/CUvk17MqqY89exbfv/gVKlZgwLBXePqxvuz7aT/DBo3k4kZtyLj0\nenbu2MVTz/X09ZiC/uzBmtgllRLxuitwlar2JjyDRpEXCCMnz8zL2x+jw4udrZlZpKcdmio/LbU2\nWVk7LH6CxU9OTmbA0FcY/f4EJo2fBsB3u74nLy8PVeXdoR/Q0GlGZ23bTp3UQzW52nVqsWP7zkL3\nezSC/uzBmtgl3q8zrdBJgKjqLgBV3Q/kFLWhqvZX1caq2jgUqhijw4ud+QuWUL9+PerWTSclJYWO\nHTswbvwUi59g8V987RnWrdlA//8MLSg7uWb1gtdXt7uS1SvXAjBl0qd0+ENbypVLIf03qdQ77Tcs\nXviN58cU9GcP4RprtEs8i9V9kFWAhYRn8VURqaWq20WkEoXP7OuZd95+nctaXET16tXYtGEBvZ95\ngcFDRsQy5GFyc3Pp3uMJJk54j6RQiCFDR7JixRqLn0Dxm1x4Pjd06sCK5auZMutDAPo++wrXXd+W\ns849A1Ulc/M2HvnL3wFYs2o94z6ezKdzxpKbk8vfej5HXp73daegP3tIvMe+is/XhyoQfpbExpKs\nn1wuNbBPuyw/VTDo+PZUwzh4qqFqqSoy7X/TLuq/2XGbx8e00nQ0fB1Jo6oHgBIlR2PMsSfeO12i\nZUMNjTGeSbQmtiVIY4xn4r3TJVqWII0xnon323aiZQnSGOOZRLsGmXBjsY0xxitWgzTGeMY6aYwx\nxoV10hhjjAurQRpjjItE66SxBGmM8Yw9tMsYY1wkVnq0BGmM8VCiXYO0+yCNMZ6J1SMXRGSTiHwj\nIktEZIFTVk1EporIWufniU65iEg/EVknIktF5PzSnk9c1yDzp52y+BY/CPnTjgUl6PMvjRjf5nO5\nqn4X8fujwHRV7es8yuVR4BHgaqCBszQD3nB+Rs1qkMYYz/j80K4OQP6U7kOB6yLKh2nYHKCqiNQu\nTYC4rkGW1Qljy3r8eJkwN+j4v6t5YSDxl+6YU+ptS3Obj4h0Jfzsqnz9VbX/r3YNU0REgTed92uq\nahaAqmaJSP6zd1OBLRHbZjplWdEeW1wnSGPMsaU0TWwn2R2ZEI/UXFW3OUlwqoisKmLdwmYoL1VV\n1RKkMcYzserFVtVtzs+dIjIaaArsEJHaTu2xNpD/qMhMID1i8zSgVBd07RqkMcYzsXiqoYhUFJHK\n+a8JPz56GTAW6Oys1hkY47weC9zh9GZfCOzNb4pHy2qQxhjPxKgGWRMYLSIQzlnvqepkEZkPjBKR\nLsBm4EZn/YlAW2AdcAC4q7SBLUEaYzwTi7HYqroBOK+Q8u+BKwopV6CbF7GtiW2MMS6sBmmM8YxN\nVmGMMS5sujNjjHFhNUhjjHFhNUhjjHFhNUhjjHFhNUhjjHGRaDXIhL0PMhQKMX/eJ4wZPbT4lT3W\nOqMly5fNYtWK2fTq6cn9qhb/GInvZ+xQKMTIqUN57e0XAPj7S4/z/vRhfDDjbV58qw/lK5QH4PZ7\nOzF61nt8MONtBrz/GrXTasXsmLQU/8WzhE2QDz5wN6tWrfU9bigUot+rfWjX/jbOPe9ybrrpOs48\ns4HFLwPx/Y596z0d2bh2U8Hv/3rqFW684g5uaHU7WZk7uPmPNwCwatkabm59Fze0up2p42fwlydj\nl7hV86Je4llMEqSINBORE5zX5UWkt4iME5F/iEiVWMSMlJpam7ZXX8GgQcNjHepXmjZpxPr1m9i4\ncTPZ2dmMGjWGa9u3tvhlIL6fsWvWrkGLK5vz0btjC8r27ztQ8Pr48scV1M7mf7GIn/93EIClC5dT\ns/bJxIrPE+bGXKxqkIMIDxIHeBWoAvzDKRsco5gFXnqxN48+9hx5ef7/61QntRZbMg/NrJS5NYs6\ndWLXpLH48RPfz9i9nu3BS8/+m7wjamDPvPI3Pv1mAnXrn8Lwge//arvf39Ke2TO+iskxQWxm8wlS\nrBJkSFVznNeNVbWHqs5W1d7AqUVtKCJdRWSBiCzIy9sfdeBr2l7Jzp3fsWjxN6U47KPnzDhyGD//\nJ7D4wcX3K3aLq5qz+7s9rFy6+lfvPdWjD1ec156NazfRusOVh713zfWtOfu8Mxjyn3c9P6Z8VoMs\nmWUikj/F0Nci0hhARE4HsovaUFX7q2pjVW0cClWMOvDFFzemfbsM1q2Zw7vv/IfLL2/O0CH9ot5P\naW3NzCI97dBU/WmptcnK2mHxy0B8v2I3bPI7WmZcyqT5H/HP/z5L0+YX8Py/ny54Py8vj8ljpnPl\nNZcXlDW7tAn3dL+TBzv3IvuXIv8Ej4rVIEvmbuAyEVkPnAV8JSIbgAHOezHztyf6UvfUxtQ//UJu\nve3PfPrpF3S+88FYhjzM/AVLqF+/HnXrppOSkkLHjh0YN36KxS8D8f2K3e/5N7jq/A5c3eQP9Lrv\nSeZ9sZDH7+9Net20gnVaZlzCpnXfAnDGOafz1L968WDnnuz+bo/nxxMpTzXqJZ7F5D5IVd0L3OnM\nAnyqEydTVf2rSgQkNzeX7j2eYOKE90gKhRgydCQrVqyx+GUgfpCxRYTn+j1JpcoVEYHVy9fx3CP/\nBOChp+6nQsUKvDCgDwDbt+7gwc69YnIc8X7bTrQknqu4yeVSAzu4svxUwaDjx8tTBYOOH+hTDVUL\ne/BVsWpWOSPqv9kde1eVKpYfEvY+SGOMOVo21NAY45l475WOliVIY4xn4vmSXWlYgjTGeCbee6Wj\nZQnSGOMZq0EaY4wLuwZpjDEurAZpjDEu7BqkMca4SLSRNJYgjTGesRqkMca4SLRrkDbU0BjjmVg9\nk0ZE2ojIahFZJyKPxvg0ClgN0hjjmVjUIEUkCXgduArIBOaLyFhVXeF5sCNYDdIY45kYTZjbFFin\nqhtU9RdgBNAhpifiiOsaZP60Txbf4pfF+Et3zAk0fmnE6ApkKrAl4vdMoFlsQh0urhNkaeekyyci\nXVW1v1eHc6zEtvgWP6j4Ob9sjfpvVkS6Al0jivofceyF7dOX3qBEb2J3LX6VhIxt8S1+0PFLLPI5\nVM5yZGLPBNIjfk8DfKneJ3qCNMYc++YDDUSknoiUAzoBY4vZxhPx3cQ2xpR5qpojIvcDnwBJwCBV\nXe5H7ERPkIFdAwo4tsW3+EHH95SqTgQm+h03rh/aZYwxQbJrkMYY48ISpDHGuEjIBBnUuE0n9iAR\n2Skiy/yMGxE/XUQ+FZGVIrJcRLr7HP94EZknIl878Xv7Gd85hiQRWSwi4/2O7cTfJCLfiMgSEVng\nc+yqIvKBiKxy/h+4yM/4iSbhrkE64zbXEDFuE7jZj3GbTvwWwD5gmKqe40fMI+LXBmqr6iIRqQws\nBK7z8fwFqKiq+0QkBZgNdFdV34aFiMhDQGPgBFVt51fciPibgMaq+l0AsYcCn6vqW84tMRVU9Qe/\njyNRJGINMrBxmwCqOgvY7Ve8QuJnqeoi5/VPwErCQ7X8iq+qus/5NcVZfPtXWETSgGuAt/yKGS9E\n5ASgBTAQQFV/seR4dBIxQRY2btO3BBFPRKQu0AiY63PcJBFZAuwEpqqqn/FfAXoBeT7GPJICU0Rk\noTOMzi+nAruAwc4lhrdEpKKP8RNOIibIwMZtxhMRqQR8CPRQ1R/9jK2quarakPCQsKYi4sulBhFp\nB+xU1YV+xCtCc1U9H7ga6OZcdvFDMnA+8IaqNgL2A75eg080iZggAxu3GS+ca38fAu+q6kdBHYfT\nvJsJtPEpZHPgWuca4AiglYi841PsAqq6zfm5ExhN+LKPHzKBzIga+weEE6YppURMkIGN24wHTifJ\nQGClqr4UQPwaIlLVeV0euBJY5UdsVX1MVdNUtS7h732Gqt7mR+x8IlLR6RzDad5mAL7c0aCq24Et\nIvJbp+gKwJfOuUSVcEMNgxy3CSAiw4GWQHURyQSeVtWBfsUnXIu6HfjGuQ4I8LgzVMsPtYGhzt0E\nIWCUqgZyu01AagKjw/9OkQy8p6qTfYz/APCuUznYANzlY+yEk3C3+RhjjFcSsYltjDGesARpjDEu\nLEEaY4wLS5DGGOPCEqQxxriwBGniiojUDWomJGOOZAnS+MK5L9KYY4olSFMoEXk2ci5JEekjIg8W\nsl5LEZklIqNFZIWI/FdEQs57+0TkGRGZC1wkIheIyGfOJA6fOFOz4ZR/LSJfAd38OkdjimMJ0rgZ\nCHQGcBJeJ+Bdl3WbAg8D5wKnAX9wyisCy1S1GeEZhV4DblDVC4BBQB9nvcHAg6pqk7uauJJwQw2N\nN1R1k4h8LyKNCA+fW6yq37usPk9VN0DBUMtLCE+UkEt40gyA3wLnAFOdYXhJQJaIVAGqqupnznpv\nE54Fx5jAWYI0RXkLuBOoRbjG5+bI8ar5v/+sqrnOawGWH1lLdCa2sPGuJi5ZE9sUZTThqcqaEJ78\nw01TZ/akEHAT4ccsHGk1UCP/GSkikiIiZztTou0VkUuc9W717vCNOTpWgzSuVPUXEfkU+CGiJliY\nr4C+hK9BziKcWAvb1w1AP6dZnUx49u/lhGecGSQiByg6ERvjK5vNx7hyaoSLgBtVda3LOi2Bvwbx\ncCxjYs2a2KZQInIWsA6Y7pYcjUl0VoM0JSIi5xLuYY500LmFx5iEZAnSGGNcWBPbGGNcWII0xhgX\nliCNMcaFJUhjjHFhCdIYY1z8P2QIZHTK6RiuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xg = xgb.XGBClassifier(learning_rate= 0.7340229699980686, n_estimators = 70, max_depth = 14)\n",
    "xg.fit(X_train,y_train)\n",
    "xg_score=xg.score(X_test,y_test)\n",
    "y_predict=xg.predict(X_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of XGBoost: '+ str(xg_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of XGBoost: '+(str(precision)))\n",
    "print('Recall of XGBoost: '+(str(recall)))\n",
    "print('F1-score of XGBoost: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "xg_train=xg.predict(X_train)\n",
    "xg_test=xg.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply RF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of RF: 0.9940298507462687\n",
      "Precision of RF: 0.9940428718755328\n",
      "Recall of RF: 0.9940298507462687\n",
      "F1-score of RF: 0.9940328670009781\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.99      1.00      3645\n",
      "           1       0.99      1.00      1.00       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       1.00      0.99      0.99       609\n",
      "           4       0.71      0.71      0.71         7\n",
      "           5       0.99      1.00      0.99       251\n",
      "           6       0.98      0.99      0.99       436\n",
      "\n",
      "    accuracy                           0.99      5360\n",
      "   macro avg       0.95      0.96      0.95      5360\n",
      "weighted avg       0.99      0.99      0.99      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XecFPX9x/HXZ+8OpSiIIHAlgoK9\nAAJqsCBKEUGMBTWxJRpMglGiscQaNBhirCTG/EAQMEqxIEhHFJEocDSRzlGEg6MoNiDilc/vj507\nFry5u12mLHufp495sPfdmX3PLN6H77TviKpijDHmxyJhr4AxxiQrK5DGGOPCCqQxxriwAmmMMS6s\nQBpjjAsrkMYY48IKpDHGuLACaYwxLqxAGmOMi/SwV6BCInabjzFhUJVEFiv8Yl3cv7MZDY5LKCsI\nSV0gC3esDS07o+HxpGdkhpZfVLil2uYXFW4BsPyQ802SF0hjzCGmpDjsNfCUFUhjjHe0JOw18JQV\nSGOMd0qsQBpjTLk0xXqQdpmPMcY7JSXxT5UQkcNFZJ6IfCoiy0Skn9M+TETWi8hiZ2rptIuIDBSR\nPBFZIiKtYz7rZhFZ40w3V5ZtPUhjjHf86UHuBTqq6i4RyQBmi8hk5717VfXNA+a/FGjhTGcDLwFn\ni0h94DGgDaDAAhEZr6pfuQVbD9IY452S4vinSmjULufHDGeq6HrLnsAIZ7k5QD0RaQJ0Aaar6k6n\nKE4HulaUbQXSGOMdLYl/qgIRSRORxcB2okVurvNWf2c3+jkROcxpywI2xSye77S5tbuyAmmM8U4C\nxyBFpLeIzI+Zeh/4saparKotgWygnYicBvwJOAloC9QH7ndmL+/OHK2g3ZUVSGOMZ1RLEph0kKq2\niZkGuX++fg3MBLqqaoGzG70XeAVo58yWD+TELJYNbKmg3ZUVSGOMd/w5i91QROo5r2sClwArneOK\niIgAVwBLnUXGAzc5Z7PPAb5R1QJgKtBZRI4SkaOAzk6bKzuLbYzxjj9nsZsAw0UkjWinboyqThCR\n90WkIdFd58XAb5z5JwHdgDxgD/BLAFXdKSJPALnOfI+r6s6Kgq1AGmO848O92Kq6BGhVTntHl/kV\n6OPy3lBgaFWzD6kCuXfvD9zc515+KCykuKiYThedxx233YiqMnDQcKZ9MJtIJMK1P7uMG67pyYSp\n7zPktTcAqFWzJo/88Q5OanEcAN9+t4vHBjxP3rrPQYQnHvwDLU87+aDX8bDDDmPm+29R47DDSE9P\n4+23J9Lv8WcO+nPj0aVzB5599nHSIhGGvjKSp/7+YmDZYW9/dnYmw4a+QKPGDSkpKeHll1/jH/8c\nElh+2Nt/152/5le/uh5VZenSldx6293s3bs3sPxUuxdbosU2OR04tpyq8r//fU+tWjUpLCript/+\nkQfuup11n29i3sIl9H/obiKRCF9+9TVHH1WPRZ8t57hjc6h75BF89Eku/xr6GiMHPw/Ag088Tesz\nT+Pqy7tSWFjI/77fy5FH1CnLOpjhzmrXrsXu3XtIT09n1syx/OHux5g7b2Fcn5HocGORSIQVyz6i\na7fryc8vYM4nk7jhxt+xYsWaQPLh4Lf/YIb7atz4GJo0PoZFi5dSp05t5s2dwlVX/yqu7T/Y4cbC\n2v7MzMZ8+MFYTj/zIr7//ntGvv5vJk9+nxGvjonrc4oKtyQ8HuTeFR/EXVAOO/mi6jcepIicRPSC\nzSyip9K3AONVdcVBfCa1atUEoKioiKKiIkSE0WMn8tSf7ycSiZ5zOvqoegC0Ov2UsmXPOPUktm3/\nAoBdu3ez4NOl9H/4HgAyMjLIyMhIdLV+ZPfuPc7nppOekUGQ/wi1a9uKtWs3sH79RgDGjBnH5T26\nxF0gD0aY279163a2bt0OwK5du1m5cg1ZmY2rzfanp6dTs+bhFBYWUqtmTQoKtgaWDaTcYBW+nMUW\nkfuBUUQPns4jelBUgJEi8sDBfHZxcTFX3dyHC7pfz7ltW3HGqSexaXMBk2d8SK9f3clv7nmEzzdt\n/tFyb0+YynnntAEgf/NWjqpXl4f7P8vVt/Th0b8+z57/fX8wq7WfSCTC/NxpFGxewowZs5iXu8iz\nz65MZlZjNuXvu3Ihf3MBmZmNA8uHcLc/1rHHZtPyzNOYOy/Y/LC2f8uWrTz73L9Zv3Ye+RsX8c23\n3zL9vVmBZJfx6ULxsPh1mc+tQFtVHaCq/3GmAUSvU7r1YD44LS2Nt4a/yIyxr/LZ8tWsWbeBHwoL\nOaxGDcYMHchVPbryyJPP7bfMvAWf8vaEadz9u18BUFRczIrVeVz7s8t4c9iL1Kx5OEPi3A2pSElJ\nCW3adubYZm1o26YVp556omefXZnoFQ/7C/owSpjbX6p27VqMGT2Yu//4GN99t6vyBTwU1vbXq1eX\ny3t0ofkJ55BzbGtq167Fz39+ZSDZZXy4zCdMfhXIEqC8AyhNnPdcxV5V//KIka7zHXlEHdq2PoPZ\nc+bTuGEDOnU4D4BLLvwpq9euL5tvVd56Hh3wPP8Y8Cj16h4JQONjGtCoYQPOOPUkADp3OI/lq/Pi\n28Iq+Oabb/lw1sd06dzB8892szm/gJzsfV99dlYTCgq2BZYfK4zth+hu5hujBzNy5FjeeWdy5Qv4\nJOjtv/ji81m/YSNffLGToqIixr4zmXOdvaagqBbHPSUzvwpkX2CGiEwWkUHONAWYAdxV0YKxV9Xf\ndtP1+72386uv+dbpDXy/dy9zchfR7NgcOl5wLnMXLAYgd9FnHJsTvb2yYOt2+j74BH999F6a/iS7\n7HMaHF2fxsc0ZP3n+QDMWbCY45v+xJMNb9CgPnWdQnz44YdzccfzWbUquGfr5M5fTPPmzWjaNIeM\njAx69erJuxOmBZYf9vYDDB70DCtW5vH8C643ZPgmzO3ftHEzZ5/dmpo1Dweg40XnsXJlcMdegZTb\nxfblJI2qThGRE4juUmcRPf6YD+TqQfyTsePLr3joL09TXFKClihdOp5Ph/Zn0/qMU7m/31O8Ovod\natU8nH4P9AXgpVde55tvv+MvT0cvc0lLS2PM0IEAPPiH33J/v6coLCokJ7MJTzz4h4PbaEeTJo0Y\nOuR50tIiRCIR3nzzXSZOes+Tz66K4uJi7ur7MJMmvk5aJMKw4aNZvnx1YPlhb3/7n7blxhuuZsln\ny5mfG/2H4ZFHBjB5yvuB5Ie5/fNyF/H22xPJnTeVoqIiFi9exuCXXwsku0yS7zLH65C6zCdI9lRD\ne6phtc5P8DKf7xe8E/fv7OFnXVH9LvMxxlRD9lRDY4xxkeTHFONlBdIY450UOwZpBdIY4x3rQRpj\njAvrQRpjjAsrkMYYU75kvzMmXlYgjTHesR6kMca4SLGTNPbQLmOMcWE9SGOMd2wX2xhjXKTYLrYV\nSGOMd6wHaYwxLqwHGZyMhseHml867JTlW351zE+I9SCNMcaFFcjghD1g7MkN24aWv2JHbujbX60H\njLX8xNgutjHGuLAepDHGuLAepDHGuEixHqTdamiM8Y4Pj30VkcNFZJ6IfCoiy0Skn9PeTETmisga\nERktIjWc9sOcn/Oc95vGfNafnPZVItKlsmwrkMYY75SUxD9Vbi/QUVXPBFoCXUXkHOBvwHOq2gL4\nCrjVmf9W4CtVbQ4858yHiJwCXAecCnQF/iUiaRUFW4E0xnjHhwKpUbucHzOcSYGOwJtO+3DgCud1\nT+dnnPcvFhFx2kep6l5VXQ/kAe0qyrYCaYzxjmrck4j0FpH5MVPvAz9WRNJEZDGwHZgOrAW+VtUi\nZ5Z8IMt5nQVsiq6OFgHfAEfHtpezTLnsJI0xxjsJnKRR1UHAoErmKQZaikg9YCxwcnmzOX+Ky3tu\n7a6sQBpjvOPzWWxV/VpEZgLnAPVEJN3pJWYDpVe45wM5QL6IpAN1gZ0x7aVilymX7WIbY7zjz1ns\nhk7PERGpCVwCrAA+AK52ZrsZGOe8Hu/8jPP++6qqTvt1zlnuZkALYF5F2daDNMYkuybAcOeMcwQY\no6oTRGQ5MEpE/gIsAoY48w8BXhWRPKI9x+sAVHWZiIwBlgNFQB+t5CljViCNMd7xYRdbVZcArcpp\nX0c5Z6FV9XvgGpfP6g/0r2q2FUhjjHe0wnMeh5yUPAbZpXMHli2dxcrls7nv3j6+ZNQ4rAajp7zC\n2A9e491Zo7jjvl8DcPZ5bXjrvRGM/3Akf/3HY6SlRa9D7dj1At6Z+Rpvv/8f3pg2nNZnn+nLekEw\n22/5yZedDPk+XSgeGtEkrvjpNbLiXrlIJMKKZR/Rtdv15OcXMOeTSdxw4+9YsWJNXJ9TleHOatWu\nyZ7d/yM9PY3/vDuYAY8+x7ODnuRXV/Vhw7qN/P7+3mzZtJW3Xh9fNi/ACac057nBT3JZ+16un53o\ncGdebn9Y+Qcz3FeY+V5+96Hnq5Z3SUyl/jfkj3H/zta89emEsoKQcj3Idm1bsXbtBtav30hhYSFj\nxozj8h6V3nKZkNKCl56RTkZGOiXFJfzwww9sWLcRgI9nzqNz94v2mxegVq2a+PUPU5Dbb/nJk50M\n+YAvZ7HDFEqBFJFf+vXZmVmN2ZS/79Km/M0FZGY29iUrEonw9vv/YfbyqXz84TyWLFxGRno6p54Z\nvYa1c4+ONM5sVDb/Jd06MPG/Y3jptWd5uO9ffFmnILff8pMnOxnyAbRE456SWVg9yH5ub8TedlRS\nsjvuD47ecrk/v3prJSUlXNnxBi46szuntzqFFicdxz23P8wDT/yB0VNeYc+uPRQV77uK4L1JM7ms\nfS9+f/N93PnA7b6sU5Dbb/nJk50M+UDKHYP07Sy2iCxxewto5PLefrcdJXIMcnN+ATnZ+47dZGc1\noaBgW7wfE5fvvt3FvI8Xcl7Hc3nlX69x4+XRW0l/2uFsjj3+Jz+af/6cReQcm029+nX5euc3nq5L\nGNtv+eFnJ0M+kPS7zPHyswfZCLgJ6FHO9KVfobnzF9O8eTOaNs0hIyODXr168u6EaZ7nHHV0PY44\nsg4Ahx1+GOde0I71az6nfoOjAMiokcFtd9zE6OFvA/CTZtlly55y+olk1Ej3vDhCcNtv+cmVnQz5\nAJRo/FMS8/M6yAlAHVVdfOAbzr2UviguLuauvg8zaeLrpEUiDBs+muXLV3ue07BRA+cynggRiTBl\n/HvMnD6bPz72ezp0Oo9IJMKoYW8xd/Z8ADp370jPa7pRWFTE3u/3cnfvhzxfJwhu+y0/ubKTIR9I\n+l3meKXcZT5esaca2lMNq3V+gpf57HnhN3H/zta6699Je5mP3UljjPFOEne4EmEF0hjjnRTbxbYC\naYzxTpKfdImXFUhjjHdS7DIfK5DGGO+kWA8y5e7FNsYYr1gP0hjjGbWTNMYY4yLFdrGtQBpjvGMn\naYwxxoX1II0xxoUdgzTGGBfWgzTGGBd2DNIYY1xYDzI4pcM+hWXFjtxQ88Pefsuv3vmJsOsgA1Rd\nx0Mszc8IMb/QxoOs9vkJsR6kMca4sAJpjDEuUuwkjQ1WYYzxjg8P7RKRHBH5QERWiMgyEbnLaf+z\niGwWkcXO1C1mmT+JSJ6IrBKRLjHtXZ22PBF5oLJs60EaYzyj/uxiFwH3qOpCETkCWCAi0533nlPV\np2NnFpFTgOuAU4FM4D0ROcF5+0WgE5AP5IrIeFVd7hZsBdIY4x0fCqSqFgAFzuvvRGQFkFXBIj2B\nUaq6F1gvInlAO+e9PFVdByAio5x5XQuk7WIbY7xTUhL3JCK9RWR+zNTb7eNFpCnQCpjrNN0hIktE\nZKiIHOW0ZQGbYhbLd9rc2l1ZgTTGhEpVB6lqm5hpUHnziUgd4C2gr6p+C7wEHA+0JNrDfKZ01vJi\nKmh3ZbvYxhjv+HSZj4hkEC2Or6nq2wCqui3m/cHABOfHfCAnZvFsoPTiTrf2clkP0hjjHX/OYgsw\nBFihqs/GtDeJme1nwFLn9XjgOhE5TESaAS2AeUAu0EJEmolIDaIncsZXlG09SGOMZ1R96UG2B24E\nPhORxU7bg8D1ItKS6G7yBuB2Zx2WicgYoidfioA+qloMICJ3AFOBNGCoqi6rKNgKpDHGO/6cxZ5N\n+ccPJ1WwTH+gfzntkypa7kBWII0x3rFbDY0xpnw+XSgeGiuQxhjvpFiBTMmz2F06d2DZ0lmsXD6b\n++7tk/L5gwc9w+b8T1m0aEZZ2xlnnMJHs8azaOF7jB07jCOOqOP7epSqbt9/smQnQz4lCUxJLOUK\nZCQSYeAL/ene4wZOP/Mirr32Ck4+uUVK5w8fMYbu3X+xX9v//fvvPPjQk7RqfQnj3pnMPff81td1\nKFUdv/9kyE6GfIjuYsc7JTPfCqSInCQiFztXv8e2d/UrE6Bd21asXbuB9es3UlhYyJgx47i8R5fK\nFzyE82fPnsvOr77er+2EE47no4/mAPDejI/42c+6lbeo56rj958M2cmQD/hyHWSYfCmQInInMA74\nPbBURHrGvP2kH5mlMrMasyl/38Xx+ZsLyMxs7GdkUuWXWrZsFT16dAbg6qu6k5MdzOjUYW9/mPnV\nedvL2C52lfwaOEtVrwA6AI+UjuFG+dczlYm9cb2kZHfcwdGL7vfn08WrSZlf6te97+a3v7mFuXMm\nU+eI2vzwQ2EguWFvf5j51Xnby/JSbBfbr7PYaaq6C0BVN4hIB+BNETmWSgqkc6P6IID0Gllxf3ub\n8wv26y1lZzWhoGBbBUt4K+z8UqtWraXbZT8HoEWL4+h26cWB5Ia9/WHmV+dtL5PkPcJ4+dWD3Orc\nAgSAUyy7Aw2A033KBCB3/mKaN29G06Y5ZGRk0KtXT96dMM3PyKTKL9Ww4dFAtFfx4J/uYtCgVwPJ\nDXv7w8yvztteynqQVXMT0Xsgy6hqEXCTiPyfT5kAFBcXc1ffh5k08XXSIhGGDR/N8uWr/YwMPf/V\nV1/kwgvOpUGD+qxfN5/HH3+aOnVq85vf3gLAO+9MYtjw0b6uQ6nq+P0nQ3Yy5AMp14OUMI6PVVUi\nu9hesce+2mNfq3W+aoWHwtx82ePCuH9nj373w4SygpBy10EaY4xX7FZDY4x3UmwX2wqkMcYzKfZY\nbCuQxhgPWYE0xpjyWQ/SGGNcWIE0xhgXViCNMcZNYpdPJi0rkMYYz1gP0hhjXGiJ9SCNMaZc1oM0\nxhgXCd7CnbSsQBpjPGM9SGOMcWHHIANUOuxTdc0vrObbb/nh5iciiUdPTEhSF8iwx2Os7vkdsoJ5\nTMOBZm6OPt+7Wo/HmAT5ibAepDHGuEi1AmkD5hpjkpqI5IjIByKyQkSWlT4hVUTqi8h0EVnj/HmU\n0y4iMlBE8kRkiYi0jvmsm53514jIzZVlW4E0xnhGNf6pCoqAe1T1ZOAcoI+InAI8AMxQ1RbADOdn\ngEuBFs7UG3gJogUVeAw4G2gHPFZaVN1YgTTGeEZLJO6p0s9ULVDVhc7r74AVQBbQExjuzDYcuMJ5\n3RMYoVFzgHoi0gToAkxX1Z2q+hUwHehaUbYVSGOMZ1Ql7klEeovI/Jipt9vni0hToBUwF2ikqgXR\nXC0AjnFmywI2xSyW77S5tbuq9CSNiDQCngQyVfVSp2t7rqoOqWxZY0z1ksiF4qo6CBhU2XwiUgd4\nC+irqt+KuPY+y3tDK2h3VZUe5DBgKlB6zcFqoG8VljPGVDMlKnFPVSEiGUSL42uq+rbTvM3Zdcb5\nc7vTng/kxCyeDWypoN1VVQpkA1Udg/O0CVUtAoqrsJwxpppJZBe7MhLtKg4BVqjqszFvjQdKz0Tf\nDIyLab/JOZt9DvCNsws+FegsIkc5J2c6O22uqnId5G4RORqnK1oaWIXljDHVjE/XQbYHbgQ+E5HF\nTtuDwABgjIjcCmwErnHemwR0A/KAPcAvAVR1p4g8AeQ68z2uqjsrCq5KgbybaEU+XkT+CzQErq7i\nhhljqhE/bjVU1dmUf/wQ4Ee3e6mqAn1cPmsoMLSq2ZUWSFVdKCIXAic6K7lKVQurGmCMqT5S7U6a\nqpzFvumAptYigqqO8GmdjDGHqKqedDlUVGUXu23M68OJdmkXAlYgjTH7qXYD5qrq72N/FpG6wKu+\nrZEx5pCVasOdJXInzR6i9zgmrbzVc1i08D3m505jzieTAs/v0rkDy5bOYuXy2dx3b7nHilMiv86R\nten3f48yYuZQhn8whFNan8wR9Y7g6df/xn8+GsbTr/+NOnXr7LfMiWeeyIzPp3LhZef7tl5hfv/V\n5e/ejV/XQYal0gIpIu+KyHhnmgCsYt/1Rknrkk7X0KZtZ845t1uguZFIhIEv9Kd7jxs4/cyLuPba\nKzj55OD+PQky/45+fZg3M5ebOvyKWzvfzsa8jfy8z3Us/O8ibjj/Fhb+dxE/73Pdfut2+4O3kfvh\nfF/WpzQjrO+/Ov3du/HjOsgwVaUH+TTwjDP9FbhAVR+oeBEQkXYi0tZ5fYqI3C0iwVarELRr24q1\nazewfv1GCgsLGTNmHJf36JJy+bXq1OLMs09n4sjJABQVFrHr29207/xTprwxDYApb0zjvC7ty5a5\n8pdXMGvSR3z9xdeer0+pML//6vJ3XxGfRvMJTYUFUkTSgEdU9UNn+q+q5lf2oSLyGDAQeElE/gr8\nE6gDPCAiD3mx4hVRVSZPGsncOZO57dZf+B23n8ysxmzK33f3Uv7mAjIzG6dcfuZPmvD1zm944Nl7\nGTzl39z797s5vObh1G9wFDu3R6+93bl9J0cdXQ+ABo2P5rxL2zP+1Qmer8t+6xXi919d/u4rkmq7\n2BWepFHVYhHZIyJ1VTWeu2euBloChwFbgWzn5vK/Ex2Fo7/bgs5IHr0BJK0ukUjtOGKjLuhwBQUF\n22jY8GimTB7FqlV5fDR7btyfk4jybqDXAP+ZDCo/LT2NE05rwcBH/smKRSu5o9/v9tudPtAdf/4d\ng558mZISfx97F+b3X13+7iuS7LvM8arKZT7fE73FZzqwu7RRVe+sYJkiVS0G9ojIWlX91lnmfyJS\n4W9I7Mge6TWyEvrbLSjYBsCOHV8ybtxk2rZtGViB3JxfQE72vmeJZGc1KVufVMrfUbCDHQU7WLFo\nJQAfTpzFz/tcz84vvqL+MfXZuX0n9Y+pz1dfRnenTzzjBB59MbrzULd+Xc7u2I7iomJmT/3Y0/UK\n8/uvLn/31UlVjkFOBB4BZgELnKmyo+w/iEgt5/VZpY3OJUK+diFq1apJnTq1y153uuRCli1b5Wfk\nfnLnL6Z582Y0bZpDRkYGvXr15N0J01Iuf+eOr9i+ZQc5x2UDcNZ5rfl8zed8PP0Tul7TGYCu13Tm\nv9OiBfD6n97IdefewHXn3sCHE2fx/EMDPS+OEO73X13+7itSrXaxHfVU9YXYhtJnQlTgAlXdC6C6\n3whxGewbfcMXjRo15M03okNVpqenMWrUO0ydNtPPyP0UFxdzV9+HmTTxddIiEYYNH83y5atTMn/g\nI//k4X/8ifQaGRR8XsCAe/5ORCI89u+H6XZdV7Zt3s6ff/OEL9luwvz+q9PfvZskP+cSN6nsGIWI\nLFTV1ge0LVLVVr6uGYnvYnshGR67Gna+Pfa1GucneDDx4yZXxf07+9OCt5K2G+nagxSR64GfA81E\nZHzMW0cAX/q9YsaYQ091OknzMVAANCB6DWSp74Alfq6UMebQ5O81CsFzLZCq+jnwOXBuRR8gIp+o\naoXzGGOqB3UdtvHQVJWTNJU53IPPMMakgJIUO0vjRYFMsa/EGJOoEutBGmNM+VJtF7sqo/nc4TwB\nzHUWD9fHGHMIK0lgSmZVuZOmMZArImNEpKv8+IbPG31YL2PMIUiRuKdkVmmBVNWHiQ6QOwS4BVgj\nIk+KyPHO+0t9XUNjzCGjOvYgSx+juNWZioCjgDdF5Ckf180Yc4hJtQJZlaca3kn0/ukvgJeBe1W1\nUEQiwBrgPn9X0RhzqEj2XeZ4VeUsdgPgSufC8TKqWiIi3f1ZLWPMoSjFHotdpacaPlrBeyu8XR1j\nzKEs1a6DTOSphsYYUy0k9YXipcM+WX44SocdC0vY21/d8xORarfVJXWBDHs8RMuvxuMhWn5C/Dgr\nLSJDge7AdlU9zWn7M/BrYIcz24OqOsl570/ArUAxcKeqTnXauwIvAGnAy6o6oLLspC6QxphDS0k5\nDw7zwDCiT0YdcUD7c6r6dGyDiJwCXAecCmQC74nICc7bLwKdgHyiN7+MV9XlFQVbgTTGeMaPXWxV\nnSUiTas4e09glPPIl/Uikge0c97LU9V1ACIyypm3wgJpJ2mMMZ5J5EJxEektIvNjpt5VjLtDRJaI\nyNCY8SKygE0x8+Q7bW7tFbICaYzxTInEP6nqIFVtEzMNqkLUS8DxQEuiTz4ofepBefv4WkF7hWwX\n2xjjmaCug1TVsgd+i8hgYILzYz6QEzNrNlB61smt3ZX1II0xntEEpkSISJOYH38GlA6aMx64TkQO\nE5FmRAfamQfkAi1EpJmI1CB6Iif2YYTlsh6kMcYzftxqKCIjgQ5AAxHJBx4DOohIS6I1dgNwO4Cq\nLhORMURPvhQBfVS12PmcO4CpRC/zGaqqyyrLtgJpjPGMH9dBqur15TQPqWD+/kD/ctonAZPiybYC\naYzxjN1JY4wxLqrdaD7GGFNVyT4AbrysQBpjPGMF0hhjXGiK7WKn3HWQ2dmZvDftDT5bMpNPF7/P\n7++4NfB16NK5A8uWzmLl8tncd28fyw9Y3uo5LFr4HvNzpzHnk7hOWh60sLc97Pxq90yaQ01RURH3\n3tePRYuXUqdObebNncJ7M2axYsWaQPIjkQgDX+hP127Xk59fwJxPJvHuhGmWH1B+qUs6XcOXX34V\naGbY2x52fipKuR7k1q3bWbQ4elH9rl27WblyDVmZjQPLb9e2FWvXbmD9+o0UFhYyZsw4Lu/RxfKr\ngbC3Pex8SL0eZGAFUkQOHMvNd8cem03LM09j7rxFgWVmZjVmU/6+WzzzNxeQGWCBru75AKrK5Ekj\nmTtnMrfd+ovAcsPe9rDzIbhbDYPiyy62iBx4j6MAF4lIPQBVvdyP3Fi1a9dizOjB3P3Hx/juu11+\nx5WRcgYMjT5W3PKDckGHKygo2EbDhkczZfIoVq3K46PZc33PDXvbw84Huw6yqrKJ3gv5MvuGGmrD\nviGJXDljwfUGkLS6RCK14w6+ZffUAAASfklEQVRPT0/njdGDGTlyLO+8Mznu5Q/G5vwCcrL3DZWf\nndWEgoJtFSxh+V4rzdux40vGjZtM27YtAymQYW972PmQ/LvM8fJrF7sNsAB4CPhGVWcC/1PVD1X1\nw4oWjB0bLpHiCDB40DOsWJnH8y9UZVg5b+XOX0zz5s1o2jSHjIwMevXqybsTpll+QGrVqkmdOrXL\nXne65EKWLVsVSHbY2x52PqTeMUhfepCqWgI8JyJvOH9u8yvrQO1/2pYbb7iaJZ8tZ35u9H+ORx4Z\nwOQp7wcRT3FxMXf1fZhJE18nLRJh2PDRLF++OpBsy4dGjRry5hvRcQzS09MYNeodpk6bGUh22Nse\ndj4k/zHFeEkQxyhE5DKgvao+GM9y6TWyQvu+q/NTBcPOT5an+lXrfE3sku+njr0h7t/Z+z7/T9Ie\nuQykV6eqE4GJQWQZY8KT7LvM8Uq5C8WNMeFJtV1sK5DGGM+UpFiJtAJpjPGM7WIbY4yL1Oo/WoE0\nxnjIepDGGOPCbjU0xhgXdpLGGGNcpFZ5TMHxII0xxivWgzTGeMZO0hhjjAs7BmmMMS5SqzxagTTG\neMh2sQNUOuyT5Vu+5R8aUm0X285iG2M848dDu0RkqIhsF5GlMW31RWS6iKxx/jzKaRcRGSgieSKy\nRERaxyxzszP/GhG5uSrbk9Q9yOo6YGx1z0+KAWOBrHqnhJK/+evlQPjbnwifdrGHAf8EYp+M+gAw\nQ1UHiMgDzs/3A5cCLZzpbOAl4GwRqQ88RvRxMAosEJHxqlrhw9OtB2mM8Ywm8F+ln6k6C9h5QHNP\nYLjzejhwRUz7CI2aA9QTkSZAF2C6qu50iuJ0oGtl2VYgjTGeSeShXSLSW0Tmx0y9qxDVSFULAJw/\nj3Has4BNMfPlO21u7RVK6l1sY8yhJZGTNKo6CPDqEaTlDZehFbRXyHqQxhjP+HGSxsU2Z9cZ58/t\nTns+kBMzXzawpYL2ClmBNMZ4pgSNe0rQeKD0TPTNwLiY9pucs9nnAN84u+BTgc4icpRzxruz01Yh\n28U2xnjGj7PYIjIS6AA0EJF8omejBwBjRORWYCNwjTP7JKAbkAfsAX4JoKo7ReQJINeZ73FVPfDE\nz49YgTTGeKYqZ6Xj/kzV613euriceRXo4/I5Q4Gh8WRbgTTGeMZuNTTGGBd+9CDDZCdpjDHGhfUg\njTGesV1sY4xxUaKptYttBdIY45nUKo8peAxy8KBn2JL/KYsXzQhtHbp07sCypbNYuXw2991b7hUH\nln8I52dmNeaN8a8wc8543v94HLfefgMAd9//O+Yve59ps95i2qy36Njp/LJl7vjDbcxeMJlZ8yZw\nYcf2nq9TqbC/+wAvFA9EyhXIESPGcFn3X4SWH4lEGPhCf7r3uIHTz7yIa6+9gpNPbmH5KZRfVFRE\nv4efosM5l9Oj8/Xcctv1tDjxeAAGvzSCzhdcRecLruL96R8B0OLE4+l5ZTc6nns5v7j6dp58+mEi\nEe9/9cL+7sGf0XzClHIF8qPZc9n51deh5bdr24q1azewfv1GCgsLGTNmHJf36GL5KZS/fdsXLF2y\nAoDdu/awZvU6Gjc5xnX+Lt0uYtzbk/jhh0I2bdzMhnWbaHXW6Z6uE4T/3UNio/kks0AKpIicJyJ3\ni0jnIPLClJnVmE35++6Bz99cQGZmY8tP0fzsnExOO+NkFi1YAsAvf/1zps9+m2f+8QR16x4JQOMm\njdiyeWvZMgVbttK4SSPP1yXs7x5sF7tKRGRezOtfEx0N+AjgMWf035Ql8uNRlTTAM3uWH1x+rdq1\nGDzieR770wB2fbebEUNH89NWXel8/lVs37aDR/9yb6DrFPZ3D7aLXVUZMa97A51UtR/RETQqPEAY\nO3hmSclun1bPP5vzC8jJ3jdUfnZWEwoKtll+iuWnp6czePjzjH1jIpMnvAfAFzu+pKSkBFXlteFv\n0tLZjS7YspXMrH09uSaZjdm2dXu5n3swwv7uwXaxq/y5zrBCRwOiqjsAVHU3UFTRgqo6SFXbqGqb\nSKS2T6vnn9z5i2nevBlNm+aQkZFBr149eXfCNMtPsfxn/vE4eavXMehfw8vajmnUoOz1pd0vYdWK\nNQBMm/wBPa/sRo0aGeT8JItmx/+ERQs+83ydwv7uIdpjjXdKZn5dB1kXWEB0FF8VkcaqulVE6lD+\nyL6e+c+rL3LhBefSoEF9NqybT7/Hn+aVYaP8jNxPcXExd/V9mEkTXyctEmHY8NEsX77a8lMov+05\nrbn6up4sX7aKabPeAmDAE89zxVXdOOX0k1BV8jdu4f4//BmA1SvX8u47U/hgzniKi4p56N6/UFLi\nfd8p7O8eUu+xrxLw8aFaRJ8lsb4q86fXyArt267OTxUMO9+eapgETzVUTagj0+Mn3eP+nX134wRf\nO00HI9A7aVR1D1Cl4miMOfQk+0mXeNmthsYYz6TaLrYVSGOMZ5L9pEu8rEAaYzyT7JftxMsKpDHG\nM6l2DDLl7sU2xhivWA/SGOMZO0ljjDEu7CSNMca4sB6kMca4SLWTNFYgjTGesYd2GWOMi9Qqj1Yg\njTEeSrVjkHYdpDHGM349ckFENojIZyKyWETmO231RWS6iKxx/jzKaRcRGSgieSKyRERaJ7o9Sd2D\nLB12yvItPwylw46FJeztT4TPl/lcpKpfxPz8ADBDVQc4j3J5ALgfuBRo4UxnAy85f8bNepDGGM8E\n/NCunkDpkO7DgSti2kdo1Bygnog0SSQgqXuQ1XXA2OqenywD5oadf0ajc0LJX7JtTsLLJnKZj4j0\nJvrsqlKDVHXQjz4apomIAv/nvN9IVQsAVLVAREqfvZsFbIpZNt9pK4h33ZK6QBpjDi2J7GI7xe7A\ngnig9qq6xSmC00VkZQXzljdCeUJdVSuQxhjP+HUWW1W3OH9uF5GxQDtgm4g0cXqPTYDSR0XmAzkx\ni2cDCR3QtWOQxhjP+PFUQxGpLSJHlL4m+vjopcB44GZntpuBcc7r8cBNztnsc4BvSnfF42U9SGOM\nZ3zqQTYCxooIRGvW66o6RURygTEiciuwEbjGmX8S0A3IA/YAv0w02AqkMcYzftyLrarrgDPLaf8S\nuLicdgX6eJFtu9jGGOPCepDGGM/YYBXGGOPChjszxhgX1oM0xhgX1oM0xhgX1oM0xhgX1oM0xhgX\nqdaDTMnrIPNWz2HRwveYnzuNOZ9MCjy/S+cOLFs6i5XLZ3PfvZ5cr2r5h0h+kNmRSITR04fzj1ef\nBuDPzz7IGzNG8Ob7r/LMy/2pWasmAGed05LR04axMP8jOnW/yNd10gT+S2YpWSABLul0DW3aduac\nc7sFmhuJRBj4Qn+697iB08+8iGuvvYKTT25h+dUgP+jsX/y6F+vXbCj7+e+PPs81F9/E1R1vpCB/\nG9f/6moACjZv5eG7nmDy2Om+rUsp1ZK4p2TmS4EUkbNF5EjndU0R6Sci74rI30Skrh+ZyaJd21as\nXbuB9es3UlhYyJgx47i8RxfLrwb5QWY3atKQCy5pz9uvjS9r271rT9nrw2seVtY727JpK2tWrKWk\nxP9iFPCAub7zqwc5lOhN4gAvAHWBvzltr/iUWUZVmTxpJHPnTOa2W3/hd9x+MrMasyl/38hK+ZsL\nyMxsbPnVID/I7Pue6MuzT/yTkgN6YI8//xAffDaRps2PZeSQN3zJrogfo/mEya8CGVHVIud1G1Xt\nq6qzVbUfcFxFC4pIbxGZLyLzS0p2JxR+QYcraHd2V7r3uIHf/vYWzj8vocdRJMQZcWQ/Qf5PYPnh\n5QeVfUGn9uz84itWLFn1o/ce7dufi8/swfo1G+jS8xLPsytjPciqWSoipUMMfSoibQBE5ASgsKIF\nVXWQqrZR1TaRSO2EwgsKtgGwY8eXjBs3mbZtWyb0OYnYnF9ATva+ofqzs5qUrY/lp3Z+UNkt255B\nh87nMzn3bZ769xO0a38WT/7zsbL3S0pKmDJuBpdc5u8JmfJYD7JqbgMuFJG1wCnAJyKyDhjsvOeb\nWrVqUqdO7bLXnS65kGXLfvwvrV9y5y+mefNmNG2aQ0ZGBr169eTdCdMsvxrkB5U98MmX6NS6J5e2\nvZL7fvMI8/67gAfv6EdO0+yyeTp0Po8NeZ97nl2ZEtW4p2Tmy3WQqvoNcIszCvBxTk6+qvr+T3mj\nRg15840hAKSnpzFq1DtMnTbT79gyxcXF3NX3YSZNfJ20SIRhw0ezfPlqy68G+WFmiwh/GfgIdY6o\njQisWpbHX+5/CoBTW57M80MHcGS9I7iw03n89t7buPJCf47NJ/tlO/GSZO7iptfICm3lqvNTBcPO\nT5anCoadH+pTDVXLe/BVpRrVPSnu39lt36xMKCsIKXsdpDHGHCy71dAY45lkPysdLyuQxhjPJPMh\nu0RYgTTGeCbZz0rHywqkMcYz1oM0xhgXdgzSGGNcWA/SGGNc2DFIY4xxkWp30liBNMZ4xnqQxhjj\nItWOQdqthsYYz/j1TBoR6Soiq0QkT0Qe8HkzylgP0hjjGT96kCKSBrwIdALygVwRGa+qyz0PO4D1\nII0xnvFpwNx2QJ6qrlPVH4BRQE9fN8SR1D3I0mGfLN/yq2P+km1zQs1PhE9HILOATTE/5wOBPEcl\nqQtkomPSlRKR3qo6yKvVOVSyLd/yw8ov+mFz3L+zItIb6B3TNOiAdS/vMwM5G5Tqu9i9K58lJbMt\n3/LDzq+y2OdQOdOBhT0fyIn5ORsIpHuf6gXSGHPoywVaiEgzEakBXAeMr2QZTyT3LrYxptpT1SIR\nuQOYCqQBQ1V1WRDZqV4gQzsGFHK25Vt+2PmeUtVJwKSgc5P6oV3GGBMmOwZpjDEurEAaY4yLlCyQ\nYd236WQPFZHtIrI0yNyY/BwR+UBEVojIMhG5K+D8w0Vknoh86uT3CzLfWYc0EVkkIhOCznbyN4jI\nZyKyWETmB5xdT0TeFJGVzv8D5waZn2pS7hikc9/mamLu2wSuD+K+TSf/AmAXMEJVTwsi84D8JkAT\nVV0oIkcAC4ArAtx+AWqr6i4RyQBmA3epamC3hYjI3UAb4EhV7R5Ubkz+BqCNqn4RQvZw4CNVfdm5\nJKaWqn4d9HqkilTsQYZ23yaAqs4CdgaVV05+gaoudF5/B6wgeqtWUPmqqrucHzOcKbB/hUUkG7gM\neDmozGQhIkcCFwBDAFT1ByuOBycVC2R5920GViCSiYg0BVoBcwPOTRORxcB2YLqqBpn/PHAfUBJg\n5oEUmCYiC5zb6IJyHLADeMU5xPCyiNQOMD/lpGKBDO2+zWQiInWAt4C+qvptkNmqWqyqLYneEtZO\nRAI51CAi3YHtqrogiLwKtFfV1sClQB/nsEsQ0oHWwEuq2grYDQR6DD7VpGKBDO2+zWThHPt7C3hN\nVd8Oaz2c3buZQNeAItsDlzvHAEcBHUXkPwFll1HVLc6f24GxRA/7BCEfyI/psb9JtGCaBKVigQzt\nvs1k4JwkGQKsUNVnQ8hvKCL1nNc1gUuAlUFkq+qfVDVbVZsS/Xt/X1VvCCK7lIjUdk6O4ezedgYC\nuaJBVbcCm0TkRKfpYiCQk3OpKuVuNQzzvk0AERkJdAAaiEg+8JiqDgkqn2gv6kbgM+c4IMCDzq1a\nQWgCDHeuJogAY1Q1lMttQtIIGBv9d4p04HVVnRJg/u+B15zOwTrglwFmp5yUu8zHGGO8koq72MYY\n4wkrkMYY48IKpDHGuLACaYwxLqxAGmOMCyuQJqmISNOwRkIy5kBWIE0gnOsijTmkWIE05RKRJ2LH\nkhSR/iJyZznzdRCRWSIyVkSWi8i/RSTivLdLRB4XkbnAuSJyloh86AziMNUZmg2n/VMR+QToE9Q2\nGlMZK5DGzRDgZgCn4F0HvOYybzvgHuB04HjgSqe9NrBUVc8mOqLQP4CrVfUsYCjQ35nvFeBOVbXB\nXU1SSblbDY03VHWDiHwpIq2I3j63SFW/dJl9nqqug7JbLc8jOlBCMdFBMwBOBE4Dpju34aUBBSJS\nF6inqh86871KdBQcY0JnBdJU5GXgFqAx0R6fmwPvVy39+XtVLXZeC7DswF6iM7CF3e9qkpLtYpuK\njCU6VFlbooN/uGnnjJ4UAa4l+piFA60CGpY+I0VEMkTkVGdItG9E5Dxnvl94t/rGHBzrQRpXqvqD\niHwAfB3TEyzPJ8AAoscgZxEtrOV91tXAQGe3Op3o6N/LiI44M1RE9lBxITYmUDaaj3Hl9AgXAteo\n6hqXeToAfwzj4VjG+M12sU25ROQUIA+Y4VYcjUl11oM0VSIipxM9wxxrr3MJjzEpyQqkMca4sF1s\nY4xxYQXSGGNcWIE0xhgXViCNMcaFFUhjjHHx/xMWbxh54SxPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(random_state = 0)\n",
    "rf.fit(X_train,y_train) \n",
    "rf_score=rf.score(X_test,y_test)\n",
    "y_predict=rf.predict(X_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of RF: '+ str(rf_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of RF: '+(str(precision)))\n",
    "print('Recall of RF: '+(str(recall)))\n",
    "print('F1-score of RF: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter optimization (HPO) of random forest using Bayesian optimization with tree-based Parzen estimator (BO-TPE)\n",
    "Based on the GitHub repo for HPO: https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████| 20/20 [01:34<00:00,  4.72s/trial, best loss: -0.9955223880597015]\n",
      "Random Forest: Hyperopt estimated optimum {'n_estimators': 71.0, 'min_samples_leaf': 1.0, 'max_depth': 46.0, 'min_samples_split': 9.0, 'max_features': 20.0, 'criterion': 1}\n"
     ]
    }
   ],
   "source": [
    "# Hyperparameter optimization of random forest\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'max_features': int(params['max_features']),\n",
    "        \"min_samples_split\":int(params['min_samples_split']),\n",
    "        \"min_samples_leaf\":int(params['min_samples_leaf']),\n",
    "        \"criterion\":str(params['criterion'])\n",
    "    }\n",
    "    clf = RandomForestClassifier( **params)\n",
    "    clf.fit(X_train,y_train)\n",
    "    score=clf.score(X_test,y_test)\n",
    "\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 10, 200, 1),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"max_features\":hp.quniform('max_features', 1, 20, 1),\n",
    "    \"min_samples_split\":hp.quniform('min_samples_split',2,11,1),\n",
    "    \"min_samples_leaf\":hp.quniform('min_samples_leaf',1,11,1),\n",
    "    \"criterion\":hp.choice('criterion',['gini','entropy'])\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"Random Forest: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of RF: 0.9951492537313433\n",
      "Precision of RF: 0.9951646455154706\n",
      "Recall of RF: 0.9951492537313433\n",
      "F1-score of RF: 0.9951217831414103\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3645\n",
      "           1       0.99      1.00      0.99       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       1.00      1.00      1.00       609\n",
      "           4       1.00      0.71      0.83         7\n",
      "           5       0.98      1.00      0.99       251\n",
      "           6       0.99      0.99      0.99       436\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       0.99      0.96      0.97      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8FPX5wPHPs0m4EUQohAQFBQ+8\nwAIeWETEgAhiq6LWA/2p1BYPqhVRUYuKpdYT29qCIODBpSL3JYiIyg0i9y0EwqEIClRMss/vj52E\nBTNJdpndWTbP29e8svvdmXlmNubhO/M9RlQVY4wxvxTw+wCMMSZRWYI0xhgXliCNMcaFJUhjjHFh\nCdIYY1xYgjTGGBeWII0xxoUlSGOMcWEJ0hhjXKT6fQDFErFhPsb4QVWi2Sz3240R/82m1Tw1qljx\nkNAJMnf3Bt9ip9U6jdS0ur7Fz8vdXmbj5+VuB7D4Psc3CZ4gjTHHmWC+30fgKUuQxhjvaNDvI/CU\nJUhjjHeCliCNMaZImmQ1SOvmY4zxTjAY+VICEakgIvNF5CsRWSEifZzyISKySUSWOksTp1xEpL+I\nrBeRZSJyQdi+uorIOmfpWlJsq0EaY7wTmxrkIaCNqu4XkTRgjohMdj57RFXfP2r9q4BGznIh8AZw\noYjUAJ4GmgEKLBKRcar6vVtgq0EaY7wTzI98KYGG7HfepjlLcf0tOwPDnO3mAtVFJB1oB0xX1T1O\nUpwOtC8utiVIY4x3NBj5UgoikiIiS4FdhJLcPOejvs5l9CsiUt4pywC2hm2e7ZS5lbuyBGmM8U4U\n9yBFpJuILAxbuh29W1XNV9UmQCbQQkTOAR4DzgSaAzWAR53VixqZo8WUu7IEaYzxjGowikUHqGqz\nsGWA+/51LzALaK+qOc5l9CHgLaCFs1o2UC9ss0xgezHlrixBGmO8E5tW7FoiUt15XRFoC6x27isi\nIgJcCyx3NhkH3O60Zl8E7FPVHGAqkCUiJ4rIiUCWU+bKWrGNMd6JTSt2OjBURFIIVepGqeoEEZkp\nIrUIXTovBe511p8EdADWAweBOwFUdY+IPAsscNZ7RlX3FBfYEqQxxjsxGIutqsuApkWUt3FZX4Hu\nLp8NBgaXNvZxlSAPHfqZrt0f4efcXPLz8rny8ku57+7bUFX6DxjKtE/mEAgEuPG3V3PrDZ2ZMHUm\ng94dDUClihV58i/3cWajUwF4e9RHfDBuCqrK9de057Ybf+vpsQYCAebNncz2bTvo/NsS+6N6ql1W\na15++RlSAgEGvzWcF/7xr7jFLl++PLNmfkC58uVJTU3hww8n0ueZl+IWH/w9f/D3d79+7Vx+3L+f\n/PwgeXl5XHRxh7jGt7HYPipXLo3B/ftRqVJFcvPyuP2Pf+E3FzVj4zdb2bHrW8a/N4BAIMB33+8F\nIKNuHYb88wWqnVCVz75cQJ8X+jN84Kus27iZD8ZNYfibr5KWmsa9D/em1SUtOKVesS3+EXng/rtZ\nvXodJ1St6tk+SyMQCND/tb6073Az2dk5zP1yEuMnTGPVqnVxiX/o0CHaZnXhwIGDpKamMnvWGKZM\n+YR58xfHJb7f5w/+/e4LtL3yBr77zrXvs4lAzBppRORMEXnUGfLzmvP6rGPcJ5UqVQQgLy+PvLw8\nRISRYybyxzt/TyAQOp2TTqwOQNNzG1PthND/pOedfSY7d30LwMbNWznv7DOpWKECqakpNGtyLjNm\nf3Esh3aEjIx0Olx1BYMHD/dsn6XVonlTNmzYzKZNW8jNzWXUqLFc06ldXI/hwIGDAKSlpZKalkbo\niic+/D5/P3/3CSEGjTR+ikmCFJFHgRGEbp7OJ3RTVIDhItLrWPadn5/PdV2706rjzVzcvCnnnX0m\nW7flMHnGp3T5vwe49+En+Wbrtl9s9+GEqVx6UTMAGp56Cou+Ws7efT/wv59+4rMvF7Bj5+5jOawj\nvPxSH3o99hxBH375dTPqsDX7cM+F7G051K1bJ67HEAgEWLhgGjnbljFjxmzmL1gSt9h+n7+fv3sA\nVWXypOHMmzuZu++6xYcDiE1Hcb/E6hL7LuBsVc0NLxSRl4EVQL9od5ySksIHQ//FDz/u58HHnmXd\nxs38nJtL+XLlGDW4P9Nnfc6Tz7/CsDdeLNxm/qKv+HDCNN52yk6rfzL/d8sN3NPjcSpVrMjpDU8l\nJSUl2kM6wtUd2rJr17csXvI1l7W62JN9RiLU4+FI8azBAQSDQZo1z6JatRP4YPQgzj77DFasWBOX\n2H6ev9+/e4BWra8lJ2cntWqdxJTJI1izZj2fzZlX8oZeSfAaYaRidYkdBIqaLz7d+cxVeK/6N4e5\nX6acULUKzS84jzlzF1KnVk2ubH0pAG0vu4S1GzYVrrdm/Sae6vcqr/d7iurVTigsv65TO0a/9U+G\n/vsfVDuhqmf3Hy+5pBmdOmaxfu1c3n3n31x+eUuGDunvyb5LY1t2DvUyD3/1mRnp5OTsjFv8cPv2\n/cCns7+gXVbruMX08/z9/t0Dhee6e/d3jB07mebNm8Q1vmp+xEsii1WC7AHMEJHJIjLAWaYAM4AH\ni9swvFf93bfffMRne77fyw8/hsas/3ToEHMXLKHBKfVo0+pi5i1aCsCCJV8XJrucHbvo8fiz/O2p\nR6h/cuYR+ypoyMnZsYsZn37OVW0v8+C04Yne/ah/ajMann4Rt9z6Jz755HO63vGAJ/sujQULl9Kw\nYQPq169HWloaXbp0ZvyEaXGLX7NmDao5/xBVqFCBK9r8hjVr4vdsIT/P3+/ffaVKFalSpXLh6yvb\nXha3mnshu8QumapOEZHTCQ39ySB0/zEbWKDH8E/G7u++54nnXiQ/GESDSrs2v6F1ywu54LyzebTP\nC7w98iMqVaxAn149AHjjrffY98OPPPdiqJtHSkoKowaH/kX/8+PPsfeHH0hNTeWJh/9U2JhzvMvP\nz+fBHr2ZNPE9UgIBhgwdycqVa+MWPz29NoMHvUpKSoBAIMD7749n4qSP4xbf7/P3U+3atXh/9CAA\nUlNTGDHiI6ZOmxXfg0iyS2yJ9/2pSETzCEmv2FMN7amGZTp+lI99/WnRRxH/zVb49bX22FdjTBlg\nTzU0xhgXCX5PMVKWII0x3kmye5CWII0x3rEapDHGuLAapDHGuLAEaYwxRUv0kTGRsgRpjPGO1SCN\nMcZFkjXS2EO7jDHGhdUgjTHesUtsY4xxkWSX2JYgjTHesRqkMca4sBpk/KTVOs3X+AXTTll8i18W\n40fFapDGGOPCEmT8+D1h7Fm1mvsWf9XuBb6ff5meMNbiR8cusY0xxoXVII0xxoXVII0xxkWS1SBt\nqKExxjsxeOyriFQQkfki8pWIrBCRPk55AxGZJyLrRGSkiJRzyss779c7n9cP29djTvkaEWlXUmxL\nkMYY7wSDkS8lOwS0UdXzgSZAexG5CPg78IqqNgK+B+5y1r8L+F5VGwKvOOshIo2Bm4CzgfbAv0Uk\npbjAliCNMd6JQYLUkP3O2zRnUaAN8L5TPhS41nnd2XmP8/kVIiJO+QhVPaSqm4D1QIviYluCNMZ4\nRzXiRUS6icjCsKXb0bsVkRQRWQrsAqYDG4C9qprnrJINZDivM4CtocPRPGAfcFJ4eRHbFMkaaYwx\n3omikUZVBwADSlgnH2giItWBMcBZRa3m/BSXz9zKXVmCNMZ4J8at2Kq6V0RmARcB1UUk1aklZgIF\nPdyzgXpAtoikAtWAPWHlBcK3KZJdYhtjvBObVuxaTs0REakItAVWAZ8A1zurdQXGOq/HOe9xPp+p\nquqU3+S0cjcAGgHzi4ttNUhjTKJLB4Y6Lc4BYJSqThCRlcAIEXkOWAIMctYfBLwtIusJ1RxvAlDV\nFSIyClgJ5AHdtYSnjFmCNMZ4JwaX2Kq6DGhaRPlGimiFVtWfgBtc9tUX6Fva2JYgjTHe0WLbPI47\nSXkPsl1Wa1Ysn83qlXPo+Uj3mMQoV74cI6e8xZhP3mX87BHc1/MeAC68tBkffDyMcZ8O52+vP01K\nSqgfapv2rfho1rt8OPMdRk8bygUXnh+T44L4nL/FT7zYiRA/Rh3FfSOawBk/tVxGxAcXCARYteIz\n2ne4mezsHOZ+OYlbb/sTq1ati2g/pZnurFLlihw88D9SU1N4Z/xA+j31Ci8PeJ7/u647mzdu4f5H\nu7F96w4+eG9c4boApzduyCsDn+fqll1c9x3tdGdenr9f8Y9lui8/43v53fseX7WoLjEl+t+gv0T8\nN1vxrhejihUPSVeDbNG8KRs2bGbTpi3k5uYyatRYrulU4pDLqBQkvNS0VNLSUgnmB/n555/ZvHEL\nAF/Mmk9Wx8uPWBegUqWKxOofpniev8VPnNiJEB+ISSu2n3xJkCJyZ6z2XTejDluzD3dtyt6WQ926\ndWISKxAI8OHMd5izcipffDqfZYtXkJaaytnnh/qwZnVqQ526tQvXb9uhNRM/H8Ub775M7x7PxeSY\n4nn+Fj9xYidCfAANasRLIvOrBtnH7YPwYUfB4IGIdxwacnmkWNXWgsEgv2tzK5ef35Fzmzam0Zmn\n8vAfetPr2T8zcspbHNx/kLz8w70IPp40i6tbduH+rj15oNcfYnJM8Tx/i584sRMhPpB09yBj1oot\nIsvcPgJqu3x2xLCjaO5BbsvOoV7m4Xs3mRnp5OTsjHQ3Efnxh/3M/2Ixl7a5mLf+/S63XRMaSnpJ\n6ws55bSTf7H+wrlLqHdKJtVrVGPvnn2eHosf52/x/Y+dCPGBhL9kjlQsa5C1gduBTkUs38Uq6IKF\nS2nYsAH169cjLS2NLl06M37CNM/jnHhSdaqeUAWA8hXKc3GrFmxa9w01ap4IQFq5NO6+73ZGDv0Q\ngJMbZBZu2/jcM0grl+p5coT4nb/FT6zYiRAfgKBGviSwWPaDnABUUdWlR3/gjKWMifz8fB7s0ZtJ\nE98jJRBgyNCRrFy51vM4tWrXdLrxBAhIgCnjPmbW9Dn85en7aX3lpQQCAUYM+YB5cxYCkNWxDZ1v\n6EBuXh6HfjrEQ92e8PyYIH7nb/ETK3YixAcS/pI5UknXzccr9lRDe6phmY4fZTefg6/dG/HfbKUH\n/5Ow3XxsJI0xxjsJXOGKhiVIY4x3kuwS2xKkMcY7Cd7oEilLkMYY7yRZNx9LkMYY7yRZDTLpxmIb\nY4xXrAZpjPGMWiONMca4SLJLbEuQxhjvWCONMca4sBqkMca4sHuQxhjjwmqQxhjjwu5BGmOMC6tB\nxk/BtE9+WbV7ga/x/T5/i1+240fD+kHGUVmdD7EgfpqP8XNtPsgyHz8qVoM0xhgXliCNMcZFkjXS\n2GQVxhjvxOChXSJST0Q+EZFVIrJCRB50yv8qIttEZKmzdAjb5jERWS8ia0SkXVh5e6dsvYj0Kim2\n1SCNMZ7R2Fxi5wEPq+piEakKLBKR6c5nr6jqi+Eri0hj4CbgbKAu8LGInO58/C/gSiAbWCAi41R1\npVtgS5DGGO/EIEGqag6Q47z+UURWARnFbNIZGKGqh4BNIrIeaOF8tl5VNwKIyAhnXdcEaZfYxhjv\nBIORLxEQkfpAU2CeU3SfiCwTkcEicqJTlgFsDdss2ylzK3dlCdIY4ysR6SYiC8OWbi7rVQE+AHqo\n6g/AG8BpQBNCNcyXClYtYnMtptyVXWIbY7wTxSW2qg4ABhS3joikEUqO76rqh852O8M+HwhMcN5m\nA/XCNs8ECjp3upUXyWqQxhjvxKYVW4BBwCpVfTmsPD1std8Cy53X44CbRKS8iDQAGgHzgQVAIxFp\nICLlCDXkjCsuttUgjTGeUY1JK3ZL4DbgaxFZ6pQ9DtwsIk0IXSZvBv7gHMMKERlFqPElD+iuqvkA\nInIfMBVIAQar6oriAluCNMZ4Jzat2HMo+v7hpGK26Qv0LaJ8UnHbHc0SpDHGOzbU0BhjihajjuK+\nsQRpjPFOkiXIpGzFbpfVmhXLZ7N65Rx6PtI96eMPHPAS27K/YsmSGYVl553XmM9mj2PJ4o8ZM2YI\nVatWiflxFChr33+ixE6E+ASjWBJY0iXIQCBA/9f60rHTrZx7/uXceOO1nHVWo6SOP3TYKDp2vOWI\nsv/+5x88/sTzNL2gLWM/mszDD/8xpsdQoCx+/4kQOxHiQ+gSO9IlkcUsQYrImSJyhdP7Pby8faxi\nArRo3pQNGzazadMWcnNzGTVqLNd0alfyhsdx/Dlz5rHn+71HlJ1++ml89tlcAD6e8Rm//W2Hojb1\nXFn8/hMhdiLEB2LSD9JPMUmQIvIAMBa4H1guIp3DPn4+FjEL1M2ow9bsw53js7flULdunViGTKj4\nBVasWEOnTlkAXH9dR+plxmd2ar/P38/4ZfncC9kldqncA/xaVa8FWgNPFszhRtH9mQqFj8sMBg9E\nHDjU6f5IMeq8mpDxC9zT7SH+eO8dzJs7mSpVK/Pzz7lxiev3+fsZvyyfe2G8JLvEjlUrdoqq7gdQ\n1c0i0hp4X0ROoYQEGT4uM7VcRsTf3rbsnCNqS5kZ6eTk7CxmC2/5Hb/AmjUb6HD17wFo1OhUOlx1\nRVzi+n3+fsYvy+deKMFrhJGKVQ1yhzMECAAnWXYEagLnxigmAAsWLqVhwwbUr1+PtLQ0unTpzPgJ\n02IZMqHiF6hV6yQgVKt4/LEHGTDg7bjE9fv8/Yxfls+9gNUgS+d2QmMgC6lqHnC7iPw3RjEByM/P\n58EevZk08T1SAgGGDB3JypVrYxnS9/hvv/0vLmt1MTVr1mDTxoU888yLVKlSmXv/eAcAH300iSFD\nR8b0GAqUxe8/EWInQnwg6WqQ4sf9sdKK5hLbK/bYV3vsa5mOr1rsrTA333W6LOK/2ZPGfxpVrHhI\nun6QxhjjFRtqaIzxTpJdYluCNMZ4Jskei20J0hjjIUuQxhhTNKtBGmOMC0uQxhjjwhKkMca4ia77\nZMKyBGmM8YzVII0xxoUGrQZpjDFFshqkMca4iHIId8KyBGmM8YzVII0xxoXdg4yjgmmfymr83DJ+\n/hbf3/jRSODZE6OS0AnS7/kYy3r81hnxeUzD0WZtCz3fu0zPx5gA8aNhNUhjjHGRbAnSJsw1xiQ0\nEaknIp+IyCoRWVHwhFQRqSEi00VknfPzRKdcRKS/iKwXkWUickHYvro6668Tka4lxbYEaYzxjGrk\nSynkAQ+r6lnARUB3EWkM9AJmqGojYIbzHuAqoJGzdAPegFBCBZ4GLgRaAE8XJFU3liCNMZ7RoES8\nlLhP1RxVXey8/hFYBWQAnYGhzmpDgWud152BYRoyF6guIulAO2C6qu5R1e+B6UD74mJbgjTGeEZV\nIl5EpJuILAxburntX0TqA02BeUBtVc0JxdUc4FfOahnA1rDNsp0yt3JXJTbSiEht4Hmgrqpe5VRt\nL1bVQSVta4wpW6LpKK6qA4ABJa0nIlWAD4AeqvqDiGvts6gPtJhyV6WpQQ4BpgIFfQ7WAj1KsZ0x\npowJqkS8lIaIpBFKju+q6odO8U7n0hnn5y6nPBuoF7Z5JrC9mHJXpUmQNVV1FM7TJlQ1D8gvxXbG\nmDImmkvskkioqjgIWKWqL4d9NA4oaInuCowNK7/dac2+CNjnXIJPBbJE5ESncSbLKXNVmn6QB0Tk\nJJyqaEHAUmxnjCljYtQPsiVwG/C1iCx1yh4H+gGjROQuYAtwg/PZJKADsB44CNwJoKp7RORZYIGz\n3jOquqe4wKVJkA8RysinicjnQC3g+lKemDGmDInFUENVnUPR9w8BfjHcS1UV6O6yr8HA4NLGLjFB\nqupiEbkMOMM5yDWqmlvaAMaYsiPZRtKUphX79qOKLhARVHVYjI7JGHOcKm2jy/GiNJfYzcNeVyBU\npV0MWII0xhyhzE2Yq6r3h78XkWrA2zE7ImPMcSvZpjuLZiTNQUJjHBNS+fLl+fLzCSxaOJ2vls7k\n6acejvsxtMtqzYrls1m9cg49HynyXvFxH7/KCZXp89+nGDZrMEM/GUTjC86iavWqvPje33nnsyG8\n+N7fqVKtSuH6TS4+nzen/oe3ZrzJq++/FJNjKuDn918WfvfFiVU/SL+IlpDyRWQ8h3ubB4DGwChV\n7eW+lTdSy2VE9e9R5cqVOHDgIKmpqcyeNYY/P/Q08+Yvjmgf0c7HGAgEWLXiM9p3uJns7BzmfjmJ\nW2/7E6tWrTvu4hc3H2SvV3ry9fyvmTh8MqlpqVSoWJ5b7v89P+79kff+NYLfd7+JKtWqMOD5N6ly\nQmX++VF/et76GLu276L6SdXZ+91e130fy3yQXpx/tPMxevnd+x4/ymvlJSd3jvhvtumWsQmbJUtT\ng3wReMlZ/ga0Kk1yFJEWItLced1YRB4SkQ7HdLSldODAQQDS0lJJTUujpH8EvNSieVM2bNjMpk1b\nyM3NZdSosVzTqV1Sxa9UpRLnX3guE4dPBiAvN4/9PxygZdYlTBk9DYApo6dxabuWAFxx7RV8NnkO\nu7aHBjoUlxyPlZ/ff1n43ZckRrP5+KbYBCkiKcCTqvqps3yuqtkl7VREngb6A2+IyN+AfwJVgF4i\n8oQXB16cQCDAwgXTyNm2jBkzZjN/wZJYhyxUN6MOW7MPj17K3pZD3bp1kip+3ZPT2btnH71efoSB\nU/7DI/94iAoVK1Cj5ons2RXqd7tn1x5OPKk6APVOzaBKtSq8Ovol/jvp32Rdd6Wnx3PEsfn4/ZeF\n331Jku0Su9gEqar5wEGnYSYS1xPq/d6KUIfNa1X1GULTDd1Y3IbhM3sEgwciDBsSDAZp1jyLUxo0\no3mzppx99hlR7ScaRQ2gj2cNNh7xU1JTOP2cRox9ezz3tL+X/x38id93v6nY9c8473R63f4EPW/p\nxe09biGzQbGTqETNz++/LPzuSxKLoYZ+Ks0l9k+EhvgMcmbp7S8i/UvYJk9V81X1ILBBVX8AUNX/\n4YzpdqOqA1S1mao2CwQql+ok3Ozb9wOfzv6Cdlmtj2k/kdiWnUO9zMP3jjIz0snJ2ZlU8Xfn7GZ3\nzm5WLVkNwKcTZ9Po3Ebs+fZ7avyqBgA1flWD751L6d053zJ/1gJ++t9P7Pv+B76a9zWnNT7N02Mq\n4Of3XxZ+92VNaRLkROBJYDawyFkWlrDNzyJSyXn964JCpyYa0yfn1qxZg2rVTgCgQoUKXNHmN6xZ\nsyGWIY+wYOFSGjZsQP369UhLS6NLl86MnzAtqeLv2f09u7bvpt6pmQD8+tIL+GbdN3wx/Uva35AF\nQPsbsvh82hcAzJn6Bee2OIeUlADlK5SncZMz2bJ+i6fHVMDP778s/O5LkmyX2KXpKF5dVV8LLyh4\nJkQxWqnqIQDVI2aIS+Pw7BsxkZ5em8GDXiUlJUAgEOD998czcdLHsQx5hPz8fB7s0ZtJE98jJRBg\nyNCRrFy5Nuni93/yn/R+/TFSy6WR800O/R7+BwEJ8PR/etPhpvbs3LaLv977LABb1m9h/qyFDJo+\nEA0GmTh8MpvWbPb8mMDf77+s/O6Lk+BtLhErTTefxap6wVFlS1S1aUyPjOi7+XghER676nd8e+xr\nGY4f5c3BL9Kvi/hv9pKcDxK2GulagxSRm4HfAw1EZFzYR1WB72J9YMaY40+iN7pEqrhL7C+AHKAm\noT6QBX4ElsXyoIwxx6eYNjD4wDVBquo3wDfAxcXtQES+VNVi1zHGlA3qOm3j8ak0jTQlqeDBPowx\nSSCYZK00XiTIJPtKjDHRCloN0hhjipZsl9gldhQXkfucJ4C5ruLh8RhjjmPBKJZEVpqRNHWABSIy\nSkTayy8HfN4Wg+MyxhyHFIl4SWQlJkhV7U1ogtxBwB3AOhF5XkROcz5fHtMjNMYcN8piDbLgMYo7\nnCUPOBF4X0ReiOGxGWOOM8mWIEvzVMMHCI2f/hZ4E3hEVXNFJACsA3rG9hCNMceLRL9kjlRpWrFr\nAr9zOo4XUtWgiHSMzWEZY45HSfZY7FI91fCpYj5b5e3hGGOOZ8nWDzKapxoaY0yZkNAdxQumfbL4\n/iiYdswvfp9/WY8fjWQbVpfQCdLv+RAtfhmeD9HiRyUWrdIiMhjoCOxS1XOcsr8C9wC7ndUeV9VJ\nzmePAXcB+cADqjrVKW8PvAakAG+qar+SYid0gjTGHF+CRTw4zANDCD0ZddhR5a+o6ovhBSLSGLgJ\nOBuoC3wsIqc7H/8LuBLIJjT4ZZyqriwusCVIY4xnYnGJraqzRaR+KVfvDIxwHvmySUTWAy2cz9ar\n6kYAERnhrFtsgrRGGmOMZ+LcUfw+EVkmIoPD5ovIALaGrZPtlLmVF8sSpDHGM0GJfBGRbiKyMGzp\nVopQbwCnAU0IPfmg4KkHRV3jazHlxbJLbGOMZ6LpB6mqA4ABEW5T+MBvERkITHDeZgP1wlbNBApa\nndzKXVkN0hjjGY1iiYaIpIe9/S1QMGnOOOAmESkvIg0ITbQzH1gANBKRBiJSjlBDTvjDCItkNUhj\njGdiMdRQRIYDrYGaIpINPA20FpEmhHLsZuAPAKq6QkRGEWp8yQO6q2q+s5/7gKmEuvkMVtUVJcW2\nBGmM8Uws+kGq6s1FFA8qZv2+QN8iyicBkyKJbQnSGOMZG0ljjDEuytxsPsYYU1qJPgFupCxBGmM8\nYwnSGGNcaJJdYiddP8jMzLp8PG00Xy+bxVdLZ3L/fXfF/RjaZbVmxfLZrF45h56PdLf4cbZ+7VyW\nLP6YhQumMffLiBotj5nf5+53/DL3TJrjTV5eHo/07MOSpcupUqUy8+dN4eMZs1m1al1c4gcCAfq/\n1pf2HW4mOzuHuV9OYvyEaRY/TvELtL3yBr777vu4xvT73P2On4ySrga5Y8culiwNdarfv/8Aq1ev\nI6NunbjFb9G8KRs2bGbTpi3k5uYyatRYrunUzuKXAX6fu9/xIflqkHFLkCJy9FxuMXfKKZk0Of8c\n5s1fEreYdTPqsDX78BDP7G051I1jgi7r8QFUlcmThjNv7mTuvuuWuMX1+9z9jg/xG2oYLzG5xBaR\no8c4CnC5iFQHUNVrYhE3XOXKlRg1ciAP/eVpfvxxf6zDFZIiJgwNPVbc4sdLq9bXkpOzk1q1TmLK\n5BGsWbOez+bMi3lcv8/d7/hg/SBLK5PQWMg3OTzVUDMOT0nkypnqqBuApFQjEKgccfDU1FRGjxzI\n8OFj+OijyRFvfyy2ZedQL/P3x/fMAAASs0lEQVTwVPmZGenk5OwsZguL77WCeLt3f8fYsZNp3rxJ\nXBKk3+fud3xI/EvmSMXqErsZsAh4AtinqrOA/6nqp6r6aXEbquoAVW2mqs2iSY4AAwe8xKrV63n1\ntYhmUPLEgoVLadiwAfXr1yMtLY0uXTozfsI0ix8nlSpVpEqVyoWvr2x7GStWrIlLbL/P3e/4kHz3\nIGNSg1TVIPCKiIx2fu6MVayjtbykObfdej3Lvl7JwgWh/zmefLIfk6fMjEd48vPzebBHbyZNfI+U\nQIAhQ0eycuXauMS2+FC7di3eHx2axyA1NYURIz5i6rRZcYnt97n7HR8S/55ipCQe9yhE5Gqgpao+\nHsl2qeUyfPu+y/JTBf2OnyhP9SvT8TW6Lt8vnHJrxH+zPb95J2HvXMalVqeqE4GJ8YhljPFPol8y\nRyrpOoobY/yTbJfYliCNMZ4JJlmKtARpjPGMXWIbY4yL5Ko/WoI0xnjIapDGGOPChhoaY4wLa6Qx\nxhgXyZUek3A+SGOM8YrVII0xnrFGGmOMcWH3II0xxkVypUdLkMYYD9kldhwVTPtk8S2+xT8+JNsl\ntrViG2M8E4uHdonIYBHZJSLLw8pqiMh0EVnn/DzRKRcR6S8i60VkmYhcELZNV2f9dSLStTTnk9A1\nyLI6YWxZj58QE8YCGdUb+xJ/296VgP/nH40YXWIPAf4JhD8ZtRcwQ1X7iUgv5/2jwFVAI2e5EHgD\nuFBEagBPE3ocjAKLRGScqhb78HSrQRpjPKNR/FfiPlVnA3uOKu4MDHVeDwWuDSsfpiFzgeoikg60\nA6ar6h4nKU4H2pcU2xKkMcYz0Ty0S0S6icjCsKVbKULVVtUcAOfnr5zyDGBr2HrZTplbebES+hLb\nGHN8iaaRRlUHAF49grSo6TK0mPJiWQ3SGOOZWDTSuNjpXDrj/NzllGcD9cLWywS2F1NeLEuQxhjP\nBNGIlyiNAwpaorsCY8PKb3dasy8C9jmX4FOBLBE50WnxznLKimWX2MYYz8SiFVtEhgOtgZoikk2o\nNbofMEpE7gK2ADc4q08COgDrgYPAnQCqukdEngUWOOs9o6pHN/z8giVIY4xnStMqHfE+VW92+eiK\nItZVoLvLfgYDgyOJbQnSGOMZG2pojDEuYlGD9JM10hhjjAurQRpjPGOX2MYY4yKoyXWJbQnSGOOZ\n5EqPSXgPcuCAl9ie/RVLl8zw7RjaZbVmxfLZrF45h56PFNnjwOIfx/HrZtRh9Li3mDV3HDO/GMtd\nf7gVgIce/RMLV8xk2uwPmDb7A9pc+ZvCbe77893MWTSZ2fMncFmblp4fUwG/v/s4dhSPi6RLkMOG\njeLqjrf4Fj8QCND/tb507HQr555/OTfeeC1nndXI4idR/Ly8PPr0foHWF11Dp6ybuePum2l0xmkA\nDHxjGFmtriOr1XXMnP4ZAI3OOI3Ov+tAm4uv4Zbr/8DzL/YmEPD+T8/v7x5iM5uPn5IuQX42Zx57\nvt/rW/wWzZuyYcNmNm3aQm5uLqNGjeWaTu0sfhLF37XzW5YvWwXAgf0HWbd2I3XSf+W6frsOlzP2\nw0n8/HMuW7dsY/PGrTT99bmeHhP4/91DdLP5JLK4JEgRuVREHhKRrHjE81PdjDpszT48Bj57Ww51\n69ax+EkaP7NeXc457yyWLFoGwJ33/J7pcz7kpdefpVq1EwCok16b7dt2FG6Ts30HddJre34sfn/3\nYJfYpSIi88Ne30NoNuCqwNPO7L9JS+SXsyppHFv2LH784leqXImBw17l6cf6sf/HAwwbPJJLmrYn\n6zfXsWvnbp567pG4HpPf3z3YJXZppYW97gZcqap9CM2gUewNwvDJM4PBAzE6vNjZlp1DvczDU+Vn\nZqSTk7PT4idZ/NTUVAYOfZUxoycyecLHAHy7+zuCwSCqyrtD36eJcxmds30HdTMO1+TS69Zh545d\nRe73WPj93YNdYpd6v860QicBoqq7AVT1AJBX3IaqOkBVm6lqs0CgcowOL3YWLFxKw4YNqF+/Hmlp\naXTp0pnxE6ZZ/CSL/9Lrz7B+7UYG/HtoYdmvatcsfH1Vx7asWbUOgGmTP6Hz7zpQrlwa9U7OoMFp\nJ7Nk0deeH5Pf3z2EaqyRLoksVv0gqwGLCM3iqyJSR1V3iEgVip7Z1zPvvP0vLmt1MTVr1mDzxoX0\neeZF3hoyIpYhj5Cfn8+DPXozaeJ7pAQCDBk6kpUr11r8JIrf/KILuP6mzqxcsYZpsz8AoN+zr3Lt\ndR1ofO6ZqCrZW7bz6J//CsDa1RsY/9EUPpk7jvy8fJ545DmCQe/rTn5/95B8j32VON8fqkToWRKb\nSrN+arkM377tsvxUQb/j21MNE+CphqpRVWQ6ndwx4r/Z8VsmxLTSdCziOpJGVQ8CpUqOxpjjT6I3\nukTKhhoaYzyTbJfYliCNMZ5J9EaXSFmCNMZ4JtG77UTKEqQxxjPJdg8y6cZiG2OMV6wGaYzxjDXS\nGGOMC2ukMcYYF1aDNMYYF8nWSGMJ0hjjGXtolzHGuEiu9GgJ0hjjoWS7B2n9II0xnonVIxdEZLOI\nfC0iS0VkoVNWQ0Smi8g65+eJTrmISH8RWS8iy0TkgmjPJ6FrkAXTTll8i++HgmnH/OL3+Ucjxt18\nLlfVb8Pe9wJmqGo/51EuvYBHgauARs5yIfCG8zNiVoM0xngmzg/t6gwUTOk+FLg2rHyYhswFqotI\nejQBEroGWVYnjC3r8RNlwly/459X+yJf4i/bOTfqbaPp5iMi3Qg9u6rAAFUd8ItdwzQRUeC/zue1\nVTUHQFVzRKTg2bsZwNawbbOdspxIjy2hE6Qx5vgSzSW2k+yOTohHa6mq250kOF1EVhezblEzlEdV\nVbUEaYzxTKxasVV1u/Nzl4iMAVoAO0Uk3ak9pgMFj4rMBuqFbZ4JRHVD1+5BGmM8E4unGopIZRGp\nWvCa0OOjlwPjgK7Oal2Bsc7rccDtTmv2RcC+gkvxSFkN0hjjmRjVIGsDY0QEQjnrPVWdIiILgFEi\nchewBbjBWX8S0AFYDxwE7ow2sCVIY4xnYjEWW1U3AucXUf4dcEUR5Qp09yK2XWIbY4wLq0EaYzxj\nk1UYY4wLm+7MGGNcWA3SGGNcWA3SGGNcWA3SGGNcWA3SGGNcJFsNMmn7QQYCARbMn8rYMUNLXtlj\n7bJas2L5bFavnEPPRzzpr2rxj5P48YwdCAQYOX0or7/9IgB/fflxRs8Yxvsz3+alN/tSsVJFAG77\nw02Mmf0e7898m4GjXyc9s07Mjkmj+C+RJW2CfOD+u1m9el3c4wYCAfq/1peOnW7l3PMv58Ybr+Ws\nsxpZ/DIQP96xb7mnC5vWbS58/4+nXuWGK27n+ja3kZO9k5v/73oAVi9fy83t7uT6NrcxfcJM/vxk\n7BK3ajDiJZHFJEGKyIUicoLzuqKI9BGR8SLydxGpFouY4TIy0ulw1RUMHjw81qF+oUXzpmzYsJlN\nm7aQm5vLqFFjuaZTO4tfBuLHM3bt9Fq0atuSD98dV1h2YP/BwtcVKpYvrJ0t+HwxP/3vEADLFq2g\ndvqviJU4T5gbc7GqQQ4mNEgc4DWgGvB3p+ytGMUs9PJLfej12HMEg/H/16luRh22Zh+eWSl7Ww51\n68buksbiJ078eMbu+WwPXn72nwSPqoE98+oTfPL1ROo3PIXhg0b/Yrvf/r4Tc2Z+GZNjgtjM5uOn\nWCXIgKrmOa+bqWoPVZ2jqn2AU4vbUES6ichCEVkYDB6IOPDVHdqya9e3LF7ydRSHfeycGUeOEM//\nCSy+f/HjFbvVlS3Z8+33rFq25hefPdWjL1ec34lN6zbTrnPbIz67+rp2nH3+mQz597ueH1MBq0GW\nznIRKZhi6CsRaQYgIqcDucVtqKoDVLWZqjYLBCpHHPiSS5rRqWMW69fO5d13/s3ll7dk6JD+Ee8n\nWtuyc6iXeXiq/syMdHJydlr8MhA/XrGbND+P1lm/YfKCD3nhP8/SouWvef6fTxd+HgwGmTJ2Bm2v\nvryw7MLfNOeeB+/gga49yf252D/BY2I1yNK5G7hMRDYAjYEvRWQjMND5LGae6N2P+qc2o+HpF3HL\nrX/ik08+p+sdD8Qy5BEWLFxKw4YNqF+/HmlpaXTp0pnxE6ZZ/DIQP16x+z//Blde0Jmrmv+Onvc+\nyfzPF/H4fX2oVz+zcJ3WWZeyef03AJx5zuk89Y+ePND1EfZ8+73nxxMuqBrxkshi0g9SVfcBdziz\nAJ/qxMlW1fhVJXySn5/Pgz16M2nie6QEAgwZOpKVK9da/DIQ38/YIsJz/Z+kStXKiMCaFet57tEX\nAHjoqfuoVLkSLw7sC8CObTt5oGvPmBxHonfbiZQkchU3tVyGbwdXlp8q6Hf8RHmqoN/xfX2qoWpR\nD74qUe1qZ0b8N7tz3+qoYsVD0vaDNMaYY2VDDY0xnkn0VulIWYI0xngmkW/ZRcMSpDHGM4neKh0p\nS5DGGM9YDdIYY1zYPUhjjHFhNUhjjHFh9yCNMcZFso2ksQRpjPGM1SCNMcZFst2DtKGGxhjPxOqZ\nNCLSXkTWiMh6EekV49MoZDVIY4xnYlGDFJEU4F/AlUA2sEBExqnqSs+DHcVqkMYYz8RowtwWwHpV\n3aiqPwMjgM4xPRFHQtcgC6Z9svgWvyzGX7Zzrq/xoxGjO5AZwNaw99nAhbEJdaSETpDRzklXQES6\nqeoArw7neIlt8S2+X/Hzft4W8d+siHQDuoUVDTjq2IvaZ1xag5L9ErtbyaskZWyLb/H9jl9q4c+h\ncpajE3s2UC/sfSYQl+p9sidIY8zxbwHQSEQaiEg54CZgXAnbeCKxL7GNMWWequaJyH3AVCAFGKyq\nK+IRO9kTpG/3gHyObfEtvt/xPaWqk4BJ8Y6b0A/tMsYYP9k9SGOMcWEJ0hhjXCRlgvRr3KYTe7CI\n7BKR5fGMGxa/noh8IiKrRGSFiDwY5/gVRGS+iHzlxO8Tz/jOMaSIyBIRmRDv2E78zSLytYgsFZGF\ncY5dXUTeF5HVzv8DF8czfrJJunuQzrjNtYSN2wRujse4TSd+K2A/MExVz4lHzKPipwPpqrpYRKoC\ni4Br43j+AlRW1f0ikgbMAR5U1bgNCxGRh4BmwAmq2jFeccPibwaaqeq3PsQeCnymqm86XWIqqere\neB9HskjGGqRv4zYBVHU2sCde8YqIn6Oqi53XPwKrCA3Vild8VdX9zts0Z4nbv8IikglcDbwZr5iJ\nQkROAFoBgwBU9WdLjscmGRNkUeM245YgEomI1AeaAvPiHDdFRJYCu4DpqhrP+K8CPYFgHGMeTYFp\nIrLIGUYXL6cCu4G3nFsMb4pI5TjGTzrJmCB9G7eZSESkCvAB0ENVf4hnbFXNV9UmhIaEtRCRuNxq\nEJGOwC5VXRSPeMVoqaoXAFcB3Z3bLvGQClwAvKGqTYEDQFzvwSebZEyQvo3bTBTOvb8PgHdV9UO/\njsO5vJsFtI9TyJbANc49wBFAGxF5J06xC6nqdufnLmAMods+8ZANZIfV2N8nlDBNlJIxQfo2bjMR\nOI0kg4BVqvqyD/FriUh153VFoC2wOh6xVfUxVc1U1fqEfu8zVfXWeMQuICKVncYxnMvbLCAuPRpU\ndQewVUTOcIquAOLSOJeskm6ooZ/jNgFEZDjQGqgpItnA06o6KF7xCdWibgO+du4DAjzuDNWKh3Rg\nqNObIACMUlVfutv4pDYwJvTvFKnAe6o6JY7x7wfedSoHG4E74xg76SRdNx9jjPFKMl5iG2OMJyxB\nGmOMC0uQxhjjwhKkMca4sARpjDEuLEGahCIi9f2aCcmYo1mCNHHh9Is05rhiCdIUSUSeDZ9LUkT6\nisgDRazXWkRmi8gYEVkpIv8RkYDz2X4ReUZE5gEXi8ivReRTZxKHqc7UbDjlX4nIl0D3eJ2jMSWx\nBGncDAK6AjgJ7ybgXZd1WwAPA+cCpwG/c8orA8tV9UJCMwq9Dlyvqr8GBgN9nfXeAh5QVZvc1SSU\npBtqaLyhqptF5DsRaUpo+NwSVf3OZfX5qroRCodaXkpoooR8QpNmAJwBnANMd4bhpQA5IlINqK6q\nnzrrvU1oFhxjfGcJ0hTnTeAOoA6hGp+bo8erFrz/SVXzndcCrDi6luhMbGHjXU1CsktsU5wxhKYq\na05o8g83LZzZkwLAjYQes3C0NUCtgmekiEiaiJztTIm2T0Qudda7xbvDN+bYWA3SuFLVn0XkE2Bv\nWE2wKF8C/Qjdg5xNKLEWta/rgf7OZXUqodm/VxCacWawiByk+ERsTFzZbD7GlVMjXAzcoKrrXNZp\nDfzFj4djGRNrdoltiiQijYH1wAy35GhMsrMapCkVETmXUAtzuENOFx5jkpIlSGOMcWGX2MYY48IS\npDHGuLAEaYwxLixBGmOMC0uQxhjj4v8BNPlwhcOloN8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rf_hpo = RandomForestClassifier(n_estimators = 71, min_samples_leaf = 1, max_depth = 46, min_samples_split = 9, max_features = 20, criterion = 'entropy')\n",
    "rf_hpo.fit(X_train,y_train)\n",
    "rf_score=rf_hpo.score(X_test,y_test)\n",
    "y_predict=rf_hpo.predict(X_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of RF: '+ str(rf_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of RF: '+(str(precision)))\n",
    "print('Recall of RF: '+(str(recall)))\n",
    "print('F1-score of RF: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rf_train=rf_hpo.predict(X_train)\n",
    "rf_test=rf_hpo.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply DT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of DT: 0.9930970149253732\n",
      "Precision of DT: 0.9936778376607716\n",
      "Recall of DT: 0.9930970149253732\n",
      "F1-score of DT: 0.9933227091323931\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.99      0.99      3645\n",
      "           1       0.99      1.00      0.99       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       0.99      1.00      1.00       609\n",
      "           4       0.42      0.71      0.53         7\n",
      "           5       0.98      1.00      0.99       251\n",
      "           6       0.98      0.99      0.98       436\n",
      "\n",
      "    accuracy                           0.99      5360\n",
      "   macro avg       0.91      0.96      0.93      5360\n",
      "weighted avg       0.99      0.99      0.99      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4FFXWwOHfaRKUTQFBCEkUFBx3\nRQEXHETUgAjijIrghn4q6qCIOuKGOqg4jLvMuAwIsrhAXJBFUBAXRGUHEQKyC4GwKLgAo2Y53x9d\nCQ2mO+lQS9M5r0896b5dVaeqYw636ta9V1QVY4wxfxQK+gCMMSZRWYI0xpgoLEEaY0wUliCNMSYK\nS5DGGBOFJUhjjInCEqQxxkRhCdIYY6KwBGmMMVGkBH0AMYlYNx9jgqAqFdks//vVcf/NptY7okKx\n/JDQCTJ/66rAYqfWP5KU1EaBxS/I31hp4xfkbwSw+AHHNwmeII0x+5miwqCPwFWWII0x7tGioI/A\nVZYgjTHuKUquBGmt2MYY16gWxb2URUQOFJHZIvK1iCwRkf5O+XARWSMiC53lZKdcRGSQiKwUkUUi\nckrEvnqIyApn6VFWbKtBGmPc400N8jegnaruEJFUYIaITHY+u1tV395r/QuAZs5yGvAScJqI1AUe\nBloACswTkfGquj1aYKtBGmPco0XxL2XtMmyH8zbVWWI9TtQFGOlsNxOoLSJpQHtgqqpuc5LiVKBD\nrNiWII0x7ikqjH8pBxGpIiILgS2Ek9ws56MBzmX0syJygFOWDqyP2DzXKYtWHpUlSGOMeypQgxSR\nniIyN2Lp+Yfdqhaq6slABtBKRI4H7gOOBloCdYF7nNVLe/BcY5RHZQnSGOOeoqK4F1UdrKotIpbB\n0Xavqj8CnwIdVDXPuYz+DXgVaOWslgtkRmyWAWyMUR6VJUhjjGs8asWuLyK1ndfVgPOAZc59RURE\ngIuBxc4m44FrnNbs04GfVDUP+BDIEpE6IlIHyHLKorJWbGOMe7xpxU4DRohIFcKVumxVnSgiH4tI\nfcKXzguBm531JwEdgZXALuA6AFXdJiKPAnOc9R5R1W2xAluCNMa4x4OeNKq6CGheSnm7KOsr0CvK\nZ8OAYeWNbQnSGOOeJOuLvV/dg/ztt9/pdsPt/LXH3+hy5U3855VRAKgqz/93OBd2u4HOV/TktbfG\nAbD6u/Vc2fMOmrftzKtv7Pks6c+/7OCOBx6jc/cb6XxFTxYuXurqsYZCIebM/pBxY0e4ut/yaJ/V\nliWLp7MsZwZ97y71H1LPBXX+Rx11JHPnTClZtn2/jN633eDrMfh97kMGP83G3K9ZuGBaSVmdOrX5\nYNKbLF0ygw8mvUnt2gf7cixePAcZpP2qBlm1airDBg2kevVq5BcUcM0tf+fPp7dg9Xfr2bTleya8\nMZhQKMQP238E4OCDanHvHTfz8fSv/rCvgc+9TOvTWvDsgH7k5+fzv19/c/VYe992A8uWreCgWrVc\n3W9ZQqEQg54fQIeO3cnNzWPmV5OYMHEKS5eu8PU4gjr/5ctX0aJlFhD+Ltatncd74yaXsZW7/D73\nkSOzefHFV3n11edLyu7p24uPP5nBE0++QN+7e3FP317cd//jvhxPMvGsBikiR4vIPU6fyOed18fs\n4z6pXr0aAAUFBRQUFCAijBn7PrdcdwWhUPh0DqlTu+TnCcf8iZSUPf8d2LFzJ/O+XswlndsDkJqa\nykG1au7Loe0hPT2Njhecy7Bhb7q2z/Jq1bI5q1atZc2adeTn55OdPY6LnPP0S5DnH+ncdmexevV3\nrFu3wbeYQZz75zNmsc2pFBTr3Lk9I0e9BcDIUW9x0UUxO4y4pwKP+SQyTxKkiNwDjCbcujSbcKuR\nAG+KyL37su/CwkIu6dGLNp26c0bL5px43NGs35DH5Gmf0fX/enPzXQ/y3frYfxC5GzZRp/bB9Bvw\nDJde24uH/vkcu/73674c1h6eebo/9973GEUB/PIbpTdkfe7uR7tyN+TRqFFDX48hyPOP1LVrF0aP\nec/XmIly7g0OrcemTVsA2LRpC4fWP8SfwEl2ie1VDfJ6oKWqDlTV15xlIOEHOa/flx1XqVKFd0a8\nwLSxo/gmZzkrVq/l9/x8Dqhalexhg7ikcwcefPzZmPsoKCxk6fKVXP6XC3l7+AtUq3YgQ0dl78th\nlbiw43ls2fI98xd848r+4hV+JGxP4UY9fwR9/sVSU1Pp3CmLt9+Z6FvMRDn3QFkNslyKgNLGi09z\nPosqstvRKyOjX6YcVKsmLU85kRkz59Kwfj3Ob3sWAOedfSbLV62JeXAND61Hg/r1OPG4owHIansW\nOctXxtymvM48swWdO2WxcvlMXn/tRc45pzUjhg9yZd/lsSE3j8yM3V99RnoaeXmbfYsf9PkX69Dh\nHBYs+IYtW773LWainDvA5i3f07DhoQA0bHgoW7b+4Etc1cK4l0TmVYLsA0wTkckiMthZPgCmAbfH\n2jCy29EN13Tf47Nt23/k51/Cg3r8+ttvzJyzgCaHZ9KuzRnMmrcQgDkLvuHwzJj9z6l3SF0aHlqf\nNd/lAjBz3kKObHxYxc50Lw/0G0jjI1rQ9KjTufKqv/HJJ1/Q49reruy7PObMXUjTpk1o3DiT1NRU\nunbtwoSJU3yLH/T5F+t2+cW+X14nyrkDTJwwhWuuvgyAa66+jAkTYnYYcU+SXWJ70oqtqh+IyFGE\nL6nTCd9/zAXm6D78k7H1h+088NhTFBYVoUVK+3Z/pm3r0zjlxOO4p/8TjBrzHtWrHUj/e/sA8P0P\n27j8+t7s2LmLUCjEa9nvMe71/1KzRg3uv+MW7un/BPkF+WQ2SuPR++9w49QDV1hYyO19+jHp/Teo\nEgoxfMQYcnKWB31YvqpW7UDOO7cNt/ztnrJXTgKvjXqBs9ucQb16dVm7ei79H3mKfz35AqPfeJnr\nru3O+vUbuLz7Tf4cTIJfMsdL/Lw/Fa+KTCHpFpvV0GY1rNTxKzjt66/z3ov7b/bAUy+2aV+NMZVA\nkvWksQRpjHFPgt9TjJclSGOMe5LsHqQlSGOMe6wGaYwxUVgN0hhjorAEaYwxpUv0njHxsgRpjHGP\n1SCNMSaKJGuk2a9GFDfGGD9ZDdIY4x67xDbGmCiS7BLbEqQxxj1WgzTGmCisBumf1PpHBhq/eNgp\ni2/xK2P8CrEapDHGRGEJ0j9BDxh7TP2WgcVfunVO4OdfqQeMtfgVY5fYxhgThdUgjTEmiiSrQVpP\nGmOMezyYF1tEDhSR2SLytYgsEZH+TnkTEZklIitEZIyIVHXKD3Der3Q+bxyxr/uc8m9FpH1ZsS1B\nGmPc4820r78B7VT1JOBkoIOInA78C3hWVZsB24HrnfWvB7aralPgWWc9RORYoBtwHNABeFFEqsQK\nbAnSGOMeD2qQGrbDeZvqLAq0A952ykcAFzuvuzjvcT4/V0TEKR+tqr+p6hpgJeGpqaOyBGmMcU8F\nEqSI9BSRuRFLz713KyJVRGQhsAWYCqwCflTVAmeVXCDdeZ0OrAdwPv8JOCSyvJRtSmWNNMYY92j8\nU9mr6mBgcBnrFAIni0htYCxwTGmrOT9Lm2dbY5RHZQnSGOMejx/zUdUfReRT4HSgtoikOLXEDKD4\nAc5cIBPIFZEU4GBgW0R5schtSmWX2MYY93jTil3fqTkiItWA84ClwCfApc5qPYBxzuvxznuczz9W\nVXXKuzmt3E2AZsDsWLGtBmmMcY83z0GmASOcFucQkK2qE0UkBxgtIo8BC4ChzvpDgVEispJwzbEb\ngKouEZFsIAcoAHppGZPoWII0xiQ0VV0ENC+lfDWltEKr6q/AZVH2NQAYUN7YliCNMe6xrobGGBNF\nBVqxE1lSNtK0z2rLksXTWZYzg7539/IkRtUDqjLmg1cZ+8nrTJg+mlv73gjAaWe14J2PRjL+szf5\n578fpkqV8IP67Tq04b1PX+fdj1/jrSkjOOW0kzw5LvDn/C1+4sVOhPheNNIESTSBM35K1fS4Dy4U\nCrF0yed06Nid3Nw8Zn41iauu/htLl66Iaz/lGe6seo1q7Nr5P1JSqvDahCEMfOhZnhn8OP93SS/W\nrl7Hbff0ZOP6TbzzxviSdQGOOrYpzw55nAtbd42674oOd+bm+QcVf1+G+woyvpvffeDxVUt7ZrBM\n/xv697j/Zqtd/1SFYvkh6WqQrVo2Z9WqtaxZs478/Hyys8dxUecy+6RXSHHCS0lNITU1haLCIn7/\n/XfWrl4HwJefziar0zl7rAtQvXo1vPqHyc/zt/iJEzsR4gNe9cUOTCAJUkSu82rfjdIbsj5397Of\nuRvyaNSooSexQqEQ7378GjNyPuTLz2azaP4SUlNSOO6k8EP+WZ3b0bBRg5L1z+vYlve/yOal15+h\nX5/HPDkmP8/f4idO7ESID6BFGveSyIKqQfaP9kFkv8yiop1x7zjcJ31PXtXWioqK+Gu7qzjnpE6c\n0PxYmh19BHfd1I97H72DMR+8yq4duygo3P2Y1UeTPuXC1l25rUdfet97kyfH5Of5W/zEiZ0I8YGk\nuwfpWSu2iCyK9hHQIMpne/TLrMg9yA25eWRm7L53k5GeRl7e5nh3E5dfft7B7C/nc1a7M3j1xde5\n+qJwX/sz257G4Uce9of1585cQObhGdSuezA/bvvJ1WMJ4vwtfvCxEyE+kPCXzPHysgbZALgG6FzK\n8oNXQefMXUjTpk1o3DiT1NRUunbtwoSJU1yPU+eQ2tQ6qCYABxx4AGe0acWaFd9Rt14dAFKrpnLD\nrdcwZsS7ABzWJKNk22NP+BOpVVNcT47g3/lb/MSKnQjxASjS+JcE5uVzkBOBmqq6cO8PnM7mnigs\nLOT2Pv2Y9P4bVAmFGD5iDDk5y12PU79BPecxnhAhCfHB+I/4dOoM/v7wbbQ9/yxCoRCjh7/DrBlz\nAcjq1I4ul3Ukv6CA3379jTt7PuD6MYF/52/xEyt2IsQHEv6SOV5J95iPW2xWQ5vVsFLHr+BjPrue\nvznuv9nqt7+csI/5WE8aY4x7ErjCVRGWII0x7kmyS2xLkMYY9yR4o0u8LEEaY9yTZI/5WII0xrgn\nyWqQSdcX2xhj3GI1SGOMa9QaaYwxJooku8S2BGmMcY810hhjTBRWgzTGmCjsHqQxxkRhNUhjjInC\n7kEaY0wUVoP0T/GwT0FZunVOoPGDPn+LX7njV4Q9B+mjyjoeYnH81ADj59t4kJU+foVYDdIYY6Kw\nBGmMMVEkWSONDVZhjHGPB5N2iUimiHwiIktFZImI3O6U/0NENojIQmfpGLHNfSKyUkS+FZH2EeUd\nnLKVInJvWbGtBmmMcY16c4ldANylqvNFpBYwT0SmOp89q6pPRa4sIscC3YDjgEbARyJylPPxC8D5\nQC4wR0TGq2pOtMCWII0x7vEgQapqHpDnvP5FRJYC6TE26QKMVtXfgDUishJo5Xy2UlVXA4jIaGfd\nqAnSLrGNMe4pKop7EZGeIjI3YukZbfci0hhoDsxyim4VkUUiMkxE6jhl6cD6iM1ynbJo5VFZgjTG\nBEpVB6tqi4hlcGnriUhN4B2gj6r+DLwEHAmcTLiG+XTxqqWFiVEelV1iG2Pc49FjPiKSSjg5vq6q\n7wKo6uaIz4cAE523uUBmxOYZQPHDndHKS2U1SGOMe7xpxRZgKLBUVZ+JKE+LWO0vwGLn9Xigm4gc\nICJNgGbAbGAO0ExEmohIVcINOeNjxbYapDHGNaqe1CBbA1cD34jIQqfsfqC7iJxM+DJ5LXCTcwxL\nRCSbcONLAdBLVQsBRORW4EOgCjBMVZfECmwJ0hjjHm9asWdQ+v3DSTG2GQAMKKV8Uqzt9mYJ0hjj\nHutqaIwxpfPoQfHAWII0xrgnyRJkUrZit89qy5LF01mWM4O+d/dK+vhDBj/NhtyvWbBgWknZiSce\ny+fTx7Ng/keMHTucWrVqen4cxSrb958osRMhPkUVWBJY0iXIUCjEoOcH0KnzVZxw0jlcfvnFHHNM\ns6SOP2JkNp06XblH2X9ffpL7H3ic5qecx7j3JnPXXbd4egzFKuP3nwixEyE+hC+x410SmWcJUkSO\nFpFznaffI8s7eBUToFXL5qxatZY1a9aRn59PdvY4LurcvuwN9+P4M2bMYtv2H/coO+qoI/n885kA\nfDTtc/7yl46lbeq6yvj9J0LsRIgPePIcZJA8SZAi0hsYB9wGLBaRLhEfP+5FzGKN0huyPnf3w/G5\nG/Jo1KihlyETKn6xJUu+pXPnLAAuvaQTmRn+jE4d9PkHGb8yn3sJu8QulxuBU1X1YqAt8GDxGG6U\n/jxTiciO60VFO+MOHH7ofk8ePbyakPGL3djzTm65+VpmzZxMzVo1+P33fF/iBn3+QcavzOdeEi/J\nLrG9asWuoqo7AFR1rYi0Bd4WkcMpI0E6HdUHA6RUTY/729uQm7dHbSkjPY28vM0xtnBX0PGLffvt\nKjpeeAUAzZodQccLzvUlbtDnH2T8ynzuJRK8Rhgvr2qQm5wuQAA4ybITUA84waOYAMyZu5CmTZvQ\nuHEmqampdO3ahQkTp3gZMqHiF6tf/xAgXKu4/77bGTx4lC9xgz7/IONX5nMvZjXI8rmGcB/IEqpa\nAFwjIv/1KCYAhYWF3N6nH5Pef4MqoRDDR4whJ2e5lyEDjz9q1Auc3eYM6tWry5rVc3nkkaeoWbMG\nN99yLQDvvTeJ4SPGeHoMxSrj958IsRMhPpB0NUgJ4v5YeVXkEtstNu2rTftaqeOrxrwVFs0Pnc+O\n+2/2kAmfVSiWH5LuOUhjjHGLdTU0xrgnyS6xLUEaY1yTZNNiW4I0xrjIEqQxxpTOapDGGBOFJUhj\njInCEqQxxkRTsccnE5YlSGOMa6wGaYwxUWiR1SCNMaZUVoM0xpgoKtiFO2FZgjTGuMZqkMYYE4Xd\ng/RR8bBPlTV+fiU/f4sfbPyKSODREyskoRNk0OMxVvb456SfF0jsTzZ8BFTy8RgTIH5FJFsN0saD\nNMa4Rosk7qUsIpIpIp+IyFIRWVI8AaCI1BWRqSKywvlZxykXERkkIitFZJGInBKxrx7O+itEpEdZ\nsS1BGmMSXQFwl6oeA5wO9BKRY4F7gWmq2gyY5rwHuABo5iw9gZcgnFCBh4HTgFbAw8VJNRpLkMYY\n16jGv5S9T81T1fnO61+ApUA60AUY4aw2ArjYed0FGKlhM4HaIpIGtAemquo2Vd0OTAU6xIqd0Pcg\njTH7F6/vQYpIY6A5MAtooKp5EE6iInKos1o6sD5is1ynLFp5VFaDNMa4RlXiXkSkp4jMjVh6lrZv\nEakJvAP0UdWfYxxGaVlaY5RHVWaCFJEGIjJURCY7748VkevL2s4YU/loUQUW1cGq2iJiGbz3fkUk\nlXByfF1V33WKNzuXzjg/tzjluUBmxOYZwMYY5VGVpwY5HPgQKH7mYDnQpxzbGWMqmSKVuJeyiIgA\nQ4GlqvpMxEfjgeKW6B7AuIjya5zW7NOBn5xL8Q+BLBGp4zTOZDllUZXnHmQ9Vc0WkfsAVLVARArL\nsZ0xppLxqC92a+Bq4BsRWeiU3Q8MBLKdK9p1wGXOZ5OAjsBKYBdwXfjYdJuIPArMcdZ7RFW3xQpc\nngS5U0QOwblWL87I5TwxY0wl4kUjjarOoPT7hwDnlrK+Ar2i7GsYMKy8scuTIO8kXGU9UkS+AOoD\nl5Y3gDGm8qh0XQ1Vdb6InA38iXAW/1ZV8z0/MmPMfifZuhqWmSBF5Jq9ik4REVR1pEfHZIzZT5Wn\n0WV/Up5L7JYRrw8kfM0/H7AEaYzZQ6UbMFdVb4t8LyIHA6M8OyJjzH4r2e5BVqQnzS7CncAT0pDB\nT7Mx92sWLpgW2DG0z2rLksXTWZYzg753l9qYlhTxaxxUg3/890FGfDqU4Z8M5dhTjqFW7Vo8+cZA\nRn0+nCffGEjNg2uG161VnQGvPsIrU17m1WlD6NC1vWfHFeT3X1l+99F48RxkkMrTk2aCiIx3lonA\nt+x+IDPhjByZzYWdrgwsfigUYtDzA+jU+SpOOOkcLr/8Yo45xr9/T/yMf1v/vzH707n0aHs9N2Td\nxHcr13FFr8uZ/8UCrv7ztcz/YgFX9OoGwMU9uvDdinXckHUzfS77O7c81JOUVPeHAgjy+69Mv/to\nKtLVMJGVpwb5FPC0s/wTaKOq98beBESklYi0dF4fKyJ3ikjHfTracvh8xiy2bf/R6zBRtWrZnFWr\n1rJmzTry8/PJzh7HRZ29qy0FFb96zeqceNoJTHpzMgAF+QXs/HknZ2adyYdvTQXgw7em0rr9mQCo\nKtVrVAOgWo1q/PLjLxQWuN/fIMjvv7L87mPxYjSfIMVMkCJSBXhQVT9zli9UNbesnYrIw8Ag4CUR\n+SfwH6AmcK+IPODGgSeqRukNWZ+7u3tn7oY8GjVqmHTx0w5L48dtP3HPM3cz+IOX+PuTd3JgtQOp\nW68O27aEOyds27KNOofUBmDs8HEc1uww3p43mmEfDeY/D72IevDXEeT3X1l+97FUqktsVS0EdjkN\nM/G4lHD3oDaEn2i/WFUfITwe2+WxNowc2aOoaGecYYMX7ja6Jy8SQdDxq6RU4ajjmzF+1AR6driF\nX3f9Svde0X+1Ldu2YOWSVVx6ajduaH8zvR+7leo1q7t+XEF+/5Xldx9LZbzE/pVwH8ihzjDmg0Rk\nUBnbFKhqoaruAlYVD02kqv8DYk4MGTmyRyhUo1wnkUg25OaRmbF7LpGM9DTy8jYnXfyteVvZmreV\npQuWAfDZ+9M56oRmbPt+O3UPrQtA3UPrsv2H8O2OC7q25/PJMwDYuHYjees3cVjTzNJ3vg+C/P4r\ny+++MilPgnwfeBCYDsxzlrllbPO7iBRXD04tLnRqokk2c+6e5sxdSNOmTWjcOJPU1FS6du3ChIlT\nki7+9q3b2bJxK5lHZABwylnNWbviO76c+hXtLzsfgPaXnc+XU74EYPOGLZxyVnMA6tSrTeaRmWz8\nLs/14wry+68sv/tYku0SuzzNiLVV9fnIguJJc2Joo6q/AajuMZV4KruHJ/LEa6Ne4Ow2Z1CvXl3W\nrp5L/0ee4tXho70MuYfCwkJu79OPSe+/QZVQiOEjxpCTszwp4w968AUe+Pd9pFRNIe+7PP5111OE\nRHj45Qfp2O0CtmzYwj9ufhSAUc+/zj3P3M3QjwYjwODHX+Hn7bHGPK2YIL//yvS7jybB21ziJmXd\noxCR+ap6yl5lC1S1uadHBqRUTQ/s+06EaVeDjm/Tvlbi+BW8Ofhl2iVx/82emfdOwlYjo9YgRaQ7\ncAXQRETGR3xUC/jB6wMzxux/Er3RJV6xLrG/BPKAeoSfgSz2C7DIy4Myxuyfkq2BIWqCVNXvgO+A\nM2LtQES+UtWY6xhjKgeNOq7t/smNvl4HurAPY0wSKEqyVho3EmSSfSXGmIoqshqkMcaULtkuscsz\nms+tzhSJUVdx8XiMMfuxogosiaw8PWkaAnNEJFtEOsgfO3xe7cFxGWP2Q4rEvSSyMhOkqvYjPEDu\nUOBaYIWIPC4iRzqfL/b0CI0x+43KWIMsnmd2k7MUAHWAt0XkCQ+PzRizn0m2BFmeWQ17E+4//T3w\nCnC3quaLSAhYAfT19hCNMfuLRL9kjld5WrHrAX91HhwvoapFItLJm8MyxuyPkmxa7HLNavhQjM+W\nuns4xpj9WbI9B1mRWQ2NMaZSSOgHxYuHfbL4wSgediwoQZ9/ZY9fEcnWrS6hE2TQ4yFa/Eo8HqLF\nrxAvWqVFZBjQCdiiqsc7Zf8AbgS2Oqvdr6qTnM/uA64HCoHeqvqhU94BeB6oAryiqgPLip3QCdIY\ns38pKmXiMBcMJzwz6si9yp9V1aciC0TkWKAbcBzQCPhIRI5yPn4BOB/IJdz5Zbyq5sQKbAnSGOMa\nLy6xVXW6iDQu5+pdgNHOlC9rRGQl0Mr5bKWqrgYQkdHOujETpDXSGGNcU5EHxSOnenaWnuUMd6uI\nLBKRYRHjRaQD6yPWyXXKopXHZAnSGOOaIol/iZzq2VkGlyPUS8CRwMmEZz4onvWgtGt8jVEek11i\nG2Nc49dzkKpaMuG3iAwBJjpvc4HICdczgOJWp2jlUVkN0hjjGq3AUhEikhbx9i9A8aA544FuInKA\niDQhPNDObGAO0ExEmohIVcINOZGTEZbKapDGGNd40dVQRN4E2gL1RCQXeBhoKyInE86xa4GbAFR1\niYhkE258KQB6qWqhs59bgQ8JP+YzTFWXlBXbEqQxxjVePAepqt1LKR4aY/0BwIBSyicBk+KJbQnS\nGOMa60ljjDFRVLrRfIwxprwSfQDceFmCNMa4xhKkMcZEoUl2iZ10z0FmZDTioylv8c2iT/l64cfc\nduv1vh9D+6y2LFk8nWU5M+h7dy+L77OVy2eyYP5HzJ0zhZlfxdVouc+CPveg41e6OWn2NwUFBdzd\ntz8LFi6mZs0azJ71AR9Nm87SpSt8iR8KhRj0/AA6dOxObm4eM7+axISJUyy+T/GLnXf+Zfzww3Zf\nYwZ97kHHT0ZJV4PctGkLCxaGH6rfsWMny5atIL1RQ9/it2rZnFWr1rJmzTry8/PJzh7HRZ3bW/xK\nIOhzDzo+JF8N0rcEKSJ7j+XmucMPz+Dkk45n1uwFvsVslN6Q9bm7u3jmbsijkY8JurLHB1BVJk96\nk1kzJ3PD9Vf6Fjfocw86PvjX1dAvnlxii8jefRwFOEdEagOo6kVexI1Uo0Z1sscM4c6/P8wvv+zw\nOlwJKWXA0PC04hbfL23aXkxe3mbq1z+EDyaP5ttvV/L5jFmexw363IOOD/YcZHllEO4L+Qq7hxpq\nwe4hiaJyxoLrCSBVDiYUqhF38JSUFN4aM4Q33xzLe+9Njnv7fbEhN4/MjN1D5Wekp5GXtznGFhbf\nbcXxtm79gXHjJtOy5cm+JMigzz3o+JD4l8zx8uoSuwUwD3gA+ElVPwX+p6qfqepnsTaMHBuuIskR\nYMjgp1m6bCXPPV+eYeXcNWfuQpo2bULjxpmkpqbStWsXJkycYvF9Ur16NWrWrFHy+vzzzmbJkm99\niR30uQcdH5LvHqQnNUhVLQKeFZG3nJ+bvYq1t9ZntuTqqy5l0Tc5zJ0T/p/jwQcHMvmDj/0IT2Fh\nIbf36cek99+gSijE8BFjyMnupXwCAAAR6UlEQVRZ7ktsiw8NGtTn7bfC4xikpFRh9Oj3+HDKp77E\nDvrcg44PiX9PMV7ixz0KEbkQaK2q98ezXUrV9MC+78o8q2DQ8RNlVr9KHV8r9sj3E4dfFfffbN/v\nXkvYO5e+1OpU9X3gfT9iGWOCk+iXzPFKugfFjTHBSbZLbEuQxhjXFCVZirQEaYxxjV1iG2NMFMlV\nf7QEaYxxkdUgjTEmCutqaIwxUVgjjTHGRJFc6TEJx4M0xhi3WA3SGOMaa6Qxxpgo7B6kMcZEkVzp\n0RKkMcZFdonto+Jhnyy+xbf4+wcvLrFFZBjQCdiiqsc7ZXWBMUBjYC3QVVW3S3jeieeBjsAu4FpV\nne9s0wPo5+z2MVUdUVZsa8U2xrjGo0m7hgMd9iq7F5imqs2Aac57gAuAZs7SE3gJShLqw8BpQCvg\nYRGpU1bghK5BVtYBYyt7/IQYMBZIr31sIPE3/JgDBH/+FeHFJbaqTheRxnsVdwHaOq9HAJ8C9zjl\nIzU8EvhMEaktImnOulNVdRuAiEwlnHTfjBU7oROkMWb/ov410zRQ1TwAVc0TkUOd8nRgfcR6uU5Z\ntPKY7BLbGOOaikzaJSI9RWRuxNJzHw6htN7gGqM8JqtBGmNcU5FGGlUdDMQ7BelmEUlzao9pwBan\nPBfIjFgvA9jolLfdq/zTsoJYDdIY4xqPGmlKMx7o4bzuAYyLKL9Gwk4nPO10HvAhkCUidZzGmSyn\nLCarQRpjXOPRYz5vEq791RORXMKt0QOBbBG5HlgHXOasPonwIz4rCT/mcx2Aqm4TkUeBOc56jxQ3\n2MRiCdIY4xqPWrG7R/no3FLWVaBXlP0MA4bFE9sSpDHGNT62YvvCEqQxxjXW1dAYY6JIthqktWIb\nY0wUVoM0xrjGLrGNMSaKIk2uS2xLkMYY1yRXekzCe5BDBj/NxtyvWbhgWmDH0D6rLUsWT2dZzgz6\n3l3qI1kWfz+O3yi9IW+Nf5VPZ47n4y/Hcf1NVwFw5z1/Y+6Sj5ky/R2mTH+Hduf/uWSbW++4gRnz\nJjN99kTObtfa9WMqFvR3X4TGvSSypEuQI0dmc2GnKwOLHwqFGPT8ADp1vooTTjqHyy+/mGOOaWbx\nkyh+QUEB/fs9QdvTL6JzVneuvaE7zf50JABDXhpJVptLyGpzCR9P/RyAZn86ki5/7Ui7My7iyktv\n4vGn+hEKuf+nF/R3D+FW7Hj/S2RJlyA/nzGLbdt/DCx+q5bNWbVqLWvWrCM/P5/s7HFc1Lm9xU+i\n+Fs2f8/iRUsB2LljFyuWr6Zh2qFR12/f8RzGvTuJ33/PZ/26DaxdvZ7mp57g6jFB8N89VGw0n0Tm\nS4IUkbNE5E4RyfIjXpAapTdkfe7uAUdzN+TRqFFDi5+k8TMyG3H8icewYN4iAK678QqmzniXp//9\nKAcffBAADdMasHHDppJt8jZuomFaA9ePJejvHuwSu1xEZHbE6xuB/wC1CA9zfm/UDZNAeEqMPamP\nLXsW37/41WtUZ8jI53j4voHs+GUnI4eN4czmHcj68yVs2byVhx6729djCvq7B7vELq/UiNc9gfNV\ntT/hIYZi3iCMHDyzqGinR4fnnQ25eWRm7B4qPyM9jby8zRY/yeKnpKQwZMRzjH3rfSZP/AiA77f+\nQFFREarK6yPe5mTnMjpv4yYape+uyaU1asjmTVtK3e++CPq7B7vELvd+nXHXDgFEVbcCqOpOoCDW\nhqo6WFVbqGqLUKiGR4fnnTlzF9K0aRMaN84kNTWVrl27MGHiFIufZPGf/vcjrFy+msEv7p4Y79AG\n9UpeX9DpPL5dugKAKZM/octfO1K1aiqZh6XT5MjDWDDvG9ePKejvHsI11niXRObVc5AHA/MID3Ou\nItJQVTeJSE1KH/rcNa+NeoGz25xBvXp1Wbt6Lv0feYpXh4/2MuQeCgsLub1PPya9/wZVQiGGjxhD\nTs5yi59E8VuefgqXdutCzpJvmTL9HQAGPvocF1/SkWNPOBpVJXfdRu654x8ALF+2ignvfcAnM8dT\nWFDIA3c/RlGR+3WnoL978GY8yCCJz/eHqhOebGdNedZPqZoe2LddmWcVDDq+zWqYALMaqlaoItP5\nsE5x/81OWDfR00rTvvC1J42q7gLKlRyNMfufRG90iZd1NTTGuCbZLrEtQRpjXJPojS7xsgRpjHFN\noj+2Ey9LkMYY1yTbPcik64ttjDFusRqkMcY11khjjDFRWCONMcZEYTVIY4yJItkaaSxBGmNcY5N2\nGWNMFMmVHi1BGmNclGz3IO05SGOMa7yackFE1orINyKyUETmOmV1RWSqiKxwftZxykVEBonIShFZ\nJCKnVPR8EroGWTzslMW3+EEoHnYsKEGff0V4/JjPOar6fcT7e4FpqjrQmcrlXuAe4AKgmbOcBrzk\n/Iyb1SCNMa7xedKuLkDxkO4jgIsjykdq2EygtoikVSRAQtcgK+uAsZU9fqIMmBt0/BMbnB5I/EWb\nZ1Z424o85iMiPQnPXVVssKoO/sOuYYqIKPBf5/MGqpoHoKp5IlI89246sD5i21ynLC/eY0voBGmM\n2b9U5BLbSXZ7J8S9tVbVjU4SnCoiy2KsW9oI5RWqqlqCNMa4xqtWbFXd6PzcIiJjgVbAZhFJc2qP\naUDxVJG5QGbE5hlAhW7o2j1IY4xrvJjVUERqiEit4teEp49eDIwHejir9QDGOa/HA9c4rdmnAz8V\nX4rHy2qQxhjXeFSDbACMFREI56w3VPUDEZkDZIvI9cA64DJn/UlAR2AlsAu4rqKBLUEaY1zjRV9s\nVV0NnFRK+Q/AuaWUK9DLjdh2iW2MMVFYDdIY4xobrMIYY6Kw4c6MMSYKq0EaY0wUVoM0xpgorAZp\njDFRWA3SGGOiSLYaZNI+BxkKhZgz+0PGjR1R9soua5/VliWLp7MsZwZ973bleVWLv5/E9zN2KBRi\nzNQR/HvUUwD845n7eWvaSN7+eBRPvzKAatWrAXD1Td0YO/0N3v54FEPe+jdpGQ09OyatwH+JLGkT\nZO/bbmDZshW+xw2FQgx6fgCdOl/FCSedw+WXX8wxxzSz+JUgvt+xr7yxK2tWrC15/+RDz3HZuddw\naburycvdTPf/uxSAZYuX0739dVza7mqmTvyYOx70LnGrFsW9JDJPEqSInCYiBzmvq4lIfxGZICL/\nEpGDvYgZKT09jY4XnMuwYW96HeoPWrVszqpVa1mzZh35+flkZ4/jos7tLX4liO9n7AZp9WlzXmve\nfX18SdnOHbtKXh9Y7YCS2tmcL+bz6/9+A2DRvCU0SDsUr/g8YK7nvKpBDiPcSRzgeeBg4F9O2ase\nxSzxzNP9ufe+xygq8v9fp0bpDVmfu3tkpdwNeTRq5N0ljcVPnPh+xu77aB+eefQ/FO1VA3vkuQf4\n5Jv3adz0cN4c+tYftvvLFZ2Z8fFXnhwTeDOaT5C8SpAhVS1wXrdQ1T6qOkNV+wNHxNpQRHqKyFwR\nmVtUtDPuwBd2PI8tW75n/oJvKnDY+84ZcWQPfv5PYPGDi+9X7Dbnt2bb99tZuujbP3z2UJ8BnHtS\nZ9asWEv7Luft8dmFl7TnuJOOZviLr7t+TMWsBlk+i0WkeIihr0WkBYCIHAXkx9pQVQeragtVbREK\n1Yg78JlntqBzpyxWLp/J66+9yDnntGbE8EFx76eiNuTmkZmxe6j+jPQ08vI2W/xKEN+v2Ce3PJG2\nWX9m8px3eeLlR2nV+lQe/8/DJZ8XFRXxwbhpnHfhOSVlp/25JTfefi29e/Ql//eYf4L7xGqQ5XMD\ncLaIrAKOBb4SkdXAEOczzzzQbyCNj2hB06NO58qr/sYnn3xBj2t7exlyD3PmLqRp0yY0bpxJamoq\nXbt2YcLEKRa/EsT3K/agx1/i/FO6cEHLv9L35geZ/cU87r+1P5mNM0rWaZt1FmtXfgfA0ccfxUNP\n9qV3j7vZ9v12148nUpFq3Esi8+Q5SFX9CbjWGQX4CCdOrqr6V5UISGFhIbf36cek99+gSijE8BFj\nyMlZbvErQfwgY4sIjw16kJq1aiAC3y5ZyWP3PAHAnQ/dSvUa1XlqyAAANm3YTO8efT05jkR/bCde\nkshV3JSq6YEdXGWeVTDo+Ikyq2DQ8QOd1VC1tImvytTg4KPj/pvd/NOyCsXyQ9I+B2mMMfvKuhoa\nY1yT6K3S8bIEaYxxTSLfsqsIS5DGGNckeqt0vCxBGmNcYzVIY4yJwu5BGmNMFFaDNMaYKOwepDHG\nRJFsPWksQRpjXGM1SGOMiSLZ7kFaV0NjjGu8mpNGRDqIyLcislJE7vX4NEpYDdIY4xovapAiUgV4\nATgfyAXmiMh4Vc1xPdherAZpjHGNRwPmtgJWqupqVf0dGA108fREHAldgywe9sniW/zKGH/R5pmB\nxq8Ij+5ApgPrI97nAqd5E2pPCZ0gKzomXTER6amqg906nP0ltsW3+EHFL/h9Q9x/syLSE+gZUTR4\nr2MvbZ++tAYl+yV2z7JXScrYFt/iBx2/3CLnoXKWvRN7LpAZ8T4D8KV6n+wJ0hiz/5sDNBORJiJS\nFegGjC9jG1ck9iW2MabSU9UCEbkV+BCoAgxT1SV+xE72BBnYPaCAY1t8ix90fFep6iRgkt9xE3rS\nLmOMCZLdgzTGmCgsQRpjTBRJmSCD6rfpxB4mIltEZLGfcSPiZ4rIJyKyVESWiMjtPsc/UERmi8jX\nTvz+fsZ3jqGKiCwQkYl+x3birxWRb0RkoYjM9Tl2bRF5W0SWOf8PnOFn/GSTdPcgnX6by4notwl0\n96PfphO/DbADGKmqx/sRc6/4aUCaqs4XkVrAPOBiH89fgBqqukNEUoEZwO2q6lu3EBG5E2gBHKSq\nnfyKGxF/LdBCVb8PIPYI4HNVfcV5JKa6qv7o93Eki2SsQQbWbxNAVacD2/yKV0r8PFWd77z+BVhK\nuKuWX/FVVXc4b1Odxbd/hUUkA7gQeMWvmIlCRA4C2gBDAVT1d0uO+yYZE2Rp/TZ9SxCJREQaA82B\nWT7HrSIiC4EtwFRV9TP+c0BfoMjHmHtTYIqIzHO60fnlCGAr8Kpzi+EVEanhY/ykk4wJMrB+m4lE\nRGoC7wB9VPVnP2OraqGqnky4S1grEfHlVoOIdAK2qOo8P+LF0FpVTwEuAHo5t138kAKcArykqs2B\nnYCv9+CTTTImyMD6bSYK597fO8DrqvpuUMfhXN59CnTwKWRr4CLnHuBooJ2IvOZT7BKqutH5uQUY\nS/i2jx9ygdyIGvvbhBOmqaBkTJCB9dtMBE4jyVBgqao+E0D8+iJS23ldDTgPWOZHbFW9T1UzVLUx\n4d/7x6p6lR+xi4lIDadxDOfyNgvw5YkGVd0ErBeRPzlF5wK+NM4lq6Trahhkv00AEXkTaAvUE5Fc\n4GFVHepXfMK1qKuBb5z7gAD3O121/JAGjHCeJggB2aoayOM2AWkAjA3/O0UK8IaqfuBj/NuA153K\nwWrgOh9jJ52ke8zHGGPckoyX2MYY4wpLkMYYE4UlSGOMicISpDHGRGEJ0hhjorAEaRKKiDQOaiQk\nY/ZmCdL4wnku0pj9iiVIUyoReTRyLEkRGSAivUtZr62ITBeRsSKSIyIvi0jI+WyHiDwiIrOAM0Tk\nVBH5zBnE4UNnaDac8q9F5Cugl1/naExZLEGaaIYCPQCchNcNeD3Kuq2Au4ATgCOBvzrlNYDFqnoa\n4RGF/g1cqqqnAsOAAc56rwK9VdUGdzUJJem6Ghp3qOpaEflBRJoT7j63QFV/iLL6bFVdDSVdLc8i\nPFBCIeFBMwD+BBwPTHW64VUB8kTkYKC2qn7mrDeK8Cg4xgTOEqSJ5RXgWqAh4RpfNHv3Vy1+/6uq\nFjqvBViydy3RGdjC+ruahGSX2CaWsYSHKmtJePCPaFo5oyeFgMsJT7Owt2+B+sVzpIhIqogc5wyJ\n9pOInOWsd6V7h2/MvrEapIlKVX8XkU+AHyNqgqX5ChhI+B7kdMKJtbR9XQoMci6rUwiP/r2E8Igz\nw0RkF7ETsTG+stF8TFROjXA+cJmqroiyTlvg70FMjmWM1+wS25RKRI4FVgLToiVHY5Kd1SBNuYjI\nCYRbmCP95jzCY0xSsgRpjDFR2CW2McZEYQnSGGOisARpjDFRWII0xpgoLEEaY0wU/w8EByZlGT0Z\nywAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dt = DecisionTreeClassifier(random_state = 0)\n",
    "dt.fit(X_train,y_train) \n",
    "dt_score=dt.score(X_test,y_test)\n",
    "y_predict=dt.predict(X_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of DT: '+ str(dt_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of DT: '+(str(precision)))\n",
    "print('Recall of DT: '+(str(recall)))\n",
    "print('F1-score of DT: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter optimization (HPO) of decision tree using Bayesian optimization with tree-based Parzen estimator (BO-TPE)\n",
    "Based on the GitHub repo for HPO: https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████| 50/50 [00:04<00:00, 11.13trial/s, best loss: -0.9936567164179104]\n",
      "Decision tree: Hyperopt estimated optimum {'min_samples_leaf': 2.0, 'max_depth': 47.0, 'min_samples_split': 3.0, 'max_features': 19.0, 'criterion': 0}\n"
     ]
    }
   ],
   "source": [
    "# Hyperparameter optimization of decision tree\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'max_features': int(params['max_features']),\n",
    "        \"min_samples_split\":int(params['min_samples_split']),\n",
    "        \"min_samples_leaf\":int(params['min_samples_leaf']),\n",
    "        \"criterion\":str(params['criterion'])\n",
    "    }\n",
    "    clf = DecisionTreeClassifier( **params)\n",
    "    clf.fit(X_train,y_train)\n",
    "    score=clf.score(X_test,y_test)\n",
    "\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"max_features\":hp.quniform('max_features', 1, 20, 1),\n",
    "    \"min_samples_split\":hp.quniform('min_samples_split',2,11,1),\n",
    "    \"min_samples_leaf\":hp.quniform('min_samples_leaf',1,11,1),\n",
    "    \"criterion\":hp.choice('criterion',['gini','entropy'])\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=50)\n",
    "print(\"Decision tree: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of DT: 0.9936567164179104\n",
      "Precision of DT: 0.9940667447648622\n",
      "Recall of DT: 0.9936567164179104\n",
      "F1-score of DT: 0.9938179408993949\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.99      1.00      3645\n",
      "           1       0.99      1.00      0.99       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       1.00      1.00      1.00       609\n",
      "           4       0.45      0.71      0.56         7\n",
      "           5       0.98      1.00      0.99       251\n",
      "           6       0.99      0.98      0.98       436\n",
      "\n",
      "    accuracy                           0.99      5360\n",
      "   macro avg       0.92      0.95      0.93      5360\n",
      "weighted avg       0.99      0.99      0.99      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8FVXawPHfc5OgNCnCAikKCq5d\nUZqiiKiACMJaUNeCLoq6oPLqgl0W27LWBdd1Bak2iAXpAqKIqHQQpVchEIoCKrBqyvP+cSfhgrlJ\nbpxyuXm+fuaTe8+dmWfmxjycMzPnHFFVjDHG/FYo6AMwxph4ZQnSGGOisARpjDFRWII0xpgoLEEa\nY0wUliCNMSYKS5DGGBOFJUhjjInCEqQxxkSRHPQBFEvEuvkYEwRVKctmOd+tj/lvNqXWcWWK5Ye4\nTpA5O9cFFjul9vEkp6QGFj83Z2u5jZ+bsxXA4gcc38R5gjTGHGby84I+AldZgjTGuEfzgz4CV1mC\nNMa4J98SpDHGFEkTrAZpj/kYY9yTnx/7UgIROVJE5onIVyKyTET6O+UjRGSDiCxxljOdchGRQSKy\nVkSWishZEfvqJiJrnKVbSbGtBmmMcY83NchfgDaquldEUoDZIjLF+ayPqr57yPqXAo2cpTnwCtBc\nRGoC/YAmgAILRWS8qu6OFthqkMYY9+Tnxb6UQMP2Om9TnKW45y07A6Oc7eYA1UWkHtAOmK6qu5yk\nOB1oX1xsS5DGGPdofsyLiPQQkQURS49DdysiSSKyBNhBOMnNdT56ymlGvygiRzhlacDmiM2znLJo\n5VFZE9sY454y3MVW1cHA4BLWyQPOFJHqwFgRORV4ENgGVHC2vx94HCiqZ44WUx6V1SCNMa5RzY95\niW3/ugeYCbRX1WynGf0LMBxo5qyWBWREbJYObC2mPCpLkMYY93hzF7u2U3NERCoCFwMrneuKiIgA\nXYBvnE3GAzc5d7NbAD+oajYwFWgrIjVEpAbQ1imLyprYxhj3eHMXux4wUkSSCFfqMlV1ooh8LCK1\nCTedlwB3OOtPBjoAa4H9wC0AqrpLRJ4A5jvrPa6qu4oLbAnSGOMeD/piq+pSoHER5W2irK9Azyif\nDQOGlTb2YZUgf/nlV7r17MOvOTnk5eZxyYXn0evWG1FVBg0eybRPZhMKhbjmT5dxw9WdmTj1Y4a+\n+Q4AlSpW5NG/9eLERscV7i8vL49rut/NH2rX4j/P9nf1WEOhEHPnTGHrlm10/lOJz6O6ql3b1rzw\nwuMkhUIMG/42zzz7sm+x09NTGTFsIHXq1iY/P5/XXnuTl/491Lf4ANWqHcXgV5/jlFP+iKpy2233\nMWfuQt/iB/m7X7t6Dj/t3UteXj65ubm0OKeDr/GtL3aAKlRIYdigAVSqVJGc3FxuuvNvnN+iCeu/\n3cy2Hd8x4a3BhEIhvt+9B4C01LqM+PczVDuqKp99OZ/+zwzi7SH/KtzfG++M47j6x7B3337Xj/Xu\nu25l5co1HFW1quv7Lk4oFGLQwKdo3+E6srKymfPlZCZMnMaKFWt8iZ+bm0ufvv1ZvOQbqlSpzLy5\nH/LRjFm+xQd48YXHmTr1E665tgcpKSlUqlTRt9gQ3O++wMWXXM3330d99tnEwLObNCJyoojc73T5\nGei8Pul37rPwf/bc3Fxyc3MREcaMncSdt/yZUCh8OkfXqA5A49NOptpR4f9JTz/lRLbv+K5wX9t2\n7GTWF/O4slO733NIRUpLq0eHSy9i2LC3Xd93SZo1bcy6dRvZsGETOTk5ZGaO43IPzjGabdt2sHhJ\n+Fr53r37WLlyDWmpdX2LX7VqFc4/rznDhoe/+5ycHH744Uff4gf5u48LHtykCZInCVJE7gdGE754\nOo/wRVEB3haRB37PvvPy8riyW09adbyOc5o25vRTTmTzlmymzPiUrn+5mzvue5RvN2/5zXbvT5zK\neS2aFL7/58BXufev3RFx/yt44fn+PPDgk+QH8MtPTavL5qwDTy5kbckm1ccEFenYY9M584xTmTtv\nsW8xjzvuWL777nuGvvYi8+dN5dX/PutrDTLI3z2AqjJl8tvMnTOFW7tfH8ABxP6geDzzqgbZHWiq\nqgNU9Q1nGUD4OaXuv2fHSUlJvDfyZWaMfZ2vl69mzfqN/JqTwxEVKpA5bBBXdmrPo0+/eNA28xZ+\nxfsTp3HvX/8CwMzP51KzRnVOObHR7zmUIl3W4WJ27PiORYu/dn3fpRF+4uFg4WvW/qpcuRKZY4Zw\n79/68dNPe0vewCXJSUk0bnwar746iqbN2rFv337u79vLl9hB/+4BWrXuQrPm7enY6QbuvPNmzj+v\nub8HYDXIUskHihovvp7zWVSR3Y5eGxW9mXJU1So0Pet0Zs9ZQN3atbik9XkAXHzBuaxet6FwvVVr\nN/DYgH/x0oDHqF7tKAAWL13OzNlzaHtlN/r0G8C8hV9xf/9nYj7Jopx7bhM6dWzL2tVzePON/3Dh\nhS0ZOWKQK/sujS1Z2WSkH/jq09PqkZ293bf4AMnJybwzZghvvz2WDz6YUvIGLsrakk1WVjbz5odr\nre+/P4nGZ57mS+ygf/dA4e96587vGTduCk2bnulrfNW8mJd45lWC7A3MEJEpIjLYWT4EZgD3FLeh\nqg5W1Saq2uTWm6476LNdu/fwo1Mb+fmXX5gzfzENjs2gTatzmLtwCQDzF3/NsRnh7pXZ23bQ+6En\n+Mdjfah/THrhfv7vzluY8cEbTHtvJM/2f4BmZ5/BP/v1deXEH35kAPWPa0LDE1pw/Q1/5ZNPPqfb\nzXe7su/SmL9gCQ0bNqB+/QxSUlLo2rUzEyZO8y0+wJDBz7Ni5Vr+NbDY3mOe2L59J1lZWznhhOMB\naNPmPFasWO1L7KB/95UqVaRKlcqFry+5+AKWLVvlW3wg4ZrYntzFVtUPReQEwk3qNMLXH7OA+fo7\n/snY+f1uHn7yOfLy89F8pV2b82ndsjlnnX4K9/d/htfHfEClikfS/4HeALwy/C1++PEnnnwu/JhL\nUlISmcP8/Rfdb3l5edzT+xEmT3qLpFCIESPHsHy5PwkCoOW5TbnxhqtY+vVyFswPJ+ZHHx3AlA8/\n9u0Y7vm/Rxk18iUqVEhhw4ZNdL/1Xt9iB6lOndq8+074kark5CRGj/6AqdNm+nsQcd5kjpUEcX2q\ntMoyhaRbbFZDm9WwXMcv47SvPy/8IOa/2SPP7mLTvhpjygGb1dAYY6KI82uKsbIEaYxxT4Jdg7QE\naYxxj9UgjTEmCqtBGmNMFJYgjTGmaPHeMyZWliCNMe6xGqQxxkSRYDdpbNIuY4yJwmqQxhj3WBPb\nGGOiSLAmtiVIY4x7rAZpjDFRWA3SPym1jw80fsGwUxbf4pfH+GViNUhjjInCEqR/gh4w9qTaTQOL\nv2Ln/MDPv1wPGGvxy8aa2MYYE4XVII0xJgqrQRpjTBQJVoO0robGGPd4MO2riBwpIvNE5CsRWSYi\n/Z3yBiIyV0TWiMgYEanglB/hvF/rfF4/Yl8POuWrRKRdSbEtQRpj3JOfH/tSsl+ANqp6BnAm0F5E\nWgD/BF5U1UbAbqC7s353YLeqNgRedNZDRE4GrgVOAdoD/xGRpOICW4I0xrjHgwSpYXudtynOokAb\n4F2nfCTQxXnd2XmP8/lFIiJO+WhV/UVVNwBrgWbFxbYEaYxxj2rMi4j0EJEFEUuPQ3crIkkisgTY\nAUwH1gF7VDXXWSULSHNepwGbw4ejucAPwNGR5UVsUyS7SWOMcU8ZbtKo6mBgcAnr5AFnikh1YCxw\nUlGrOT8lymfRyqOyBGmMcY/Hd7FVdY+IzARaANVFJNmpJaYDBU+4ZwEZQJaIJAPVgF0R5QUitymS\nNbGNMe7x5i52bafmiIhUBC4GVgCfAFc5q3UDxjmvxzvvcT7/WFXVKb/WucvdAGgEzCsuttUgjTHx\nrh4w0rnjHAIyVXWiiCwHRovIk8BiYKiz/lDgdRFZS7jmeC2Aqi4TkUxgOZAL9NQSZhmzBGmMcY8H\nTWxVXQo0LqJ8PUXchVbVn4Gro+zrKeCp0sa2BGmMcY8We8/jsJOQ1yDbtW3Nsm9msXL5bPr26elJ\njApHVGDMh8MZ+8mbTJg1ml59bwOg+XlNeO+jUYz/9G3+8VI/kpLCz6G2ad+KD2a+yfsfv8E700Zy\nVvMzPDku8Of8LX78xY6H+B49KB4Y0TjO+MkV0mI+uFAoxIpln9G+w3VkZWUz58vJ3HDjX1mxYk1M\n+ynNcGeVKldk/77/kZycxBsThjDgsRd5YfDT/OXKnmxcv4m77u/B1s3beO+t8YXrApxwckNeHPI0\nl7XsGnXfZR3uzM3zDyr+7xnuK8j4bn73gcdXLeqRmBL9b+jfYv6brdj9uTLF8kPC1SCbNW3MunUb\n2bBhEzk5OWRmjuPyTiV2uSyTgoSXnJJMSkoy+Xn5/Prrr2xcvwmAL2bOo23HCw9aF6BSpYp49Q+T\nn+dv8eMndjzEBzy5ix2kQBKkiNzi1b5T0+qyOevAo01ZW7JJTa3rSaxQKMT7H7/B7OVT+eLTeSxd\ntIyU5GROOSP8DGvbTm2om1qncP2LO7Rm0ueZvPLmCzzS+0lPjsnP87f48RM7HuIDaL7GvMSzoGqQ\n/aN9ENntKD9/X8w7Dne5PJhXtbX8/HyuaHMDF57RkdMan0yjE4/jvtsf4YEn/o8xHw5n/9795OYd\neIrgo8kzuaxlV+7q1pe7H7jdk2Py8/wtfvzEjof4QMJdg/TsLraILI32EVAnymcHdTsqyzXILVnZ\nZKQfuHaTnlaP7Oztse4mJj/9uJd5XyzivDbnMPw/b3Lj5eGupOe2bs6xxx/zm/UXzFlMxrHpVK9Z\njT27fnD1WII4f4sffOx4iA/EfZM5Vl7WIOsANwGdili+9yro/AVLaNiwAfXrZ5CSkkLXrp2ZMHGa\n63FqHF2dqkdVAeCII4/gnFbN2LDmW2rWqgFASoUUbu11E2NGvg/AMQ3SC7c9+bQ/klIh2fXkCP6d\nv8WPr9jxEB+AfI19iWNePgc5EaiiqksO/cDpS+mJvLw87un9CJMnvUVSKMSIkWNYvny163Fq16nl\nPMYTIiQhPhz/ETOnz+Zv/e6i9SXnEQqFGD3iPebOXgBA245t6Hx1B3Jyc/nl51+4t8fDrh8T+Hf+\nFj++YsdDfCDum8yxSrjHfNxisxrarIblOn4ZH/PZP/COmP9mK93z37h9zMd60hhj3BPHFa6ysARp\njHFPgjWxLUEaY9wT5zddYmUJ0hjjngR7zMcSpDHGPQlWg0y4vtjGGOMWq0EaY1yjdpPGGGOiSLAm\ntiVIY4x77CaNMcZEYTVIY4yJwq5BGmNMFFaDNMaYKOwapDHGRGE1SP8UDPsUlBU75wcaP+jzt/jl\nO35Z2HOQPiqv4yEWxE8JMH6OjQdZ7uOXidUgjTEmCkuQxhgTRYLdpLHBKowx7vFg0i4RyRCRT0Rk\nhYgsE5F7nPK/i8gWEVniLB0itnlQRNaKyCoRaRdR3t4pWysiD5QU22qQxhjXqDdN7FzgPlVdJCJV\ngYUiMt357EVVfS5yZRE5GbgWOAVIBT4SkROcj18GLgGygPkiMl5Vl0cLbAnSGOMeDxKkqmYD2c7r\nn0RkBZBWzCadgdGq+guwQUTWAs2cz9aq6noAERntrBs1QVoT2xjjnvz8mBcR6SEiCyKWHtF2LyL1\ngcbAXKeol4gsFZFhIlLDKUsDNkdsluWURSuPyhKkMSZQqjpYVZtELIOLWk9EqgDvAb1V9UfgFeB4\n4EzCNcznC1YtKkwx5VFZE9sY4x6PHvMRkRTCyfFNVX0fQFW3R3w+BJjovM0CMiI2TwcKHu6MVl4k\nq0EaY9zjzV1sAYYCK1T1hYjyehGr/Qn4xnk9HrhWRI4QkQZAI2AeMB9oJCINRKQC4Rs544uLbTVI\nY4xrVD2pQbYEbgS+FpElTtlDwHUicibhZvJG4HbnGJaJSCbhmy+5QE9VzQMQkV7AVCAJGKaqy4oL\nbAnSGOMeb+5iz6bo64eTi9nmKeCpIsonF7fdoSxBGmPcY10NjTGmaB49KB4YS5DGGPckWIJMyLvY\n7dq2Ztk3s1i5fDZ9+/RM+PhDBj/PlqyvWLx4RmHZ6aefzGezxrN40UeMHTuCqlWreH4cBcrb9x8v\nseMhPvllWOJYwiXIUCjEoIFP0bHTDZx2xoVcc00XTjqpUULHHzkqk44drz+o7NX/PstDDz9N47Mu\nZtwHU7jvvjs9PYYC5fH7j4fY8RAfwk3sWJd45lmCFJETReQi5+n3yPL2XsUEaNa0MevWbWTDhk3k\n5OSQmTmOyzu1K3nDwzj+7Nlz2bV7z0FlJ5xwPJ99NgeAj2Z8xp/+1KGoTV1XHr//eIgdD/EBT56D\nDJInCVJE7gbGAXcB34hI54iPn/YiZoHUtLpszjrwcHzWlmxSU+t6GTKu4hdYtmwVnTq1BeCqKzuS\nke7P6NRBn3+Q8cvzuReyJnap3AacrapdgNbAowVjuFH080yFIjuu5+fvizlw+KH7g3n08Gpcxi9w\nW497ufOOm5k7ZwpVqlbm119zfIkb9PkHGb88n3thvARrYnt1FztJVfcCqOpGEWkNvCsix1JCgnQ6\nqg8GSK6QFvO3tyUr+6DaUnpaPbKztxezhbuCjl9g1ap1dLjszwA0anQcHS69yJe4QZ9/kPHL87kX\nivMaYay8qkFuc7oAAeAky45ALeA0j2ICMH/BEho2bED9+hmkpKTQtWtnJkyc5mXIuIpfoHbto4Fw\nreKhB+9h8ODXfYkb9PkHGb88n3sBq0GWzk2E+0AWUtVc4CYRedWjmADk5eVxT+9HmDzpLZJCIUaM\nHMPy5au9DBl4/Ndff5kLWp1DrVo12bB+AY8//hxVqlTmjjtvBuCDDyYzYuQYT4+hQHn8/uMhdjzE\nBxKuBilBXB8rrbI0sd1i077atK/lOr5qsZfCovm+0wUx/80ePeHTMsXyQ8I9B2mMMW6xrobGGPck\nWBPbEqQxxjUJNi22JUhjjIssQRpjTNGsBmmMMVFYgjTGmCgsQRpjTDRle3wyblmCNMa4xmqQxhgT\nheZbDdIYY4pkNUhjjImijF2445YlSGOMa6wGaYwxUdg1SB8VDPtUXuPnlPPzt/jBxi+LOB49sUzi\nOkEGPR5jeY/fOs2faRoONXNLeH7vcj0eYxzEL4tEq0HaeJDGGNdovsS8lEREMkTkExFZISLLCiYA\nFJGaIjJdRNY4P2s45SIig0RkrYgsFZGzIvbVzVl/jYh0Kym2JUhjTLzLBe5T1ZOAFkBPETkZeACY\noaqNgBnOe4BLgUbO0gN4BcIJFegHNAeaAf0Kkmo0liCNMa5RjX0peZ+araqLnNc/ASuANKAzMNJZ\nbSTQxXndGRilYXOA6iJSD2gHTFfVXaq6G5gOtC8udlxfgzTGHF68vgYpIvWBxsBcoI6qZkM4iYrI\nH5zV0oDNEZtlOWXRyqOyGqQxxjWqEvMiIj1EZEHE0qOofYtIFeA9oLeq/ljMYRSVpbWY8qhKTJAi\nUkdEhorIFOf9ySLSvaTtjDHlj+aXYVEdrKpNIpbBh+5XRFIIJ8c3VfV9p3i703TG+bnDKc8CMiI2\nTwe2FlMeVWlqkCOAqUDBMwergd6l2M4YU87kq8S8lEREBBgKrFDVFyI+Gg8U3InuBoyLKL/JuZvd\nAvjBaYpPBdqKSA3n5kxbpyyq0lyDrKWqmSLyIICq5opIXim2M8aUMx71xW4J3Ah8LSJLnLKHgAFA\nptOi3QRc7Xw2GegArAX2A7eEj013icgTwHxnvcdVdVdxgUuTIPeJyNE4bfWCjFzKEzPGlCNe3KRR\n1dkUff0Q4De9GVRVgZ5R9jUMGFba2KVJkPcSrrIeLyKfA7WBq0obwBhTfpS7roaqukhELgD+SDiL\nr1LVHM+PzBhz2Em0roYlJkgRuemQorNEBFUd5dExGWMOU6W56XI4KU0Tu2nE6yMJt/kXAZYgjTEH\nKXcD5qrqXZHvRaQa8LpnR2SMOWwl2jXIsvSk2U+4E3hcGjL4ebZmfcWSxTMCO4Z2bVuz7JtZrFw+\nm759iryZlhDxqxxVmf6vPsaomcMY+clQTj7rJKpWr8pzb/2TNz4bwXNv/ZMq1aoAULlqZZ4e/gSv\nTXuV4TNeo33Xdp4dV5Dff3n53UfjxXOQQSpNT5oJIjLeWSYCqzjwQGbcGTUqk8s6Xh9Y/FAoxKCB\nT9Gx0w2cdsaFXHNNF046yb9/T/yM36t/T+bNnM9Nrf9C97a3s2ntJv7c81oWfb6YG86/mUWfL+bP\nPa8FoEu3y9m45ltubXs7va++j78+djvJKe4PBRDk91+efvfRlKWrYTwrTQ3yOeB5Z/kH0EpVHyh+\nExCRZiLS1Hl9sojcKyIdftfRlsJns+eya/cer8NE1axpY9at28iGDZvIyckhM3Mcl3fyrrYUVPxK\nVSpxRvPTmPT2FAByc3LZ++M+WrY9lw/fmQbAh+9M47x2LYFw06tS5UoAVKxckZ/2/ERervv9DYL8\n/svL7744XozmE6RiE6SIJAGPquqnzvK5qmaVtFMR6QcMAl4RkX8A/waqAA+IyMNuHHi8Sk2ry+as\nA907s7Zkk5paN+Hipx5Tjz27fuCBF/ow5MP/0ufZezmy4pHUrFWDXTvCnRN27dhFjaOrAzB2xAcc\n2+gY3ls4huEfDeGlx/6DevDXEeT3X15+98UpV01sVc0D9js3ZmJxFeHuQa0IP9HeRVUfJzwe2zXF\nbRg5skd+/r4YwwYv3G30YF4kgqDjJyUnccKpjRj3+gRua38H/9v/c2FzuijNWjdh7bJ1XHn2Ndza\n7nbuebIXlapUcv24gvz+y8vvvjjlsYn9M+E+kEOdYcwHicigErbJVdU8Vd0PrCsYmkhV/wcUOzFk\n5MgeoVDlUp1EPNmSlU1G+oG5RNLT6pGdvT3h4u/M3snO7J2sWLwSgE8nzaLRaY3Y9d1uav6hJgA1\n/1CT3d+HL3e079qeWVM+Cx/jxq1kb97GMQ0zit757xDk919efvflSWkS5CTgUWAWsNBZFpSwza8i\nUlA9OLug0KmJJtjMuQebv2AJDRs2oH79DFJSUujatTMTJk5LuPi7du5mx9adZByXDsDZ553Ft2u+\n5YvpX9L+6rYAtL+6LZ9P+wKAHVt2cPZ54alBatSqTsbxGWR/m+36cQX5/ZeX331xEq2JXZrbiNVV\ndWBkQcGkOcVopaq/AKgeNJV4CgeGJ/LEG6+/zAWtzqFWrZpsXL+A/o8/x/ARo70MeZC8vDzu6f0I\nkye9RVIoxIiRY1i+fHVCxh/06L955KUHSa6QQva32Qy471lCEqLffx+hw7Xt2b5lB3+/4wkARg18\ngwde6MOwj4YgwOCnh/DD7uLGPC2bIL//8vS7jybO77nETEq6RiEii1T1rEPKFqtqY0+PDEiukBbY\n9x0P064GHd+mfS3H8ct4cfCLelfG/Dd7bvZ7cVuNjFqDFJHrgD8DDURkfMRHVYHvvT4wY8zhJ95v\nusSquCb2F0A2UIvwM5AFfgKWenlQxpjDU6LdYIiaIFX1W+Bb4JzidiAiX6pqsesYY8oHjTqu7eHJ\njb5eR7qwD2NMAshPsLs0biTIBPtKjDFllW81SGOMKVqiNbFLM5pPL2eKxKiruHg8xpjDWH4ZlnhW\nmp40dYH5IpIpIu3ltx0+b/TguIwxhyFFYl7iWYkJUlUfITxA7lDgZmCNiDwtIsc7n3/j6REaYw4b\n5bEGWTDP7DZnyQVqAO+KyDMeHpsx5jCTaAmyNLMa3k24//R3wGtAH1XNEZEQsAbo6+0hGmMOF/He\nZI5Vae5i1wKucB4cL6Sq+SLS0ZvDMsYcjhJsWuxSzWr4WDGfrXD3cIwxh7NEew6yLLMaGmNMuRDX\nD4oXDPtk8YNRMOxYUII+//IevywSrVtdXCfIoMdDtPjleDxEi18mXtyVFpFhQEdgh6qe6pT9HbgN\n2Oms9pCqTnY+exDoDuQBd6vqVKe8PTAQSAJeU9UBJcWO6wRpjDm85BcxcZgLRhCeGXXUIeUvqupz\nkQUicjJwLXAKkAp8JCInOB+/DFwCZBHu/DJeVZcXF9gSpDHGNV40sVV1lojUL+XqnYHRzpQvG0Rk\nLdDM+Wytqq4HEJHRzrrFJki7SWOMcU1ZHhSPnOrZWXqUMlwvEVkqIsMixotIAzZHrJPllEUrL5Yl\nSGOMa/Il9iVyqmdnGVyKUK8AxwNnEp75oGDWg6La+FpMebGsiW2McY1fz0GqauGE3yIyBJjovM0C\nIidcTwcK7jpFK4/KapDGGNdoGZayEJF6EW//BBQMmjMeuFZEjhCRBoQH2pkHzAcaiUgDEalA+EZO\n5GSERbIapDHGNV50NRSRt4HWQC0RyQL6Aa1F5EzCOXYjcDuAqi4TkUzCN19ygZ6qmufspxcwlfBj\nPsNUdVlJsS1BGmNc48VzkKp6XRHFQ4tZ/yngqSLKJwOTY4ltCdIY4xrrSWOMMVGUu9F8jDGmtOJ9\nANxYWYI0xrjGEqQxxkShCdbETrjnINPTU/lo2jt8vXQmXy35mLt6dff9GNq1bc2yb2axcvls+vbp\nafF9tnb1HBYv+ogF86cx58uYblr+bkGfe9Dxy92cNIeb3Nxc+vTtz+Il31ClSmXmzf2Qj2bMYsWK\nNb7ED4VCDBr4FO07XEdWVjZzvpzMhInTLL5P8QtcfMnVfP/9bl9jBn3uQcdPRAlXg9y2bQeLl4Qf\nqt+7dx8rV64hLbWub/GbNW3MunUb2bBhEzk5OWRmjuPyTu0sfjkQ9LkHHR8SrwbpW4IUkUPHcvPc\nscemc+YZpzJ33mLfYqam1WVz1oEunllbskn1MUGX9/gAqsqUyW8zd84Ubu1+vW9xgz73oOODf10N\n/eJJE1tEDu3jKMCFIlIdQFUv9yJupMqVK5E5Zgj3/q0fP/201+twhaSIAUPD04pbfL+0at2F7Ozt\n1K59NB9OGc2qVWv5bPZcz+MGfe5Bxwd7DrK00gn3hXyNA0MNNeHAkERROWPB9QCQpGqEQpVjDp6c\nnMw7Y4bw9ttj+eCDKTFv/3tsycomI/3AUPnpafXIzt5ezBYW320F8Xbu/J5x46bQtOmZviTIoM89\n6PgQ/03mWHnVxG4CLAQeBn5mYAOTAAASnklEQVRQ1ZnA/1T1U1X9tLgNI8eGK0tyBBgy+HlWrFzL\nvwaWZlg5d81fsISGDRtQv34GKSkpdO3amQkTp1l8n1SqVJEqVSoXvr7k4gtYtmyVL7GDPveg40Pi\nXYP0pAapqvnAiyLyjvNzu1exDtXy3KbceMNVLP16OQvmh//nePTRAUz58GM/wpOXl8c9vR9h8qS3\nSAqFGDFyDMuXr/YltsWHOnVq8+474XEMkpOTGD36A6ZOm+lL7KDPPej4EP/XFGMlflyjEJHLgJaq\n+lAs2yVXSAvs+y7PswoGHT9eZvUr1/G1bI98P3PsDTH/zfb99o24vXLpS61OVScBk/yIZYwJTrw3\nmWOVcA+KG2OCk2hNbEuQxhjX5CdYirQEaYxxjTWxjTEmisSqP1qCNMa4yGqQxhgThXU1NMaYKOwm\njTHGRJFY6TEBx4M0xhi3WA3SGOMau0ljjDFR2DVIY4yJIrHSoyVIY4yLrInto4Jhnyy+xbf4hwcv\nmtgiMgzoCOxQ1VOdsprAGKA+sBHoqqq7JTzvxECgA7AfuFlVFznbdAMecXb7pKqOLCm23cU2xrjG\no0m7RgDtDyl7AJihqo2AGc57gEuBRs7SA3gFChNqP6A50AzoJyI1Sgoc1zXI8jpgbHmPHxcDxgJp\n1U8OJP6WPcuB4M+/LLxoYqvqLBGpf0hxZ6C183okMBO43ykfpeGRwOeISHURqeesO11VdwGIyHTC\nSfft4mLHdYI0xhxe1L/bNHVUNRtAVbNF5A9OeRqwOWK9LKcsWnmxrIltjHFNWSbtEpEeIrIgYunx\nOw6hqN7gWkx5sawGaYxxTVlu0qjqYCDWKUi3i0g9p/ZYD9jhlGcBGRHrpQNbnfLWh5TPLCmI1SCN\nMa7x6CZNUcYD3ZzX3YBxEeU3SVgLwtNOZwNTgbYiUsO5OdPWKSuW1SCNMa7x6DGftwnX/mqJSBbh\nu9EDgEwR6Q5sAq52Vp9M+BGftYQf87kFQFV3icgTwHxnvccLbtgUxxKkMcY1Ht3Fvi7KRxcVsa4C\nPaPsZxgwLJbYliCNMa7x8S62LyxBGmNcY10NjTEmikSrQdpdbGOMicJqkMYY11gT2xhjosjXxGpi\nW4I0xrgmsdJjAl6DHDL4ebZmfcWSxTMCO4Z2bVuz7JtZrFw+m759inwky+IfxvFT0+ryzvjhzJwz\nno+/GEf3228A4N77/8qCZR8zbdZ7TJv1Hm0uOb9wm17/dyuzF05h1ryJXNCmpevHVCDo7z4fjXmJ\nZwmXIEeNyuSyjtcHFj8UCjFo4FN07HQDp51xIddc04WTTmpk8RMofm5uLv0feYbWLS6nU9vruPnW\n62j0x+MBGPLKKNq2upK2ra7k4+mfAdDoj8fT+YoOtDnncq6/6naefu4RQiH3//SC/u4hfBc71v/i\nWcIlyM9mz2XX7j2BxW/WtDHr1m1kw4ZN5OTkkJk5jss7tbP4CRR/x/bv+GbpCgD27d3PmtXrqVvv\nD1HXb9fhQsa9P5lff81h86YtbFy/mcZnn+bqMUHw3z2UbTSfeOZLghSR80TkXhFp60e8IKWm1WVz\n1oEBR7O2ZJOaWtfiJ2j89IxUTj39JBYvXArALbf9memz3+f5l56gWrWjAKhbrw5bt2wr3CZ76zbq\n1qvj+rEE/d2DNbFLRUTmRby+Dfg3UJXwMOcPRN0wAYSnxDiY+nhnz+L7F79S5UoMGfUv+j04gL0/\n7WPUsDGc27g9bc+/kh3bd/LYk318Paagv3uwJnZppUS87gFcoqr9CQ8xVOwFwsjBM/Pz93l0eN7Z\nkpVNRvqBofLT0+qRnb3d4idY/OTkZIaM/Bdj35nElIkfAfDdzu/Jz89HVXlz5Luc6TSjs7duIzXt\nQE2uXmpdtm/bUeR+f4+gv3uwJnap9+uMu3Y0IKq6E0BV9wG5xW2oqoNVtYmqNgmFKnt0eN6Zv2AJ\nDRs2oH79DFJSUujatTMTJk6z+AkW//mXHmft6vUM/s+BifH+UKdW4etLO17MqhVrAJg25RM6X9GB\nChVSyDgmjQbHH8PihV+7fkxBf/cQrrHGusQzr56DrAYsJDzMuYpIXVXdJiJVKHroc9e88frLXNDq\nHGrVqsnG9Qvo//hzDB8x2suQB8nLy+Oe3o8wedJbJIVCjBg5huXLV1v8BIrftMVZXHVtZ5YvW8W0\nWe8BMOCJf9Hlyg6cfNqJqCpZm7Zy///9HYDVK9cx4YMP+WTOePJy83i4z5Pk57tfdwr6uwdvxoMM\nkvh8fagS4cl2NpRm/eQKaYF92+V5VsGg49ushnEwq6FqmSoynY7pGPPf7IRNEz2tNP0evvakUdX9\nQKmSozHm8BPvN11iZV0NjTGuSbQmtiVIY4xr4v2mS6wsQRpjXBPvj+3EyhKkMcY1iXYNMuH6Yhtj\njFusBmmMcY3dpDHGmCjsJo0xxkRhNUhjjIki0W7SWII0xrjGJu0yxpgoEis9WoI0xrgo0a5B2nOQ\nxhjXeDXlgohsFJGvRWSJiCxwymqKyHQRWeP8rOGUi4gMEpG1IrJURM4q6/nEdQ2yYNgpi2/xg1Aw\n7FhQgj7/svD4MZ8LVfW7iPcPADNUdYAzlcsDwP3ApUAjZ2kOvOL8jJnVII0xrvF50q7OQMGQ7iOB\nLhHlozRsDlBdROqVJUBc1yDL64Cx5T1+vAyYG3T80+u0CCT+0u1zyrxtWR7zEZEehOeuKjBYVQf/\nZtcwTUQUeNX5vI6qZgOoaraIFMy9mwZsjtg2yynLjvXY4jpBGmMOL2VpYjvJ7tCEeKiWqrrVSYLT\nRWRlMesWNUJ5maqqliCNMa7x6i62qm51fu4QkbFAM2C7iNRzao/1gIKpIrOAjIjN04EyXdC1a5DG\nGNd4MauhiFQWkaoFrwlPH/0NMB7o5qzWDRjnvB4P3OTczW4B/FDQFI+V1SCNMa7xqAZZBxgrIhDO\nWW+p6ociMh/IFJHuwCbgamf9yUAHYC2wH7ilrIEtQRpjXONFX2xVXQ+cUUT598BFRZQr0NON2NbE\nNsaYKKwGaYxxjQ1WYYwxUdhwZ8YYE4XVII0xJgqrQRpjTBRWgzTGmCisBmmMMVEkWg0yIZ+DvKtX\nd5YsnsFXSz7m7rtu9T1+u7atWfbNLFYun03fPq48r2rxD5P4fsYOhUKMmT6Sl15/DoB/vPx3xs8e\nzfsz36D/iw+TnJwEwM1/vZ7Mj0aS+dFI3p/5Bou3zOao6kd5ckxahv/iWcIlyFNO+SPdu/+Zc869\njLPOvoTLOlxMw4YNfIsfCoUYNPApOna6gdPOuJBrrunCSSc1svjlIL7fsa+/rSsb1mwsfD/p/alc\nft61XNH6Bo48sgJXXH85ACP+8yZdL+5G14u7MfCp/7Lwy8X8uOdHT45JNT/mJZ55kiBFpLmIHOW8\nrigi/UVkgoj8U0SqeRGzwIknNmLu3EX8738/k5eXx6zP5tClc3svQx6kWdPGrFu3kQ0bNpGTk0Nm\n5jgu79TO4peD+H7GrlOvNq0ubsn7b44vLJs948vC118vXkGden/4zXaX/ukSpoyd7skxge8D5nrO\nqxrkMMKdxAEGAtWAfzplwz2KCcCyZSs5//wW1KxZg4oVj+TS9m1IT/dv4NPUtLpszjowslLWlmxS\nU+ta/HIQ38/YfZ/ozQtP/Jv8ImpgyclJdLqqPZ9/cvDAt0dWPIKWF7Zg+qSZnhwTeDOaT5C8ukkT\nUtVc53UTVS2YNGe2iCwpbsPI0YUlqRqhUOWYAq9cuZZnn32ZD6e8zb69+/hq6XLycvNiPf4yc0Yc\nOYif/xNY/ODi+xW71SUt2fXdblYsXUWTcxv/5vOHB/Rh4ZwlLJr71UHlF7Q9jyXzl3rWvAab1bC0\nvhGRgiGGvhKRJgAicgKQU9yGqjpYVZuoapNYk2OB4SNG06x5ey686Ep2797DmrUbyrSfstiSlU1G\nRI01Pa0e2dnbLX45iO9X7DObnk7rtuczZf77PPPfJ2jW8mye/nc/AO647y/UOLo6z/Yb+Jvt2nf2\ntnkNiVeD9CpB3gpcICLrgJOBL0VkPTDE+cxTtWsfDUBGRipdulzK6DEfeB2y0PwFS2jYsAH162eQ\nkpJC166dmTBxmsUvB/H9ij3o6Ve45KzOXNr0Cvre8SjzPl/IQ736c8WfO3Fu6xbcf2e/3ySeKlUr\n0+ScxnwydZbrxxMpXzXmJZ550sRW1R+Am51RgI9z4mSpqi//lL8zZgg1j65BTk4ud9/9MHv2/OBH\nWADy8vK4p/cjTJ70FkmhECNGjmH58tUWvxzED/rcH3mmL9lZ23h9Ynh6lxmTP+XVF4YB0KbDBXzx\n6Vz+t/9nT48h3h/biZXEcxU3uUJaYAdXnmcVDDp+vMwqGHT8QGc1VC1q4qsS1al2Ysx/s9t/WFmm\nWH5IuOcgjTHGLdbV0BjjmkS7i20J0hjjmni+ZFcWliCNMa6J97vSsbIEaYxxjdUgjTEmCrsGaYwx\nUVgN0hhjorBrkMYYE0Wi9aSxBGmMcY3VII0xJopEuwZpXQ2NMa7xak4aEWkvIqtEZK2IPODxaRSy\nGqQxxjVe1CBFJAl4GbgEyALmi8h4VV3uerBDWA3SGOMajwbMbQasVdX1qvorMBro7OmJOOK6Blkw\n7JPFt/jlMf7S7XNKXinOeHQFMg3YHPE+C2juTaiDxXWCLOuYdAVEpIeqDnbrcA6X2Bbf4gcVP/fX\nLTH/zUbOQ+UYfMixF7VPX+4GJXoTu0fJqyRkbItv8YOOX2qR81A5y6GJPQvIiHifDvhSvU/0BGmM\nOfzNBxqJSAMRqQBcC4wvYRtXxHcT2xhT7qlqroj0AqYCScAwVV3mR+xET5CBXQMKOLbFt/hBx3eV\nqk4GJvsdN64n7TLGmCDZNUhjjInCEqQxxkSRkAkyqH6bTuxhIrJDRL7xM25E/AwR+UREVojIMhG5\nx+f4R4rIPBH5yonf38/4zjEkichiEZnod2wn/kYR+VpElojIAp9jVxeRd0VkpfP/wDl+xk80CXcN\n0um3uZqIfpvAdX7023TitwL2AqNU9VQ/Yh4Svx5QT1UXiUhVYCHQxcfzF6Cyqu4VkRRgNnCPqvrW\nLURE7gWaAEepake/4kbE3wg0UdXvAog9EvhMVV9zHomppKp7/D6ORJGINcjA+m0CqOosYJdf8YqI\nn62qi5zXPwErCHfV8iu+qupe522Ks/j2r7CIpAOXAa/5FTNeiMhRQCtgKICq/mrJ8fdJxARZVL9N\n3xJEPBGR+kBjYK7PcZNEZAmwA5iuqn7G/xfQF8j3MeahFJgmIgudbnR+OQ7YCQx3LjG8JiKVfYyf\ncBIxQQbWbzOeiEgV4D2gt6r+6GdsVc1T1TMJdwlrJiK+XGoQkY7ADlVd6Ee8YrRU1bOAS4GezmUX\nPyQDZwGvqGpjYB/g6zX4RJOICTKwfpvxwrn29x7wpqq+H9RxOM27mUB7n0K2BC53rgGOBtqIyBs+\nxS6kqludnzuAsYQv+/ghC8iKqLG/SzhhmjJKxAQZWL/NeODcJBkKrFDVFwKIX1tEqjuvKwIXAyv9\niK2qD6pquqrWJ/x7/1hVb/AjdgERqezcHMNp3rYFfHmiQVW3AZtF5I9O0UWALzfnElXCdTUMst8m\ngIi8DbQGaolIFtBPVYf6FZ9wLepG4GvnOiDAQ05XLT/UA0Y6TxOEgExVDeRxm4DUAcaG/50iGXhL\nVT/0Mf5dwJtO5WA9cIuPsRNOwj3mY4wxbknEJrYxxrjCEqQxxkRhCdIYY6KwBGmMMVFYgjTGmCgs\nQZq4IiL1gxoJyZhDWYI0vnCeizTmsGIJ0hRJRJ6IHEtSRJ4SkbuLWK+1iMwSkbEislxE/isiIeez\nvSLyuIjMBc4RkbNF5FNnEIepztBsOOVficiXQE+/ztGYkliCNNEMBboBOAnvWuDNKOs2A+4DTgOO\nB65wyisD36hqc8IjCr0EXKWqZwPDgKec9YYDd6uqDe5q4krCdTU07lDVjSLyvYg0Jtx9brGqfh9l\n9Xmquh4Ku1qeR3ighDzCg2YA/BE4FZjudMNLArJFpBpQXVU/ddZ7nfAoOMYEzhKkKc5rwM1AXcI1\nvmgO7a9a8P5nVc1zXguw7NBaojOwhfV3NXHJmtimOGMJD1XWlPDgH9E0c0ZPCgHXEJ5m4VCrgNoF\nc6SISIqInOIMifaDiJznrHe9e4dvzO9jNUgTlar+KiKfAHsiaoJF+RIYQPga5CzCibWofV0FDHKa\n1cmER/9eRnjEmWEisp/iE7ExvrLRfExUTo1wEXC1qq6Jsk5r4G9BTI5ljNesiW2KJCInA2uBGdGS\nozGJzmqQplRE5DTCd5gj/eI8wmNMQrIEaYwxUVgT2xhjorAEaYwxUViCNMaYKCxBGmNMFJYgjTEm\niv8H+2LyHdCVZ/oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dt_hpo = DecisionTreeClassifier(min_samples_leaf = 2, max_depth = 47, min_samples_split = 3, max_features = 19, criterion = 'gini')\n",
    "dt_hpo.fit(X_train,y_train)\n",
    "dt_score=dt_hpo.score(X_test,y_test)\n",
    "y_predict=dt_hpo.predict(X_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of DT: '+ str(dt_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of DT: '+(str(precision)))\n",
    "print('Recall of DT: '+(str(recall)))\n",
    "print('F1-score of DT: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dt_train=dt_hpo.predict(X_train)\n",
    "dt_test=dt_hpo.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply ET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of ET: 0.9957089552238806\n",
      "Precision of ET: 0.9957161969649261\n",
      "Recall of ET: 0.9957089552238806\n",
      "F1-score of ET: 0.9956792533287913\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3645\n",
      "           1       0.99      1.00      1.00       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       0.99      0.99      0.99       609\n",
      "           4       1.00      0.71      0.83         7\n",
      "           5       0.99      1.00      0.99       251\n",
      "           6       1.00      0.99      1.00       436\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       1.00      0.96      0.97      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcFNXV8PHf6ZlhVxAhMMwQIYJG\njQqERYIiIg6IIBgVJXF9NJgEo7gTN4KKr0lciT7mAUHADXBBdmVRRFQ2AZFh34SBERAUBVxmOe8f\nXTM0ODUz3VRXNd3n66c+03O7qk5Vt3O4t+6tW6KqGGOM+blQ0AdgjDGJyhKkMca4sARpjDEuLEEa\nY4wLS5DGGOPCEqQxxriwBGmMMS4sQRpjjAtLkMYY4yI96AMol4jd5mNMEFQlls0KvtoY9d9sRr1f\nxRTLDwmdIAt2bQgsdkb9E0nPaBRY/MKC7Skbv7BgO4DFDzi+SfAEaYw5yhQXBX0EnrIEaYzxjhYH\nfQSesgRpjPFOsSVIY4wpk1oN0hhjXCRZDdLGQRpjvKPF0S8VEJFqIrJQRD4TkVwRGeyUjxKRTSKy\nzFlaOOUiIkNFZL2ILBeRVhH7ulZE1jnLtRXFthqkMcY78enF/hHorKr7RCQDmCci05337lLVNw5b\n/0KgubO0A54H2olIXWAQ0BpQ4FMRmaSqX7sFthqkMcY7cahBatg+59cMZylvQHovYIyz3Xygjohk\nAl2Bmaq6x0mKM4Fu5cW2BGmM8U5xcdSLiPQTkcURS7/DdysiaSKyDNhJOMktcN4a4jSjnxKRqk5Z\nFrA1YvM8p8yt3JU1sY0xnomlF1tVhwHDKlinCGghInWACSLyG+DvwJdAFWf7e4CHgLJuXdRyyl1Z\nDdIY450YapDRUNVvgDlAN1XNd5rRPwIvAm2d1fKAxhGbZQPbyyl3ZQnSGOOd+PRi13dqjohIdaAL\nsNq5roiICNAbWOFsMgm4xunNPgvYq6r5wLtAjogcJyLHATlOmStrYhtjvBOfXuxMYLSIpBGu1I1X\n1Ski8p6I1CfcdF4G/NlZfxrQHVgPHACuB1DVPSLyMLDIWe8hVd1TXuCjKkH++ONPXNv/Ln4qKKCo\nsIgLzjubm2+8GlVl6LDRzHh/HqFQiCsuuYirLu/Fex9+wn+GjyEkIdLS0hh4az9anfkbtn+5gwH3\nPkJRUTGFhYX84bKLueKSizw5xqpVqzLnvTepUrUq6elpvPXWVAY/9IQn+66srjmdePLJh0gLhRj5\n4mv869/P+Rp//dr5fLdvX+nne1b77r7GD/L8g/7+hw97gou6d2Hnrq9o0fJ83+KWisOdNKq6HGhZ\nRnlnl/UV6O/y3khgZGVjS3hfienwueVUle+//4EaNapTUFjINX+5k4G33sTGL7aycMlyhtx3O6FQ\niN1ff8Pxx9XhwIHvqV69GiLCmvWbuPOBR5n82nAKCgpQVapUqcKBA9/T++o/8/J/n+QX9Y8vjXUk\n053VrFmD/fsPkJ6eztw5E7jt9kEsWLgkqn3EOt1YKBRiVe6HdOvel7y8fOZ/Mo2rrv4rq1at8yU+\nhBNku/YXsnu36/CyCmNDbNN9eXH+Rzrd2JF+/0cS/5yz27Fv335efPGZmBNkYcH2mOeD/HHV+1En\nlKqnnJd680GKyK8Jj0fKItxTtB2YpKqrjmCf1KhRHYDCwkIKCwsREcZNmMq//nEPoVD4kurxx9UB\nKF0X4PsffgAJfw8ZGRml5T8VFFDs8T8S+/cfcOKkk56RgZ//CLVt05INGzazadMWAMaPn8jFPbtG\nnSCPVolw/kF+/x/OW8AJJ2T7Fu9n7FbDionIPcBYwtcGFhJu8wvwmogMPJJ9FxUVcem1/enYoy/t\n27TkjNN+zdZt+Uyf/QF9/ucW/nzHA3yxdVvp+rM++Iieff/EX+98kIfvva20PH/HLi655i90ueQa\nbvjj5YfUHo9UKBRi8aIZ5G9bzuzZc1m4aKln+65Io6yGbM072DGXty2fRo0a+hYfwjX96dNeY8H8\n6dx4wx99jZ0I5x/k9x+4OHTSBClevdg3AG1U9TFVfdlZHiPcDX/Dkew4LS2NN0c/x+wJL/H5yrWs\n27iZnwoKqFqlCuNHDuXSnt144NGnStfvcm4HJr82nKGPPcizw8eUlmc2qM+EMc8zbdwIJk6fxVd7\nYmsOlqW4uJjWbXI4oWlr2rRuyWmnnezZvisi8vPWit+XUTp26k3bdt3o0fMq/vKX6zjn7Ha+xU6E\n8w/y+w9cnIf5+C1eCbIYKOsCSqbznqvIUfUvjHnNdb1jj6lFm1ZnMG/+YhrWr8cFnc4GoMu5v2Pt\nhk0/W791i9PZui2fr7/Ze0j5L+ofT7OmJ7DksxU/2+ZI7d37LR/M/ZiuOZ0837ebbXn5NM4++NFn\nZ2WSn7/Dt/hAabxdu3YzceJ02rRp4VvsRDj/EkF8/0FTLYp6SWTxSpADgNkiMl1EhjnLO8Bs4Nby\nNlTVYaraWlVb33hN30Pe2/P1N3z7XfiWzB9+/JH5i5bS9ITGdO7YngWfLgNg0dLPOaFx+O6hLXnb\nS2sPK9esp6CgkDq1j+XLnbv44ccfAdj77Xcs/XwlTX7pzXWbevXqUrv2sQBUq1aN8zufw5o1/j1b\nZ9HiZTRr1pQmTRqTkZFBnz69mDxlhm/xa9SoTq1aNUtfX9DlXHJz1/gWP+jzD/r7D1ySNbHj0kmj\nqu+IyEmEm9RZhK8/5gGL9Aj+ydi1+2vue+RxioqL0WKla+dz6NShHa3OOI17Bv+Ll8a9TY3q1Rg8\ncAAAM+fMY9L02aSnp1OtahUef2ggIsLGzVv597PDERFUlev6/p6TTmzqxamTmdmAkSOeJi0tRCgU\n4o03JjN12ixP9l0ZRUVF3DrgfqZNfZW0UIhRo8excuVa3+I3aFCfN14fAUB6ehpjx77NuzPm+BY/\n6PMP+vt/+aXnOLdje+rVq8vmjYsZ/NDjvDhqrG/xE73JHK2japiPn+yphvZUw5SOH+Mwnx8+fTvq\nv9lqv+2desN8jDEpyJ5qaIwxLhL8mmK0LEEaY7yTZNcgLUEaY7xjNUhjjHFhNUhjjHFhCdIYY8qW\n6HfGRMsSpDHGO1aDNMYYF0nWSWPPpDHGGBdWgzTGeMea2MYY4yLJmtiWII0x3rEapDHGuLAapH8y\n6p8YaPySaacsvsVPxfgxsRqkMca4sATpn6AnjD2lfpvA4q/atSjw80/pCWMtfmysiW2MMS6sBmmM\nMS6sBmmMMS6SrAZptxoaY7wTh8e+ikg1EVkoIp+JSK6IDHbKm4rIAhFZJyLjRKSKU17V+X29836T\niH393SlfIyJdK4ptCdIY453i4uiXiv0IdFbVM4EWQDcROQv4J/CUqjYHvgZucNa/AfhaVZsBTznr\nISKnAlcCpwHdgP8VkbTyAluCNMZ4Jw4JUsP2Ob9mOIsCnYE3nPLRQG/ndS/nd5z3zxcRccrHquqP\nqroJWA+0LS+2JUhjjHdUo15EpJ+ILI5Y+h2+WxFJE5FlwE5gJrAB+EZVC51V8oAs53UWsDV8OFoI\n7AWOjywvY5syWSeNMcY7MXTSqOowYFgF6xQBLUSkDjABOKWs1Zyf4vKeW7krS5DGGO/EuRdbVb8R\nkTnAWUAdEUl3aonZQMkI9zygMZAnIulAbWBPRHmJyG3KZE1sY4x34tOLXd+pOSIi1YEuwCrgfeAy\nZ7VrgYnO60nO7zjvv6eq6pRf6fRyNwWaAwvLi201SGNMossERjs9ziFgvKpOEZGVwFgReQRYCoxw\n1h8BvCQi6wnXHK8EUNVcERkPrAQKgf5awVPGLEEaY7wThya2qi4HWpZRvpEyeqFV9Qfgcpd9DQGG\nVDa2JUhjjHe03D6Po05SXoPsmtOJ3BVzWb1yHnff1T8uMapUrcK4d15kwvuvMHnuWG6++08AtDu7\nNW/OGsOkD17j//1nEGlp4XGonbt15O05r/DWey/z+ozRtGp3ZlyOC/w5f4ufeLETIX6cBooHRjSB\nM356layoDy4UCrEq90O6de9LXl4+8z+ZxlVX/5VVq9ZFtZ/KTHdWo2Z1Duz/nvT0NF6ePJzHHnyK\nJ4c9yv9c2p/NG7fwt3v6sX3rl7z56qTSdQFOOrUZTw1/lIs69HHdd6zTnXl5/kHFP5LpvoKM7+Vn\nH3h81bKGxFTo+xF3Rv03W/2Gx2OK5Yekq0G2bdOSDRs2s2nTFgoKChg/fiIX96zwlsuYlCS89Ix0\nMjLSKS4q5qeffmLzxi0AfDxnITk9zjtkXYAaNaoTr3+Y/Dx/i584sRMhPhCXXuwgBZIgReT6eO27\nUVZDtuYdHNqUty2fRo0axiVWKBTirfdeZt7Kd/n4g4UsX5JLRno6p50ZHsOa07MzDRs1KF2/S/dO\nTP1oPM+/8iT3D3gkLsfk5/lb/MSJnQjxAbRYo14SWVA1yMFub0TedlRcvD/qHYdvuTxUvGprxcXF\n/L7zVZx3Zg9Ob3kqzX/9K+646X4GPnwb4955kQP7DlBYdHAUwaxpc7ioQx/+du3d3DLwprgck5/n\nb/ETJ3YixAeS7hpk3HqxRWS521tAA5f3DrntKJZrkNvy8mmcffDaTXZWJvn5O6LdTVS++3YfCz9e\nwtmd2/Pi/77C1ReHbyX9Xad2nHDiL3+2/uL5S2l8QjZ16tbmmz17PT2WIM7f4gcfOxHiAwnfZI5W\nPGuQDYBrgJ5lLLvjFXTR4mU0a9aUJk0ak5GRQZ8+vZg8ZYbncY47vg7HHFsLgKrVqtK+Y1s2rfuC\nuvWOAyCjSgY33nwN40a/BcAvm2aXbnvq6SeTUSXd8+QI/p2/xU+s2IkQH4BijX5JYPEcBzkFqKWq\nyw5/w7mXMi6Kioq4dcD9TJv6KmmhEKNGj2PlyrWex6nfoJ4zjCdESEK8M2kWc2bO485Bf6PTBWcT\nCoUYO+pNFsxbDEBOj870urw7BYWF/PjDj9ze7z7Pjwn8O3+Ln1ixEyE+kPBN5mgl3TAfr9hTDe2p\nhikdP8ZhPgee+XPUf7M1bv1vwg7zsTtpjDHeSeAKVywsQRpjvJNkTWxLkMYY7yR4p0u0LEEaY7yT\nZMN8LEEaY7yTZDXIpLsX2xhjvGI1SGOMZ9Q6aYwxxkWSNbEtQRpjvGOdNMYY48JqkMYY48KuQRpj\njAurQRpjjAu7BmmMMS6sBumfkmmfgrJq16JA4wd9/hY/tePHwsZB+ihV50MsiZ8RYPwCmw8y5ePH\nxGqQxhjjwhKkMca4SLJOGpuswhjjnTg8tEtEGovI+yKySkRyReRWp/wfIrJNRJY5S/eIbf4uIutF\nZI2IdI0o7+aUrReRgRXFthqkMcYzGp8mdiFwh6ouEZFjgE9FZKbz3lOq+njkyiJyKnAlcBrQCJgl\nIic5bz8HXADkAYtEZJKqrnQLbAnSGOOdOCRIVc0H8p3X34nIKiCrnE16AWNV9Udgk4isB9o6761X\n1Y0AIjLWWdc1QVoT2xjjneLi6JcoiEgToCWwwCm6WUSWi8hIETnOKcsCtkZslueUuZW7sgRpjAmU\niPQTkcURSz+X9WoBbwIDVPVb4HngRKAF4RrmEyWrlrG5llPuyprYxhjvxNDEVtVhwLDy1hGRDMLJ\n8RVVfcvZbkfE+8OBKc6veUDjiM2zgZLBnW7lZbIapDHGO/HpxRZgBLBKVZ+MKM+MWO0SYIXzehJw\npYhUFZGmQHNgIbAIaC4iTUWkCuGOnEnlxbYapDHGM6px6cXuAFwNfC4iy5yye4G+ItKCcDN5M3CT\ncwy5IjKecOdLIdBfVYsARORm4F0gDRipqrnlBbYEaYzxTnx6sedR9vXDaeVsMwQYUkb5tPK2O5wl\nSGOMd+xWQ2OMKVucBooHxhKkMcY7SZYgk7IXu2tOJ3JXzGX1ynncfVf/pI8/fNgTbMv7jKVLZ5eW\nnXHGqXw4dxJLl8xiwoRRHHNMrbgfR4lU+/wTJXYixKc4hiWBJV2CDIVCDH1mCD16XsXpZ57HFVf0\n5pRTmid1/NFjxtOjxx8PKfu///6be+97lJatujDx7encccdf4noMJVLx80+E2IkQH8JN7GiXRBa3\nBCkivxaR853R75Hl3eIVE6Btm5Zs2LCZTZu2UFBQwPjxE7m4Z9eKNzyK48+bt4A9X39zSNlJJ53I\nhx/OB2DW7A+55JLuZW3quVT8/BMhdiLEB+IyDjJIcUmQInILMBH4G7BCRHpFvP1oPGKWaJTVkK15\nBwfH523Lp1GjhvEMmVDxS+TmrqFnzxwALru0B42z/ZmdOujzDzJ+Kp97KWtiV8qfgN+qam+gE/BA\nyRxulD2eqVTkfZnFxfujDhwedH+oOA1eTcj4Jf7U73b+8ufrWDB/OrWOqclPPxX4Ejfo8w8yfiqf\ne2m8JGtix6sXO01V9wGo6mYR6QS8ISInUEGCjLwvM71KVtSf3ra8/ENqS9lZmeTn7yhnC28FHb/E\nmjUb6H7RHwBo3vxXdL/wfF/iBn3+QcZP5XMvleA1wmjFqwb5pXMLEABOsuwB1ANOj1NMABYtXkaz\nZk1p0qQxGRkZ9OnTi8lTZsQzZELFL1G//vFAuFZx799vZdiwl3yJG/T5Bxk/lc+9hNUgK+cawvdA\nllLVQuAaEfm/OMUEoKioiFsH3M+0qa+SFgoxavQ4Vq5cG8+Qgcd/6aXnOLdje+rVq8umjYt56KHH\nqVWrJn/+y3UAvP32NEaNHhfXYyiRip9/IsROhPhA0tUgJYjrY5UVSxPbK/bYV3vsa0rHVy33Upib\n3T3Pjfpv9vjJH8QUyw9JNw7SGGO8YrcaGmO8k2RNbEuQxhjPJNljsS1BGmM8ZAnSGGPKZjVIY4xx\nYQnSGGNcWII0xhg3sQ2fTFiWII0xnrEapDHGuNBiq0EaY0yZrAZpjDEuYryFO2FZgjTGeMZqkMYY\n48KuQfqoZNqnVI1fkOLnb/GDjR+LBJ49MSYJnSCDno8x1eN3yvLnMQ2Hm7Mt/HzvlJ6PMQHix8Jq\nkMYY4yLZEqRNmGuMSWgi0lhE3heRVSKSW/KEVBGpKyIzRWSd8/M4p1xEZKiIrBeR5SLSKmJf1zrr\nrxORayuKbQnSGOMZ1eiXSigE7lDVU4CzgP4iciowEJitqs2B2c7vABcCzZ2lH/A8hBMqMAhoB7QF\nBpUkVTeWII0xntFiiXqpcJ+q+aq6xHn9HbAKyAJ6AaOd1UYDvZ3XvYAxGjYfqCMimUBXYKaq7lHV\nr4GZQLfyYluCNMZ4RlWiXkSkn4gsjlj6ue1fRJoALYEFQANVzQ/H1XzgF85qWcDWiM3ynDK3clcV\ndtKISAPgUaCRql7oVG3bq+qIirY1xqSWWAaKq+owYFhF64lILeBNYICqfiviWvss6w0tp9xVZWqQ\no4B3gZIxB2uBAZXYzhiTYopVol4qQ0QyCCfHV1T1Lad4h9N0xvm50ynPAxpHbJ4NbC+n3FVlEmQ9\nVR2P87QJVS0EiiqxnTEmxcTSxK6IhKuKI4BVqvpkxFuTgJKe6GuBiRHl1zi92WcBe50m+LtAjogc\n53TO5DhlriozDnK/iByPUxUtCViJ7YwxKSZO4yA7AFcDn4vIMqfsXuAxYLyI3ABsAS533psGdAfW\nAweA6wFUdY+IPAwsctZ7SFX3lBe4MgnydsIZ+UQR+QioD1xWyRMzxqSQeNxqqKrzKPv6IcDPbvdS\nVQX6u+xrJDCysrErTJCqukREzgVOdg5yjaoWVDaAMSZ1JNudNJXpxb7msKJWIoKqjonTMRljjlKV\n7XQ5WlSmid0m4nU1wlXaJYAlSGPMIVJuwlxV/Vvk7yJSG3gpbkdkjDlqJdt0Z7HcSXOA8D2OCWv9\n2vksXTKLxYtmMP+Tab7H75rTidwVc1m9ch5331XmteKkiF/r2JoM/r8HGTNnJKPfH8GprU7hmDrH\n8Pir/+TlD0fx+Kv/pFbtWodsc/KZJzP7i3c596Jz4nZcQX7+qfLdu4nXOMigVJggRWSyiExylinA\nGg6ON0pYXS64nNZtcjirfXdf44ZCIYY+M4QePa/i9DPP44orenPKKf79e+Jn/JsH92fhnEVc0+l/\nuCHnJras38If+l/Jko+WctU517Hko6X8of+VhxzbTffeyKIPFsfleEpiBPX5p9J37yYe4yCDVJka\n5OPAE87y/4COqjqw/E1ARNqKSBvn9akicruI+JutAtC2TUs2bNjMpk1bKCgoYPz4iVzcs2vSxa9R\nqwZntjudqa9NB6CwoJB93+6nQ87veOf1GQC88/oMzu7aoXSb31/fm7nTPuSbr77x/HhKBPn5p8p3\nX544zeYTmHITpIikAQ+o6gfO8pGq5lW0UxEZBAwFnheR/wc8C9QCBorIfV4ceHlUlenTXmPB/Onc\neMMf4x3uEI2yGrI17+DdS3nb8mnUqGHSxW/0y0y+2bOXgU/exfB3/std/76datWrUbfecezZGR57\nu2fnHo47vg4A9Roez9kXdmDSS1M8P5ZDjivAzz9VvvvyJFsTu9xOGlUtEpEDIlJbVaO5e+YyoAVQ\nFfgSyHZuLv834Vk4hrht6Mzk0Q9A0moTCtWMImxYx069yc/fQf36x/PO9LGsWbOeD+ctiHo/sSjr\nBnr18Z9Jv+Knpadx0m+aM/SBZ1m1dDU3D/7rIc3pw938j78y7NEXKC6O72Pvgvz8U+W7L0+iN5mj\nVZlhPj8QvsVnJrC/pFBVbylnm0JVLQIOiMgGVf3W2eZ7ESn3LyRyZo/0Klkxfbv5+TsA2LVrNxMn\nTqdNmxa+Jchtefk0zj74LJHsrMzS40mm+Lvyd7Erfxerlq4G4IOpc/lD/77s+epr6v6iLnt27qHu\nL+ry9e5wc/rkM07iwefCjYfadWvTrnNbigqLmPfux54eV5Cff6p896mkMtcgpwIPAHOBT52loqvs\nP4lIDef1b0sKnSFCca1C1KhRnVq1apa+vqDLueTmrolnyEMsWryMZs2a0qRJYzIyMujTpxeTp8xI\nuvh7dn3Nzu27aPyrbAB+e3Yrvlj3BR/P/IRul+cA0O3yHD6aEU6AfX93NVe2v4or21/FB1Pn8vR9\nQz1PjhDs558q3315UqqJ7aijqs9EFpQ8E6IcHVX1RwDVQ2aIy+Dg7Btx0aBBfd54PTxVZXp6GmPH\nvs27M+bEM+QhioqKuHXA/Uyb+ippoRCjRo9j5cq1SRl/6APPcv9//k56lQzyv8jnsTv+TUhCDPrv\n/XS/shs7tu3kH39+OC6x3QT5+afSd+8mwftcoiYVXaMQkSWq2uqwsqWq2jKuR0bsTWwvJMJjV4OO\nb499TeH4MV5M/Djz0qj/Zn+X/2bCViNda5Ai0hf4A9BURCZFvHUMsDveB2aMOfqkUifNx0A+UI/w\nGMgS3wHL43lQxpijU3zHKPjPNUGq6hfAF0D78nYgIp+oarnrGGNSg7pO23h0qkwnTUWqebAPY0wS\nKE6yXhovEmSSfSTGmFgVWw3SGGPKlmxN7MrM5nOz8wQw11U8PB5jzFGsOIYlkVXmTpqGwCIRGS8i\n3eTnN3xeHYfjMsYchRSJeklkFSZIVb2f8AS5I4DrgHUi8qiInOi8vyKuR2iMOWqkYg2y5DGKXzpL\nIXAc8IaI/CuOx2aMOcokW4KszFMNbyF8//RXwAvAXapaICIhYB1wd3wP0RhztEj0JnO0KtOLXQ/4\nvTNwvJSqFotIj/gcljHmaJRkj8Wu1FMNHyznvVXeHo4x5miWbOMgY3mqoTHGpISEHiheMu2TxQ9G\nybRjQQn6/FM9fiyS7ba6hE6QQc+HaPFTeD5Eix+TePRKi8hIoAewU1V/45T9A/gTsMtZ7V5Vnea8\n93fgBqAIuEVV33XKuwHPAGnAC6r6WEWxEzpBGmOOLsVlPDjMA6MIPxl1zGHlT6nq45EFInIqcCVw\nGtAImCUiJzlvPwdcAOQRvvllkqquLC+wJUhjjGfi0cRW1bki0qSSq/cCxjqPfNkkIuuBts5761V1\nI4CIjHXWLTdBWieNMcYzPg8Uv1lElovIyIj5IrKArRHr5DllbuXlsgRpjPFMsUS/iEg/EVkcsfSr\nRKjngROBFoSffFDy1IOy2vhaTnm5rIltjPFMLOMgVXUYMCzKbUof+C0iw4Epzq95QOOIVbOBkl4n\nt3JXVoM0xnhGY1hiISKZEb9eApRMmjMJuFJEqopIU8IT7SwEFgHNRaSpiFQh3JET+TDCMlkN0hjj\nmXjcaigirwGdgHoikgcMAjqJSAvCOXYzcBOAquaKyHjCnS+FQH9VLXL2czPwLuFhPiNVNbei2JYg\njTGeicc4SFXtW0bxiHLWHwIMKaN8GjAtmtiWII0xnrE7aYwxxkXKzeZjjDGVlegT4EbLEqQxxjOW\nII0xxoUmWRM76cZBZmc3YtaM1/l8+Rw+W/Yef7v5Bt+PoWtOJ3JXzGX1ynncfVd/i++z9Wvns3TJ\nLBYvmsH8T6LqtDxiQZ970PFT7pk0R5vCwkLuunswS5etoFatmixc8A6zZs9l1ap1vsQPhUIMfWYI\n3br3JS8vn/mfTGPylBkW36f4JbpccDm7d3/ta8ygzz3o+Mko6WqQX365k6XLwoPq9+3bz+rV68hq\n1NC3+G3btGTDhs1s2rSFgoICxo+fyMU9u1r8FBD0uQcdH5KvBulbghSRw+dyi7sTTsimxZm/YcHC\npb7FbJTVkK15B2/xzNuWTyMfE3SqxwdQVaZPe40F86dz4w1/9C1u0OcedHzw71ZDv8SliS0ih9/j\nKMB5IlIHQFUvjkfcSDVr1mD8uOHcfucgvvtuX7zDlZIyJgwNP1bc4vulY6fe5OfvoH7943ln+ljW\nrFnPh/MWxD1u0OcedHywcZCVlU34XsgXODjVUGsOTknkypnqqB+ApNUmFKoZdfD09HReHzec116b\nwNtvT496+yOxLS+fxtkHp8rPzsokP39HOVtYfK+VxNu1azcTJ06nTZsWviTIoM896PiQ+E3maMWr\nid0a+BS4D9irqnOA71X1A1X9oLwNVXWYqrZW1daxJEeA4cOeYNXq9Tz9TFQzKHli0eJlNGvWlCZN\nGpORkUGfPr2YPGWGxfdJjRrVqVWrZunrC7qcS27uGl9iB33uQceH5LsGGZcapKoWA0+JyOvOzx3x\ninW4Dr9rw9VXXcbyz1eyeFHHAFy7AAASQ0lEQVT4f44HHniM6e+850d4ioqKuHXA/Uyb+ippoRCj\nRo9j5cq1vsS2+NCgQX3eeD08j0F6ehpjx77NuzPm+BI76HMPOj4k/jXFaIkf1yhE5CKgg6reG812\n6VWyAvu8U/mpgkHHT5Sn+qV0fI1tyPe/Trgq6r/Zu794OWGvXPpSq1PVqcBUP2IZY4KT6E3maCXd\nQHFjTHCSrYltCdIY45niJEuRliCNMZ6xJrYxxrhIrvqjJUhjjIesBmmMMS7sVkNjjHFhnTTGGOMi\nudJjEs4HaYwxXrEapDHGM9ZJY4wxLuwapDHGuEiu9GgJ0hjjIWti+6hk2ieLb/Et/tEh2ZrY1ott\njPFMPB7aJSIjRWSniKyIKKsrIjNFZJ3z8zinXERkqIisF5HlItIqYptrnfXXici1lTmfhK5BpuqE\nsakePyEmjAWy6pwaSPxt36wEgj//WMSpiT0KeBaIfDLqQGC2qj4mIgOd3+8BLgSaO0s74HmgnYjU\nBQYRfhyMAp+KyCRVLffh6VaDNMZ4RmP4r8J9qs4F9hxW3AsY7bweDfSOKB+jYfOBOiKSCXQFZqrq\nHicpzgS6VRTbEqQxxjOxPLRLRPqJyOKIpV8lQjVQ1XwA5+cvnPIsYGvEenlOmVt5uRK6iW2MObrE\n0kmjqsMArx5BWtZ0GVpOebmsBmmM8Uw8Omlc7HCazjg/dzrleUDjiPWyge3llJfLEqQxxjPFaNRL\njCYBJT3R1wITI8qvcXqzzwL2Ok3wd4EcETnO6fHOccrKZU1sY4xn4tGLLSKvAZ2AeiKSR7g3+jFg\nvIjcAGwBLndWnwZ0B9YDB4DrAVR1j4g8DCxy1ntIVQ/v+PkZS5DGGM9Uplc66n2q9nV56/wy1lWg\nv8t+RgIjo4ltCdIY4xm71dAYY1zEowYZJOukMcYYF1aDNMZ4xprYxhjjoliTq4ltCdIY45nkSo9J\neA1y+LAn2J73GcuWzg7sGLrmdCJ3xVxWr5zH3XeVOeLA4h/F8RtlNeT1SS8yZ/4k3vt4IjfcdBUA\nt9/zVxbnvseMuW8yY+6bdL7gnNJtbr7tRuZ9Op25C6dwbucOnh9TiaA/ex8Hivsi6RLkmDHjuajH\nHwOLHwqFGPrMEHr0vIrTzzyPK67ozSmnNLf4SRS/sLCQwff/i05nXUzPnL5cd2Nfmp98IgDDnx9D\nTsdLyel4Ke/N/BCA5iefSK/fd6dz+4v542U38ejj9xMKef+nF/RnD/GZzSdISZcgP5y3gD1ffxNY\n/LZtWrJhw2Y2bdpCQUEB48dP5OKeXS1+EsXfueMrVixfBcD+fQdYt3YjDTN/4bp+1+7nMfGtafz0\nUwFbt2xj88attPzt6Z4eEwT/2UNss/kkMl8SpIicLSK3i0iOH/GC1CirIVvzDt4Dn7ctn0aNGlr8\nJI2f3bgRvznjFJZ+uhyA6//0B2bOe4sn/vMwtWsfC0DDzAZs3/Zl6Tb527+kYWYDz48l6M8erIld\nKSKyMOL1nwjPBnwMMMiZ/Tdpifx8ViX1sWfP4vsXv0bNGgwf8zSD/v4Y+77bz5iR4/hdy27knHMp\nO3fs4sFH7vL1mIL+7MGa2JWVEfG6H3CBqg4mPINGuRcIIyfPLC7eH6fDi59tefk0zj44VX52Vib5\n+TssfpLFT09PZ/jop5nw+lSmT5kFwFe7dlNcXIyq8sroN2jhNKPzt39Jo6yDNbnMRg3Z8eXOMvd7\nJIL+7MGa2JXerzOt0PGAqOouAFXdDxSWt6GqDlPV1qraOhSqGafDi59Fi5fRrFlTmjRpTEZGBn36\n9GLylBkWP8niP/Gfh1i/diPD/nd0adkvGtQrfX1hjy6sWbUOgBnT36fX77tTpUoGjX+ZRdMTf8nS\nTz/3/JiC/uwhXGONdklk8RoHWRv4lPAsvioiDVX1SxGpRdkz+3rm5Zee49yO7alXry6bNy5m8EOP\n8+KosfEMeYiioiJuHXA/06a+SlooxKjR41i5cq3FT6L4bc5qxWVX9mJl7hpmzH0TgMcefprel3bn\n1NN/jaqSt2U799z2DwDWrt7A5Lff4f35kygqLOK+ux6huNj7ulPQnz0k32NfxefrQzUIP0tiU2XW\nT6+SFdinncpPFQw6vj3VMAGeaqgaU0Wm5y97RP03O3nLlLhWmo6Er3fSqOoBoFLJ0Rhz9En0Tpdo\n2a2GxhjPJFsT2xKkMcYzid7pEi1LkMYYzyT6sJ1oWYI0xngm2a5BJt292MYY4xWrQRpjPGOdNMYY\n48I6aYwxxoXVII0xxkWyddJYgjTGeMYe2mWMMS6SKz1agjTGeCjZrkHaOEhjjGfi9cgFEdksIp+L\nyDIRWeyU1RWRmSKyzvl5nFMuIjJURNaLyHIRaRXr+SR0DbJk2imLb/GDUDLtWFCCPv9YxHmYz3mq\n+lXE7wOB2ar6mPMol4HAPcCFQHNnaQc87/yMmtUgjTGe8fmhXb2AkindRwO9I8rHaNh8oI6IZMYS\nIKFrkKk6YWyqx0+UCXODjn9Gg7MCib98x/yYt41lmI+I9CP87KoSw1R12M92DTNERIH/c95voKr5\nAKqaLyIlz97NArZGbJvnlOVHe2wJnSCNMUeXWJrYTrI7PCEeroOqbneS4EwRWV3OumXNUB5TVdUS\npDHGM/HqxVbV7c7PnSIyAWgL7BCRTKf2mAmUPCoyD2gcsXk2ENMFXbsGaYzxTDyeaigiNUXkmJLX\nhB8fvQKYBFzrrHYtMNF5PQm4xunNPgvYW9IUj5bVII0xnolTDbIBMEFEIJyzXlXVd0RkETBeRG4A\ntgCXO+tPA7oD64EDwPWxBrYEaYzxTDzuxVbVjcCZZZTvBs4vo1yB/l7Etia2Mca4sBqkMcYzNlmF\nMca4sOnOjDHGhdUgjTHGhdUgjTHGhdUgjTHGhdUgjTHGRbLVIJNuHGTVqlX55KMpfLp4Jp8te49B\nD97h+zF0zelE7oq5rF45j7vv8mS8qsU/SuL7GTsUCjFu5mj+89LjAPzjyXt5ffYY3njvJZ54YQjV\na1QH4PJrLuHN919m/KzRjJr4X351UpO4HZPG8F8ik0R+jm16layYDq5mzRrs33+A9PR05s6ZwG23\nD2LBwiVR7SPW6b5CoRCrcj+kW/e+5OXlM/+TaVx19V9ZtWqdxY8iNsQ23ViQ8b387KHi6c6uvulK\nTjvzFGoeU5O/XX0nNWvVYP++AwDc+Y9b2PPV14x89qVDyjvlnM0V113KX/5wm+t+l++YD6plzYhT\noabHnxn13+ym3Z/FFMsPcalBikg7ETnWeV1dRAaLyGQR+aeI1I5HzEj794f/Z8jISCc9I8PXh5m3\nbdOSDRs2s2nTFgoKChg/fiIX9+xq8VMgvp+xG2TWp2OXDrz1yqTSspIkCFCtetXS2llkefUa1eNa\na/N5wty4i1cTeyThm8QBngFqA/90yl6MU8xSoVCIxYtmkL9tObNnz2XhoqXxDlmqUVZDtuYdnFkp\nb1s+jRo1tPgpEN/P2Hc/PIAnH36WYi0+pPyhp+/j/c+n0qTZCbw24vXS8iuuv5Sp81/ntgf689h9\nT8blmCA+s/kEKV4JMqSqhc7r1qo6QFXnqepg4FflbSgi/URksYgsLi7eH1Pw4uJiWrfJ4YSmrWnT\nuiWnnXZyTPuJhTPjyCH8/J/A4gcX36/YHS/owJ6vvmbV8jU/e+/BAUM4/8yebFq3ma69upSWj3vx\nTS4663KefuR/6XdbzJPbVMhqkJWzQkRKvoXPRKQ1gIicBBSUt6GqDlPV1qraOhSqeUQHsXfvt3ww\n92O65nQ6ov1EY1tePo2zD167ys7KJD9/h8VPgfh+xW7R5gw65ZzD9EVv8a//PkzbDr/l0WcHlb5f\nXFzMOxNn0+Wi83627fS3Z3Jet46eH1MJq0FWzo3AuSKyATgV+ERENgLDnffipl69utSufSwA1apV\n4/zO57BmzYZ4hjzEosXLaNasKU2aNCYjI4M+fXoxecoMi58C8f2KPfTR57mgVS8ubPN77v7zAyz8\n6FPuvXkwjZtkl67TKedsNq//AoBfNj1Y3rFLB7Zs2vqzfXqlWDXqJZHFZRykqu4FrnNmAf6VEydP\nVeP+T3lmZgNGjniatLQQoVCIN96YzNRps+IdtlRRURG3DrifaVNfJS0UYtTocaxcudbip0D8IGOL\nCI8MfYBax9REBNbkrueRe/4FQN//uYx2HdtQWFDIt3u/4/5bHo7bcST6sJ1oJeUwHy+k8lMFg46f\nKE8VDDp+oE81jHGYT4Pav476b3bH3tWpNczHGGOSgd1qaIzxTKL3SkfLEqQxxjOJfMkuFpYgjTGe\nSfRe6WhZgjTGeMZqkMYY48KuQRpjjAurQRpjjAu7BmmMMS6S7U4aS5DGGM9YDdIYY1wk2zVIu9XQ\nGOOZeD2TRkS6icgaEVkvIgPjfBqlrAZpjPFMPGqQIpIGPAdcAOQBi0Rkkqqu9DzYYawGaYzxTJwm\nzG0LrFfVjar6EzAW6BXXE3EkdA2yZNoni2/xUzH+8h3zA40fizhdgcwCImf5zQPaxSfUoRI6QcY6\nJ10JEemnqsO8OpyjJbbFt/hBxS/8aVvUf7Mi0g/oF1E07LBjL2ufvvQGJXsTu1/FqyRlbItv8YOO\nX2mRz6FylsMTex7QOOL3bMCX6n2yJ0hjzNFvEdBcRJqKSBXgSmBSBdt4IrGb2MaYlKeqhSJyM/Au\nkAaMVNVcP2Ine4IM7BpQwLEtvsUPOr6nVHUaMM3vuAn90C5jjAmSXYM0xhgXliCNMcZFUibIoO7b\ndGKPFJGdIrLCz7gR8RuLyPsiskpEckXkVp/jVxORhSLymRN/sJ/xnWNIE5GlIjLF79hO/M0i8rmI\nLBORxT7HriMib4jIauf/gfZ+xk82SXcN0rlvcy0R920Cff24b9OJ3xHYB4xR1d/4EfOw+JlApqou\nEZFjgE+B3j6evwA1VXWfiGQA84BbVdW320JE5HagNXCsqvbwK25E/M1Aa1X9KoDYo4EPVfUFZ0hM\nDVX9xu/jSBbJWIMM7L5NAFWdC+zxK14Z8fNVdYnz+jtgFeFbtfyKr6q6z/k1w1l8+1dYRLKBi4AX\n/IqZKETkWKAjMAJAVX+y5HhkkjFBlnXfpm8JIpGISBOgJbDA57hpIrIM2AnMVFU/4z8N3A0U+xjz\ncArMEJFPndvo/PIrYBfwonOJ4QURqelj/KSTjAkysPs2E4mI1ALeBAao6rd+xlbVIlVtQfiWsLYi\n4sulBhHpAexU1U/9iFeODqraCrgQ6O9cdvFDOtAKeF5VWwL7AV+vwSebZEyQgd23mSica39vAq+o\n6ltBHYfTvJsDdPMpZAfgYuca4Figs4i87FPsUqq63fm5E5hA+LKPH/KAvIga+xuEE6aJUTImyMDu\n20wETifJCGCVqj4ZQPz6IlLHeV0d6AKs9iO2qv5dVbNVtQnh7/09Vb3Kj9glRKSm0zmG07zNAXwZ\n0aCqXwJbReRkp+h8wJfOuWSVdLcaBnnfJoCIvAZ0AuqJSB4wSFVH+BWfcC3qauBz5zogwL3OrVp+\nyARGO6MJQsB4VQ1kuE1AGgATwv9OkQ68qqrv+Bj/b8ArTuVgI3C9j7GTTtIN8zHGGK8kYxPbGGM8\nYQnSGGNcWII0xhgXliCNMcaFJUhjjHFhCdIkFBFpEtRMSMYczhKk8YUzLtKYo4olSFMmEXk4ci5J\nERkiIreUsV4nEZkrIhNEZKWI/FdEQs57+0TkIRFZALQXkd+KyAfOJA7vOlOz4ZR/JiKfAP39Okdj\nKmIJ0rgZAVwL4CS8K4FXXNZtC9wBnA6cCPzeKa8JrFDVdoRnFPoPcJmq/hYYCQxx1nsRuEVVbXJX\nk1CS7lZD4w1V3Swiu0WkJeHb55aq6m6X1Req6kYovdXybMITJRQRnjQD4GTgN8BM5za8NCBfRGoD\ndVT1A2e9lwjPgmNM4CxBmvK8AFwHNCRc43Nz+P2qJb//oKpFzmsBcg+vJToTW9j9riYhWRPblGcC\n4anK2hCe/MNNW2f2pBBwBeHHLBxuDVC/5BkpIpIhIqc5U6LtFZGznfX+6N3hG3NkrAZpXKnqTyLy\nPvBNRE2wLJ8AjxG+BjmXcGIta1+XAUOdZnU64dm/cwnPODNSRA5QfiI2xlc2m49x5dQIlwCXq+o6\nl3U6AXcG8XAsY+LNmtimTCJyKrAemO2WHI1JdlaDNJUiIqcT7mGO9KMzhMeYpGQJ0hhjXFgT2xhj\nXFiCNMYYF5YgjTHGhSVIY4xxYQnSGGNc/H/LTjASiJ6+CgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "et = ExtraTreesClassifier(random_state = 0)\n",
    "et.fit(X_train,y_train) \n",
    "et_score=et.score(X_test,y_test)\n",
    "y_predict=et.predict(X_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of ET: '+ str(et_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of ET: '+(str(precision)))\n",
    "print('Recall of ET: '+(str(recall)))\n",
    "print('F1-score of ET: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter optimization (HPO) of extra trees using Bayesian optimization with tree-based Parzen estimator (BO-TPE)\n",
    "Based on the GitHub repo for HPO: https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████| 20/20 [00:25<00:00,  1.28s/trial, best loss: -0.9955223880597015]\n",
      "Random Forest: Hyperopt estimated optimum {'n_estimators': 53.0, 'min_samples_leaf': 1.0, 'max_depth': 31.0, 'min_samples_split': 5.0, 'max_features': 20.0, 'criterion': 1}\n"
     ]
    }
   ],
   "source": [
    "# Hyperparameter optimization of extra trees\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'max_features': int(params['max_features']),\n",
    "        \"min_samples_split\":int(params['min_samples_split']),\n",
    "        \"min_samples_leaf\":int(params['min_samples_leaf']),\n",
    "        \"criterion\":str(params['criterion'])\n",
    "    }\n",
    "    clf = ExtraTreesClassifier( **params)\n",
    "    clf.fit(X_train,y_train)\n",
    "    score=clf.score(X_test,y_test)\n",
    "\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 10, 200, 1),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"max_features\":hp.quniform('max_features', 1, 20, 1),\n",
    "    \"min_samples_split\":hp.quniform('min_samples_split',2,11,1),\n",
    "    \"min_samples_leaf\":hp.quniform('min_samples_leaf',1,11,1),\n",
    "    \"criterion\":hp.choice('criterion',['gini','entropy'])\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"Random Forest: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of ET: 0.9955223880597015\n",
      "Precision of ET: 0.9955353802920419\n",
      "Recall of ET: 0.9955223880597015\n",
      "F1-score of ET: 0.9954932494250629\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3645\n",
      "           1       0.99      1.00      1.00       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       0.99      1.00      0.99       609\n",
      "           4       1.00      0.71      0.83         7\n",
      "           5       0.98      1.00      0.99       251\n",
      "           6       1.00      0.99      0.99       436\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       0.99      0.96      0.97      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XecFPX9x/HXZ++OjqJCKHdEiGBi\nB6Qpiog0EYRExYbtpxINFmJE0WgMGv2RxEqK+YEgYAHOgvSuiERpCgJ39CIcHEVQQ4l45fP7Y+eO\nBW/ubpfZmWXv8/QxD/a+O7PvmcX78J32HVFVjDHG/Fgo6BUwxphEZQXSGGNcWIE0xhgXViCNMcaF\nFUhjjHFhBdIYY1xYgTTGGBdWII0xxoUVSGOMcZEa9AqUSsRu8zEmCKoSy2J5X2+K+nc2rfbPYsry\nQ0IXyLw9GwPLTqtzBqlpDQLLz8/bUWHz8/N2AFh+wPkmwQukMeYEU1gQ9Bp4ygqkMcY7Whj0GnjK\nCqQxxjuFViCNMaZEaj1IY4xxkWQ9SLsO0hjjHS2MfiqDiFQRkcUi8qWIZInIYKd9lIhsFpHlztTM\naRcRGSoiG0RkhYi0iPis20RkvTPdVla29SCNMd6Jz1nsw0BHVT0gImnAAhGZ7rw3UFXfPWb+K4Gm\nztQGeBVoIyKnAk8BLQEFPheRSar6jVuw9SCNMd6JQw9Sww44P6Y5U2kXpPcCxjjLLQRqiUh9oCsw\nW1X3OUVxNtCttGwrkMYY7xQWRj2JSD8RWRox9Tv2Y0UkRUSWA7sJF7lFzlvPOrvRL4lIZactHdgW\nsXiO0+bW7sp2sY0xnonlLLaqDgOGlTFPAdBMRGoBE0TkXOAxYCdQyVn+UeBpoKRbF7WUdlfWgzTG\neCeGHmQ0VPVbYB7QTVVznd3ow8DrQGtnthygYcRiGcCOUtpdWYE0xngnPmex6zg9R0SkKtAJWOMc\nV0REBOgNrHIWmQTc6pzNbgt8p6q5wEygi4icIiKnAF2cNle2i22M8U58zmLXB0aLSArhTl2mqk4R\nkQ9FpA7hXeflwD3O/NOA7sAG4BBwB4Cq7hORZ4AlznxPq+q+0oJPqAJ5+PAP3NZ/ID/k5VGQX0Dn\nyy/hvrtuQVUZOmw0sz5aQCgU4vpfXkXf63rx4Sef8bfhYwhJiJSUFAY92I8WF5wLwK8feoIVWWto\nfv45/POvgz1bx8qVKzPvw/eoVLkyqakpvP/+VAY//YJnn18eXbt04MUXnyYlFGLk62P5y1//4Wv+\nhnUL2X/gAAUFheTn59P2ou6+5ge9/aFQiEULp7Nj+056/bLMS+08k5HRgFEjX6FuvToUFhby2mtv\n8be/j/AtH4jLvdiqugJoXkJ7R5f5Fejv8t5IYGR5s0+oAlmpUhojhw6hWrWq5OXnc+u9D3Np25Zs\n+mobO3d/zeS3hxEKhdj7zbcAtL2wGZdf0hYRYe2GzTz85HNMHjscgDtuuobvvz9M5sTppUVG7fDh\nw3Tq0oeDBw+RmprK/HkTmDHjIxYt/sLTHDehUIihrzxLt+43kpOTy8LPpjF5yixWr17vS36RTp2v\nY+9e18vL4iYRtv+B++9izZr1nFSzpm+ZAPn5+Qx8ZDDLlq+iRo3qLF40gzlz5/v+d59M4nYMUkR+\nISKPOle0v+K8Pus4P5Nq1aoC4f8Z8vPzERHGT5jKvXfcRCgU3pzTTqkFQLVqVQkfnoD/fv89yJGT\nWG1bNqdatWrHszquDh48BEBaWiqpaWmE/0HzR+tWzdm4cQubN28lLy+PzMyJXN2zq2/5QQt6+9PT\n69P9yisYOXKsb5lFdu7czbLl4cNwBw4cZM2a9aQ3qOfvSsT5JI3f4lIgReRRYBzhYwOLCe/zCzBW\nRAYdz2cXFBRwzW39ad/jRi5q1Zzzz/kF27bnMn3ux/T5nwe453dP8tW27cXzz/n43/S88W5+8/Af\neObx3x5PdLmFQiGWLplF7vYVzJ07n8VLlvmSC9AgvR7bco6cmMvZnksDn39JVJXp08ayaOF07rrz\nZl+zg97+F18YzKDH/kRhwL/4p5+eQbMLzmXRYv/+3wPicpImSPHqQd4JtFLVIar6pjMNIXwa/s7j\n+eCUlBTeG/0P5k54g5XZ61i/aQs/5OVRuVIlMkcO5Zqe3XjyuZeK5+90WTsmjx3O0CF/4O/Dxxzf\nVpVTYWEhLVt14fTGLWnVsjnnnPNzX3KB4h5zJD97sADtO/SmdZtu9OjZl3vvvZ1LL2njW3aQ239V\n907s3v01Xyxb6Uuem+rVq5E5fjgPPfwU+/cfKHsBL1kPslwKgZLGi6/vvOcq8qr618a476acVLMG\nrVqcz4KFS6lXpzadO1wCQKfLLmbdxs0/mr9ls/PYtj2Xb779LprtOC7fffcfPp7/KV27dPAtc3tO\nLg0zjnz1Gen1yc3d5Vs+UJy3Z89eJk6cTqtWzXzLDnL7L764JT17dGHDuoW89eY/ufzydoweNdSX\n7CKpqam8M344Y8dO4IMPvD2+Xh6qBVFPiSxeBXIAMFdEpovIMGeaAcwFHixtQVUdpqotVbXlXbfe\neNR7+775lv84/yJ+f/gwC5cso/HpDenY/iIWfb4cgCXLVnJ6w/DdQ1tzdhT3HrLXbiAvL59aJ5/k\n7ZYeo3btUznZyahSpQpXdLyUtWv9e7bOkqXLadKkMY0aNSQtLY0+fXoxecos3/KrVatKjRrVi193\n7nQZWVlrfcsPcvt//8QQGv2sJU3ObMvNfX/DRx/9m9tuf8CX7CLDh73A6jUbePmVUm9MiZ8k28WO\ny1lsVZ0hImcS3qVOJ3z8MQdYosfxT8aevd/w+z89T0FhIVqodO14KR3ataHF+efw6OC/8Mb4D6hW\ntQqDBw0AYPa8BUyaPpfU1FSqVK7E808PKt4Fu/Xeh9m8dRuHDn3PFb378vRjv6VdmwuPe9vr16/L\nyBEvk5ISIhQK8e67k5k6bc5xf255FRQU8OCAJ5g29W1SQiFGjR5PdvY63/Lr1q3Du++ELy1JTU1h\n3LgPmDlrnm/5QW9/kNpd3Ipb+l7LipXZLF0S/kfhySeHMH3Gh/6tRILvMkdL/D4+FY1YHiHpFXuq\noT3VsELnx/jY1+8//yDq39kqF/a2x74aYyoAe6qhMca4SPBjitGyAmmM8U6SHYO0AmmM8Y71II0x\nxoX1II0xxoUVSGOMKVmi3xkTLSuQxhjvWA/SGGNcJNlJGnsmjTHGuLAepDHGO7aLbYwxLpJsF9sK\npDHGO9aDNMYYF9aD9E9anTMCzS8adsryLb8i5sfEepDGGOPCCqR/gh4w9qw6rQLLX71nSeDbX6EH\njLX82NgutjHGuLAepDHGuLAepDHGuEiyHqTdamiM8U4cHvsqIlVEZLGIfCkiWSIy2GlvLCKLRGS9\niIwXkUpOe2Xn5w3O+40iPusxp32tiHQtK9sKpDHGO4WF0U9lOwx0VNULgGZANxFpC/wZeElVmwLf\nAHc6898JfKOqTYCXnPkQkbOBG4BzgG7AP0UkpbRgK5DGGO/EoUBq2AHnxzRnUqAj8K7TPhro7bzu\n5fyM8/4VIiJO+zhVPayqm4ENQOvSsq1AGmO8oxr1JCL9RGRpxNTv2I8VkRQRWQ7sBmYDG4FvVTXf\nmSUHSHdepwPbwquj+cB3wGmR7SUsUyI7SWOM8U4MJ2lUdRgwrIx5CoBmIlILmACcVdJszp/i8p5b\nuysrkMYY78T5LLaqfisi84C2QC0RSXV6iRlA0RXuOUBDIEdEUoGTgX0R7UUilymR7WIbY7wTn7PY\ndZyeIyJSFegErAY+Aq51ZrsNmOi8nuT8jPP+h6qqTvsNzlnuxkBTYHFp2daDNMYkuvrAaOeMcwjI\nVNUpIpINjBORPwHLgBHO/COAN0RkA+Ge4w0AqpolIplANpAP9NcynjJmBdIY45047GKr6gqgeQnt\nmyjhLLSqfg9c5/JZzwLPljfbCqQxxjta6jmPE05SHoPs2qUDWavmsyZ7AY8M7B+XjEqVKzF+xutM\n+OgtJs8fx32P3A1Am0ta8t6cMUz6eCz/+7enSEkJX4fasVt7Ppj3Fu9/+CbvzBpNizYXxGW9wJ/t\nt/zEy06E/DhdKB4Y0QSu+KmV0qNeuVAoxOqsT+jW/UZycnJZ+Nk0+t7yG1avXh/V55RnuLNq1aty\n6OB/SU1N4c3Jwxnyh5d4cdhz/M81/dmyaSv3P9qPHdt28t7bk4rnBTjz7Ca8NPw5rmrXx/WzYx3u\nzMvtDyr/eIb7CjLfy+8+8HzVki6JKdN/Rzwc9e9s1TufjynLD0nXg2zdqjkbN25h8+at5OXlkZk5\nkat7lnnLZUyKCl5qWippaakUFhTyww8/sGXTVgA+nbeYLj0uP2pegGrVqhKvf5j83H7LT5zsRMgH\n4nIWO0iBFEgRuSNen90gvR7bco5c2pSzPZcGDerFJSsUCvH+h2+yIHsmn368mBVfZJGWmso5F4Sv\nYe3SsyP1GtQtnr9T9w5M/Xcmr771Ik8M+FNc1snP7bf8xMlOhHwALdSop0QWVA9ysNsbkbcdFRYe\njPqDw7dcHi1evbXCwkJ+1bEvl1/Qg/Oan03TX/yM3/36CQY981vGz3idQwcOkV9w5CqCOdPmcVW7\nPtx/2yM8MOjXcVknP7ff8hMnOxHygaQ7Bhm3s9gissLtLaCuy3tH3XYUyzHI7Tm5NMw4cuwmI70+\nubm7ov2YqOz/zwEWf/oFl3S8iNf/+Ra3XB2+lfTiDm04/Yyf/mj+pQuX0fD0DGqdejLf7vvO03UJ\nYvstP/jsRMgHEn6XOVrx7EHWBW4FepYw7Y1X6JKly2nSpDGNGjUkLS2NPn16MXnKLM9zTjmtFjVP\nqgFA5SqVuah9azav/4pTa58CQFqlNO6671bGj34fgJ82zihe9uzzfk5apVTPiyP4t/2Wn1jZiZAP\nQKFGPyWweF4HOQWooarLj33DuZcyLgoKCnhwwBNMm/o2KaEQo0aPJzt7nec5derWdi7jCRGSEDMm\nzWHe7AU8/NT9dOh8CaFQiHGj3mPRgqUAdOnRkV7XdScvP5/D3x/moX6/93ydwL/tt/zEyk6EfCDh\nd5mjlXSX+XjFnmpoTzWs0PkxXuZz6JV7ov6drfbgvxL2Mh+7k8YY450E7nDFwgqkMcY7SbaLbQXS\nGOOdBD/pEi0rkMYY7yTZZT5WII0x3kmyHmTS3YttjDFesR6kMcYzaidpjDHGRZLtYluBNMZ4x07S\nGGOMC+tBGmOMCzsGaYwxLqwHaYwxLuwYpDHGuLAepH+Khn0Kyuo9SwLND3r7Lb9i58fCroP0UUUd\nD7EoPy3A/DwbD7LC58fEepDGGOPCCqQxxrhIspM0NliFMcY7cXhol4g0FJGPRGS1iGSJyINO+x9F\nZLuILHem7hHLPCYiG0RkrYh0jWjv5rRtEJFBZWVbD9IY4xmNzy52PvA7Vf1CRGoCn4vIbOe9l1T1\n+ciZReRs4AbgHKABMEdEznTe/gfQGcgBlojIJFXNdgu2AmmM8U4cCqSq5gK5zuv9IrIaSC9lkV7A\nOFU9DGwWkQ1Aa+e9Daq6CUBExjnzuhZI28U2xninsDD6KQoi0ghoDixymu4TkRUiMlJETnHa0oFt\nEYvlOG1u7a6sQBpjAiUi/URkacTUz2W+GsB7wABV/Q/wKnAG0IxwD/OFollLWFxLaXdlu9jGGO/E\nsIutqsOAYaXNIyJphIvjW6r6vrPcroj3hwNTnB9zgIYRi2cARRd3urWXyHqQxhjvxOcstgAjgNWq\n+mJEe/2I2X4JrHJeTwJuEJHKItIYaAosBpYATUWksYhUInwiZ1Jp2daDNMZ4RjUuZ7HbAbcAK0Vk\nudP2OHCjiDQjvJu8Bfi1sw5ZIpJJ+ORLPtBfVQsAROQ+YCaQAoxU1azSgq1AGmO8E5+z2Aso+fjh\ntFKWeRZ4toT2aaUtdywrkMYY79ithsYYU7I4XSgeGCuQxhjvJFmBTMqz2F27dCBr1XzWZC/gkYH9\nkz5/+LAX2J7zJcuWzS1uO//8s/lk/iSWfTGHCRNGUbNmjbivR5GK9v0nSnYi5FMYw5TAkq5AhkIh\nhr7yLD169uW8Cy7n+ut7c9ZZTZM6f/SYTHr0uPmotv/71195/PfP0bxFJyZ+MJ3f/e7euK5DkYr4\n/SdCdiLkQ3gXO9opkcWtQIrIL0TkCufq98j2bvHKBGjdqjkbN25h8+at5OXlkZk5kat7di17wRM4\nf8GCRez75tuj2s488ww++WQhAHPmfsIvf9m9pEU9VxG//0TIToR8IC7XQQYpLgVSRB4AJgL3A6tE\npFfE28/FI7NIg/R6bMs5cnF8zvZcGjSoF8/IhMovkpW1lp49uwBw7TU9aJjhz+jUQW9/kPkVeduL\n2S52udwNXKiqvYEOwJNFY7hR8vVMxSLvyywsPBh1cPii+6PF6eLVhMwvcne/h7j3nttZtHA6NWpW\n54cf8nzJDXr7g8yvyNtenJdku9jxOoudoqoHAFR1i4h0AN4VkdMpo0BG3peZWik96m9ve07uUb2l\njPT65ObuKmUJbwWdX2Tt2o10v+omAJo2/Rndr7zCl9ygtz/I/Iq87cUSvEcYrXj1IHc6twAB4BTL\nHkBt4Lw4ZQKwZOlymjRpTKNGDUlLS6NPn15MnjIrnpEJlV+kTp3TgHCv4vHHHmTYsDd8yQ16+4PM\nr8jbXsR6kOVzK+F7IIupaj5wq4j8X5wyASgoKODBAU8wberbpIRCjBo9nuzsdfGMDDz/jTf+wWXt\nL6J27VPZvGkpTz/9PDVqVOeee28H4IMPpjFq9Pi4rkORivj9J0J2IuQDSdeDlCCOj5VXLLvYXrHH\nvtpjXyt0vmqph8Lc7O15WdS/s6dN/jimLD8k3XWQxhjjFbvV0BjjnSTbxbYCaYzxTJI9FtsKpDHG\nQ1YgjTGmZNaDNMYYF1YgjTHGhRVIY4xxE9vlkwnLCqQxxjPWgzTGGBdaaD1IY4wpkfUgjTHGRYy3\ncCcsK5DGGM9YD9IYY1zYMUgfFQ37VFHz8yr49lt+sPmxSODRE2OS0AUy6PEYK3p+h3R/HtNwrHnb\nw8/3rtDjMSZAfiysB2mMMS6SrUDagLnGmIQmIg1F5CMRWS0iWUVPSBWRU0Vktoisd/48xWkXERkq\nIhtEZIWItIj4rNuc+deLyG1lZVuBNMZ4RjX6qRzygd+p6llAW6C/iJwNDALmqmpTYK7zM8CVQFNn\n6ge8CuGCCjwFtAFaA08VFVU3ViCNMZ7RQol6KvMzVXNV9Qvn9X5gNZAO9AJGO7ONBno7r3sBYzRs\nIVBLROoDXYHZqrpPVb8BZgPdSsu2AmmM8YyqRD2JSD8RWRox9XP7fBFpBDQHFgF1VTU3nKu5wE+c\n2dKBbRGL5Thtbu2uyjxJIyJ1geeABqp6pdO1vUhVR5S1rDGmYonlQnFVHQYMK2s+EakBvAcMUNX/\niLj2Pkt6Q0tpd1WeHuQoYCZQdM3BOmBAOZYzxlQwhSpRT+UhImmEi+Nbqvq+07zL2XXG+XO3054D\nNIxYPAPYUUq7q/IUyNqqmonztAlVzQcKyrGcMaaCiWUXuywS7iqOAFar6osRb00Cis5E3wZMjGi/\n1Tmb3Rb4ztkFnwl0EZFTnJMzXZw2V+W5DvKgiJyG0xUtCizHcsaYCiZO10G2A24BVorIcqftcWAI\nkCkidwJbgeuc96YB3YENwCHgDgBV3ScizwBLnPmeVtV9pQWXp0A+RLginyEi/wbqANeWc8OMMRVI\nPG41VNUFlHz8EOBHt3upqgL9XT5rJDCyvNllFkhV/UJELgN+7qzkWlXNK2+AMabiSLY7acpzFvvW\nY5paiAiqOiZO62SMOUGV96TLiaI8u9itIl5XIdyl/QKwAmmMOUqFGzBXVe+P/FlETgbeiNsaGWNO\nWMk23Fksd9IcInyPY0LKyGjAnFnvsHLFPL5c/iH333en7+vQtUsHslbNZ032Ah4ZWOKx4hM+v8ZJ\n1Rn8f39gzLyRjP5oBGe3OIuatWry/Nt/5s1PRvH823+mxsk1ALj+nj68NvNfvDbzX7w+Zzhzv5pJ\nzVo147JeEOz3XxH+7ksTr+sggyJaRskXkckcudo8BJwNZKrqIPelvJFaKT3qf4/q1fsJ9ev9hGXL\nV1GjRnUWL5rBNdf+D6tXr4/qc2IdjzEUCrE66xO6db+RnJxcFn42jb63/OaEzC9tPMhBLz3CysUr\nmTp2OqlpqVSpWpmb77+J/d/u5+1/jOOm/jdQ4+QaDHvutaOWu6hTW667+xoeun6g62cfz3iQXmx/\nrOMxevndB54f477ysp/2ivp3tvnWiQlbJcvTg3weeMGZ/hdoX57iKCKtRaSV8/psEXlIRLof19qW\nw86du1m2fBUABw4cZM2a9aQ3qBfv2GKtWzVn48YtbN68lby8PDIzJ3J1z65JlV+tRjUuaHMeU8dO\nByA/L58D/zlIuy4XM+OdWQDMeGcWl3Rt96Nlr+jdkbkTP/J0fSIF+f1XhL/7ssRpNJ/AlFogRSQF\neFJVP3amf6tqTlkfKiJPAUOBV0Xkf4G/AzWAQSLyey9WvDxOPz2DZhecy6LFy/yKpEF6PbblHLl7\nKWd7Lg18LNB+5Df4aX2+3fcdg14cyPAZ/2LgXx+iStUqnFr7FPbtDl93u2/3Pk45rdZRy1WuUpnW\nHVoyf9onnq7PUesW4PdfEf7uy5Jsu9ilFkhVLQAOOSdmonEt4avf2xO+YLO3qj5NeLih60tbMHJk\nj8LCg1HGHlG9ejUyxw/noYefYv/+AzF/TrRKuoG+rMMYJ1p+SmoKZ57blIlvTObubvfw30Pfc1P/\nG8pc7uLOF7FqSRb7v93v6fpECvL7rwh/92WJx62GQSrPZT7fE77FZzZQXLFU9YFSlsmPKK4bVfU/\nzjL/FZFSx/uIHNkjlmOQAKmpqbwzfjhjx07ggw+mx/IRMduek0vDjCPHjjLS65Obuyup8vfk7mFP\n7h5WL1sDwMdT53NT/xvZ9/U3nPqTU9m3ex+n/uRUvtn77VHLdezVIa671xDs918R/u4rmvIcg5wK\nPAnMBz53pqVlLPODiFRzXl9Y1Oj0ROP+5Nzhw15g9ZoNvPxKmSMoeW7J0uU0adKYRo0akpaWRp8+\nvZg8ZVZS5e/b8w27d+yh4c8yALjwkhZ8tf4rPp39Gd2u6wJAt+u68O9ZnxYvU71mdS5oez7/nvlp\niZ/plSC//4rwd1+WZNvFLk8PspaqvhLZUPRMiFK0V9XDAKpHjRCXxpHRN+Ki3cWtuKXvtaxYmc3S\nJeH/OZ58cgjTZ3wYz9hiBQUFPDjgCaZNfZuUUIhRo8eTnb3Ol2w/84c++Xee+NtjpFZKI/erXIb8\n7q+EJMRT/3qC7jd0Y9f23fzxnmeK57+0WzuWfvw53//3e8/XJVKQ339F+bsvTYKfc4laeS7z+UJV\nWxzTtkxVm8d1zYh9F9sLifDY1aDz7bGvFTg/xoODn9a/Jurf2Ytz30vYbqRrD1JEbgRuAhqLyKSI\nt2oCe+O9YsaYE0+in3SJVmm72J8CuUBtwtdAFtkPrIjnShljTkxxP8HgM9cCqapfAV8BF5X2ASLy\nmaqWOo8xpmJQ12EbT0zlOUlTlioefIYxJgkUJtlZGi8KZJJ9JcaYWBVaD9IYY0qWbLvYZV4oLiL3\nOU8Ac53Fw/UxxpzACmOYEll57qSpBywRkUwR6SY/vuHzljislzHmBKRI1FMiK7NAquoThAfIHQHc\nDqwXkedE5Azn/VVxXUNjzAmjIvYgix6juNOZ8oFTgHdF5C9xXDdjzAkm2QpkeZ5q+ADh+6e/Bl4D\nBqpqnoiEgPXAI/FdRWPMiSLRd5mjVZ6z2LWBXzkXjhdT1UIR6RGf1TLGnIiS7LHY5Xqq4R9KeW+1\nt6tjjDmRJdt1kLE81dAYYyqEhL5QvGjYJ8sPRtGwY0EJevsren4sku22uoQukEGPh2j5FXg8RMuP\nSTzOSovISKAHsFtVz3Xa/gjcDexxZntcVac57z0G3AkUAA+o6kynvRvwCpACvKaqQ8rKTugCaYw5\nsRSW8OAwD4wi/GTUMce0v6Sqz0c2iMjZwA3AOUADYI6InOm8/Q+gM5BD+OaXSaqaXVqwFUhjjGfi\nsYutqvNFpFE5Z+8FjHMe+bJZRDYArZ33NqjqJgARGefMW2qBtJM0xhjP+Hyh+H0iskJERkaMF5EO\nbIuYJ8dpc2svlRVIY4xnCiX6SUT6icjSiKlfOaJeBc4AmhF+8kHRUw9K2sfXUtpLZbvYxhjPxHId\npKoOA6J6RrOqFj/wW0SGA1OcH3OAhhGzZgBFZ53c2l1ZD9IY4xmNYYqFiNSP+PGXQNGgOZOAG0Sk\nsog0JjzQzmJgCdBURBqLSCXCJ3IiH0ZYIutBGmM8E49bDUVkLNABqC0iOcBTQAcRaUa4xm4Bfg2g\nqlkikkn45Es+0F9VC5zPuQ+YSfgyn5GqmlVWthVIY4xn4nEdpKreWELziFLmfxZ4toT2acC0aLKt\nQBpjPGN30hhjjIsKN5qPMcaUV6IPgBstK5DGGM9YgTTGGBeaZLvYSXcdZEZGA+bMeoeVK+bx5fIP\nuf++O31fh65dOpC1aj5rshfwyMD+lu+zDesWsuyLOSxdMouFn0V10vK4Bb3tQedXuGfSnGjy8/MZ\n+Mhgli1fRY0a1Vm8aAZz5s5n9er1vuSHQiGGvvIs3brfSE5OLgs/m8bkKbMs36f8Ip06X8fevd/4\nmhn0tgedn4ySrge5c+duli0PX1R/4MBB1qxZT3qDer7lt27VnI0bt7B581by8vLIzJzI1T27Wn4F\nEPS2B50PydeD9K1AisixY7nF3emnZ9DsgnNZtHiZb5kN0uuxLefILZ4523Np4GOBruj5AKrK9Glj\nWbRwOnfdebNvuUFve9D54N+thn6Jyy62iBx7j6MAl4tILQBVvToeuZGqV69G5vjhPPTwU+zffyDe\nccWkhAFDw48Vt3y/tO/Qm9zcXdSpcxozpo9j7doNfLJgUdxzg972oPPBroMsrwzC90K+xpGhhlpy\nZEgiV85QR/0AJOVkQqHqUYenpqbyzvjhjB07gQ8+mB718sdje04uDTOODJWfkV6f3NxdpSxh+V4r\nytuzZy8TJ06nVatmvhTIoLerbcSnAAASTElEQVQ96HxI/F3maMVrF7sl8Dnwe+A7VZ0H/FdVP1bV\nj0tbUFWHqWpLVW0ZS3EEGD7sBVav2cDLr0Q1gpInlixdTpMmjWnUqCFpaWn06dOLyVNmWb5PqlWr\nSo0a1Ytfd+50GVlZa33JDnrbg86H5DsGGZcepKoWAi+JyDvOn7vilXWsdhe34pa+17JiZTZLl4T/\n53jyySFMn/GhH/EUFBTw4IAnmDb1bVJCIUaNHk929jpfsi0f6tatw7vvhMcxSE1NYdy4D5g5a54v\n2UFve9D5kPjHFKMlfhyjEJGrgHaq+ng0y6VWSg/s+67ITxUMOj9RnupXofM1tku+/3J636h/Zx/5\n6s2EPXLpS69OVacCU/3IMsYEJ9F3maOVdBeKG2OCk2y72FYgjTGeKUyyEmkF0hjjGdvFNsYYF8nV\nf7QCaYzxkPUgjTHGhd1qaIwxLuwkjTHGuEiu8piE40EaY4xXrAdpjPGMnaQxxhgXdgzSGGNcJFd5\ntAJpjPGQ7WL7qGjYJ8u3fMs/MSTbLradxTbGeCYeD+0SkZEisltEVkW0nSois0VkvfPnKU67iMhQ\nEdkgIitEpEXEMrc5868XkdvKsz0J3YOsqAPGVvT8hBgwFkivdXYg+du/zQaC3/5YxGkXexTwdyDy\nyaiDgLmqOkREBjk/PwpcCTR1pjbAq0AbETkVeIrw42AU+FxEJqlqqQ9Ptx6kMcYzGsN/ZX6m6nxg\n3zHNvYDRzuvRQO+I9jEathCoJSL1ga7AbFXd5xTF2UC3srKtQBpjPBPLQ7tEpJ+ILI2Y+pUjqq6q\n5gI4f/7EaU8HtkXMl+O0ubWXKqF3sY0xJ5ZYTtKo6jDAq0eQljRchpbSXirrQRpjPBOPkzQudjm7\nzjh/7nbac4CGEfNlADtKaS+VFUhjjGcK0ainGE0Cis5E3wZMjGi/1Tmb3Rb4ztkFnwl0EZFTnDPe\nXZy2UtkutjHGM/E4iy0iY4EOQG0RySF8NnoIkCkidwJbgeuc2acB3YENwCHgDgBV3ScizwBLnPme\nVtVjT/z8iBVIY4xnynNWOurPVL3R5a0rSphXgf4unzMSGBlNthVIY4xn7FZDY4xxEY8eZJDsJI0x\nxriwHqQxxjO2i22MMS4KNbl2sa1AGmM8k1zlMQmPQQ4f9gI7cr5k+bK5ga1D1y4dyFo1nzXZC3hk\nYIlXHFj+CZzfIL0e70x6nXkLJ/HhpxO589d9AXjo0d+wNOtDZs1/j1nz36Nj50uLl7nvt3ex4PPp\nzF88hcs6tvN8nYoE/d37eKG4L5KuQI4Zk8lVPW4OLD8UCjH0lWfp0bMv511wOddf35uzzmpq+UmU\nn5+fz+An/kKHtlfTs8uN3H7XjTT9+RkADH91DF3aX0OX9tfw4exPAGj68zPo9avudLzoam6+9tc8\n9/wThELe/+oF/d1DfEbzCVLSFchPFixi3zffBpbfulVzNm7cwubNW8nLyyMzcyJX9+xq+UmUv3vX\n16xasRqAgwcOsX7dJurV/4nr/F27X87E96fxww95bNu6nS2bttH8wvM8XScI/ruH2EbzSWS+FEgR\nuUREHhKRLn7kBalBej225Ry5Bz5ney4NGtSz/CTNz2jYgHPPP4tln68A4I67b2L2gvd54W/PcPLJ\nJwFQr35ddmzfWbxM7o6d1Ktf1/N1Cfq7B9vFLhcRWRzx+m7CowHXBJ5yRv9NWiI/HlVJfTyzZ/n+\n5VerXo3hY17mqceGcGD/QcaMHM/FzbvR5dJr2L1rD3/400Bf1yno7x5sF7u80iJe9wM6q+pgwiNo\nlHqAMHLwzMLCg3FavfjZnpNLw4wjQ+VnpNcnN3eX5SdZfmpqKsNHv8yEd6YyfcocAL7es5fCwkJU\nlbdGv0szZzc6d8dOGqQf6cnVb1CPXTt3l/i5xyPo7x5sF7vcn+sMK3QaIKq6B0BVDwL5pS2oqsNU\ntaWqtgyFqsdp9eJnydLlNGnSmEaNGpKWlkafPr2YPGWW5SdZ/gt/e5oN6zYx7J+ji9t+Urd28esr\ne3Ri7er1AMya/hG9ftWdSpXSaPjTdBqf8VOWfb7S83UK+ruHcI812imRxes6yJOBzwmP4qsiUk9V\nd4pIDUoe2dczb77xDy5rfxG1a5/Klk1LGfz087w+alw8I49SUFDAgwOeYNrUt0kJhRg1ejzZ2ess\nP4nyW7VtwbU39CI7ay2z5r8HwJBnXqb3Nd05+7xfoKrkbN3Bo7/9IwDr1mxk8gcz+GjhJAryC/j9\nwD9RWOh93yno7x6S77Gv4vPxoWqEnyWxuTzzp1ZKD+zbrshPFQw6355qmABPNVSNqSPT86c9ov6d\nnbx1Slw7TcfD1ztpVPUQUK7iaIw58ST6SZdo2a2GxhjPJNsuthVIY4xnEv2kS7SsQBpjPJPol+1E\nywqkMcYzyXYMMunuxTbGGK9YD9IY4xk7SWOMMS7sJI0xxriwHqQxxrhItpM0ViCNMZ6xh3YZY4yL\n5CqPViCNMR5KtmOQdh2kMcYz8XrkgohsEZGVIrJcRJY6baeKyGwRWe/8eYrTLiIyVEQ2iMgKEWkR\n6/YkdA+yaNgpy7f8IBQNOxaUoLc/FnG+zOdyVf064udBwFxVHeI8ymUQ8ChwJdDUmdoArzp/Rs16\nkMYYz/j80K5eQNGQ7qOB3hHtYzRsIVBLROrHEpDQPciKOmBsRc9PlAFzg84/v27bQPJX7FoY87Kx\nXOYjIv0IP7uqyDBVHfajj4ZZIqLA/znv11XVXABVzRWRomfvpgPbIpbNcdpyo123hC6QxpgTSyy7\n2E6xO7YgHqudqu5wiuBsEVlTyrwljVAeU1fVCqQxxjPxOoutqjucP3eLyASgNbBLROo7vcf6QNGj\nInOAhhGLZwAxHdC1Y5DGGM/E46mGIlJdRGoWvSb8+OhVwCTgNme224CJzutJwK3O2ey2wHdFu+LR\nsh6kMcYzcepB1gUmiAiEa9bbqjpDRJYAmSJyJ7AVuM6ZfxrQHdgAHALuiDXYCqQxxjPxuBdbVTcB\nF5TQvhe4ooR2Bfp7kW272MYY48J6kMYYz9hgFcYY48KGOzPGGBfWgzTGGBfWgzTGGBfWgzTGGBfW\ngzTGGBfJ1oNMyusgN6xbyLIv5rB0ySwWfjbN9/yuXTqQtWo+a7IX8MhAT65XtfwTJN/P7FAoxPjZ\no/nbG88D8McXH+eduWN498M3eOG1Z6larSoAF7ZtxvhZo/gi5xM697g8ruukMfyXyJKyQAJ06nwd\nLVt1oe1F3X3NDYVCDH3lWXr07Mt5F1zO9df35qyzmlp+Bcj3O/vmu/uwef2W4p//+oeXue6KW7m2\n4y3k5uzixv+5FoDc7Tt54sFnmD5hdtzWpYhqYdRTIotLgRSRNiJykvO6qogMFpHJIvJnETk5HpmJ\nonWr5mzcuIXNm7eSl5dHZuZEru7Z1fIrQL6f2XXr16F9p3a8/9ak4raDBw4Vv65StXJx72zHtp2s\nX72RwsL4FyOfB8yNu3j1IEcSvkkc4BXgZODPTtvrccospqpMnzaWRQunc9edN8c77igN0uuxLefI\nyEo523Np0KCe5VeAfD+zH3lmAC8+83cKj+mBPf3y7/lo5VQaNTmdsSPeiUt2aeIxmk+Q4lUgQ6qa\n77xuqaoDVHWBqg4GflbagiLST0SWisjSwsKDMYW379Cb1m260aNnX+6993YuvSSmx1HExBlx5Ch+\n/k9g+cHl+5XdvnM79n39DatXrP3Re38Y8CxXXNCTzeu30LVXJ8+zy2I9yPJZJSJFQwx9KSItAUTk\nTCCvtAVVdZiqtlTVlqFQ9ZjCc3N3AbBnz14mTpxOq1bNYvqcWGzPyaVhxpGh+jPS6xevj+Und75f\n2c1anU+HLpcyfcn7/OVfz9C63YU89/enit8vLCxkxsS5dLoqvidkSmI9yPK5C7hMRDYCZwOficgm\nYLjzXtxUq1aVGjWqF7/u3OkysrJ+/C9tvCxZupwmTRrTqFFD0tLS6NOnF5OnzLL8CpDvV/bQ516l\nc4teXNnqVzxyz5Ms/vfnPH7fYBo2yiiep0OXS9iy4SvPs8tSqBr1lMjich2kqn4H3O6MAvwzJydH\nVeP+T3ndunV4950RAKSmpjBu3AfMnDUv3rHFCgoKeHDAE0yb+jYpoRCjRo8nO3ud5VeA/CCzRYQ/\nDX2SGjWrIwJrszbwp0f/AsA5zc7i5ZFDOKlWTS7rfAn3DryLX10Wn2PziX7ZTrQkkbu4qZXSA1u5\nivxUwaDzE+WpgkHnB/pUQ9WSHnxVpron/yLq39ld362JKcsPSXsdpDHGHC+71dAY45lEPysdLSuQ\nxhjPJPIhu1hYgTTGeCbRz0pHywqkMcYz1oM0xhgXdgzSGGNcWA/SGGNc2DFIY4xxkWx30liBNMZ4\nxnqQxhjjItmOQdqthsYYz8TrmTQi0k1E1orIBhEZFOfNKGY9SGOMZ+LRgxSRFOAfQGcgB1giIpNU\nNdvzsGNYD9IY45k4DZjbGtigqptU9QdgHNArrhviSOgeZNGwT5Zv+RUxf8WuhYHmxyJORyDTgW0R\nP+cAvjxHJaELZKxj0hURkX6qOsyr1TlRsi3f8oPKz/9he9S/syLSD+gX0TTsmHUv6TN9ORuU7LvY\n/cqeJSmzLd/yg84vt8jnUDnTsYU9B2gY8XMG4Ev3PtkLpDHmxLcEaCoijUWkEnADMKmMZTyR2LvY\nxpgKT1XzReQ+YCaQAoxU1Sw/spO9QAZ2DCjgbMu3/KDzPaWq04Bpfucm9EO7jDEmSHYM0hhjXFiB\nNMYYF0lZIIO6b9PJHikiu0VklZ+5EfkNReQjEVktIlki8qDP+VVEZLGIfOnkD/Yz31mHFBFZJiJT\n/M528reIyEoRWS4iS33OriUi74rIGuf/gYv8zE82SXcM0rlvcx0R920CN/px36aT3x44AIxR1XP9\nyDwmvz5QX1W/EJGawOdAbx+3X4DqqnpARNKABcCDqurbbSEi8hDQEjhJVXv4lRuRvwVoqapfB5A9\nGvhEVV9zLomppqrf+r0eySIZe5CB3bcJoKrzgX1+5ZWQn6uqXziv9wOrCd+q5Ve+quoB58c0Z/Lt\nX2ERyQCuAl7zKzNRiMhJQHtgBICq/mDF8fgkY4Es6b5N3wpEIhGRRkBzYJHPuSkishzYDcxWVT/z\nXwYeAQp9zDyWArNE5HPnNjq//AzYA7zuHGJ4TUSq+5ifdJKxQAZ232YiEZEawHvAAFX9j5/Zqlqg\nqs0I3xLWWkR8OdQgIj2A3ar6uR95pWinqi2AK4H+zmEXP6QCLYBXVbU5cBDw9Rh8sknGAhnYfZuJ\nwjn29x7wlqq+H9R6OLt384BuPkW2A652jgGOAzqKyJs+ZRdT1R3On7uBCYQP+/ghB8iJ6LG/S7hg\nmhglY4EM7L7NROCcJBkBrFbVFwPIryMitZzXVYFOwBo/slX1MVXNUNVGhP/eP1TVvn5kFxGR6s7J\nMZzd2y6AL1c0qOpOYJuI/NxpugLw5eRcskq6Ww2DvG8TQETGAh2A2iKSAzylqiP8yifci7oFWOkc\nBwR43LlVyw/1gdHO1QQhIFNVA7ncJiB1gQnhf6dIBd5W1Rk+5t8PvOV0DjYBd/iYnXSS7jIfY4zx\nSjLuYhtjjCesQBpjjAsrkMYY48IKpDHGuLACaYwxLqxAmoQiIo2CGgnJmGNZgTS+cK6LNOaEYgXS\nlEhEnokcS1JEnhWRB0qYr4OIzBeRCSKSLSL/EpGQ894BEXlaRBYBF4nIhSLysTOIw0xnaDac9i9F\n5DOgv1/baExZrEAaNyOA2wCcgncD8JbLvK2B3wHnAWcAv3LaqwOrVLUN4RGF/gZcq6oXAiOBZ535\nXgceUFUb3NUklKS71dB4Q1W3iMheEWlO+Pa5Zaq612X2xaq6CYpvtbyE8EAJBYQHzQD4OXAuMNu5\nDS8FyBWRk4FaqvqxM98bhEfBMSZwViBNaV4DbgfqEe7xuTn2ftWin79X1QLntQBZx/YSnYEt7H5X\nk5BsF9uUZgLhocpaER78w01rZ/SkEHA94ccsHGstUKfoGSkikiYi5zhDon0nIpc4893s3eobc3ys\nB2lcqeoPIvIR8G1ET7AknwFDCB+DnE+4sJb0WdcCQ53d6lTCo39nER5xZqSIHKL0QmyMr2w0H+PK\n6RF+AVynqutd5ukAPBzEw7GMiTfbxTYlEpGzgQ3AXLfiaEyysx6kKRcROY/wGeZIh51LeIxJSlYg\njTHGhe1iG2OMCyuQxhjjwgqkMca4sAJpjDEurEAaY4yL/wcn6Rvqt2eC3wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "et_hpo = ExtraTreesClassifier(n_estimators = 53, min_samples_leaf = 1, max_depth = 31, min_samples_split = 5, max_features = 20, criterion = 'entropy')\n",
    "et_hpo.fit(X_train,y_train) \n",
    "et_score=et_hpo.score(X_test,y_test)\n",
    "y_predict=et_hpo.predict(X_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of ET: '+ str(et_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of ET: '+(str(precision)))\n",
    "print('Recall of ET: '+(str(recall)))\n",
    "print('F1-score of ET: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "et_train=et_hpo.predict(X_train)\n",
    "et_test=et_hpo.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply stacking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DecisionTree</th>\n",
       "      <th>ExtraTrees</th>\n",
       "      <th>RandomForest</th>\n",
       "      <th>XgBoost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   DecisionTree  ExtraTrees  RandomForest  XgBoost\n",
       "0             0           0             0        0\n",
       "1             0           0             0        0\n",
       "2             1           1             1        1\n",
       "3             0           0             0        0\n",
       "4             3           3             3        3"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base_predictions_train = pd.DataFrame( {\n",
    "    'DecisionTree': dt_train.ravel(),\n",
    "        'RandomForest': rf_train.ravel(),\n",
    "     'ExtraTrees': et_train.ravel(),\n",
    "     'XgBoost': xg_train.ravel(),\n",
    "    })\n",
    "base_predictions_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dt_train=dt_train.reshape(-1, 1)\n",
    "et_train=et_train.reshape(-1, 1)\n",
    "rf_train=rf_train.reshape(-1, 1)\n",
    "xg_train=xg_train.reshape(-1, 1)\n",
    "dt_test=dt_test.reshape(-1, 1)\n",
    "et_test=et_test.reshape(-1, 1)\n",
    "rf_test=rf_test.reshape(-1, 1)\n",
    "xg_test=xg_test.reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(23334, 1)"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x_train = np.concatenate(( dt_train, et_train, rf_train, xg_train), axis=1)\n",
    "x_test = np.concatenate(( dt_test, et_test, rf_test, xg_test), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of Stacking: 0.9955223880597015\n",
      "Precision of Stacking: 0.9955023011268291\n",
      "Recall of Stacking: 0.9955223880597015\n",
      "F1-score of Stacking: 0.99550369632154\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3645\n",
      "           1       0.99      1.00      1.00       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       1.00      1.00      1.00       609\n",
      "           4       0.83      0.71      0.77         7\n",
      "           5       0.99      1.00      0.99       251\n",
      "           6       0.99      0.99      0.99       436\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       0.97      0.96      0.96      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8FGX+wPHPd5MgVRBBSsIJCp5d\nUJpiQVRABPFORbChh6J3WDgVRE/kUPHH3Vm5ogeCgIWmIr0jIiodpIRehEAoiiLlxJTv74+dhAWz\nSXaZnVk237eveWXz7Mx8Z3bNl+eZZ55nRFUxxhjzawG/D8AYY+KVJUhjjAnDEqQxxoRhCdIYY8Kw\nBGmMMWFYgjTGmDAsQRpjTBiWII0xJgxLkMYYE0ay3wdQKBEb5mOMH1Qlms2yvtsc8d9sSpWzoorl\nhbhOkFl7N/kWO6Xq2SSn1PQtfnbWzhIbPztrJ4DF9zm+ifMEaYw5yeTm+H0ErrIEaYxxj+b6fQSu\nsgRpjHFPriVIY4wpkCZYDdJu8zHGuCc3N/KlCCJSWkQWisg3IrJaRPo65UNFZIuILHeW+k65iMgA\nEdkoIitE5NKQfXUWkQ3O0rmo2FaDNMa4JzY1yCNAC1U9KCIpwDwRmeK810NVPzpu/RuBes7SBHgL\naCIilYE+QENAgSUiMl5VfwgX2GqQxhj35OZEvhRBgw46v6Y4S2H3W7YHhjvbzQcqiUgNoBUwQ1X3\nOUlxBtC6sNiWII0x7tHcyJdiEJEkEVkO7CGY5BY4b/VzmtGvi8gpTlkqsD1k8wynLFx5WJYgjTHu\nieIapIh0FZHFIUvX43erqjmqWh9IAxqLyIXAM8C5QCOgMvC0s3pBI3O0kPKwLEEaY1yjmhvFogNV\ntWHIMjD8/vVHYA7QWlUznWb0EeBdoLGzWgZQK2SzNGBnIeVhWYI0xrgnNr3YVUWkkvO6DHA9sNa5\nroiICHALsMrZZDxwr9Ob3RTYr6qZwDSgpYicJiKnAS2dsrCsF9sY457Y9GLXAIaJSBLBSt1oVZ0o\nIrNFpCrBpvNy4GFn/clAG2AjcBi4H0BV94nIi8AiZ70XVHVfYYEtQRpj3BODsdiqugJoUEB5izDr\nK9AtzHtDgCHFjX1SNbGPHPmFjg88zu87/4n2dz3Ev955DwBV5c3/DuWmjg/Q7s6uvD9mHACzv/ia\n3937R27t3I0Of3iMpd+syt/XuMkzaHNHF9rc0YVxk2e4doynnHIKX385kSWLZ/DN8tn0ef5J1/Zd\nXK1aNmf1qrmsTZ9Hzx4F/n8SM2lpNZk5fQwrV8zhm+WzefSRLp7GHzTwVXZmfMPyZbM8jZvH7++/\nYsVTGTVyIKtWfs7KFXNo2uQyT+PHqhfbLydVDbJUqRSGDOhP2bJlyMrO5t4/PsVVTRuy+dvt7Nrz\nHRM+HEggEOD7H34EoOll9bn2yqaICOs2buGp3i8zYcQg9v90gLfe/ZBRgwcAcEeXx2h+ZVMqnlrh\nhI/xyJEjXN+yA4cOHSY5OZm5c8YydepnLFi49IT3XRyBQIABb/ajdZtOZGRkMv/ryUyYOJ01azZ4\nEj87O5sePfuybPkqypcvx8IFU5k5a65n8YcPH81//vMu7777pifxjuf39//6ay8wbdpn3NGxKykp\nKZQtW8aTuIkqZjVIETlXRJ52hvy86bw+7wT3mf+FZ2dnk52djYgwauwk/nj/nQQCwdM5/bRKAJQt\nW4bg9Vv4388/g/P6ywVLuLxRAyqeWoGKp1bg8kYN+HLBkhM5tGMcOnQYgJSUZJJTUgjW+L3RuFED\nNm3aypYt28jKymL06HHc3K6VZ/F37drDsuXBmvrBg4dYu3YDqTWrexb/i3kL2Of8A+kXv77/ChXK\nc9WVTRjy7ggAsrKy2L//J09i54tBJ42fYpIgReRpYCTBi6cLCV4UFWCEiPQ6kX3n5ORwa+duXN22\nE5c3asDFF5zL9h2ZTJn1OR3+8BgPP9mbb7fvyF9/5udf0q7Tg/zpqed58dk/A7B773dUP6Nq/jrV\nqlZh997vTuSwjhEIBFi8aDqZO1Ywa9ZcFi5a5tq+i1IztTrbM47euZCxI5OaHiaoUGeemUb9Sy5k\nwULvzj8e+PX9n3XWmXz33fcMfud1Fi2cxn/f/of3NcgEa2LHqgbZBWikqv1V9X1n6U/wPqUTuiiV\nlJTEx8P+zayx77EyfT0bNm/ll6wsTilVitFDBnBru9b0fvn1/PWvv6YZE0YMYkD/5/nXoOEAFPQP\nel5N0w25ubk0bNSSM+s0pFHDBlxwwW9d23dRCjoPL2uwecqVK8voUYN44qk+HDhwsOgNEohf339y\nUhINGlzEf/87nEaNW3Ho0GGe7vmIJ7HzWQ2yWHKBguaLr+G8F1boXfXvDB8Rdr1TK5Sn0aUXM2/+\nYqpXrcINza8E4PprrmD9pi2/Wr9h/YvYviOTH37cT/UzqrBrz97893bv/Y4zqpxerBOLxP79P/H5\n3K9o1bK56/sOZ0dGJrXSjn70aak1yMzc7Vl8gOTkZMaMGsSIEWP59NMpRW+QoLz+/jN2ZJKRkZlf\nY/3kk0k0qH+RJ7HzqOZEvMSzWCXI7sAsEZkiIgOdZSowC3i8sA1D76p/4N5Ox7y374cf+cmpjfx8\n5AjzFy2jzpm1aHH15SxYshyARctWcmat4PDKbRk782tP6es2kpWVTaWKp9KsyWV8tXAp+386wP6f\nDvDVwqU0c6m3r0qVylSseCoApUuX5roWV7FunXfP1lm0eDl169ahdu1apKSk0KFDeyZMnO5ZfAj2\nJK9Zu5E33gw7ICJh+fn97969l4yMnZxzztkAtGhxJWvWrPckdr4Ea2LHpBdbVaeKyDkEm9SpBK8/\nZgCL9AT+ydj7/Q/85aVXyMnNRXOVVi2uonmzJlx68QU83ffvvDfqU8qWKU3fXt0BmDFnHuOnzCI5\nOZnSp5TilRd6ISJUPLUCD93XiY4PBHP1w/ff6UoPNkCNGtUYMvgNkpICBAIBPvpoApMmz3Rl38WR\nk5PD492fY/KkD0kKBBg6bBTp6d79kTS7ohH33H0bK1ams3hRMDH37t2fKVNnexL//ff+zTVXX06V\nKpXZunkxfV94hXeHjvQkNvj//T/+594MH/ZPSpVKYcuWbXR54AnPYgNx32SOlPhxfaq4onmEpFvs\nqYb2VMMSHT/Kx77+vOTTiP9mS192iz321RhTAthTDY0xJow4v6YYKUuQxhj3JNg1SEuQxhj3WA3S\nGGPCsBqkMcaEYQnSGGMKFu8jYyJlCdIY4x6rQRpjTBgJ1klzUs0obowxXrIapDHGPdbENsaYMBKs\niW0J0hjjHqtBGmNMGFaD9E5K1bN9jZ837ZTFt/glMX5UrAZpjDFhWIL0jt8Txp5XtZFv8dfsXeT7\n+ZfoCWMtfnSsiW2MMWFYDdIYY8KwGqQxxoSRYDVIG2pojHFPDB77KiKlRWShiHwjIqtFpK9TXkdE\nFojIBhEZJSKlnPJTnN83Ou/XDtnXM075OhFpVVRsS5DGGPfk5ka+FO0I0EJVLwHqA61FpCnwN+B1\nVa0H/AB0cdbvAvygqnWB1531EJHzgY7ABUBr4D8iklRYYEuQxhj3xCBBatBB59cUZ1GgBfCRUz4M\nuMV53d75Hef960REnPKRqnpEVbcAG4HGhcW2BGmMcY9qxIuIdBWRxSFL1+N3KyJJIrIc2APMADYB\nP6pqtrNKBpDqvE4FtgcPR7OB/cDpoeUFbFMg66Qxxrgnik4aVR0IDCxinRygvohUAsYC5xW0mvNT\nwrwXrjwsS5DGGPfEuBdbVX8UkTlAU6CSiCQ7tcQ0IO8O9wygFpAhIslARWBfSHme0G0KZE1sY4x7\nYtOLXdWpOSIiZYDrgTXAZ8BtzmqdgXHO6/HO7zjvz1ZVdco7Or3cdYB6wMLCYlsN0hgT72oAw5we\n5wAwWlUnikg6MFJEXgKWAYOd9QcD74nIRoI1x44AqrpaREYD6UA20E2LeMqYJUhjjHti0MRW1RVA\ngwLKN1NAL7Sq/gzcHmZf/YB+xY1tCdIY4x4ttM/jpJOQ1yBbtWzO6lVzWZs+j549usUkRqlTSjFq\n6ruM/ewDJswdySM9HwSgyZUN+XjmcMZ/PoL/+2cfkpKC96G2aH01n875gE9mv8+Y6cO4tMklMTku\n8Ob8LX78xY6H+DG6Udw3onGc8ZNLpUZ8cIFAgDWrv6B1m05kZGQy/+vJ3H3Pn1izZkNE+ynOdGdl\ny5Xh8KH/kZycxPsTBtH/+dd5beDL/OHWbmzdvI1Hn+7Kzu27+PjD8fnrApxzfl1eH/QyNzXrEHbf\n0U535ub5+xX/RKb78jO+m5+97/FVC7olpkj/G/xUxH+zZbq8ElUsLyRcDbJxowZs2rSVLVu2kZWV\nxejR47i5XZFDLqOSl/CSU5JJSUkmNyeXX375ha2btwHw1ZyFtGx77THrApQtW4ZY/cPk5flb/PiJ\nHQ/xgZj0YvvJlwQpIvfHat81U6uzPePorU0ZOzKpWbN6TGIFAgE+mf0+89Kn8dXnC1mxdDUpyclc\ncEnwHtaW7VpQvWa1/PWvb9OcSV+O5q0PXuO57i/F5Ji8PH+LHz+x4yE+gOZqxEs886sG2TfcG6HD\njnJzD0W84+CQy2PFqraWm5vL71vczbWXtOWiBudT79yzePKh5+j14p8ZNfVdDh88THbO0bsIZk6e\nw03NOvBo55481uuhmByTl+dv8eMndjzEBxLuGmTMerFFZEW4t4BqYd47ZthRNNcgd2RkUivt6LWb\ntNQaZGbujnQ3ETnw00EWfrWUK1tczrv/+YB7bg4OJb2ieRPOPPs3v1p/8fxl1DozjUqVK/Ljvv2u\nHosf52/x/Y8dD/GBuG8yRyqWNchqwL1AuwKW72MVdNHi5dStW4fatWuRkpJChw7tmTBxuutxTju9\nEhVOLQ/AKaVP4fKrG7Nlw7dUrnIaACmlUnjgkXsZNewTAH5TJy1/2/Mv+i0ppZJdT47g3flb/PiK\nHQ/xAcjVyJc4Fsv7ICcC5VV1+fFvOGMpYyInJ4fHuz/H5EkfkhQIMHTYKNLT17sep2q1Ks5tPAEC\nEmDq+JnMmTGPp/o8SvMbriQQCDBy6McsmLcYgJZtW9D+9jZkZWdz5OcjPNH1L64fE3h3/hY/vmLH\nQ3wg7pvMkUq423zcYk81tKcaluj4Ud7mc/jNhyP+my37+Ntxe5uPjaQxxrgnjitc0bAEaYxxT4I1\nsS1BGmPcE+edLpGyBGmMcU+C3eZjCdIY454Eq0Em3FhsY4xxi9UgjTGuUeukMcaYMBKsiW0J0hjj\nHuukMcaYMKwGaYwxYdg1SGOMCcNqkMYYE4ZdgzTGmDCsBumdvGmf/LJm7yJf4/t9/ha/ZMePht0H\n6aGSOh9iXvwUH+Nn2XyQJT5+VKwGaYwxYViCNMaYMBKsk8YmqzDGuCcGD+0SkVoi8pmIrBGR1SLy\nuFP+VxHZISLLnaVNyDbPiMhGEVknIq1Cyls7ZRtFpFdRsa0GaYxxjcamiZ0NPKmqS0WkArBERGY4\n772uqq+Eriwi5wMdgQuAmsBMETnHefvfwA1ABrBIRMaranq4wJYgjTHuiUGCVNVMINN5fUBE1gCp\nhWzSHhipqkeALSKyEWjsvLdRVTcDiMhIZ92wCdKa2MYY9+TmRr5EQERqAw2ABU7RIyKyQkSGiMhp\nTlkqsD1kswynLFx5WJYgjTG+EpGuIrI4ZOkaZr3ywMdAd1X9CXgLOBuoT7CG+WreqgVsroWUh2VN\nbGOMe6JoYqvqQGBgYeuISArB5PiBqn7ibLc75P1BwETn1wygVsjmaUDezZ3hygtkNUhjjHti04st\nwGBgjaq+FlJeI2S13wGrnNfjgY4icoqI1AHqAQuBRUA9EakjIqUIduSMLyy21SCNMa5RjUkvdjPg\nHmCliCx3yp4FOolIfYLN5K3AQ84xrBaR0QQ7X7KBbqqaAyAijwDTgCRgiKquLiywJUhjjHti04s9\nj4KvH04uZJt+QL8CyicXtt3xLEEaY9xjQw2NMaZgMbpR3DeWII0x7kmwBJmQvditWjZn9aq5rE2f\nR88e3RI+/qCBr7Ij4xuWLZuVX3bxxefzxdzxLFs6k7Fjh1KhQvmYH0eekvb5x0vseIhPbhRLHEu4\nBBkIBBjwZj/atrubiy65ljvuuIXzzquX0PGHDR9N27Z3HVP237f/wbN/eZkGl17PuE+n8OSTf4zp\nMeQpiZ9/PMSOh/gQbGJHusSzmCVIETlXRK5z7n4PLW8dq5gAjRs1YNOmrWzZso2srCxGjx7Hze1a\nFb3hSRx/3rwF7Pvhx2PKzjnnbL74Yj4AM2d9we9+16agTV1XEj//eIgdD/GBmNwH6aeYJEgReQwY\nBzwKrBKR9iFvvxyLmHlqplZne8bRm+MzdmRSs2b1WIaMq/h5Vq9eR7t2LQG47da21ErzZnZqv8/f\nz/gl+dzzWRO7WB4ELlPVW4DmQO+8Odwo+H6mfKHjMnNzD0UcOHjT/bFidPNqXMbP82DXJ/jjw/ex\nYP4Uylcoxy+/ZHkS1+/z9zN+ST73/HgJ1sSOVS92kqoeBFDVrSLSHPhIRM6kiAQZOi4zuVRqxJ/e\njozMY2pLaak1yMzcXcgW7vI7fp516zbR5qY7AahX7yza3HidJ3H9Pn8/45fkc88X5zXCSMWqBrnL\nGQIEgJMs2wJVgItiFBOARYuXU7duHWrXrkVKSgodOrRnwsTpsQwZV/HzVK16OhCsVTz7zOMMHPie\nJ3H9Pn8/45fkc89jNcjiuZfgGMh8qpoN3Csi/41RTABycnJ4vPtzTJ70IUmBAEOHjSI9fX0sQ/oe\n/733/s01V19OlSqV2bJ5MS+88Arly5fj4T/eB8Cnn05m6LBRMT2GPCXx84+H2PEQH0i4GqT4cX2s\nuKJpYrvFHvtqj30t0fFVC70UFs737a6J+G/29AmfRxXLCwl3H6QxxrjFhhoaY9yTYE1sS5DGGNck\n2GOxLUEaY1xkCdIYYwpmNUhjjAnDEqQxxoRhCdIYY8KJ7vbJuGUJ0hjjGqtBGmNMGJprNUhjjCmQ\n1SCNMSaMKIdwxy1LkMYY11gN0hhjwrBrkB7Km/appMbPKuHnb/H9jR+NOJ49MSpxnSD9no+xpMdv\nnurNYxqON2dH8PneJXo+xjiIHw2rQRpjTBiJliBtwlxjTFwTkVoi8pmIrBGR1XlPSBWRyiIyQ0Q2\nOD9Pc8pFRAaIyEYRWSEil4bsq7Oz/gYR6VxUbEuQxhjXqEa+FEM28KSqngc0BbqJyPlAL2CWqtYD\nZjm/A9wI1HOWrsBbEEyoQB+gCdAY6JOXVMOxBGmMcY3mSsRLkftUzVTVpc7rA8AaIBVoDwxzVhsG\n3OK8bg8M16D5QCURqQG0Amao6j5V/QGYAbQuLLYlSGOMa1Ql4kVEuorI4pCla7j9i0htoAGwAKim\nqpnBuJoJnOGslgpsD9kswykLVx5WkZ00IlINeBmoqao3OlXby1V1cFHbGmNKlmhuFFfVgcDAotYT\nkfLAx0B3Vf1JJGzts6A3tJDysIpTgxwKTAPy7jlYD3QvxnbGmBImVyXipThEJIVgcvxAVT9xinc7\nTWecn3uc8gygVsjmacDOQsrDKk6CrKKqo3GeNqGq2UBOMbYzxpQw0TSxiyLBquJgYI2qvhby1ngg\nrye6MzAupPxepze7KbDfaYJPA1qKyGlO50xLpyys4twHeUhETsepiuYFLMZ2xpgSJkb3QTYD7gFW\nishyp+xZoD8wWkS6ANuA2533JgNtgI3AYeB+AFXdJyIvAouc9V5Q1X2FBS5OgnyCYEY+W0S+BKoC\ntxXzxIwxJUgshhqq6jwKvn4I8KvhXqqqQLcw+xoCDClu7CITpKouFZFrgN86B7lOVbOKG8AYU3Ik\n2kia4vRi33tc0aUigqoOj9ExGWNOUsXtdDlZFKeJ3SjkdWmCVdqlgCVIY8wxStyEuar6aOjvIlIR\neC9mR2SMOWkl2nRn0YykOUxwjGNcSkuryczpY1i5Yg7fLJ/No4908fwYWrVszupVc1mbPo+ePQq8\nVnzSxy9/ajn6/vd5hs8ZwrDPBnP+pedRoVIFXvnwb7z/xVBe+fBvlK9YHoA7Hu7AO9Pe5p1pb/Pu\nzEHM+nYaFSpViMlxgb+ff0n47gsTq/sg/SJaRMoXkQkcvds8AJwPjFbVXuG3ckdyqdSI/z2qXv0M\nalQ/g2XLV1G+fDkWLpjKrbf9gTVrNkS0n2jnYwwEAqxZ/QWt23QiIyOT+V9P5u57/nRSxi9sPshe\nr/dk5cKVTBoxheSUZEqXOYW7Hr2TAz8e4MN/j+TObh0pX7E8A19+55jtLr++Kbc/eCtP3NEj7L5P\nZD5IN84/2vkY3fzsfY8fZVt52W/aR/w322DbuLjNksWpQb4CvOos/wdcXZzkKCKNRaSR8/p8EXlC\nRNqc0NEWw65de1i2fBUABw8eYu3aDaTWrB7rsPkaN2rApk1b2bJlG1lZWYwePY6b27VKqPhly5fl\nkiYXMWnEFACys7I5+NMhmrW8gqljpgMwdcx0rmzV7FfbXndLC2aN+8zV4wnl5+dfEr77osRoNh/f\nFJogRSQJ6K2qnzvLl6qaUdRORaQPMAB4S0T+D/gXUB7oJSJ/cePAi+PMM9Oof8mFLFi4zKuQ1Eyt\nzvaMo6OXMnZkUtPDBO1F/Jq/qcGP+/bT67UeDJr6Nj3+8QSly5SmcpXT2LcneN/tvj37OO30Ssds\nd0rpU2jcvCFzJ3/h6vEcc2w+fv4l4bsvSqI1sQtNkKqaAxx2OmYicRvBu9+vJnjD5i2q+gLB6Ybu\nKGzD0Jk9cnMPRRj2qHLlyjJ61CCeeKoPBw4cjHo/kSpoAH1RlzFOtvhJyUmcc2E9xr03gQdbP8z/\nDv/Mnd06FrndFTdczqpFqznw4wFXjyeUn59/SfjuixKLoYZ+Ks5tPj8THOIzA8jPWKr6WCHbZIck\n102q+pOzzf9EpND5PkJn9ojmGiRAcnIyY0YNYsSIsXz66ZRodhG1HRmZ1Eo7eu0oLbUGmZm7Eyr+\n3sy97M3cy5plawH4fNJc7uzWiX3f/UDlMyqzb88+Kp9RmR++//GY7Vq0bx7T5jX4+/mXhO++pCnO\nNchJQG9gLrDEWRYXsc0vIlLWeX1ZXqFTE435k3MHDXyVNWs38sabRc6g5LpFi5dTt24dateuRUpK\nCh06tGfCxOkJFX/f3h/Ys3Mvtc5KA+CyKy/l2w3f8tWMr2l9e0sAWt/eki+nf5W/TbkK5bik6cV8\nOe2rAvfpFj8//5Lw3Rcl0ZrYxalBVlLVN0ML8p4JUYirVfUIgOoxM8SlcHT2jZhodkUj7rn7Nlas\nTGfxouD/HL1792fK1NmxDJsvJyeHx7s/x+RJH5IUCDB02CjS09d7EtvL+AN6/4vn/vkMyaVSyPw2\nk/5P/oOABOjz9nO06dia3Tv28NeHX8xf/6rWzVj8+RJ+/t/Prh9LKD8//5Ly3RcmzvtcIlac23yW\nquqlx5UtU9UGMT0yom9iuyEeHrvqd3x77GsJjh/lxcGvatwa8d/sFZkfx201MmwNUkQ6AXcCdURk\nfMhbFYDvY31gxpiTT7x3ukSqsCb2V0AmUIXgPZB5DgArYnlQxpiTU8w7GDwWNkGq6rfAt8Dlhe1A\nRL5W1ULXMcaUDBp22saTU3E6aYpS2oV9GGMSQG6C9dK4kSAT7CMxxkQr12qQxhhTsERrYhd5o7iI\nPOI8ASzsKi4ejzHmJJYbxRLPijOSpjqwSERGi0hr+fWAz3ticFzGmJOQIhEv8azIBKmqzxGcIHcw\ncB+wQUReFpGznfdXxfQIjTEnjZJYg8x7jOIuZ8kGTgM+EpG/x/DYjDEnmURLkMV5quFjBMdPfwe8\nA/RQ1SwRCQAbgJ6xPURjzMki3pvMkSpOL3YV4PfOjeP5VDVXRNrG5rCMMSejBHssdrGeavh8Ie+t\ncfdwjDEns0S7DzKapxoaY0yJENc3iudN+2Tx/ZE37Zhf/D7/kh4/Gok2rC6uE6Tf8yFa/BI8H6LF\nj0oseqVFZAjQFtijqhc6ZX8FHgT2Oqs9q6qTnfeeAboAOcBjqjrNKW8NvAkkAe+oav+iYsd1gjTG\nnFxyC3hwmAuGEnwy6vDjyl9X1VdCC0TkfKAjcAFQE5gpIuc4b/8buAHIIDj4ZbyqphcW2BKkMcY1\nsWhiq+pcEaldzNXbAyOdR75sEZGNQGPnvY2quhlAREY66xaaIK2TxhjjGo9vFH9ERFaIyJCQ+SJS\nge0h62Q4ZeHKC2UJ0hjjmlyJfBGRriKyOGTpWoxQbwFnA/UJPvkg76kHBbXxtZDyQlkT2xjjmmju\ng1TVgUBEz2hW1fwHfovIIGCi82sGUCtk1TQgr9cpXHlYVoM0xrhGo1iiISI1Qn79HZA3ac54oKOI\nnCIidQhOtLMQWATUE5E6IlKKYEdO6MMIC2Q1SGOMa2Ix1FBERgDNgSoikgH0AZqLSH2COXYr8BCA\nqq4WkdEEO1+ygW6qmuPs5xFgGsHbfIao6uqiYluCNMa4Jhb3QapqpwKKBxeyfj+gXwHlk4HJkcS2\nBGmMcY2NpDHGmDBK3Gw+xhhTXPE+AW6kLEEaY1xjCdIYY8LQBGtiJ9x9kGlpNZk5fQwrV8zhm+Wz\nefSRLp4fQ6uWzVm9ai5r0+fRs0c3i++xjevns2zpTBYvms78ryPqtDxhfp+73/FL3DNpTjbZ2dn0\n6NmXZctXUb58ORYumMrMWXNZs2aDJ/EDgQAD3uxH6zadyMjIZP7Xk5kwcbrF9yh+nutvuJ3vv//B\n05h+n7vf8RNRwtUgd+3aw7LlwZvqDx48xNq1G0itWd2z+I0bNWDTpq1s2bKNrKwsRo8ex83tWln8\nEsDvc/c7PiReDdKzBCkix8/lFnNnnplG/UsuZMHCZZ7FrJlane0ZR4d4ZuzIpKaHCbqkxwdQVaZM\nHsGC+VN4oMtdnsX1+9z9jg/eDTX0Skya2CJy/BhHAa4VkUoAqnpzLOKGKleuLKNHDeKJp/pw4MDB\nWIfLJwVMGBp8rLjF98rVzW8hM3M3VaueztQpI1m3biNfzFsQ87h+n7vf8cHugyyuNIJjId/h6FRD\nDTk6JVFYzlRHXQEkqSKBQLnHkOTgAAASy0lEQVSIgycnJzNm1CBGjBjLp59OiXj7E7EjI5NaaUen\nyk9LrUFm5u5CtrD4bsuLt3fv94wbN4VGjep7kiD9Pne/40P8N5kjFasmdkNgCfAXYL+qzgH+p6qf\nq+rnhW2oqgNVtaGqNowmOQIMGvgqa9Zu5I03I5pByRWLFi+nbt061K5di5SUFDp0aM+EidMtvkfK\nli1D+fLl8l/fcP01rF69zpPYfp+73/Eh8a5BxqQGqaq5wOsiMsb5uTtWsY7X7IpG3HP3baxYmc7i\nRcH/OXr37s+UqbO9CE9OTg6Pd3+OyZM+JCkQYOiwUaSnr/cktsWHatWq8tGY4DwGyclJjBz5KdOm\nz/Ektt/n7nd8iP9ripESL65RiMhNQDNVfTaS7ZJLpfr2eZfkpwr6HT9enupXouNrdLd8//3MuyP+\nm+357ftxe+XSk1qdqk4CJnkRyxjjn3hvMkcq4W4UN8b4J9Ga2JYgjTGuyU2wFGkJ0hjjGmtiG2NM\nGIlVf7QEaYxxkdUgjTEmDBtqaIwxYVgnjTHGhJFY6TEB54M0xhi3WA3SGOMa66Qxxpgw7BqkMcaE\nkVjp0RKkMcZF1sT2UN60Txbf4lv8k0OiNbGtF9sY45pYPLRLRIaIyB4RWRVSVllEZojIBufnaU65\niMgAEdkoIitE5NKQbTo7628Qkc7FOZ+4rkGW1AljS3r8uJgwFkitdL4v8Xf8mA74f/7RiFETeyjw\nLyD0yai9gFmq2l9Eejm/Pw3cCNRzlibAW0ATEakM9CH4OBgFlojIeFUt9OHpVoM0xrhGo/ivyH2q\nzgX2HVfcHhjmvB4G3BJSPlyD5gOVRKQG0AqYoar7nKQ4A2hdVGxLkMYY10Tz0C4R6Soii0OWrsUI\nVU1VMwGcn2c45anA9pD1MpyycOWFiusmtjHm5BJNJ42qDgTcegRpQdNlaCHlhbIapDHGNbHopAlj\nt9N0xvm5xynPAGqFrJcG7CykvFCWII0xrslFI16iNB7I64nuDIwLKb/X6c1uCux3muDTgJYicprT\n493SKSuUNbGNMa6JRS+2iIwAmgNVRCSDYG90f2C0iHQBtgG3O6tPBtoAG4HDwP0AqrpPRF4EFjnr\nvaCqx3f8/IolSGOMa4rTKx3xPlU7hXnrugLWVaBbmP0MAYZEEtsSpDHGNTbU0BhjwohFDdJP1klj\njDFhWA3SGOMaa2IbY0wYuZpYTWxLkMYY1yRWekzAa5CDBr7KzoxvWL5slm/H0Kplc1avmsva9Hn0\n7FHgHQcW/ySOXzO1OmPGv8uc+eOZ/dU4ujx0NwBPPP0nFq+ezfS5HzN97se0uOGq/G0e+fMDzFsy\nhbkLJ3JNi2auH1Mevz97D28U90TCJcjhw0dzU9u7fIsfCAQY8GY/2ra7m4suuZY77riF886rZ/ET\nKH52djZ9n/s7zZveTLuWnbjvgU7U++3ZAAx6azgtr76VllffyuwZXwBQ77dn0/73bWhx+c3cddtD\nvPzKcwQC7v/p+f3ZQ2xm8/FTwiXIL+YtYN8PP/oWv3GjBmzatJUtW7aRlZXF6NHjuLldK4ufQPH3\n7P6OVSvWAHDo4GE2rN9M9RpnhF2/VZtrGffJZH75JYvt23awdfN2Glx2kavHBP5/9hDdbD7xzJME\nKSJXisgTItLSi3h+qplane0ZR8fAZ+zIpGbN6hY/QeOn1arJhRefx7IlKwC4/8E7mTHvE17954tU\nrHgqANVrVGPnjl3522Tu3EX1GtVcPxa/P3uwJnaxiMjCkNcPEpwNuALQx5n9N2GJ/HpWJfWwZ8/i\nexe/bLmyDBr+Bn2e6c/BA4cYPmQUVzRoTcurbmXP7r08/1IPT4/J788erIldXCkhr7sCN6hqX4Iz\naBR6gTB08szc3EMxOrzY2ZGRSa20o1Plp6XWIDNzt8VPsPjJyckMGvYGY8dMYsrEmQB8t/d7cnNz\nUVU+GPYR9Z1mdObOXdRMPVqTq1GzOrt37SlwvyfC788erIld7P060wqdDoiq7gVQ1UNAdmEbqupA\nVW2oqg0DgXIxOrzYWbR4OXXr1qF27VqkpKTQoUN7JkycbvETLP6r/3yBjes3M/A/w/LLzqhWJf/1\njW2vZ92aDQBMn/IZ7X/fhlKlUqj1m1TqnP0bli1Z6fox+f3ZQ7DGGukSz2J1H2RFYAnBWXxVRKqr\n6i4RKU/BM/u65v33/s01V19OlSqV2bp5MX1feIV3h46MZchj5OTk8Hj355g86UOSAgGGDhtFevp6\ni59A8Rs1vZTbOrYnffU6ps/9GID+L77BLbe24fyLzkVVydi2k6f//FcA1q/dxIRPp/LZ/PHkZOfw\nlx4vkZvrft3J788eEu+xr+Lx9aGyBJ8lsaU46yeXSvXt0y7JTxX0O7491TAOnmqoGlVFpt1v2kb8\nNzth28SYVppOhKcjaVT1MFCs5GiMOfnEe6dLpGyooTHGNYnWxLYEaYxxTbx3ukTKEqQxxjXxfttO\npCxBGmNck2jXIBNuLLYxxrjFapDGGNdYJ40xxoRhnTTGGBOG1SCNMSaMROuksQRpjHGNPbTLGGPC\nSKz0aAnSGOOiRLsGafdBGmNcE6tHLojIVhFZKSLLRWSxU1ZZRGaIyAbn52lOuYjIABHZKCIrROTS\naM8nrmuQedNOWXyL74e8acf84vf5RyPGt/lcq6rfhfzeC5ilqv2dR7n0Ap4GbgTqOUsT4C3nZ8Ss\nBmmMcY3HD+1qD+RN6T4MuCWkfLgGzQcqiUiNaALEdQ2ypE4YW9Ljx8uEuX7Hv7haU1/ir9g9P+pt\no7nNR0S6Enx2VZ6BqjrwV7uG6SKiwH+d96upaiaAqmaKSN6zd1OB7SHbZjhlmZEeW1wnSGPMySWa\nJraT7I5PiMdrpqo7nSQ4Q0TWFrJuQTOUR1VVtQRpjHFNrHqxVXWn83OPiIwFGgO7RaSGU3usAeQ9\nKjIDqBWyeRoQ1QVduwZpjHFNLJ5qKCLlRKRC3muCj49eBYwHOjurdQbGOa/HA/c6vdlNgf15TfFI\nWQ3SGOOaGNUgqwFjRQSCOetDVZ0qIouA0SLSBdgG3O6sPxloA2wEDgP3RxvYEqQxxjWxGIutqpuB\nSwoo/x64roByBbq5Edua2MYYE4bVII0xrrHJKowxJgyb7swYY8KwGqQxxoRhNUhjjAnDapDGGBOG\n1SCNMSaMRKtBJux9kIFAgEULpzFu7LCiV3ZZq5bNWb1qLmvT59Gzhyv3q1r8kyS+l7EDgQCjZgzj\nn++9AsBfX3uWMbOG89Hs93j1nX6UKVsGgHse6sjYuR/y0ez3GDTmn9RIqx6zY9Io/otnCZsgH3v0\nAdau3eB53EAgwIA3+9G23d1cdMm13HHHLZx3Xj2LXwLiex37rgc7sGXD1vzf//H8G9x+3b3c1uIe\nMjN20+kPtwGwdtV6OrW6n9ta3MOMibP5c+/YJW7V3IiXeBaTBCkiTUTkVOd1GRHpKyITRORvIlIx\nFjFDpabWoM2N1zFkyIhYh/qVxo0asGnTVrZs2UZWVhajR4/j5natLH4JiO9l7Go1qnL19c345IPx\n+WWHDh7Of126zCn5tbNFXy7l5/8dAWDFktVUq3EGseLxhLkxF6sa5BCCg8QB3gQqAn9zyt6NUcx8\nr73al17PvERurvf/OtVMrc72jKMzK2XsyKRmzdg1aSx+/MT3MnbPF7vz2ov/Ive4GtgLb/yFz1ZO\nonbdMxkxeMyvtvvdne2YN/vrmBwTxGY2Hz/FKkEGVDXbed1QVbur6jxV7QucVdiGItJVRBaLyOLc\n3EMRB76pzfXs2fMdS5etjOKwT5wz48gxvPyfwOL7F9+r2Fff0Ix93/3AmhXrfvXe8937cd0l7diy\nYSut2l9/zHs33dqKCy45l6H/+cD1Y8pjNcjiWSUieVMMfSMiDQFE5Bwgq7ANVXWgqjZU1YaBQLmI\nA19xRUPatW3JxvXz+eD9/3Dttc0YNnRAxPuJ1o6MTGqlHZ2qPy21BpmZuy1+CYjvVez6jS6mecur\nmLLoE/7+9os0bnYZL/+rT/77ubm5TB03i+tvuja/rMlVjXjw8ft4rHNPsn4p9E/whFgNsngeAK4R\nkU3A+cDXIrIZGOS8FzN/ea4/tc9qSN1zmnLX3X/is8++pPN9j8Uy5DEWLV5O3bp1qF27FikpKXTo\n0J4JE6db/BIQ36vYA15+ixsubc+NjX5Pz4d7s/DLJTz7SF9q1U7LX6d5yyvZuvFbAM698Bye/0dP\nHuvcg33f/eD68YTKVY14iWcxuQ9SVfcD9zmzAJ/lxMlQVe+qEj7Jycnh8e7PMXnShyQFAgwdNor0\n9PUWvwTE9zO2iPDSgN6Ur1AOEVi3eiMvPf13AJ54/hHKlivLK4P6AbBrx24e69wzJscR77ftREri\nuYqbXCrVt4MryU8V9Dt+vDxV0O/4vj7VULWgB18VqVrFcyP+m929f21UsbyQsPdBGmPMibKhhsYY\n18R7r3SkLEEaY1wTz5fsomEJ0hjjmnjvlY6UJUhjjGusBmmMMWHYNUhjjAnDapDGGBOGXYM0xpgw\nEm0kjSVIY4xrrAZpjDFhJNo1SBtqaIxxTayeSSMirUVknYhsFJFeMT6NfFaDNMa4JhY1SBFJAv4N\n3ABkAItEZLyqprse7DhWgzTGuCZGE+Y2Bjaq6mZV/QUYCbSP6Yk44roGmTftk8W3+CUx/ord832N\nH40YXYFMBbaH/J4BNIlNqGPFdYKMdk66PCLSVVUHunU4J0tsi2/x/Yqf/cuOiP9mRaQr0DWkaOBx\nx17QPj3pDUr0JnbXoldJyNgW3+L7Hb/YQp9D5SzHJ/YMoFbI72mAJ9X7RE+QxpiT3yKgnojUEZFS\nQEdgfBHbuCK+m9jGmBJPVbNF5BFgGpAEDFHV1V7ETvQE6ds1IJ9jW3yL73d8V6nqZGCy13Hj+qFd\nxhjjJ7sGaYwxYViCNMaYMBIyQfo1btOJPURE9ojIKi/jhsSvJSKficgaEVktIo97HL+0iCwUkW+c\n+H29jO8cQ5KILBORiV7HduJvFZGVIrJcRBZ7HLuSiHwkImud/wcu9zJ+okm4a5DOuM31hIzbBDp5\nMW7TiX81cBAYrqoXehHzuPg1gBqqulREKgBLgFs8PH8ByqnqQRFJAeYBj6uqZ8NCROQJoCFwqqq2\n9SpuSPytQENV/c6H2MOAL1T1HeeWmLKq+qPXx5EoErEG6du4TQBVnQvs8ypeAfEzVXWp8/oAsIbg\nUC2v4quqHnR+TXEWz/4VFpE04CbgHa9ixgsRORW4GhgMoKq/WHI8MYmYIAsat+lZgognIlIbaAAs\n8DhukogsB/YAM1TVy/hvAD2BXA9jHk+B6SKyxBlG55WzgL3Au84lhndEpJyH8RNOIiZI38ZtxhMR\nKQ98DHRX1Z+8jK2qOapan+CQsMYi4smlBhFpC+xR1SVexCtEM1W9FLgR6OZcdvFCMnAp8JaqNgAO\nAZ5eg080iZggfRu3GS+ca38fAx+o6id+HYfTvJsDtPYoZDPgZuca4EighYi871HsfKq60/m5BxhL\n8LKPFzKAjJAa+0cEE6aJUiImSN/GbcYDp5NkMLBGVV/zIX5VEankvC4DXA+s9SK2qj6jqmmqWpvg\n9z5bVe/2InYeESnndI7hNG9bAp7c0aCqu4DtIvJbp+g6wJPOuUSVcEMN/Ry3CSAiI4DmQBURyQD6\nqOpgr+ITrEXdA6x0rgMCPOsM1fJCDWCYczdBABitqr7cbuOTasDY4L9TJAMfqupUD+M/CnzgVA42\nA/d7GDvhJNxtPsYY45ZEbGIbY4wrLEEaY0wYliCNMSYMS5DGGBOGJUhjjAnDEqSJKyJS26+ZkIw5\nniVI4wnnvkhjTiqWIE2BROTF0LkkRaSfiDxWwHrNRWSuiIwVkXQReVtEAs57B0XkBRFZAFwuIpeJ\nyOfOJA7TnKnZcMq/EZGvgW5enaMxRbEEacIZDHQGcBJeR+CDMOs2Bp4ELgLOBn7vlJcDVqlqE4Iz\nCv0TuE1VLwOGAP2c9d4FHlNVm9zVxJWEG2po3KGqW0XkexFpQHD43DJV/T7M6gtVdTPkD7W8kuBE\nCTkEJ80A+C1wITDDGYaXBGSKSEWgkqp+7qz3HsFZcIzxnSVIU5h3gPuA6gRrfOEcP1417/efVTXH\neS3A6uNric7EFjbe1cQla2KbwowlOFVZI4KTf4TT2Jk9KQDcQfAxC8dbB1TNe0aKiKSIyAXOlGj7\nReRKZ7273Dt8Y06M1SBNWKr6i4h8BvwYUhMsyNdAf4LXIOcSTKwF7es2YIDTrE4mOPv3aoIzzgwR\nkcMUnoiN8ZTN5mPCcmqES4HbVXVDmHWaA0/58XAsY2LNmtimQCJyPrARmBUuORqT6KwGaYpFRC4i\n2MMc6ohzC48xCckSpDHGhGFNbGOMCcMSpDHGhGEJ0hhjwrAEaYwxYViCNMaYMP4fWdOU5kaLIXEA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "stk = xgb.XGBClassifier().fit(x_train, y_train)\n",
    "y_predict=stk.predict(x_test)\n",
    "y_true=y_test\n",
    "stk_score=accuracy_score(y_true,y_predict)\n",
    "print('Accuracy of Stacking: '+ str(stk_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of Stacking: '+(str(precision)))\n",
    "print('Recall of Stacking: '+(str(recall)))\n",
    "print('F1-score of Stacking: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter optimization (HPO) of the stacking ensemble model (XGBoost) using Bayesian optimization with tree-based Parzen estimator (BO-TPE)\n",
    "Based on the GitHub repo for HPO: https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████| 20/20 [00:08<00:00,  2.34trial/s, best loss: -0.9957089552238806]\n",
      "XGBoost: Hyperopt estimated optimum {'learning_rate': -0.19229249758051492, 'n_estimators': 30.0, 'max_depth': 36.0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'learning_rate':  abs(float(params['learning_rate'])),\n",
    "\n",
    "    }\n",
    "    clf = xgb.XGBClassifier( **params)\n",
    "    clf.fit(x_train, y_train)\n",
    "    y_pred = clf.predict(x_test)\n",
    "    score = accuracy_score(y_test, y_pred)\n",
    "\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 10, 100, 5),\n",
    "    'max_depth': hp.quniform('max_depth', 4, 100, 1),\n",
    "    'learning_rate': hp.normal('learning_rate', 0.01, 0.9),\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"XGBoost: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of XGBoost: 0.9957089552238806\n",
      "Precision of XGBoost: 0.9956893766598436\n",
      "Recall of XGBoost: 0.9957089552238806\n",
      "F1-score of XGBoost: 0.9956902750637269\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      3645\n",
      "           1       0.99      1.00      1.00       393\n",
      "           2       1.00      1.00      1.00        19\n",
      "           3       1.00      1.00      1.00       609\n",
      "           4       0.83      0.71      0.77         7\n",
      "           5       0.99      1.00      0.99       251\n",
      "           6       0.99      0.99      0.99       436\n",
      "\n",
      "    accuracy                           1.00      5360\n",
      "   macro avg       0.97      0.96      0.96      5360\n",
      "weighted avg       1.00      1.00      1.00      5360\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFBCAYAAAAVGzb7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8FVXawPHfc5OgNEEEKUlWUHCt\nKyhFRRFRAyKIuypiRRdFd1FhdcGylkXFl9214rquINVCsSAdaSKi0kGkdyEQioIosGLK8/5xJ+GC\nmSQ3zJ253DxfP/PJzbkz88zcax7OmTPnjKgqxhhjfi0U9AEYY0y8sgRpjDEuLEEaY4wLS5DGGOPC\nEqQxxriwBGmMMS4sQRpjjAtLkMYY48ISpDHGuEgO+gCKJGLDfIwJgqqUZrPs7zZE/TebUv3UUsXy\nQ1wnyOxd6wOLnVLjNJJT6gQWPyd7W5mNn5O9DcDiBxzfxHmCNMYcY/Jygz4CT1mCNMZ4R/OCPgJP\nWYI0xngnzxKkMcYUShOsBmm3+RhjvJOXF/1SDBE5XkTmicjXIrJcRHo75UNEZKOILHGWhk65iEg/\nEVknIktF5PyIfXUWkbXO0rm42FaDNMZ4JzY1yINAK1XdJyIpwGwRmeS811NVPzhi/auBBs7SDHgD\naCYi1YCngcaAAgtFZKyq7nELbDVIY4x38nKjX4qhYfucX1Ocpaj7LTsAw5zt5gBVRaQ20BqYqqq7\nnaQ4FWhTVGxLkMYY72he9EsJiEiSiCwBdhJOcnOdt/o4zeiXReQ4pywV2BKxeaZT5lbuyhKkMcY7\npbgGKSJdRWRBxNL1yN2qaq6qNgTSgKYicg7wGHAG0ASoBjzirF7YyBwtotyVJUhjjGdU80qxaH9V\nbRyx9Hffv/4AzATaqGqW04w+CAwGmjqrZQLpEZulAduKKHdlCdIY453Y9GLXEJGqzuvywJXAKue6\nIiIiwHXAMmeTscAdTm/2hcBeVc0CPgEyROREETkRyHDKXFkvtjHGO7Hpxa4NDBWRJMKVulGqOl5E\nZohIDcJN5yXAfc76E4G2wDrgAHAXgKruFpFngfnOes+o6u6iAluCNMZ4JwZjsVV1KdCokPJWLusr\n0M3lvUHAoJLGPqaa2AcP/kKnu7vzh85/psOt9/Lvt94GQFV59c0hXNPpbtrf0pV33h8DwIzPv+L3\nd/yJ6zt3o+MfH2TR18sK9jVm4lTa3tSFtjd1YczEqZ4d43HHHcdXX4xn4YKpfL1kBk8/9bBn+y6p\n1hktWb5sFqtWzKZXz0L/P4mZtLQ6TJvyPt8sncnXS2bwwP1dfI0/oP+LbMv8miWLp/saN1/Q33+V\nKicwckR/ln3zGd8sncmFzS7wNX6serGDckzVIMuVS2FQv75UqFCe7Jwc7vjTX7n0wsZs+HYL23d+\nx7j3+hMKhfh+zw8AXHhBQy6/5EJEhNXrNvLXJ59n3PAB7P3xJ94Y/B4jB/YD4KYuD9LykgupckLl\noz7GgwcPcmVGR/bvP0BycjKzZo5m8uRPmTtv0VHvuyRCoRD9Xu1Dm7Y3k5mZxZyvJjJu/BRWrlzr\nS/ycnBx69urN4iXLqFSpIvPmTmba9Fm+xR82bBT/+c9gBg9+1Zd4Rwr6+3/5pWf45JNPualTV1JS\nUqhQobwvcRNVzGqQInKGiDziDPl51Xl95lHus+ALz8nJIScnBxFh5OgJ/OmuWwiFwqdz0olVAahQ\noTzh67fwv59/Buf1F3MXclGTRlQ5oTJVTqjMRU0a8cXchUdzaIfZv/8AACkpySSnpBCu8fujaZNG\nrF+/iY0bN5Odnc2oUWO4tn1r3+Jv376TxUvCNfV9+/azatVaUuvU8i3+57Pnstv5BzIoQX3/lStX\n4tJLmjFo8HAAsrOz2bv3R19iF4hBJ02QYpIgReQRYAThi6fzCF8UFWC4iDx6NPvOzc3l+s7daNHu\nZi5q0ojfnX0GW7ZmMWn6Z3T844Pc9/CTfLtla8H60z77gvY338Of//oUzz7+FwB27PqOWifXKFin\nZo3q7Nj13dEc1mFCoRAL5k8ha+tSpk+fxbz5iz3bd3HqpNZiS+ahOxcyt2ZRx8cEFemUU9JoeN45\nzJ3n3/nHg6C+/1NPPYXvvvuegW+9zPx5n/Dmf//lfw0ywZrYsapBdgGaqGpfVX3HWfoSvk/pqC5K\nJSUl8eHQ15k++m2+WbGGtRs28Ut2NseVK8eoQf24vn0bnnz+5YL1r7ysOeOGD6Bf36f494BhABT2\nD3p+TdMLeXl5NG6SwSn1GtOkcSPOPvu3nu27OIWdh5812HwVK1Zg1MgBPPTXp/npp33Fb5BAgvr+\nk5OSaNToXN58cxhNmrZm//4DPNLrfl9iF7AaZInkAYXNF1/bec9V5F31bw0b7rreCZUr0eT83zF7\nzgJq1ajOVS0vAeDKyy5mzfqNv1q/ccNz2bI1iz0/7KXWydXZvnNXwXs7dn3HydVPKtGJRWPv3h/5\nbNaXtM5o6fm+3WzNzCI97dBHn5Zam6ysHb7FB0hOTub9kQMYPnw0H388qfgNEpTf33/m1iwyM7MK\naqwffTSBRg3P9SV2PtXcqJd4FqsE2QOYLiKTRKS/s0wGpgPdi9ow8q76u++4+bD3du/5gR+d2sjP\nBw8yZ/5i6p2STqsWFzF34RIA5i/+hlPSw8MrN2duK6g9rVi9juzsHKpWOYHmzS7gy3mL2PvjT+z9\n8Se+nLeI5h719lWvXo0qVU4A4Pjjj+eKVpeyerV/z9aZv2AJ9evXo27ddFJSUujYsQPjxk/xLT6E\ne5JXrlrHK6+6DohIWEF+/zt27CIzcxunn34aAK1aXcLKlWt8iV0gwZrYMenFVtXJInI64SZ1KuHr\nj5nAfD2KfzJ2fb+Hvz33Arl5eWie0rrVpbRs3ozzf3c2j/T+J2+P/JgK5Y+n96M9AJg6czZjJ00n\nOTmZ448rxwvPPIqIUOWEytx75810ujucq++76xZPerABateuyaCBr5CUFCIUCvHBB+OYMHGaJ/su\nidzcXLr3eIKJE94jKRRiyNCRrFjh3x9J84ubcPttN7D0mxUsmB9OzE8+2ZdJk2f4Ev+dt1/nshYX\nUb16NTZtWEDvZ15g8JARvsSG4L//7n95kmFDX6NcuRQ2btxMl7sf8i02EPdN5mhJENenSqo0j5D0\nij3V0J5qWKbjl/Kxrz8v/Djqv9njL7jOHvtqjCkD7KmGxhjjIs6vKUbLEqQxxjsJdg3SEqQxxjtW\ngzTGGBdWgzTGGBeWII0xpnDxPjImWpYgjTHesRqkMca4SLBOmmNqRnFjjPGT1SCNMd6xJrYxxrhI\nsCa2JUhjjHesBmmMMS6sBumflBqnBRo/f9opi2/xy2L8UrEapDHGuLAE6Z+gJ4w9s0aTwOKv3DU/\n8PMv0xPGWvzSsSa2Mca4sBqkMca4sBqkMca4SLAapA01NMZ4JwaPfRWR40Vknoh8LSLLRaS3U15P\nROaKyFoRGSki5Zzy45zf1znv143Y12NO+WoRaV1cbEuQxhjv5OVFvxTvINBKVc8DGgJtRORC4B/A\ny6raANgDdHHW7wLsUdX6wMvOeojIWUAn4GygDfAfEUkqKrAlSGOMd2KQIDVsn/NrirMo0Ar4wCkf\nClznvO7g/I7z/hUiIk75CFU9qKobgXVA06JiW4I0xnhHNepFRLqKyIKIpeuRuxWRJBFZAuwEpgLr\ngR9UNcdZJRNIdV6nAlvCh6M5wF7gpMjyQrYplHXSGGO8U4pOGlXtD/QvZp1coKGIVAVGA2cWtprz\nU1zecyt3ZQnSGOOdGPdiq+oPIjITuBCoKiLJTi0xDci/wz0TSAcyRSQZqALsjijPF7lNoayJbYzx\nTmx6sWs4NUdEpDxwJbAS+BS4wVmtMzDGeT3W+R3n/Rmqqk55J6eXux7QAJhXVGyrQRpj4l1tYKjT\n4xwCRqnqeBFZAYwQkeeAxcBAZ/2BwNsiso5wzbETgKouF5FRwAogB+imxTxlzBKkMcY7MWhiq+pS\noFEh5RsopBdaVX8GbnTZVx+gT0ljW4I0xnhHi+zzOOYk5DXI1hktWb5sFqtWzKZXz24xiVHuuHKM\nnDyY0Z++y7hZI7i/1z0ANLukMR9OG8bYz4bzf689TVJS+D7UVm1a8PHMd/loxju8P2Uo5zc7LybH\nBf6cv8WPv9jxED9GN4oHRjSOM35yudSoDy4UCrFy+ee0aXszmZlZzPlqIrfd/mdWrlwb1X5KMt1Z\nhYrlObD/fyQnJ/HOuAH0feplXur/PH+8vhubNmzmgUe6sm3Ldj58b2zBugCnn1Wflwc8zzXNO7ru\nu7TTnXl5/kHFP5rpvoKM7+VnH3h81cJuiSnW/wb+Neq/2fJdXihVLD8kXA2yaZNGrF+/iY0bN5Od\nnc2oUWO4tn2xQy5LJT/hJackk5KSTF5uHr/88gubNmwG4MuZ88hod/lh6wJUqFCeWP3D5Of5W/z4\niR0P8YGY9GIHKZAEKSJ3xWrfdVJrsSXz0K1NmVuzqFOnVkxihUIhPprxDrNXfMKXn81j6aLlpCQn\nc/Z54XtYM9q3oladmgXrX9m2JRO+GMUb777EEz2ei8kx+Xn+Fj9+YsdDfADN06iXeBZUDbK32xuR\nw47y8vZHvePwkMvDxaq2lpeXxx9a3cbl57Xj3EZn0eCMU3n43id49Nm/MHLyYA7sO0BO7qG7CKZN\nnMk1zTvyQOdePPjovTE5Jj/P3+LHT+x4iA8k3DXImPVii8hSt7eAmi7vHTbsqDTXILdmZpGeduja\nTVpqbbKydkS7m6j89OM+5n25iEtaXcTg/7zL7deGh5Je3LIZp5z2m1+tv2DOYtJPSaNqtSr8sHuv\np8cSxPlb/OBjx0N8IO6bzNGKZQ2yJnAH0L6Q5ftYBZ2/YAn169ejbt10UlJS6NixA+PGT/E8zokn\nVaXyCZUAOO7447ioRVM2rv2WatVPBCClXAp3338HI4d+BMBv6qUVbHvWub8lpVyy58kR/Dt/ix9f\nseMhPgB5Gv0Sx2J5H+R4oJKqLjnyDWcsZUzk5ubSvccTTJzwHkmhEEOGjmTFijWex6lRs7pzG0+I\nkISYPHYaM6fO5q9PP0DLqy4hFAoxYsiHzJ29AICMdq3ocGNbsnNyOPjzQR7q+jfPjwn8O3+LH1+x\n4yE+EPdN5mgl3G0+XrGnGtpTDct0/FLe5nPg1fui/put0P2/cXubj42kMcZ4J44rXKVhCdIY450E\na2JbgjTGeCfOO12iZQnSGOOdBLvNxxKkMcY7CVaDTLix2MYY4xWrQRpjPKPWSWOMMS4SrIltCdIY\n4x3rpDHGGBdWgzTGGBd2DdIYY1xYDdIYY1zYNUhjjHFhNUj/5E/7FJSVu+YHGj/o87f4ZTt+adh9\nkD4qq/Mh5sdPCTB+ts0HWebjl4rVII0xxoUlSGOMcZFgnTQ2WYUxxjsxeGiXiKSLyKcislJElotI\nd6f87yKyVUSWOEvbiG0eE5F1IrJaRFpHlLdxytaJyKPFxbYapDHGMxqbJnYO8LCqLhKRysBCEZnq\nvPeyqr4QubKInAV0As4G6gDTROR05+3XgauATGC+iIxV1RVugS1BGmO8E4MEqapZQJbz+icRWQmk\nFrFJB2CEqh4ENorIOqCp8946Vd0AICIjnHVdE6Q1sY0x3snLi36JgojUBRoBc52i+0VkqYgMEpET\nnbJUYEvEZplOmVu5K0uQxphAiUhXEVkQsXR1Wa8S8CHQQ1V/BN4ATgMaEq5hvpi/aiGbaxHlrqyJ\nbYzxTima2KraH+hf1DoikkI4Ob6rqh852+2IeH8AMN75NRNIj9g8Dci/udOtvFBWgzTGeCc2vdgC\nDARWqupLEeW1I1b7PbDMeT0W6CQix4lIPaABMA+YDzQQkXoiUo5wR87YomJbDdIY4xnVmPRiNwdu\nB74RkSVO2ePAzSLSkHAzeRNwr3MMy0VkFOHOlxygm6rmAojI/cAnQBIwSFWXFxXYEqQxxjux6cWe\nTeHXDycWsU0foE8h5ROL2u5IliCNMd6xoYbGGFO4GN0oHhhLkMYY7yRYgkzIXuzWGS1ZvmwWq1bM\nplfPbgkff0D/F9ma+TWLF08vKPvd787i81ljWbxoGqNHD6Fy5UoxP458Ze3zj5fY8RCfvFIscSzh\nEmQoFKLfq31o1/42zj3vcm666TrOPLNBQscfOmwU7drdeljZm//9F4//7XkanX8lYz6exMMP/ymm\nx5CvLH7+8RA7HuJDuIkd7RLPYpYgReQMEbnCufs9srxNrGICNG3SiPXrN7Fx42ays7MZNWoM17Zv\nXfyGx3D82bPnsnvPD4eVnX76aXz++RwApk3/nN//vm1hm3quLH7+8RA7HuIDMbkPMkgxSZAi8iAw\nBngAWCYiHSLefj4WMfPVSa3FlsxDN8dnbs2iTp1asQwZV/HzLV++mvbtMwC44fp2pKf5Mzt10Ocf\nZPyyfO4FrIldIvcAF6jqdUBL4Mn8Odwo/H6mApHjMvPy9kcdOHzT/eFidPNqXMbPd0/Xh/jTfXcy\nd84kKlWuyC+/ZPsSN+jzDzJ+WT73gngJ1sSOVS92kqruA1DVTSLSEvhARE6hmAQZOS4zuVxq1J/e\n1sysw2pLaam1ycraUcQW3go6fr7Vq9fT9ppbAGjQ4FTaXn2FL3GDPv8g45flcy8Q5zXCaMWqBrnd\nGQIEgJMs2wHVgXNjFBOA+QuWUL9+PerWTSclJYWOHTswbvyUWIaMq/j5atQ4CQjXKh5/rDv9+7/t\nS9ygzz/I+GX53PNZDbJk7iA8BrKAquYAd4jImzGKCUBubi7dezzBxAnvkRQKMWToSFasWBPLkIHH\nf/vt17msxUVUr16NjRsW8MwzL1CpUkXu+9OdAHz88USGDB0Z02PIVxY//3iIHQ/xgYSrQUoQ18dK\nqjRNbK/YY1/tsa9lOr5qkZfC3Hzf/rKo/2ZPGvdZqWL5IeHugzTGGK/YUENjjHcSrIltCdIY45kE\neyy2JUhjjIcsQRpjTOGsBmmMMS4sQRpjjAtLkMYY46Z0t0/GLUuQxhjPWA3SGGNcaJ7VII0xplBW\ngzTGGBelHMIdtyxBGmM8YzVIY4xxYdcgfZQ/7VNZjZ9dxs/f4gcbvzTiePbEUonrBBn0fIxlPX7L\nVH8e03CkmVvDz/cu0/MxxkH80rAapDHGuEi0BGkT5hpj4pqIpIvIpyKyUkSW5z8hVUSqichUEVnr\n/DzRKRcR6Sci60RkqYicH7Gvzs76a0Wkc3GxLUEaYzyjGv1SAjnAw6p6JnAh0E1EzgIeBaaragNg\nuvM7wNVAA2fpCrwB4YQKPA00A5oCT+cnVTeWII0xntE8iXopdp+qWaq6yHn9E7ASSAU6AEOd1YYC\n1zmvOwDDNGwOUFVEagOtgamqultV9wBTgTZFxbYEaYzxjKpEvYhIVxFZELF0ddu/iNQFGgFzgZqq\nmhWOq1nAyc5qqcCWiM0ynTK3clfFdtKISE3geaCOql7tVG0vUtWBxW1rjClbSnOjuKr2B/oXt56I\nVAI+BHqo6o8irrXPwt7QIspdlaQGOQT4BMi/52AN0KME2xljypg8laiXkhCRFMLJ8V1V/cgp3uE0\nnXF+7nTKM4H0iM3TgG1FlLsqSYKsrqqjcJ42oao5QG4JtjPGlDGlaWIXR8JVxYHASlV9KeKtsUB+\nT3RnYExE+R1Ob/aFwF6nCf4JkCEiJzqdMxlOmauS3Ae5X0ROwqmK5gcswXbGmDImRvdBNgduB74R\nkSVO2eNAX2CUiHQBNgM3Ou9NBNoC64ADwF0AqrpbRJ4F5jvrPaOqu4sKXJIE+RDhjHyaiHwB1ABu\nKOGJGWPKkFgMNVTV2RR+/RDgV8O9VFWBbi77GgQMKmnsYhOkqi4SkcuA3zoHuVpVs0sawBhTdiTa\nSJqS9GLfcUTR+SKCqg6L0TEZY45RJe10OVaUpIndJOL18YSrtIsAS5DGmMOUuQlzVfWByN9FpArw\ndsyOyBhzzEq06c5KM5LmAOExjnFpQP8X2Zb5NUsWTw/sGFpntGT5slmsWjGbXj0LvVacEPErnVCR\n3m8+xbCZgxj66UDOOv9MKletzAvv/YN3Ph/CC+/9g0pVKgFQsXJFnh/8LG9NeZPB09+iTcfWMTuu\nID//svLdu4nVfZBBKTZBisg4ERnrLOOB1Ry63yjuDBs2imva3RpY/FAoRL9X+9Cu/W2ce97l3HTT\ndZx5pn//nvgZ//7e3Zg3cz53tPwjXTLuZfO6zdzSrROLvljMbZfeyaIvFnNLt04AXNf5Wjat/Za7\nM+6lx40P8+en7iU5xfvZ9oL8/MvSd+8mFvdBBqkkNcgXgBed5f+AFqr6aNGbgIg0FZEmzuuzROQh\nEWl7VEdbAp/PnsvuPT/EOoyrpk0asX79JjZu3Ex2djajRo3h2vaxqy0FFb9CpQqc1+xcJgyfBEBO\ndg77ftxP84yLmfz+FAAmvz+FS1o3B8JNrwoVKwBQvmJ5fvrhJ3JzvB9vEOTnX1a++6LEaDafwBSZ\nIEUkCXhSVT9zli9UNbO4nYrI00A/4A0R+T/g30Al4FER+ZsXBx6v6qTWYkvmodFLmVuzqFOnVsLF\nr/Ob2vywey+PvtSTAZP/S89/PcTx5Y+nWvUT2b0zfO/t7p27OfGkqgCMHvIxpzT4DR8uHMngaQN4\n7an/oDH46wjy8y8r331RylQTW1VzgQNOx0w0biB893sLwjdsXqeqzxCebuimojaMnNkjL29/lGGD\nV9gA+lgkgqDjJyUncfo5DRjz9jjuaXMf/zvwc0FzujBNWzZm3fL1XH/BTdzd+l66P3c/FSpV8Py4\ngvz8y8p3X5Sy2MT+mfAQn4HOLL39RKRfMdvkqGquqh4A1qvqjwCq+j+cMd1uVLW/qjZW1cahUMUS\nnUQ82ZqZRXraoWeJpKXWJitrR8LF35W1i11Zu1i5eBUAn02YRYNzG7D7uz1UO7kaANVOrsae78OX\nO9p0bMOsSZ+Hj3HTNrK2bOc39dML3/lRCPLzLyvffVlSkgQ5AXgSmAUsdJYFxWzzi4jkVw8uyC90\naqIJ9uTcw81fsIT69etRt246KSkpdOzYgXHjpyRc/N279rBz2y7ST00D4IJLzufbtd/y5dSvaHNj\nBgBtbszgiylfArBz604uuCQ88/2J1auSflo6Wd9meX5cQX7+ZeW7L0qiNbFL0o1YVVVfjSzIfyZE\nEVqo6kEA1cNmiEvh0OwbMfHO269zWYuLqF69Gps2LKD3My8weMiIWIY8TG5uLt17PMHECe+RFAox\nZOhIVqxYk5Dx+z35b5547TGSy6WQ9W0WfR/+FyEJ8fR/n6Btpzbs2LqTv9/3LADDXn2HR1/qyaBp\nAxCg//MD2LvnR8+PKcjPvyx9927ivM8lalLcNQoRWaSq5x9RtlhVG8X0yIDkcqmBfd7x8NjVoOPb\nY1/LcPxSXhz8svb1Uf/NXpz1YdxWI11rkCJyM3ALUE9Exka8VRn4PtYHZow59sR7p0u0impifwlk\nAdUJ3wOZ7ydgaSwPyhhzbEq0DgbXBKmq3wLfAhcVtQMR+UpVi1zHGFM2qOu0jccmL8Z6He/BPowx\nCSAvwXppvEiQCfaRGGNKK89qkMYYU7hEa2KXZDaf+50ngLmu4uHxGGOOYXmlWOJZSUbS1ALmi8go\nEWkjvx7weXsMjssYcwxSJOolnhWbIFX1CcIT5A4E7gTWisjzInKa8/6ymB6hMeaYURZrkPmPUdzu\nLDnAicAHIvLPGB6bMeYYk2gJsiRPNXyQ8Pjp74C3gJ6qmi0iIWAt0Cu2h2iMOVbEe5M5WiXpxa4O\n/MG5cbyAquaJSLvYHJYx5liUYI/FLtFTDZ8q4r2V3h6OMeZYlmj3QZbmqYbGGFMmxPWN4vnTPln8\nYORPOxaUoM+/rMcvjUQbVhfXCTLo+RAtfhmeD9Hil0oseqVFZBDQDtipquc4ZX8H7gF2Oas9rqoT\nnfceA7oAucCDqvqJU94GeBVIAt5S1b7FxY7rBGmMObbkFfLgMA8MIfxk1GFHlL+sqi9EFojIWUAn\n4GygDjBNRE533n4duArIJDz4ZayqrigqsCVIY4xnYtHEVtVZIlK3hKt3AEY4j3zZKCLrgKbOe+tU\ndQOAiIxw1i0yQVonjTHGMz7fKH6/iCwVkUER80WkAlsi1sl0ytzKi2QJ0hjjmTyJfhGRriKyIGLp\nWoJQbwCnAQ0JP/kg/6kHhbXxtYjyIlkT2xjjmdLcB6mq/YH+UW5T8MBvERkAjHd+zQQiH7ieBuT3\nOrmVu7IapDHGM1qKpTREpHbEr78H8ifNGQt0EpHjRKQe4Yl25gHzgQYiUk9EyhHuyIl8GGGhrAZp\njPFMLIYaishwoCVQXUQygaeBliLSkHCO3QTcC6Cqy0VkFOHOlxygm6rmOvu5H/iE8G0+g1R1eXGx\nLUEaYzwTi/sgVfXmQooHFrF+H6BPIeUTgYnRxLYEaYzxjI2kMcYYF2VuNh9jjCmpeJ8AN1qWII0x\nnrEEaYwxLjTBmtgJdx9kWlodpk15n2+WzuTrJTN44P4uvh9D64yWLF82i1UrZtOrZzeL77N1a+aw\neNE0Fsyfwpyvouq0PGpBn3vQ8cvcM2mONTk5OfTs1ZvFS5ZRqVJF5s2dzLTps1i5cq0v8UOhEP1e\n7UObtjeTmZnFnK8mMm78FIvvU/x8V151I99/v8fXmEGfe9DxE1HC1SC3b9/J4iXhm+r37dvPqlVr\nSa1Ty7f4TZs0Yv36TWzcuJns7GxGjRrDte1bW/wyIOhzDzo+JF4N0rcEKSJHzuUWc6eckkbD885h\n7rzFvsWsk1qLLZmHhnhmbs2ijo8JuqzHB1BVJk0cztw5k7i7y62+xQ363IOOD/4NNfRLTJrYInLk\nGEcBLheRqgCqem0s4kaqWLECo0YO4KG/Ps1PP+2LdbgCUsiEoeHHilt8v7RoeR1ZWTuoUeMkJk8a\nwerV6/h89tyYxw363IOOD3YfZEmlER4L+RaHphpqzKEpiVw5Ux11BZCkKoRCFaMOnpyczPsjBzB8\n+Gg+/nhS1Nsfja2ZWaSnHZoqPy21NllZO4rYwuJ7LT/erl3fM2bMJJo0aehLggz63IOOD/HfZI5W\nrJrYjYGFwN+Avao6E/ifqn67ovrdAAASdUlEQVSmqp8VtaGq9lfVxqrauDTJEWBA/xdZuWodr7wa\n1QxKnpi/YAn169ejbt10UlJS6NixA+PGT7H4PqlQoTyVKlUseH3VlZexfPlqX2IHfe5Bx4fEuwYZ\nkxqkquYBL4vI+87PHbGKdaTmFzfh9ttuYOk3K1gwP/w/x5NP9mXS5Bl+hCc3N5fuPZ5g4oT3SAqF\nGDJ0JCtWrPEltsWHmjVr8MH74XkMkpOTGDHiYz6ZMtOX2EGfe9DxIf6vKUZL/LhGISLXAM1V9fFo\ntksulxrY512WnyoYdPx4eapfmY6vpbvl+5+n3Bb132yvb9+J2yuXvtTqVHUCMMGPWMaY4MR7kzla\nCXejuDEmOInWxLYEaYzxTF6CpUhLkMYYz1gT2xhjXCRW/dESpDHGQ1aDNMYYFzbU0BhjXFgnjTHG\nuEis9JiA80EaY4xXrAZpjPGMddIYY4wLuwZpjDEuEis9WoI0xnjImtg+yp/2yeJbfIt/bEi0Jrb1\nYhtjPBOLh3aJyCAR2SkiyyLKqonIVBFZ6/w80SkXEeknIutEZKmInB+xTWdn/bUi0rkk5xPXNciy\nOmFsWY8fFxPGAqlVzwok/tYfVgDBn39pxKiJPQT4NxD5ZNRHgemq2ldEHnV+fwS4GmjgLM2AN4Bm\nIlINeJrw42AUWCgiY1W1yIenWw3SGOMZLcV/xe5TdRaw+4jiDsBQ5/VQ4LqI8mEaNgeoKiK1gdbA\nVFXd7STFqUCb4mJbgjTGeKY0D+0Ska4isiBi6VqCUDVVNQvA+XmyU54KbIlYL9MpcysvUlw3sY0x\nx5bSdNKoan/Aq0eQFjZdhhZRXiSrQRpjPBOLThoXO5ymM87PnU55JpAesV4asK2I8iJZgjTGeCYP\njXoppbFAfk90Z2BMRPkdTm/2hcBepwn+CZAhIic6Pd4ZTlmRrIltjPFMLHqxRWQ40BKoLiKZhHuj\n+wKjRKQLsBm40Vl9ItAWWAccAO4CUNXdIvIsMN9Z7xlVPbLj51csQRpjPFOSXumo96l6s8tbVxSy\nrgLdXPYzCBgUTWxLkMYYz9hQQ2OMcRGLGmSQrJPGGGNcWA3SGOMZa2IbY4yLPE2sJrYlSGOMZxIr\nPSbgNcgB/V9kW+bXLFk8PbBjaJ3RkuXLZrFqxWx69Sz0jgOLfwzHr5Nai/fHDmbmnLHM+HIMXe69\nDYCHHvkzC5bPYMqsD5ky60NaXXVpwTb3/+VuZi+cxKx547msVXPPjylf0J+9jzeK+yLhEuSwYaO4\npt2tgcUPhUL0e7UP7drfxrnnXc5NN13HmWc2sPgJFD8nJ4feT/yTlhdeS/uMm7nz7ptp8NvTABjw\nxjAyWlxPRovrmTH1cwAa/PY0OvyhLa0uupZbb7iX5194glDI+z+9oD97iM1sPkFKuAT5+ey57N7z\nQ2DxmzZpxPr1m9i4cTPZ2dmMGjWGa9u3tvgJFH/nju9YtnQlAPv3HWDtmg3Uqn2y6/qt217OmI8m\n8ssv2WzZvJVNG7bQ6IJzPT0mCP6zh9LN5hPPfEmQInKJiDwkIhl+xAtSndRabMk8NAY+c2sWderU\nsvgJGj8tvQ7n/O5MFi9cCsBd99zC1Nkf8eJrz1KlygkA1Kpdk21btxdsk7VtO7Vq1/T8WIL+7MGa\n2CUiIvMiXt9DeDbgysDTzuy/CUvk17MqqY89exbfv/gVKlZgwLBXePqxvuz7aT/DBo3k4kZtyLj0\nenbu2MVTz/X09ZiC/uzBmtgllRLxuitwlar2JjyDRpEXCCMnz8zL2x+jw4udrZlZpKcdmio/LbU2\nWVk7LH6CxU9OTmbA0FcY/f4EJo2fBsB3u74nLy8PVeXdoR/Q0GlGZ23bTp3UQzW52nVqsWP7zkL3\nezSC/uzBmtgl3q8zrdBJgKjqLgBV3Q/kFLWhqvZX1caq2jgUqhijw4ud+QuWUL9+PerWTSclJYWO\nHTswbvwUi59g8V987RnWrdlA//8MLSg7uWb1gtdXt7uS1SvXAjBl0qd0+ENbypVLIf03qdQ77Tcs\nXviN58cU9GcP4RprtEs8i9V9kFWAhYRn8VURqaWq20WkEoXP7OuZd95+nctaXET16tXYtGEBvZ95\ngcFDRsQy5GFyc3Pp3uMJJk54j6RQiCFDR7JixRqLn0Dxm1x4Pjd06sCK5auZMutDAPo++wrXXd+W\ns849A1Ulc/M2HvnL3wFYs2o94z6ezKdzxpKbk8vfej5HXp73daegP3tIvMe+is/XhyoQfpbExpKs\nn1wuNbBPuyw/VTDo+PZUwzh4qqFqqSoy7X/TLuq/2XGbx8e00nQ0fB1Jo6oHgBIlR2PMsSfeO12i\nZUMNjTGeSbQmtiVIY4xn4r3TJVqWII0xnon323aiZQnSGOOZRLsGmXBjsY0xxitWgzTGeMY6aYwx\nxoV10hhjjAurQRpjjItE66SxBGmM8Yw9tMsYY1wkVnq0BGmM8VCiXYO0+yCNMZ6J1SMXRGSTiHwj\nIktEZIFTVk1EporIWufniU65iEg/EVknIktF5PzSnk9c1yDzp52y+BY/CPnTjgUl6PMvjRjf5nO5\nqn4X8fujwHRV7es8yuVR4BHgaqCBszQD3nB+Rs1qkMYYz/j80K4OQP6U7kOB6yLKh2nYHKCqiNQu\nTYC4rkGW1Qljy3r8eJkwN+j4v6t5YSDxl+6YU+ptS3Obj4h0Jfzsqnz9VbX/r3YNU0REgTed92uq\nahaAqmaJSP6zd1OBLRHbZjplWdEeW1wnSGPMsaU0TWwn2R2ZEI/UXFW3OUlwqoisKmLdwmYoL1VV\n1RKkMcYzserFVtVtzs+dIjIaaArsEJHaTu2xNpD/qMhMID1i8zSgVBd07RqkMcYzsXiqoYhUFJHK\n+a8JPz56GTAW6Oys1hkY47weC9zh9GZfCOzNb4pHy2qQxhjPxKgGWRMYLSIQzlnvqepkEZkPjBKR\nLsBm4EZn/YlAW2AdcAC4q7SBLUEaYzwTi7HYqroBOK+Q8u+BKwopV6CbF7GtiW2MMS6sBmmM8YxN\nVmGMMS5sujNjjHFhNUhjjHFhNUhjjHFhNUhjjHFhNUhjjHGRaDXIhL0PMhQKMX/eJ4wZPbT4lT3W\nOqMly5fNYtWK2fTq6cn9qhb/GInvZ+xQKMTIqUN57e0XAPj7S4/z/vRhfDDjbV58qw/lK5QH4PZ7\nOzF61nt8MONtBrz/GrXTasXsmLQU/8WzhE2QDz5wN6tWrfU9bigUot+rfWjX/jbOPe9ybrrpOs48\ns4HFLwPx/Y596z0d2bh2U8Hv/3rqFW684g5uaHU7WZk7uPmPNwCwatkabm59Fze0up2p42fwlydj\nl7hV86Je4llMEqSINBORE5zX5UWkt4iME5F/iEiVWMSMlJpam7ZXX8GgQcNjHepXmjZpxPr1m9i4\ncTPZ2dmMGjWGa9u3tvhlIL6fsWvWrkGLK5vz0btjC8r27ztQ8Pr48scV1M7mf7GIn/93EIClC5dT\ns/bJxIrPE+bGXKxqkIMIDxIHeBWoAvzDKRsco5gFXnqxN48+9hx5ef7/61QntRZbMg/NrJS5NYs6\ndWLXpLH48RPfz9i9nu3BS8/+m7wjamDPvPI3Pv1mAnXrn8Lwge//arvf39Ke2TO+iskxQWxm8wlS\nrBJkSFVznNeNVbWHqs5W1d7AqUVtKCJdRWSBiCzIy9sfdeBr2l7Jzp3fsWjxN6U47KPnzDhyGD//\nJ7D4wcX3K3aLq5qz+7s9rFy6+lfvPdWjD1ec156NazfRusOVh713zfWtOfu8Mxjyn3c9P6Z8VoMs\nmWUikj/F0Nci0hhARE4HsovaUFX7q2pjVW0cClWMOvDFFzemfbsM1q2Zw7vv/IfLL2/O0CH9ot5P\naW3NzCI97dBU/WmptcnK2mHxy0B8v2I3bPI7WmZcyqT5H/HP/z5L0+YX8Py/ny54Py8vj8ljpnPl\nNZcXlDW7tAn3dL+TBzv3IvuXIv8Ej4rVIEvmbuAyEVkPnAV8JSIbgAHOezHztyf6UvfUxtQ//UJu\nve3PfPrpF3S+88FYhjzM/AVLqF+/HnXrppOSkkLHjh0YN36KxS8D8f2K3e/5N7jq/A5c3eQP9Lrv\nSeZ9sZDH7+9Net20gnVaZlzCpnXfAnDGOafz1L968WDnnuz+bo/nxxMpTzXqJZ7F5D5IVd0L3OnM\nAnyqEydTVf2rSgQkNzeX7j2eYOKE90gKhRgydCQrVqyx+GUgfpCxRYTn+j1JpcoVEYHVy9fx3CP/\nBOChp+6nQsUKvDCgDwDbt+7gwc69YnIc8X7bTrQknqu4yeVSAzu4svxUwaDjx8tTBYOOH+hTDVUL\ne/BVsWpWOSPqv9kde1eVKpYfEvY+SGOMOVo21NAY45l475WOliVIY4xn4vmSXWlYgjTGeCbee6Wj\nZQnSGOMZq0EaY4wLuwZpjDEurAZpjDEu7BqkMca4SLSRNJYgjTGesRqkMca4SLRrkDbU0BjjmVg9\nk0ZE2ojIahFZJyKPxvg0ClgN0hjjmVjUIEUkCXgduArIBOaLyFhVXeF5sCNYDdIY45kYTZjbFFin\nqhtU9RdgBNAhpifiiOsaZP60Txbf4pfF+Et3zAk0fmnE6ApkKrAl4vdMoFlsQh0urhNkaeekyyci\nXVW1v1eHc6zEtvgWP6j4Ob9sjfpvVkS6Al0jivofceyF7dOX3qBEb2J3LX6VhIxt8S1+0PFLLPI5\nVM5yZGLPBNIjfk8DfKneJ3qCNMYc++YDDUSknoiUAzoBY4vZxhPx3cQ2xpR5qpojIvcDnwBJwCBV\nXe5H7ERPkIFdAwo4tsW3+EHH95SqTgQm+h03rh/aZYwxQbJrkMYY48ISpDHGuEjIBBnUuE0n9iAR\n2Skiy/yMGxE/XUQ+FZGVIrJcRLr7HP94EZknIl878Xv7Gd85hiQRWSwi4/2O7cTfJCLfiMgSEVng\nc+yqIvKBiKxy/h+4yM/4iSbhrkE64zbXEDFuE7jZj3GbTvwWwD5gmKqe40fMI+LXBmqr6iIRqQws\nBK7z8fwFqKiq+0QkBZgNdFdV34aFiMhDQGPgBFVt51fciPibgMaq+l0AsYcCn6vqW84tMRVU9Qe/\njyNRJGINMrBxmwCqOgvY7Ve8QuJnqeoi5/VPwErCQ7X8iq+qus/5NcVZfPtXWETSgGuAt/yKGS9E\n5ASgBTAQQFV/seR4dBIxQRY2btO3BBFPRKQu0AiY63PcJBFZAuwEpqqqn/FfAXoBeT7GPJICU0Rk\noTOMzi+nAruAwc4lhrdEpKKP8RNOIibIwMZtxhMRqQR8CPRQ1R/9jK2quarakPCQsKYi4sulBhFp\nB+xU1YV+xCtCc1U9H7ga6OZcdvFDMnA+8IaqNgL2A75eg080iZggAxu3GS+ca38fAu+q6kdBHYfT\nvJsJtPEpZHPgWuca4AiglYi841PsAqq6zfm5ExhN+LKPHzKBzIga+weEE6YppURMkIGN24wHTifJ\nQGClqr4UQPwaIlLVeV0euBJY5UdsVX1MVdNUtS7h732Gqt7mR+x8IlLR6RzDad5mAL7c0aCq24Et\nIvJbp+gKwJfOuUSVcEMNgxy3CSAiw4GWQHURyQSeVtWBfsUnXIu6HfjGuQ4I8LgzVMsPtYGhzt0E\nIWCUqgZyu01AagKjw/9OkQy8p6qTfYz/APCuUznYANzlY+yEk3C3+RhjjFcSsYltjDGesARpjDEu\nLEEaY4wLS5DGGOPCEqQxxriwBGniiojUDWomJGOOZAnS+MK5L9KYY4olSFMoEXk2ci5JEekjIg8W\nsl5LEZklIqNFZIWI/FdEQs57+0TkGRGZC1wkIheIyGfOJA6fOFOz4ZR/LSJfAd38OkdjimMJ0rgZ\nCHQGcBJeJ+Bdl3WbAg8D5wKnAX9wyisCy1S1GeEZhV4DblDVC4BBQB9nvcHAg6pqk7uauJJwQw2N\nN1R1k4h8LyKNCA+fW6yq37usPk9VN0DBUMtLCE+UkEt40gyA3wLnAFOdYXhJQJaIVAGqqupnznpv\nE54Fx5jAWYI0RXkLuBOoRbjG5+bI8ar5v/+sqrnOawGWH1lLdCa2sPGuJi5ZE9sUZTThqcqaEJ78\nw01TZ/akEHAT4ccsHGk1UCP/GSkikiIiZztTou0VkUuc9W717vCNOTpWgzSuVPUXEfkU+CGiJliY\nr4C+hK9BziKcWAvb1w1AP6dZnUx49u/lhGecGSQiByg6ERvjK5vNx7hyaoSLgBtVda3LOi2Bvwbx\ncCxjYs2a2KZQInIWsA6Y7pYcjUl0VoM0JSIi5xLuYY500LmFx5iEZAnSGGNcWBPbGGNcWII0xhgX\nliCNMcaFJUhjjHFhCdIYY1z8P2QIZHTK6RiuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xg = xgb.XGBClassifier(learning_rate= 0.19229249758051492, n_estimators = 30, max_depth = 36)\n",
    "xg.fit(x_train,y_train)\n",
    "xg_score=xg.score(x_test,y_test)\n",
    "y_predict=xg.predict(x_test)\n",
    "y_true=y_test\n",
    "print('Accuracy of XGBoost: '+ str(xg_score))\n",
    "precision,recall,fscore,none= precision_recall_fscore_support(y_true, y_predict, average='weighted') \n",
    "print('Precision of XGBoost: '+(str(precision)))\n",
    "print('Recall of XGBoost: '+(str(recall)))\n",
    "print('F1-score of XGBoost: '+(str(fscore)))\n",
    "print(classification_report(y_true,y_predict))\n",
    "cm=confusion_matrix(y_true,y_predict)\n",
    "f,ax=plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(cm,annot=True,linewidth=0.5,linecolor=\"red\",fmt=\".0f\",ax=ax)\n",
    "plt.xlabel(\"y_pred\")\n",
    "plt.ylabel(\"y_true\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Anomaly-based IDS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate the port-scan datasets for unknown attack detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df=pd.read_csv('./data/CICIDS2017_sample_km.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18225\n",
       "3     3042\n",
       "6     2180\n",
       "1     1966\n",
       "5     1255\n",
       "2       96\n",
       "4       36\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df1 = df[df['Label'] != 5]\n",
    "df1['Label'][df1['Label'] > 0] = 1\n",
    "df1.to_csv('./data/CICIDS2017_sample_km_without_portscan.csv',index=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df2 = df[df['Label'] == 5]\n",
    "df2['Label'][df2['Label'] == 5] = 1\n",
    "df2.to_csv('./data/CICIDS2017_sample_km_portscan.csv',index=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the generated datasets for unknown attack detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df1 = pd.read_csv('./data/CICIDS2017_sample_km_without_portscan.csv')\n",
    "df2 = pd.read_csv('./data/CICIDS2017_sample_km_portscan.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "features = df1.drop(['Label'],axis=1).dtypes[df1.dtypes != 'object'].index\n",
    "df1[features] = df1[features].apply(\n",
    "    lambda x: (x - x.mean()) / (x.std()))\n",
    "df2[features] = df2[features].apply(\n",
    "    lambda x: (x - x.mean()) / (x.std()))\n",
    "df1 = df1.fillna(0)\n",
    "df2 = df2.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18225\n",
       "1     7320\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.Label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    1255\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.Label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df2p=df1[df1['Label']==0]\n",
    "df2pp=df2p.sample(n=None, frac=1255/18225, replace=False, weights=None, random_state=None, axis=0)\n",
    "df2=pd.concat([df2, df2pp])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    1255\n",
       "0    1255\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.Label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df1.append(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    19480\n",
       "1     8575\n",
       "dtype: int64"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df.drop(['Label'],axis=1) .values\n",
    "y = df.iloc[:, -1].values.reshape(-1,1)\n",
    "y=np.ravel(y)\n",
    "pd.Series(y).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature engineering (IG, FCBF, and KPCA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature selection by information gain (IG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import mutual_info_classif\n",
    "importances = mutual_info_classif(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# calculate the sum of importance scores\n",
    "f_list = sorted(zip(map(lambda x: round(x, 4), importances), features), reverse=True)\n",
    "Sum = 0\n",
    "fs = []\n",
    "for i in range(0, len(f_list)):\n",
    "    Sum = Sum + f_list[i][0]\n",
    "    fs.append(f_list[i][1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# select the important features from top to bottom until the accumulated importance reaches 90%\n",
    "f_list2 = sorted(zip(map(lambda x: round(x, 4), importances/Sum), features), reverse=True)\n",
    "Sum2 = 0\n",
    "fs = []\n",
    "for i in range(0, len(f_list2)):\n",
    "    Sum2 = Sum2 + f_list2[i][0]\n",
    "    fs.append(f_list2[i][1])\n",
    "    if Sum2>=0.9:\n",
    "        break        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_fs = df[fs].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28055, 50)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_fs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.34612159, -0.51326791, -0.44364535, ..., -0.11333586,\n",
       "        -0.13353417, -0.05349902],\n",
       "       [-0.3443274 , -0.51326791, -0.44364535, ..., -0.11333586,\n",
       "        -0.13353417, -0.05349902],\n",
       "       [-0.3443274 , -0.51326791, -0.44364535, ..., -0.11333586,\n",
       "        -0.13353417, -0.05349902],\n",
       "       ...,\n",
       "       [-0.36859622, -0.20454057, -0.32295149, ..., -0.11333586,\n",
       "        -0.13353417, -0.05349902],\n",
       "       [-0.3561313 ,  0.63721854,  0.36583358, ..., -0.11333586,\n",
       "        -0.13353417,  0.00459227],\n",
       "       [ 2.7318634 , -0.53347551, -0.44364535, ..., -0.11333586,\n",
       "        -0.13353417, -0.05349902]])"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_fs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature selection by Fast Correlation Based Filter (FCBF)\n",
    "\n",
    "The module is imported from the GitHub repo: https://github.com/SantiagoEG/FCBF_module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from FCBF_module import FCBF, FCBFK, FCBFiP, get_i\n",
    "fcbf = FCBFK(k = 20)\n",
    "#fcbf.fit(X_fs, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_fss = fcbf.fit_transform(X_fs,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28055, 20)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_fss.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.34612159, -0.53319222, -0.34935843, ..., -0.42229765,\n",
       "        -0.2803002 , -0.41947688],\n",
       "       [-0.3443274 , -0.54906516, -0.34935843, ..., -0.42229765,\n",
       "        -0.2803002 , -0.41947688],\n",
       "       [-0.3443274 , -0.55544206, -0.34935843, ..., -0.42229765,\n",
       "        -0.2803002 , -0.41947688],\n",
       "       ...,\n",
       "       [-0.36859622, -0.56375976, -0.34935843, ..., -0.42229765,\n",
       "        -0.2803002 , -0.32403604],\n",
       "       [-0.3561313 ,  0.00413109, -0.33807808, ..., -0.41021078,\n",
       "        -0.27174505,  0.36453998],\n",
       "       [ 2.7318634 , -0.53929186, -0.34935843, ..., -0.42229765,\n",
       "        -0.2803002 , -0.42271216]])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_fss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  kernel principal component analysis (KPCA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import KernelPCA\n",
    "kpca = KernelPCA(n_components = 10, kernel = 'rbf')\n",
    "kpca.fit(X_fss, y)\n",
    "X_kpca = kpca.transform(X_fss)\n",
    "\n",
    "# from sklearn.decomposition import PCA\n",
    "# kpca = PCA(n_components = 10)\n",
    "# kpca.fit(X_fss, y)\n",
    "# X_kpca = kpca.transform(X_fss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train-test split after feature selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = X_kpca[:len(df1)]\n",
    "y_train = y[:len(df1)]\n",
    "X_test = X_kpca[len(df1):]\n",
    "y_test = y[len(df1):]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solve class-imbalance by SMOTE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18225\n",
       "1     7320\n",
       "dtype: int64"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from imblearn.over_sampling import SMOTE\n",
    "smote=SMOTE(n_jobs=-1,sampling_strategy={1:18225})\n",
    "X_train, y_train = smote.fit_sample(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    18225\n",
       "0    18225\n",
       "dtype: int64"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    1255\n",
       "0    1255\n",
       "dtype: int64"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_test).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply the cluster labeling (CL) k-means method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "from sklearn.cluster import DBSCAN,MeanShift\n",
    "from sklearn.cluster import SpectralClustering,AgglomerativeClustering,AffinityPropagation,Birch,MiniBatchKMeans,MeanShift \n",
    "from sklearn.mixture import GaussianMixture, BayesianGaussianMixture\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def CL_kmeans(X_train, X_test, y_train, y_test,n,b=100):\n",
    "    km_cluster = MiniBatchKMeans(n_clusters=n,batch_size=b)\n",
    "    result = km_cluster.fit_predict(X_train)\n",
    "    result2 = km_cluster.predict(X_test)\n",
    "\n",
    "    count=0\n",
    "    a=np.zeros(n)\n",
    "    b=np.zeros(n)\n",
    "    for v in range(0,n):\n",
    "        for i in range(0,len(y_train)):\n",
    "            if result[i]==v:\n",
    "                if y_train[i]==1:\n",
    "                    a[v]=a[v]+1\n",
    "                else:\n",
    "                    b[v]=b[v]+1\n",
    "    list1=[]\n",
    "    list2=[]\n",
    "    for v in range(0,n):\n",
    "        if a[v]<=b[v]:\n",
    "            list1.append(v)\n",
    "        else: \n",
    "            list2.append(v)\n",
    "    for v in range(0,len(y_test)):\n",
    "        if result2[v] in list1:\n",
    "            result2[v]=0\n",
    "        elif result2[v] in list2:\n",
    "            result2[v]=1\n",
    "        else:\n",
    "            print(\"-1\")\n",
    "    print(classification_report(y_test, result2))\n",
    "    cm=confusion_matrix(y_test,result2)\n",
    "    acc=metrics.accuracy_score(y_test,result2)\n",
    "    print(str(acc))\n",
    "    print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.58      0.69      0.63      1255\n",
      "           1       0.62      0.51      0.56      1255\n",
      "\n",
      "    accuracy                           0.60      2510\n",
      "   macro avg       0.60      0.60      0.60      2510\n",
      "weighted avg       0.60      0.60      0.60      2510\n",
      "\n",
      "0.5984063745019921\n",
      "[[864 391]\n",
      " [617 638]]\n"
     ]
    }
   ],
   "source": [
    "CL_kmeans(X_train, X_test, y_train, y_test, 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter optimization of CL-k-means\n",
    "Tune \"k\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 0.6972111553784861\n",
      "43 0.7127490039840637\n",
      "43 0.399203187250996\n",
      "43 0.47051792828685257\n",
      "32 0.653784860557769\n",
      "20 0.34860557768924305\n",
      "16 0.9195219123505977\n",
      "5 0.4370517928286853\n",
      "15 0.6729083665338645\n",
      "25 0.7063745019920319\n",
      "2 0.47808764940239046\n",
      "50 0.4199203187250996\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Program Files\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:409: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 0.47768924302788845\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Program Files\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:409: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50 0.39282868525896414\n",
      "17 0.42828685258964144\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Program Files\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:409: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 0.47768924302788845\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Program Files\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:409: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 0.47768924302788845\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Program Files\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:409: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16 0.6992031872509961\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Program Files\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:409: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16 0.3737051792828685\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Program Files\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:409: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50 0.6250996015936255\n",
      "9.127083539962769\n",
      "Best score=0.9195\n",
      "Best parameters: n_clusters=16\n"
     ]
    }
   ],
   "source": [
    "#Hyperparameter optimization by BO-GP\n",
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "from sklearn import metrics\n",
    "\n",
    "space  = [Integer(2, 50, name='n_clusters')]\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    km_cluster = MiniBatchKMeans(batch_size=100, **params)\n",
    "    n=params['n_clusters']\n",
    "    \n",
    "    result = km_cluster.fit_predict(X_train)\n",
    "    result2 = km_cluster.predict(X_test)\n",
    "\n",
    "    count=0\n",
    "    a=np.zeros(n)\n",
    "    b=np.zeros(n)\n",
    "    for v in range(0,n):\n",
    "        for i in range(0,len(y_train)):\n",
    "            if result[i]==v:\n",
    "                if y_train[i]==1:\n",
    "                    a[v]=a[v]+1\n",
    "                else:\n",
    "                    b[v]=b[v]+1\n",
    "    list1=[]\n",
    "    list2=[]\n",
    "    for v in range(0,n):\n",
    "        if a[v]<=b[v]:\n",
    "            list1.append(v)\n",
    "        else: \n",
    "            list2.append(v)\n",
    "    for v in range(0,len(y_test)):\n",
    "        if result2[v] in list1:\n",
    "            result2[v]=0\n",
    "        elif result2[v] in list2:\n",
    "            result2[v]=1\n",
    "        else:\n",
    "            print(\"-1\")\n",
    "    cm=metrics.accuracy_score(y_test,result2)\n",
    "    print(str(n)+\" \"+str(cm))\n",
    "    return (1-cm)\n",
    "from skopt import gp_minimize\n",
    "import time\n",
    "t1=time.time()\n",
    "res_gp = gp_minimize(objective, space, n_calls=20, random_state=0)\n",
    "t2=time.time()\n",
    "print(t2-t1)\n",
    "print(\"Best score=%.4f\" % (1-res_gp.fun))\n",
    "print(\"\"\"Best parameters: n_clusters=%d\"\"\" % (res_gp.x[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23 0.34422310756972113\n",
      "15 0.6685258964143427\n",
      "46 0.450199203187251\n",
      "15 0.4896414342629482\n",
      "29 0.6824701195219124\n",
      "36 0.3888446215139442\n",
      "22 0.35776892430278884\n",
      "25 0.34860557768924305\n",
      "42 0.41832669322709165\n",
      "27 0.47051792828685257\n",
      "26 0.39402390438247015\n",
      "25 0.6824701195219124\n",
      "33 0.3848605577689243\n",
      "19 0.7191235059760956\n",
      "6 0.5824701195219123\n",
      "21 0.6697211155378486\n",
      "24 0.451394422310757\n",
      "37 0.4681274900398406\n",
      "14 0.47250996015936253\n",
      "21 0.8434262948207172\n",
      "100%|███████████████████████████████████████████████| 20/20 [00:06<00:00,  2.87trial/s, best loss: 0.15657370517928282]\n",
      "Random Forest: Hyperopt estimated optimum {'n_clusters': 21.0}\n"
     ]
    }
   ],
   "source": [
    "#Hyperparameter optimization by BO-TPE\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "from sklearn.cluster import MiniBatchKMeans\n",
    "from sklearn import metrics\n",
    "\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_clusters': int(params['n_clusters']), \n",
    "    }\n",
    "    km_cluster = MiniBatchKMeans(batch_size=100, **params)\n",
    "    n=params['n_clusters']\n",
    "    \n",
    "    result = km_cluster.fit_predict(X_train)\n",
    "    result2 = km_cluster.predict(X_test)\n",
    "\n",
    "    count=0\n",
    "    a=np.zeros(n)\n",
    "    b=np.zeros(n)\n",
    "    for v in range(0,n):\n",
    "        for i in range(0,len(y_train)):\n",
    "            if result[i]==v:\n",
    "                if y_train[i]==1:\n",
    "                    a[v]=a[v]+1\n",
    "                else:\n",
    "                    b[v]=b[v]+1\n",
    "    list1=[]\n",
    "    list2=[]\n",
    "    for v in range(0,n):\n",
    "        if a[v]<=b[v]:\n",
    "            list1.append(v)\n",
    "        else: \n",
    "            list2.append(v)\n",
    "    for v in range(0,len(y_test)):\n",
    "        if result2[v] in list1:\n",
    "            result2[v]=0\n",
    "        elif result2[v] in list2:\n",
    "            result2[v]=1\n",
    "        else:\n",
    "            print(\"-1\")\n",
    "    score=metrics.accuracy_score(y_test,result2)\n",
    "    print(str(params['n_clusters'])+\" \"+str(score))\n",
    "    return {'loss':1-score, 'status': STATUS_OK }\n",
    "space = {\n",
    "    'n_clusters': hp.quniform('n_clusters', 2, 50, 1),\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"Random Forest: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      0.90      0.94      1255\n",
      "           1       0.91      0.99      0.95      1255\n",
      "\n",
      "    accuracy                           0.95      2510\n",
      "   macro avg       0.95      0.95      0.94      2510\n",
      "weighted avg       0.95      0.95      0.94      2510\n",
      "\n",
      "0.9450199203187251\n",
      "[[1127  128]\n",
      " [  10 1245]]\n"
     ]
    }
   ],
   "source": [
    "CL_kmeans(X_train, X_test, y_train, y_test, 16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply the CL-k-means model with biased classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# needs to work on the entire dataset to generate sufficient training samples for biased classifiers\n",
    "def Anomaly_IDS(X_train, X_test, y_train, y_test,n,b=100):\n",
    "    # CL-kmeans\n",
    "    km_cluster = MiniBatchKMeans(n_clusters=n,batch_size=b)\n",
    "    result = km_cluster.fit_predict(X_train)\n",
    "    result2 = km_cluster.predict(X_test)\n",
    "\n",
    "    count=0\n",
    "    a=np.zeros(n)\n",
    "    b=np.zeros(n)\n",
    "    for v in range(0,n):\n",
    "        for i in range(0,len(y_train)):\n",
    "            if result[i]==v:\n",
    "                if y_train[i]==1:\n",
    "                    a[v]=a[v]+1\n",
    "                else:\n",
    "                    b[v]=b[v]+1\n",
    "    list1=[]\n",
    "    list2=[]\n",
    "    for v in range(0,n):\n",
    "        if a[v]<=b[v]:\n",
    "            list1.append(v)\n",
    "        else: \n",
    "            list2.append(v)\n",
    "    for v in range(0,len(y_test)):\n",
    "        if result2[v] in list1:\n",
    "            result2[v]=0\n",
    "        elif result2[v] in list2:\n",
    "            result2[v]=1\n",
    "        else:\n",
    "            print(\"-1\")\n",
    "    print(classification_report(y_test, result2))\n",
    "    cm=confusion_matrix(y_test,result2)\n",
    "    acc=metrics.accuracy_score(y2,result2)\n",
    "    print(str(acc))\n",
    "    print(cm)\n",
    "    \n",
    "    #Biased classifier construction\n",
    "    count=0\n",
    "    print(len(y))\n",
    "    a=np.zeros(n)\n",
    "    b=np.zeros(n)\n",
    "    FNL=[]\n",
    "    FPL=[]\n",
    "    for v in range(0,n):\n",
    "        al=[]\n",
    "        bl=[]\n",
    "        for i in range(0,len(y)):   \n",
    "            if result[i]==v:        \n",
    "                if y[i]==1:        #label 1\n",
    "                    a[v]=a[v]+1\n",
    "                    al.append(i)\n",
    "                else:             #label 0\n",
    "                    b[v]=b[v]+1\n",
    "                    bl.append(i)\n",
    "        if a[v]<=b[v]:\n",
    "            FNL.extend(al)\n",
    "        else:\n",
    "            FPL.extend(bl)\n",
    "        #print(str(v)+\"=\"+str(a[v]/(a[v]+b[v])))\n",
    "        \n",
    "    dffp=df.iloc[FPL, :]\n",
    "    dffn=df.iloc[FNL, :]\n",
    "    dfva0=df[df['Label']==0]\n",
    "    dfva1=df[df['Label']==1]\n",
    "    \n",
    "    dffpp=dfva1.sample(n=None, frac=len(FPL)/dfva1.shape[0], replace=False, weights=None, random_state=None, axis=0)\n",
    "    dffnp=dfva0.sample(n=None, frac=len(FNL)/dfva0.shape[0], replace=False, weights=None, random_state=None, axis=0)\n",
    "    \n",
    "    dffp_f=pd.concat([dffp, dffpp])\n",
    "    dffn_f=pd.concat([dffn, dffnp])\n",
    "    \n",
    "    Xp = dffp_f.drop(['Label'],axis=1)  \n",
    "    yp = dffp_f.iloc[:, -1].values.reshape(-1,1)\n",
    "    yp=np.ravel(yp)\n",
    "\n",
    "    Xn = dffn_f.drop(['Label'],axis=1)  \n",
    "    yn = dffn_f.iloc[:, -1].values.reshape(-1,1)\n",
    "    yn=np.ravel(yn)\n",
    "    \n",
    "    rfp = RandomForestClassifier(random_state = 0)\n",
    "    rfp.fit(Xp,yp)\n",
    "    rfn = RandomForestClassifier(random_state = 0)\n",
    "    rfn.fit(Xn,yn)\n",
    "\n",
    "    dffnn_f=pd.concat([dffn, dffnp])\n",
    "    \n",
    "    Xnn = dffn_f.drop(['Label'],axis=1)  \n",
    "    ynn = dffn_f.iloc[:, -1].values.reshape(-1,1)\n",
    "    ynn=np.ravel(ynn)\n",
    "\n",
    "    rfnn = RandomForestClassifier(random_state = 0)\n",
    "    rfnn.fit(Xnn,ynn)\n",
    "\n",
    "    X2p = df2.drop(['Label'],axis=1) \n",
    "    y2p = df2.iloc[:, -1].values.reshape(-1,1)\n",
    "    y2p=np.ravel(y2p)\n",
    "\n",
    "    result2 = km_cluster.predict(X2p)\n",
    "\n",
    "    count=0\n",
    "    a=np.zeros(n)\n",
    "    b=np.zeros(n)\n",
    "    for v in range(0,n):\n",
    "        for i in range(0,len(y)):\n",
    "            if result[i]==v:\n",
    "                if y[i]==1:\n",
    "                    a[v]=a[v]+1\n",
    "                else:\n",
    "                    b[v]=b[v]+1\n",
    "    list1=[]\n",
    "    list2=[]\n",
    "    l1=[]\n",
    "    l0=[]\n",
    "    for v in range(0,n):\n",
    "        if a[v]<=b[v]:\n",
    "            list1.append(v)\n",
    "        else: \n",
    "            list2.append(v)\n",
    "    for v in range(0,len(y2p)):\n",
    "        if result2[v] in list1:\n",
    "            result2[v]=0\n",
    "            l0.append(v)\n",
    "        elif result2[v] in list2:\n",
    "            result2[v]=1\n",
    "            l1.append(v)\n",
    "        else:\n",
    "            print(\"-1\")\n",
    "    print(classification_report(y2p, result2))\n",
    "    cm=confusion_matrix(y2p,result2)\n",
    "    print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More details are in the paper"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}