MLflowDemo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af5d1fcb",
   "metadata": {},
   "source": [
    "## What is MLFlow and its Components\n",
    "\n",
    "MLFLow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry. MLFLow currently offers four components:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee3e15be",
   "metadata": {},
   "source": [
    "<img src=\"mlflow.png\">\n",
    "\n",
    "\n",
    "https://www.mlflow.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9d61a2b",
   "metadata": {},
   "source": [
    "_First thing first_\n",
    "### Create Conda environment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d384e41",
   "metadata": {},
   "source": [
    "##### run below commands in terminal but make sure conda is installed or use anaconda prompt which you will get as part of anaconda installation\n",
    "\n",
    "1. `conda create -n envname python=3.9 ipykernel` \n",
    "it will create a conda env named envname and install python version 3.9 and a ipykernel inside this environment\n",
    "\n",
    "2. Activate the environment\n",
    "`conda activate envname`\n",
    "\n",
    "3. add newly created environment to the notebook as kernel\n",
    "`python -m ipykernel install --user --name=envname` \n",
    "\n",
    "4. install notebook inside the environment\n",
    "`pip install notebook`\n",
    "\n",
    "5. Now install all required dependencies to run this notebook\n",
    "\n",
    "* `pip install pandas`\n",
    "* `pip install numpy`\n",
    "* `pip install scikit-learn`\n",
    "* `pip install imblearn`\n",
    "* `pip install matplotlib`\n",
    "* `pip install mlflow`\n",
    "\n",
    "Now open the notebook using below command: (from the anaconda prompt inside conda environment)\n",
    "\n",
    "`jupyter notebook`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80019380",
   "metadata": {},
   "source": [
    "#### Make sure python is used from your newly created environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ef8dc001",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c:\\Users\\lokeshkumar\\python_envs\\dev\\Scripts\\python.exe\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(sys.executable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "aef7fe14",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.10.0\n"
     ]
    }
   ],
   "source": [
    "!python --version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30694612",
   "metadata": {},
   "source": [
    "### Create functions for all the steps involved in complete model training lifecycle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d293e8b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "34494947",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2c3dcd10",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data(path):\n",
    "    data = pd.read_csv(path)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "859af0ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>month</th>\n",
       "      <th>day_of_week</th>\n",
       "      <th>...</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>emp_var_rate</th>\n",
       "      <th>cons_price_idx</th>\n",
       "      <th>cons_conf_idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr_employed</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>44</td>\n",
       "      <td>blue-collar</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.4y</td>\n",
       "      <td>unknown</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>aug</td>\n",
       "      <td>thu</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.444</td>\n",
       "      <td>-36.1</td>\n",
       "      <td>4.963</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>53</td>\n",
       "      <td>technician</td>\n",
       "      <td>married</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>nov</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>93.200</td>\n",
       "      <td>-42.0</td>\n",
       "      <td>4.021</td>\n",
       "      <td>5195.8</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28</td>\n",
       "      <td>management</td>\n",
       "      <td>single</td>\n",
       "      <td>university.degree</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>jun</td>\n",
       "      <td>thu</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>success</td>\n",
       "      <td>-1.7</td>\n",
       "      <td>94.055</td>\n",
       "      <td>-39.8</td>\n",
       "      <td>0.729</td>\n",
       "      <td>4991.6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>39</td>\n",
       "      <td>services</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>apr</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>93.075</td>\n",
       "      <td>-47.1</td>\n",
       "      <td>1.405</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55</td>\n",
       "      <td>retired</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.4y</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>aug</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>success</td>\n",
       "      <td>-2.9</td>\n",
       "      <td>92.201</td>\n",
       "      <td>-31.4</td>\n",
       "      <td>0.869</td>\n",
       "      <td>5076.2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age          job  marital          education  default housing loan  \\\n",
       "0   44  blue-collar  married           basic.4y  unknown     yes   no   \n",
       "1   53   technician  married            unknown       no      no   no   \n",
       "2   28   management   single  university.degree       no     yes   no   \n",
       "3   39     services  married        high.school       no      no   no   \n",
       "4   55      retired  married           basic.4y       no     yes   no   \n",
       "\n",
       "    contact month day_of_week  ...  campaign  pdays  previous     poutcome  \\\n",
       "0  cellular   aug         thu  ...         1    999         0  nonexistent   \n",
       "1  cellular   nov         fri  ...         1    999         0  nonexistent   \n",
       "2  cellular   jun         thu  ...         3      6         2      success   \n",
       "3  cellular   apr         fri  ...         2    999         0  nonexistent   \n",
       "4  cellular   aug         fri  ...         1      3         1      success   \n",
       "\n",
       "  emp_var_rate  cons_price_idx  cons_conf_idx  euribor3m  nr_employed  y  \n",
       "0          1.4          93.444          -36.1      4.963       5228.1  0  \n",
       "1         -0.1          93.200          -42.0      4.021       5195.8  0  \n",
       "2         -1.7          94.055          -39.8      0.729       4991.6  1  \n",
       "3         -1.8          93.075          -47.1      1.405       5099.1  0  \n",
       "4         -2.9          92.201          -31.4      0.869       5076.2  1  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = load_data('banking.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "764ff655",
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_cleaning(data):\n",
    "    print(\"na values available in data \\n\")\n",
    "    print(data.isna().sum())\n",
    "    data = data.dropna()\n",
    "    print(\"after droping na values \\n\")\n",
    "    print(data.isna().sum())\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8d4a65a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessing(data):\n",
    "    data['education']=np.where(data['education'] =='basic.9y', 'Basic', data['education'])\n",
    "    data['education']=np.where(data['education'] =='basic.6y', 'Basic', data['education'])\n",
    "    data['education']=np.where(data['education'] =='basic.4y', 'Basic', data['education'])\n",
    "    \n",
    "    cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']\n",
    "    for var in cat_vars:\n",
    "        cat_list='var'+'_'+var\n",
    "        cat_list = pd.get_dummies(data[var], prefix=var)\n",
    "        data1=data.join(cat_list)\n",
    "        data=data1\n",
    "\n",
    "    cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']\n",
    "    data_vars=data.columns.values.tolist()\n",
    "    to_keep=[i for i in data_vars if i not in cat_vars]\n",
    "    \n",
    "    final_data=data[to_keep]\n",
    "    \n",
    "    \n",
    "    final_data.columns = final_data.columns.str.replace('.','_')\n",
    "    final_data.columns = final_data.columns.str.replace(' ','_')\n",
    "    return final_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c24542e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_test_split(final_data):\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    X = final_data.loc[:, final_data.columns != 'y']\n",
    "    y = final_data.loc[:, final_data.columns == 'y']\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,stratify = y, random_state=47)\n",
    "    return X_train, X_test, y_train, y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "db42c382",
   "metadata": {},
   "outputs": [],
   "source": [
    "def over_sampling_target_class(X_train, y_train):\n",
    "    ### Over-sampling using SMOTE \n",
    "    from imblearn.over_sampling import SMOTE\n",
    "    os = SMOTE(random_state=0)\n",
    "\n",
    "    columns = X_train.columns\n",
    "    os_data_X,os_data_y=os.fit_resample(X_train, y_train)\n",
    "\n",
    "    os_data_X = pd.DataFrame(data=os_data_X,columns=columns )\n",
    "    os_data_y= pd.DataFrame(data=os_data_y,columns=['y'])\n",
    "    # we can Check the numbers of our data\n",
    "    print(\"length of oversampled data is \",len(os_data_X))\n",
    "    print(\"Number of no subscription in oversampled data\",len(os_data_y[os_data_y['y']==0]))\n",
    "    print(\"Number of subscription\",len(os_data_y[os_data_y['y']==1]))\n",
    "    print(\"Proportion of no subscription data in oversampled data is \",len(os_data_y[os_data_y['y']==0])/len(os_data_X))\n",
    "    print(\"Proportion of subscription data in oversampled data is \",len(os_data_y[os_data_y['y']==1])/len(os_data_X))\n",
    "    \n",
    "    X_train = os_data_X\n",
    "    y_train = os_data_y['y']\n",
    " \n",
    "    return X_train, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ff14ff21",
   "metadata": {},
   "outputs": [],
   "source": [
    "def training_basic_classifier(X_train,y_train):\n",
    "    from sklearn.ensemble import RandomForestClassifier\n",
    "    model = RandomForestClassifier(n_estimators=101)\n",
    "    model.fit(X_train, y_train)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ce906745",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_on_test_data(model,X_test):\n",
    "    y_pred = model.predict(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6c5244a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_prob_on_test_data(model,X_test):\n",
    "    y_pred = model.predict_proba(X_test)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "08e008cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_metrics(y_true, y_pred, y_pred_prob):\n",
    "    from sklearn.metrics import accuracy_score,precision_score,recall_score,log_loss\n",
    "    acc = accuracy_score(y_true, y_pred)\n",
    "    prec = precision_score(y_true, y_pred)\n",
    "    recall = recall_score(y_true, y_pred)\n",
    "    entropy = log_loss(y_true, y_pred_prob)\n",
    "    return {'accuracy': round(acc, 2), 'precision': round(prec, 2), 'recall': round(recall, 2), 'entropy': round(entropy, 2)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e6bb8fa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_roc_auc_plot(y_true, y_pred):\n",
    "    import matplotlib.pyplot as plt\n",
    "    from sklearn import metrics     \n",
    "    fpr, tpr, thresholds = metrics.roc_curve(y_true, y_pred)\n",
    "    roc_auc = metrics.auc(fpr, tpr)\n",
    "    display = metrics.RocCurveDisplay(fpr=fpr, tpr=tpr, roc_auc=roc_auc)\n",
    "    display.plot()\n",
    "    plt.savefig('roc_auc_curve.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "126b782e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_confusion_matrix_plot(clf, y_test, y_pred):\n",
    "    import matplotlib.pyplot as plt\n",
    "    from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "    cm = confusion_matrix(y_test, y_pred, labels=clf.classes_)\n",
    "    disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=clf.classes_)\n",
    "    disp.plot()\n",
    "    plt.savefig('confusion_matrix.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "86d0d9fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hyper_parameter_tuning(X_train, y_train):\n",
    "    # define random parameters grid\n",
    "    n_estimators = [5,21,51,101] # number of trees in the random forest\n",
    "    max_features = ['auto', 'sqrt'] # number of features in consideration at every split\n",
    "    max_depth = [int(x) for x in np.linspace(10, 120, num = 12)] # maximum number of levels allowed in each decision tree\n",
    "    min_samples_split = [2, 6, 10] # minimum sample number to split a node\n",
    "    min_samples_leaf = [1, 3, 4] # minimum sample number that can be stored in a leaf node\n",
    "    bootstrap = [True, False] # method used to sample data points\n",
    "\n",
    "    random_grid = {'n_estimators': n_estimators,\n",
    "                    'max_features': max_features,\n",
    "                    'max_depth': max_depth,\n",
    "                    'min_samples_split': min_samples_split,\n",
    "                    'min_samples_leaf': min_samples_leaf,\n",
    "                    'bootstrap': bootstrap\n",
    "                  }\n",
    "    \n",
    "    from sklearn.model_selection import RandomizedSearchCV\n",
    "    from sklearn.ensemble import RandomForestClassifier\n",
    "    classifier = RandomForestClassifier()\n",
    "    model_tuning = RandomizedSearchCV(estimator = classifier, param_distributions = random_grid,\n",
    "                   n_iter = 100, cv = 5, verbose=2, random_state=35, n_jobs = -1)\n",
    "    model_tuning.fit(X_train, y_train)\n",
    "\n",
    "    print ('Random grid: ', random_grid, '\\n')\n",
    "    # print the best parameters\n",
    "    print ('Best Parameters: ', model_tuning.best_params_, ' \\n')\n",
    "\n",
    "    best_params = model_tuning.best_params_\n",
    "    \n",
    "    n_estimators = best_params['n_estimators']\n",
    "    min_samples_split = best_params['min_samples_split']\n",
    "    min_samples_leaf = best_params['min_samples_leaf']\n",
    "    max_features = best_params['max_features']\n",
    "    max_depth = best_params['max_depth']\n",
    "    bootstrap = best_params['bootstrap']\n",
    "    \n",
    "    model_tuned = RandomForestClassifier(n_estimators = n_estimators, min_samples_split = min_samples_split,\n",
    "                                         min_samples_leaf= min_samples_leaf, max_features = max_features,\n",
    "                                         max_depth= max_depth, bootstrap=bootstrap) \n",
    "    model_tuned.fit( X_train, y_train)\n",
    "    return model_tuned,best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97f39019",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "074489bd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data = load_data('banking.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "55f182ec",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "na values available in data \n",
      "\n",
      "age               0\n",
      "job               0\n",
      "marital           0\n",
      "education         0\n",
      "default           0\n",
      "housing           0\n",
      "loan              0\n",
      "contact           0\n",
      "month             0\n",
      "day_of_week       0\n",
      "duration          0\n",
      "campaign          0\n",
      "pdays             0\n",
      "previous          0\n",
      "poutcome          0\n",
      "emp_var_rate      0\n",
      "cons_price_idx    0\n",
      "cons_conf_idx     0\n",
      "euribor3m         0\n",
      "nr_employed       0\n",
      "y                 0\n",
      "dtype: int64\n",
      "after droping na values \n",
      "\n",
      "age               0\n",
      "job               0\n",
      "marital           0\n",
      "education         0\n",
      "default           0\n",
      "housing           0\n",
      "loan              0\n",
      "contact           0\n",
      "month             0\n",
      "day_of_week       0\n",
      "duration          0\n",
      "campaign          0\n",
      "pdays             0\n",
      "previous          0\n",
      "poutcome          0\n",
      "emp_var_rate      0\n",
      "cons_price_idx    0\n",
      "cons_conf_idx     0\n",
      "euribor3m         0\n",
      "nr_employed       0\n",
      "y                 0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "cleaned_data = data_cleaning(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "86948bdc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\lokeshkumar\\AppData\\Local\\Temp\\ipykernel_4948\\4067079169.py:20: FutureWarning: The default value of regex will change from True to False in a future version. In addition, single character regular expressions will *not* be treated as literal strings when regex=True.\n",
      "  final_data.columns = final_data.columns.str.replace('.','_')\n"
     ]
    }
   ],
   "source": [
    "final_data = preprocessing(cleaned_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b5f6e5bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(final_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4ce5b8d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of oversampled data is  51166\n",
      "Number of no subscription in oversampled data 25583\n",
      "Number of subscription 25583\n",
      "Proportion of no subscription data in oversampled data is  0.5\n",
      "Proportion of subscription data in oversampled data is  0.5\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train = over_sampling_target_class(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7994c6cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = training_basic_classifier(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d4f37883",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = predict_on_test_data(model,X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fc4431da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 0], dtype=int64)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6a35dc4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_prob = predict_prob_on_test_data(model,X_test) #model.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "75429e45",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 0.        ],\n",
       "       [0.97029703, 0.02970297],\n",
       "       [0.89108911, 0.10891089],\n",
       "       ...,\n",
       "       [1.        , 0.        ],\n",
       "       [0.81188119, 0.18811881],\n",
       "       [1.        , 0.        ]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c19e0010",
   "metadata": {},
   "outputs": [],
   "source": [
    "run_metrics = get_metrics(y_test, y_pred, y_pred_prob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "d76f0b28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'accuracy': 0.91, 'precision': 0.62, 'recall': 0.52, 'entropy': 0.2}\n"
     ]
    }
   ],
   "source": [
    "print(run_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7b15bd57",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQKUlEQVR4nO3deXhTZdoG8Dtpm3RvKaVrgmXfoTRIB1AZoFpAEaRVVITKKI4KyNBxARUQFXBcEEcYGRVE/ESQlk3ZlCIKiAOklJ0itEh3KEvTPW3yfn+Epo0UaNosbXL/rivXZd6ckzw5gnk89/ueIxFCCBARERE5CKm9CyAiIiKyJDY3RERE5FDY3BAREZFDYXNDREREDoXNDRERETkUNjdERETkUNjcEBERkUNxtXcBtqbX65GbmwsfHx9IJBJ7l0NEREQNIIRAcXExwsLCIJXe+tyM0zU3ubm5UCqV9i6DiIiIGiErKwsKheKW2zhdc+Pj4wPAcHB8fX3tXA0RERE1hEajgVKpNP6O34rTNTc1UZSvry+bGyIiohamIVNKOKGYiIiIHAqbGyIiInIobG6IiIjIobC5ISIiIofC5oaIiIgcCpsbIiIicihsboiIiMihsLkhIiIih8LmhoiIiBwKmxsiIiJyKHZtbn755ReMGjUKYWFhkEgk2Lhx42332b17N6KioiCXy9GxY0esXLnS6nUSERFRy2HX5qa0tBR9+vTB0qVLG7R9ZmYm7r//fgwZMgRpaWn4xz/+gaeffho7duywcqVERETUUtj1xpkjRozAiBEjGrz9smXL0K5dO3zwwQcAgG7dumHv3r348MMPERsba60yiYiIqIFyrpWjXKtDxyBvu9XQou4Kvn//fsTExJiMxcbG4h//+MdN96msrERlZaXxuUajsVZ5RERETqlcq8OOE/lIUmdj37lCxHQLxmcT+9mtnhbV3OTn5yM4ONhkLDg4GBqNBuXl5fDw8Lhhn4ULF2LevHm2KpGIiMgpCCFw6I+rSDqUjS3H8lBSWW18raJKB51ewEUqsUttLaq5aYxZs2YhMTHR+Fyj0UCpVNqxIiIiopYr51o51quzkZyajfOXy4zjygAPxEcpMTYqHMoATztW2MKam5CQEBQUFJiMFRQUwNfXt96zNgAgl8shl8ttUR4REZFDKtNWY/vxfCSnZuPXc5chhGHcS+aCkb1CEa9S4M6IAEjtdKbmz1pUczNgwABs3brVZOzHH3/EgAED7FQRERGRYxJC4OD5q0hSZ2HL0TyUanXG1wZ2aI14lQLDe4bAU9b8Wgm7VlRSUoKzZ88an2dmZiItLQ0BAQFo27YtZs2ahZycHKxatQoA8Oyzz2LJkiV4+eWX8be//Q27du3Ct99+iy1bttjrKxARETmU7KtlWJ+agyR1Ni5cqY2d2gZ4Il6lwNiocCha2Td2uh27NjeHDh3CkCFDjM9r5sYkJCRg5cqVyMvLw4ULF4yvt2vXDlu2bMGMGTPw0UcfQaFQ4PPPP+cycCIioiYo01Zj2zHDaqf9GZeN414yFzzQOwxxKgXujGgFiaR5xE63IxGiJjlzDhqNBn5+figqKoKvr6+9yyEiIrILvV7g4PkrSFJnY+ux2thJIqmNnWJ7NJ/YyZzf7+ZRMREREdlE1pUyJKcaVjtlXSk3jke0NsROD0UpEO5f/yKdloLNDRERkYMrrazGtuP5SFJn4beMK8Zxb7krHuhtWO2kuqPlxE63w+aGiIjIAen1Av/LNMRO247noaxO7HRXx0DERRliJw+Zi50rtTw2N0RERA7kwuXa2Cn7am3s1C7QyxA79Q1HWAuPnW6HzQ0REVELV1pZja3H8pCkzsb/MmtjJx+5Kx7oY4idoto6Tux0O2xuiIiIWiC9XuC3zMuG2OlYPsqrTGOnmtVO7m6OFzvdDpsbIiKiFuSPy6VITs1BsjobOddqY6f2gV6Iu36RvVA/x46dbofNDRERUTNXUlmNrUcNsdOB83ViJ3dXjOoThniVAn2V/k4TO90OmxsiIqJmSK8X+C3jeux0vDZ2kkqAuzq1QbxKgfu6Bztl7HQ7bG6IiIiakfOFpUhOzcb61BzT2KmNFx5WKfFQ33CE+LnbscLmj80NERGRnRVXVBlXOx08f9U47uPuigevx06RjJ0ajM0NERGRHej1Ar+eu4zkVMNF9iqq9AAMsdPd12Onexk7NQqbGyIiIhvKLCxFsjob61OzkVtUYRzvGORtvMhesC9jp6Zgc0NERGRlmooqbDmah2R1Ng79URs7+bq74sHIMMSrlOij8GPsZCFsboiIiKxApxf49VwhktTZ2H48H5XVtbHT4M5tEK9SYli3IMZOVsDmhoiIyIIyLpUYVzvl1YmdOtWJnYIYO1kVmxsiIqIm0lRU4fsjeUhOzYa6Tuzk5+GG0ZGG1U69whk72QqbGyIiokbQ6QX2nTXETjtO1MZOLlLJ9dhJgWHdgiB3Zexka2xuiIiIzHD2oiF22pCag3xNbezUOdgQO42JZOxkb2xuiIiIbqOovArfH81Fkjobhy9cM477e7phdB/Daqee4b6MnZoJNjdERET10OkF9vx+CUnqbPxwsgDaOrHTkC5tEBelwFDGTs0SmxsiIqI6zl4sRpI6BxsOZ6NAU2kc7xLsg4f7KTA6MhxtfOR2rJBuh80NERE5vaKyKnx3PXZKy7pmHPf3dMOYyHDEqxToEcbYqaVgc0NERE6pWqfHnuurnX68IXYKQrwqHEO6MnZqidjcEBGRU/m9oBhJ11c7XSyujZ26hvggXsXYyRGwuSEiIod3rUyL744YYqcj2UXG8QAvGUZHhiEuirGTI2FzQ0REDqlap8ee3+vETjpD7OQqlWBI1yDEqxQY0iUIMlepnSslS2NzQ0REDuVMQTGS1NnYcDgHl+rETt1DfRGnUmB0ZBgCvRk7OTI2N0RE1OJdK9Ni8/XY6eifYqcxkeGIU4WjR5ifHSskW2JzQ0RELVK1To9frl9kb+fJiyax09DrsdNfGTs5JTY3RETUoqTnFyNJnYUNh3NRWFIbO/UI80W8SoEH+4ShNWMnp8bmhoiImr2rpbWx07Gc2tiptZcMY/qGIy5Kge5hvnaskJoTNjdERNQsVen0+DndEDulnC5AlU4AANxcJBjWNRjxKgUGd2kDNxfGTmSKzQ0RETUrp/I0SFZnY2NaDgpLtMbxnuG+iI9S4MHIcAR4yexYITV3bG6IiMjurpRqsSktB0nqbJzI1RjHA71leKhvOOJUCnQNYexEDcPmhoiI7KJKp8fu9EtIUmdh1+mLJrFTTDdD7HRPZ8ZOZD42N0REZFMnczVITs3GxsM5uFxaGzv1CvczrnZqxdiJmoDNDRERWd3lkkpsSjOsdjqZVzd2kmNslGG1U5cQHztWSI6EzQ0REVmFtlqPn9IvIlmdjV2nL6Jab4idZC5SxHQ3XGTvnk5t4MrYiSyMzQ0REVnUidwiJKmzsSktF1fqxE59FIbYaVSfMPh7MnYi62FzQ0RETVZYJ3Y6VSd2auMjx9jrq506BzN2Ittgc0NERI2irdZj1+mLSE7Nxk9/ip3u7W5Y7XR3p0DGTmRzbG6IiKjBhBA4kau5Hjvl4GpZlfG1Pkp/Q+zUO5SxE9kVmxsiIrqtS8WVxovsnc4vNo4H+cjxUFQ44qMU6MTYiZoJNjdERFQvQ+xUgCR1Nn5KvwRdTezkKsV912OnuzoydqLmh80NEREZCSFwPEeDJHUWNh/JNYmd+rb1R1yUAqN6h8HP082OVRLdGpsbIiLCxeIKbDpsWO2UXlAbOwX7yjE2SoG4KAU6BnnbsUKihmNzQ0TkpCqrddh16iKS1NnYfcY0dortEWKMnVykEjtXSmQeNjdERE5ECIFjOYaL7G0+kotrdWKnqLb+iFMp8EDvMPh5MHailovNDRGRE7ioqcDG66udzhSUGMdDfN0N93ZSKdChDWMncgxsboiIHFRFlQ4ppy4iSZ2FX34vNMZOclcphvcMQVyUAoMYO5EDYnNDRORAhBA4ml0bOxWV18ZOqjtaIV6lwP29Q+HrztiJHBebGyIiB1CgqcCGw4bY6ezF2tgp1M8dcVEKjI0KR3vGTuQk2NwQEbVQFVU67DxluMjeL2cu4XrqBLmrFCN6hiBepcSADq0ZO5HTYXNDRNSCCCFwJLvIcJG9tFxoKqqNr/W7HjuNZOxETo7NDRFRC5BfVBM7ZeHcpVLjeJifO+JUCoyNUqBdoJcdKyRqPtjcEBE1UxVVOvx40hA77fm9NnZyd5NiRM9QxKsUGNC+NaSMnYhM2P1uZ0uXLkVERATc3d0RHR2NAwcO3HL7xYsXo0uXLvDw8IBSqcSMGTNQUVFho2qJiKxLCIHUC1fx6oZjuHP+Tkz75jB+vj6fpn9EAN6N642Dr8Xgw3GRGNQxkI0NUT3seuZm7dq1SExMxLJlyxAdHY3FixcjNjYW6enpCAoKumH71atXY+bMmVixYgUGDhyIM2fO4Mknn4REIsGiRYvs8A2IiCwjv6gC6w9nI0mdjYw6sVO4vwfiosIxNkqBCMZORA0iEUIIe314dHQ07rzzTixZsgQAoNfroVQqMW3aNMycOfOG7adOnYpTp04hJSXFOPbPf/4T//vf/7B37956P6OyshKVlZXG5xqNBkqlEkVFRfD19bXwNyIiariKKh12nMhHkjobe88Woua/xh5uLhjRy3Bvp7+0Y+xEBBh+v/38/Br0+223MzdarRZqtRqzZs0yjkmlUsTExGD//v317jNw4ED83//9Hw4cOID+/fsjIyMDW7duxYQJE276OQsXLsS8efMsXj8RUWMYYqdrSFJn4/ujuSius9qpf7sAw2qnXqHwlnNKJFFj2e1vT2FhIXQ6HYKDg03Gg4ODcfr06Xr3efzxx1FYWIi77roLQghUV1fj2WefxauvvnrTz5k1axYSExONz2vO3BAR2VJeUTnWp+YgWZ2NjMI/xU4qBeKiwnFHa8ZORJbQov7XYPfu3ViwYAH+85//IDo6GmfPnsX06dPx1ltvYfbs2fXuI5fLIZfLbVwpERFQrtXhh5P1x04jexlWO0W3C2DsRGRhdmtuAgMD4eLigoKCApPxgoIChISE1LvP7NmzMWHCBDz99NMAgF69eqG0tBTPPPMMXnvtNUildl/8RUROTggB9R9XkZyaje+P5KG4sjZ2ir4eO41g7ERkVXb72yWTyaBSqZCSkoIxY8YAMEwoTklJwdSpU+vdp6ys7IYGxsXFBYDhPyhERPaSc60cG1KzkZyag8w6sZMywANxUQrERSmgDPC0Y4VEzsOu/+uQmJiIhIQE9OvXD/3798fixYtRWlqKSZMmAQAmTpyI8PBwLFy4EAAwatQoLFq0CH379jXGUrNnz8aoUaOMTQ4Rka2Ua2tXO+07Vxs7ecpqY6f+EYydiGzNrs3NuHHjcOnSJcyZMwf5+fmIjIzE9u3bjZOML1y4YHKm5vXXX4dEIsHrr7+OnJwctGnTBqNGjcL8+fPt9RWIyMkIIXDoj6tIOpSNLcfyUFIndhrQvjXiVAqM6BkCL8ZORHZj1+vc2IM56+SJiGrkXCvHenU2klOzcf5ymXFcGeCB+CglxkaFM3YisqIWcZ0bIqLmrkxbje3H85Gcmo1fz102xk5edWKnOxk7ETU7bG6IiOoQQuDg+atIUmdhy9E8lGp1xtcGdmiNeJUCw3uGwFPG/3wSNVf820lEBCDrSpnhInup2bhwpTZ2uqO1J+KiFBgbFQ5FK8ZORC0Bmxsiclpl2mpsO2ZY7bQ/47Jx3Evmggd6hyG+nwL97mgFiYSxE1FLwuaGiJyKXi9w8PwVJKmzsfVYbewkkdTGTrE9GDsRtWT820tETiHrShmSUw2rnbKulBvHI1p7Il6lwENRCoT7e9ixQiKyFDY3ROSwSiurse14PpLUWfgt44px3Fvuigd6G1Y7qRg7ETkcNjdE5FD0eoH/ZRpip23H81BWJ3a6q2Mg4qIMsZOHjFc1J3JUbG6IyCFcuFwbO2VfrY2d2gV6GWKnvuEIY+xE5BTY3BBRi1VaWY2tx/KQpM7G/zJrYycfuSse6BOGeFU4otoydiJyNmxuiKhF0esFfsu8bIidjuWjvMo0dqpZ7eTuxtiJyFmxuSGiFuGPy6VITs1BsjobOddqY6f2gV6IUxkushfqx9iJiNjcEFEzVlJZja1HDbHTgfN1Yid3V4zqE4Z4lQJ9lf6MnYjIBJsbImpW9HqB3zKux07Ha2MnqQS4q1MbxKsUuK97MGMnIropNjdE1CycLyxFcmo21qfmmMROHdp4IV6lxEN9wxHi527HComopWBzQ0R2U1xRZVztdPD8VeO4j7srHrweO0UydiIiM7G5ISKb0usFfj13GUnqLGw/kY+KKj0AQ+x0T2dD7BTTjbETETUemxsisonMwlIkq7OxPjUbuUUVxvGOQd7Gi+wF+zJ2IqKmY3NDRFajqajClqN5SFZn49AftbGTr7srHowMQ7xKiT4KP8ZORGRRbG6IyKJ0eoFfzxUiSZ2N7cfzUVldGzsN7twG8SolhnULYuxERFbD5oaILCLjUolxtVNendipU53YKYixExHZAJsbImq0onJD7JSkzkLqhWvGcT8PN4yONKx26hXO2ImIbIvNDRGZRacX2HfWEDvtOFEbO7lIJddjJwWGdQuC3JWxExHZB5sbImqQsxcNsdOG1Bzka2pjp87BhthpTCRjJyJqHtjcENFNFZVX4fujuUhSZ+NwndjJ39MNo/sYVjv1DPdl7EREzQqbGyIyodML7Pn9EpLU2fjhZAG0dWKnIV3aIC5KgaGMnYioGWNzQ0QAgLMXi5GkzsGGw9ko0FQax7sE++DhfgqMjgxHGx+5HSskImoYNjdETqyorAqbj+YiWZ2NtKxrxvFWnm4YHRmOeJUCPcIYOxFRy8LmhsjJVOv02HN9tdOPN8ROQYhXKTC0axBkrlI7V0pE1DhsboicxO8FxUi6vtrpYnFt7NQ1xAfxKsZOROQ42NwQObBrZVp8d8Sw2ulIdpFxPMBLhtGRYYiLYuxERI6HzQ2Rg6nW6bHn9zqxk84QO7lKJRjS1RA7DenC2ImIHBebGyIHcaagGEnqbGw4nINLdWKn7qG+iFMpMDoyDIHejJ2IyPE1qbmpqKiAuzuvSEpkL1dLtfju+kX2jv4pdhoTGY44VTh6hPnZsUIiItszu7nR6/WYP38+li1bhoKCApw5cwbt27fH7NmzERERgaeeesoadRLRddU6PX65fpG9nScvmsROQ6/HTn9l7ERETszs5ubtt9/Gl19+iXfffReTJ082jvfs2ROLFy9mc0NkJen5xUhSZ2HD4VwUltTGTj3CfBGvUuDBPmFozdiJiMj85mbVqlX49NNPMWzYMDz77LPG8T59+uD06dMWLY7I2V0t1WLz9dVOx3JqY6fWXjKM6RuOuCgFuof52rFCIqLmx+zmJicnBx07drxhXK/Xo6qqyiJFETmzKp0eP6cbYqeU0wWo0gkAgJuLBMO6BiNepcDgLm3g5sLYiYioPmY3N927d8eePXtwxx13mIwnJSWhb9++FiuMyNmcytMgWZ2NjWk5KCzRGsd7hvsiPkqBByPDEeAls2OFREQtg9nNzZw5c5CQkICcnBzo9XqsX78e6enpWLVqFb7//ntr1EjksK6UarEpLQdJ6mycyNUYxwO9ZXiobzjiVAp0DWHsRERkDokQQpi70549e/Dmm2/iyJEjKCkpQVRUFObMmYP77rvPGjValEajgZ+fH4qKiuDryx8Nsr0qnR670y8hSZ2FXacvmsROMd0MsdM9nRk7ERHVZc7vd6Oam5aMzQ3Zy8lcDZLU2diUloPLpbWxU2+FH+JVCozqHYZWjJ2IiOplzu+32bFU+/btcfDgQbRu3dpk/Nq1a4iKikJGRoa5b0nksC6XVGJTmmG108m8urGTHGOjDKuduoT42LFCIiLHY3Zzc/78eeh0uhvGKysrkZOTY5GiiFoybbUeP6VfRLI6G7tOX0S13nByVOYiRUx3w0X27unUBq6MnYiIrKLBzc3mzZuN/7xjxw74+dVe0l2n0yElJQUREREWLY6oJTmRW3Q9dsrFlTqxU5+a2KlPGPw9GTsREVlbg5ubMWPGAAAkEgkSEhJMXnNzc0NERAQ++OADixZH1NwV1omdTtWJndr4yDH2+mqnzsGMnYiIbKnBzY1eb7h/Tbt27XDw4EEEBgZarSii5kxbrceu0xeRpM7G7nTT2OneHobVTnd3DGTsRERkJ2bPucnMzLRGHUTNmhACJ+qsdrpaVns17j5K/+urnUIZOxERNQNmNzcAUFpaip9//hkXLlyAVqs1ee2FF16wSGFEzcGl4krjRfZO5xcbx4N85HgoKhzxUQp0YuxERNSsmN3cHD58GCNHjkRZWRlKS0sREBCAwsJCeHp6IigoiM0NtXiG2KkASeps/JR+Cbqa2MlVivu6G2Knuxg7ERE1W2Y3NzNmzMCoUaOwbNky+Pn54bfffoObmxueeOIJTJ8+3Ro1ElmdEALHczRIUmdh85Fck9ipb1t/xEUZLrLn5+lmxyqJiKghzG5u0tLS8N///hdSqRQuLi6orKxE+/bt8e677yIhIQFjx461Rp1EVnGxuAKbDhtWO6UX1MZOwb5yjI1SIC5KgY5B3naskIiIzGV2c+Pm5gap1HA6PigoCBcuXEC3bt3g5+eHrKwsixdIZGmV1TqknDJcZG/3mdrYSe4qxX09Qoyxk4tUYudKiYioMcxubvr27YuDBw+iU6dOGDx4MObMmYPCwkJ89dVX6NmzpzVqJGoyIQSO5dReZK+ovDZ2imrrj3iVEvf3DoWfB2MnIqKWzuzmZsGCBSguNpy+nz9/PiZOnIjnnnsOnTp1wvLlyy1eIFFTXNRUYOP11U5nCkqM4yG+7oZ7O6kU6NCGsRMRkSPhXcHJ4VRUGWKnJHUWfvm90CR2Gt4zBHFRCgxi7ERE1KKY8/ttsbWsqampeOCBB8zeb+nSpYiIiIC7uzuio6Nx4MCBW25/7do1TJkyBaGhoZDL5ejcuTO2bt3a2LLJQQghcCTrGmZvPI7oBSmYsjrVuIxbdUcrLBzbCwdfj8FHj/bFPZ3bsLEhInJgZsVSO3bswI8//giZTIann34a7du3x+nTpzFz5kx89913iI2NNevD165di8TERCxbtgzR0dFYvHgxYmNjkZ6ejqCgoBu212q1uPfeexEUFISkpCSEh4fjjz/+gL+/v1mfS46jQFOBDYcNsdPZi7WxU6ifO+KiFBgbFY72jJ2IiJxKg2Op5cuXY/LkyQgICMDVq1fRunVrLFq0CNOmTcO4ceMwffp0dOvWzawPj46Oxp133oklS5YAMNy/SqlUYtq0aZg5c+YN2y9btgzvvfceTp8+DTe3hk38rKysRGVlpfG5RqOBUqlkLNWCVVTpsPOU4SJ7v5y5hOupE9zdpBjeIwTxKiUGdGjNszNERA7EnFiqwc1N7969MWHCBLz00ktITk7Gww8/jL/85S/49ttvoVAozC5Sq9XC09MTSUlJxjuOA0BCQgKuXbuGTZs23bDPyJEjERAQAE9PT2zatAlt2rTB448/jldeeQUuLi71fs4bb7yBefPm3TDO5qZlEULgSHaR4SJ7abnQVFQbX+t3RyvEqxQY2TsUvu5c7URE5IjMaW4aHEudO3cODz/8MABg7NixcHV1xXvvvdeoxgYACgsLodPpEBwcbDIeHByM06dP17tPRkYGdu3ahfHjx2Pr1q04e/Ysnn/+eVRVVWHu3Ln17jNr1iwkJiYan9ecuaGWIb+oJnbKwrlLpcbxMD93xKkUGBulQLtALztWSEREzU2Dm5vy8nJ4enoCACQSCeRyOUJDQ61WWH30ej2CgoLw6aefwsXFBSqVCjk5OXjvvfdu2tzI5XLI5XKb1klNU1Glw48nDbHTnt9NY6cRPUMRr1JgQPvWkDJ2IiKiepg1ofjzzz+Ht7dhcmZ1dTVWrlyJwMBAk20aeuPMwMBAuLi4oKCgwGS8oKAAISEh9e4TGhoKNzc3kwiqW7duyM/Ph1arhUwmM+frUDMihMDhrGtIUmfjuyO5KK4TO/WPCEC8SoERvULgw9iJiIhuo8HNTdu2bfHZZ58Zn4eEhOCrr74y2UYikTS4uZHJZFCpVEhJSTHOudHr9UhJScHUqVPr3WfQoEFYvXo19Hq98RYQZ86cQWhoKBubFiq/qALrD2cjSZ2NjDqxU7i/B+KiwjE2SoEIxk5ERGSGBjc358+ft/iHJyYmIiEhAf369UP//v2xePFilJaWYtKkSQCAiRMnIjw8HAsXLgQAPPfcc1iyZAmmT5+OadOm4ffff8eCBQsa3FBR81BRpcOOE/lIUmdj79lC1Exp93BzwYhehns7/aUdYyciImocs2+/YEnjxo3DpUuXMGfOHOTn5yMyMhLbt283TjK+cOGC8QwNACiVSuzYsQMzZsxA7969ER4ejunTp+OVV16x11egBhJCIPWCIXb6/kguiivrxE7tDLHTyF6h8Jbb9Y8kERE5AN5+gawqr6gc61NzkKzORkbhn2InlQJxUeG4ozVjJyIiujWrLAUnaqhyrQ4/nKw/dhrZy7DaKbpdAGMnIiKyCjY3ZBFCCKj/uIrk1Gx8fyTPJHaKblez2omxExERWR9/aahJcq6VY0NqNpJTc5BZJ3ZSBnggLkqBuCgFlAGedqyQiIicTaOam3PnzuGLL77AuXPn8NFHHyEoKAjbtm1D27Zt0aNHD0vXSM1MuVaH7SfykKzOwb5ztbGTp6w2duofwdiJiIjsw+zm5ueff8aIESMwaNAg/PLLL5g/fz6CgoJw5MgRLF++HElJSdaok+xMCIFDf1xF0qFsbDmWh5I6sdOA9q0Rr1JgeM8QeDF2IiIiOzP7l2jmzJl4++23kZiYCB8fH+P40KFDjXf3JseRc60c69XZSErNxh+Xy4zjygAPxEcpMTYqnLETERE1K2Y3N8eOHcPq1atvGA8KCkJhYaFFiiL7KtNWY/vxfCSnZuPXc5eNsZNXndjpTsZORETUTJnd3Pj7+yMvLw/t2rUzGT98+DDCw8MtVhjZlhACB89fRZI6C1uO5qFUqzO+NrBDbezkKWPsREREzZvZv1SPPvooXnnlFaxbtw4SiQR6vR779u3Diy++iIkTJ1qjRrKirCtlhovspWbjwpXa2OmO1p6Ii1JgbFQ4FK0YOxERUcthdnOzYMECTJkyBUqlEjqdDt27d4dOp8Pjjz+O119/3Ro1koWVaaux7ZjhInv7My4bx73lrri/Vyji+ynQ745WkEgYOxERUcvT6NsvXLhwAcePH0dJSQn69u2LTp06Wbo2q3DW2y/o9QIHzl9Bkjob247Vxk4SCTCoQyDiVOGI7cHYiYiImier3n5h7969uOuuu9C2bVu0bdu20UWSbWRdKUNyajaSU7ORdaXcOB7R2hPxKgUeilIg3N/DjhUSERFZltnNzdChQxEeHo7HHnsMTzzxBLp3726NuqgJSiurse14PpLUWfgt44px3Fvuigd6G1Y7qRg7ERGRgzK7ucnNzcWaNWvwzTff4J133kHv3r0xfvx4PPbYY1AoFNaokRpArxf4X+b12Ol4HsrqxE53dQxEXJQCsT1C4CFzsXOlRERE1tXoOTcAkJmZidWrV+Obb77B6dOncc8992DXrl2WrM/iHG3OzYXLtbFT9tXa2KldoJchduobjjDGTkRE1MKZ8/vdpOYGAHQ6HbZt24bZs2fj6NGj0Ol0t9/JjhyhuSmprMbWY3lIUmfjQGZt7OQjd8UDfcIQrwpHVFvGTkRE5DisOqG4xr59+/D1118jKSkJFRUVGD16NBYuXNjYt6Pb0OsFfsu8fH21Uz7Kq0xjp3iVIXZyd2PsREREzs3s5mbWrFlYs2YNcnNzce+99+Kjjz7C6NGj4enJC71Zwx+XS5GcmoNkdTZyrtXGTu0DvRCnMlxkL9SPsRMREVENs5ubX375BS+99BIeeeQRBAYGWqMmp1daWY0tR6/HTufrxE7urhjVJwzxKgX6Kv0ZOxEREdXD7OZm37591qiD6nh42X6czNMAAKQS4K5ObRCvUuC+7sGMnYiIiG6jQc3N5s2bMWLECLi5uWHz5s233PbBBx+0SGHOqqJKZ2xsXryvM+JVSoT4udu5KiIiopajQc3NmDFjkJ+fj6CgIIwZM+am20kkkma/Wqq5K9BUAADc3aSYMqQjoyciIiIzNai50ev19f4zWV5+kaG5CfF1Z2NDRETUCFJzd1i1ahUqKytvGNdqtVi1apVFinJm+dfP3AT7MooiIiJqDLObm0mTJqGoqOiG8eLiYkyaNMkiRTmzmliK82yIiIgax+zmRghRb1ySnZ0NPz8/ixTlzPKLDGfFQnjmhoiIqFEavBS8b9++kEgkkEgkGDZsGFxda3fV6XTIzMzE8OHDrVKkMykoZixFRETUFA1ubmpWSaWlpSE2Nhbe3t7G12QyGSIiIhAXF2fxAp1NQRGbGyIioqZocHMzd+5cAEBERATGjRsHd3f++FpDvnHOjdzOlRAREbVMZl+hOCEhwRp1EAzzmS5qDHNueOaGiIiocRrU3AQEBODMmTMIDAxEq1atbnn9lStXrtz0Nbq1K6VaaHWG6wgF+bC5ISIiaowGNTcffvghfHx8jP/Mi8tZR00kFegtg8zV7IVsREREhAY2N3WjqCeffNJatTi9Al7Aj4iIqMnMPj2QmpqKY8eOGZ9v2rQJY8aMwauvvgqtVmvR4pxNgYbXuCEiImoqs5ubv//97zhz5gwAICMjA+PGjYOnpyfWrVuHl19+2eIFOpOa+0oFsbkhIiJqNLObmzNnziAyMhIAsG7dOgwePBirV6/GypUrkZycbOn6nIrx1gtsboiIiBqtUbdfqLkz+M6dOzFy5EgAgFKpRGFhoWWrczK8xg0REVHTmd3c9OvXD2+//Ta++uor/Pzzz7j//vsBAJmZmQgODrZ4gc4kn1cnJiIiajKzm5vFixcjNTUVU6dOxWuvvYaOHTsCAJKSkjBw4ECLF+hMeEdwIiKipjP7CsW9e/c2WS1V47333oOLi4tFinJGFVU6XC2rAsA5N0RERE1hdnNTQ61W49SpUwCA7t27IyoqymJFOaNLxYZl4HJXKfw83OxcDRERUctldnNz8eJFjBs3Dj///DP8/f0BANeuXcOQIUOwZs0atGnTxtI1OoX8Ohfw4xWgiYiIGs/sOTfTpk1DSUkJTpw4gStXruDKlSs4fvw4NBoNXnjhBWvU6BRqJhMzkiIiImoas8/cbN++HTt37kS3bt2MY927d8fSpUtx3333WbQ4Z2K89QInExMRETWJ2Wdu9Ho93NxunBPi5uZmvP4Nma/2zA2vcUNERNQUZjc3Q4cOxfTp05Gbm2scy8nJwYwZMzBs2DCLFudM8nnTTCIiIoswu7lZsmQJNBoNIiIi0KFDB3To0AHt2rWDRqPBxx9/bI0anQKvcUNERGQZZs+5USqVSE1NRUpKinEpeLdu3RATE2Px4pwJ7whORERkGWY1N2vXrsXmzZuh1WoxbNgwTJs2zVp1ORUhBGMpIiIiC2lwc/PJJ59gypQp6NSpEzw8PLB+/XqcO3cO7733njXrcwrXyqqgrTZMxg7ihGIiIqImafCcmyVLlmDu3LlIT09HWloavvzyS/znP/+xZm1Oo+asTYCXDHJX3sKCiIioKRrc3GRkZCAhIcH4/PHHH0d1dTXy8vKsUpgzYSRFRERkOQ1ubiorK+Hl5VW7o1QKmUyG8vJyqxTmTAp4jRsiIiKLMWtC8ezZs+Hp6Wl8rtVqMX/+fPj5+RnHFi1aZLnqnEQ+l4ETERFZTIObm3vuuQfp6ekmYwMHDkRGRobxOW/42DgFjKWIiIgspsHNze7du61YhnOrucYNmxsiIqKmM/sKxdawdOlSREREwN3dHdHR0Thw4ECD9luzZg0kEgnGjBlj3QKtjHcEJyIishy7Nzdr165FYmIi5s6di9TUVPTp0wexsbG4ePHiLfc7f/48XnzxRdx99902qtR6GEsRERFZjt2bm0WLFmHy5MmYNGkSunfvjmXLlsHT0xMrVqy46T46nQ7jx4/HvHnz0L59extWa3mV1TpcLtUC4IRiIiIiS7Brc6PVaqFWq03uSyWVShETE4P9+/ffdL8333wTQUFBeOqpp277GZWVldBoNCaP5uTi9fk2MlcpWnm62bkaIiKils+uzU1hYSF0Oh2Cg4NNxoODg5Gfn1/vPnv37sXy5cvx2WefNegzFi5cCD8/P+NDqVQ2uW5Lqo2k5FxtRkREZAGNam727NmDJ554AgMGDEBOTg4A4KuvvsLevXstWtyfFRcXY8KECfjss88QGBjYoH1mzZqFoqIi4yMrK8uqNZrLeI0bzrchIiKyCLMu4gcAycnJmDBhAsaPH4/Dhw+jstIQqxQVFWHBggXYunVrg98rMDAQLi4uKCgoMBkvKChASEjIDdufO3cO58+fx6hRo4xjer3hhpOurq5IT09Hhw4dTPaRy+WQy5vvlX9rloEHsbkhIiKyCLPP3Lz99ttYtmwZPvvsM7i51c4RGTRoEFJTU816L5lMBpVKhZSUFOOYXq9HSkoKBgwYcMP2Xbt2xbFjx5CWlmZ8PPjggxgyZAjS0tKaXeTUEAU8c0NERGRRZp+5SU9Pxz333HPDuJ+fH65du2Z2AYmJiUhISEC/fv3Qv39/LF68GKWlpZg0aRIAYOLEiQgPD8fChQvh7u6Onj17muzv7+8PADeMtxS8xg0REZFlmd3chISE4OzZs4iIiDAZ37t3b6OWZY8bNw6XLl3CnDlzkJ+fj8jISGzfvt04yfjChQuQSu2+Yt1qjHcE5zJwIiIiizC7uZk8eTKmT5+OFStWQCKRIDc3F/v378eLL76I2bNnN6qIqVOnYurUqfW+drvbPqxcubJRn9lcMJYiIiKyLLObm5kzZ0Kv12PYsGEoKyvDPffcA7lcjhdffBHTpk2zRo0OSwjBWIqIiMjCzG5uJBIJXnvtNbz00ks4e/YsSkpK0L17d3h7e1ujPodWVF6FymrDaq8g3+a7oouIiKglMbu5qSGTydC9e3dL1uJ0apaB+3u6wd3Nxc7VEBEROQazm5shQ4bc8kq6u3btalJBzoQX8CMiIrI8s5ubyMhIk+dVVVVIS0vD8ePHkZCQYKm6nEJBEe8GTkREZGlmNzcffvhhveNvvPEGSkpKmlyQM+GZGyIiIsuz2AVknnjiCaxYscJSb+cUeI0bIiIiy7NYc7N//364u/NH2hwFXAZORERkcWbHUmPHjjV5LoRAXl4eDh061OiL+DkrYyzlx2XgRERElmJ2c+Pn52fyXCqVokuXLnjzzTdx3333WawwZ1CzFJwTiomIiCzHrOZGp9Nh0qRJ6NWrF1q1amWtmpxClU6Py6VsboiIiCzNrDk3Li4uuO+++xp1928ydbG4EkIAbi4SBHjK7F0OERGRwzB7QnHPnj2RkZFhjVqcSs09pYJ83CGV3vyiiERERGQes5ubt99+Gy+++CK+//575OXlQaPRmDyoYYx3A+cycCIiIotq8JybN998E//85z8xcuRIAMCDDz5ochsGIQQkEgl0Op3lq3RAvBs4ERGRdTS4uZk3bx6effZZ/PTTT9asx2nUnLnhZGIiIiLLanBzI4QAAAwePNhqxTiTAl7jhoiIyCrMmnNzq7uBk3nyeeaGiIjIKsy6zk3nzp1v2+BcuXKlSQU5C17Aj4iIyDrMam7mzZt3wxWKyXxCCE4oJiIishKzmptHH30UQUFB1qrFaWgqqlFeZVhVxqXgREREltXgOTecb2M5NZOJ/Tzc4O7mYudqiIiIHEuDm5ua1VLUdIykiIiIrKfBsZRer7dmHU7FeI0bRlJEREQWZ/btF6jpjM2ND69xQ0REZGlsbuwgn/eVIiIisho2N3aQX8Rr3BAREVkLmxs7MN56gc0NERGRxbG5sQPGUkRERNbD5sbGqnR6FJYwliIiIrIWNjc2VlhSCSEAV6kErb1k9i6HiIjI4bC5sbGaC/gF+cghlfKqz0RERJbG5sbGeAE/IiIi62JzY2O89QIREZF1sbmxsXwNJxMTERFZE5sbGyvgMnAiIiKrYnNjY4yliIiIrIvNjY0VFF+fUMzmhoiIyCrY3NhYQVFNc8M7ghMREVkDmxsbKq6oQqlWB4BzboiIiKyFzY0N1Uwm9nF3hafM1c7VEBEROSY2NzaUX2RYBs7JxERERNbD5saGeDdwIiIi62NzY0PGWy/wzA0REZHVsLmxIeMF/NjcEBERWQ2bGxvK5zJwIiIiq2NzY0OMpYiIiKyPzY0NcUIxERGR9bG5sZFqnR6XirkUnIiIyNrY3NhIYYkWegG4SCVo7c05N0RERNbC5sZGaiKpIB85XKQSO1dDRETkuNjc2AgnExMREdkGmxsbqW1uGEkRERFZE5sbG6m5xg0nExMREVkXmxsbqZlzE8xl4ERERFbF5sZGeOsFIiIi22gWzc3SpUsREREBd3d3REdH48CBAzfd9rPPPsPdd9+NVq1aoVWrVoiJibnl9s0FYykiIiLbsHtzs3btWiQmJmLu3LlITU1Fnz59EBsbi4sXL9a7/e7du/HYY4/hp59+wv79+6FUKnHfffchJyfHxpWbp0BjuIAfYykiIiLrkgghhD0LiI6Oxp133oklS5YAAPR6PZRKJaZNm4aZM2fedn+dTodWrVphyZIlmDhx4m2312g08PPzQ1FREXx9fZtcf0OUVFaj59wdAIDj82LhLXe1yecSERE5CnN+v+165kar1UKtViMmJsY4JpVKERMTg/379zfoPcrKylBVVYWAgIB6X6+srIRGozF52FrNfBtvuSsbGyIiIiuza3NTWFgInU6H4OBgk/Hg4GDk5+c36D1eeeUVhIWFmTRIdS1cuBB+fn7Gh1KpbHLd5ioo4jVuiIiIbMXuc26a4p133sGaNWuwYcMGuLvXP5dl1qxZKCoqMj6ysrJsXCXvBk5ERGRLds1IAgMD4eLigoKCApPxgoIChISE3HLf999/H++88w527tyJ3r1733Q7uVwOudy+Z0zyeesFIiIim7HrmRuZTAaVSoWUlBTjmF6vR0pKCgYMGHDT/d5991289dZb2L59O/r162eLUpukgMvAiYiIbMbus1sTExORkJCAfv36oX///li8eDFKS0sxadIkAMDEiRMRHh6OhQsXAgD+9a9/Yc6cOVi9ejUiIiKMc3O8vb3h7e1tt+9xK4yliIiIbMfuzc24ceNw6dIlzJkzB/n5+YiMjMT27duNk4wvXLgAqbT2BNMnn3wCrVaL+Ph4k/eZO3cu3njjDVuW3mDGa9zwzA0REZHV2f06N7Zmj+vcDFiYgryiCmycMgiRSn+bfCYREZEjaTHXuXEGOr3AxWLDmRvOuSEiIrI+NjdWdrmkEjq9gFQCBHrL7F0OERGRw2NzY2U1k4nb+Mjh6sLDTUREZG38tbUy3g2ciIjIttjcWFkBL+BHRERkU2xurKxmGTivcUNERGQbbG6sjLdeICIisi02N1bGWIqIiMi22NxYGScUExER2RabGyurva+Ufe9MTkRE5CzY3FhRmbYaxRXVABhLERER2QqbGyuqiaS8ZC7wcXezczVERETOgc2NFRnvBs5l4ERERDbD5saKjCulfNjcEBER2QqbGyuqnUzM5oaIiMhW2NxYUc2cG04mJiIish02N1ZUE0uF+HIZOBERka2wubEixlJERES2x+bGigoYSxEREdkcmxsr0esFLhbzjuBERES2xubGSi6XalGtF5BIgEBvzrkhIiKyFTY3VlIzmTjQWw43Fx5mIiIiW+GvrpXwbuBERET2webGSmpWSnEyMRERkW2xubES4zVu/DjfhoiIyJbY3FgJYykiIiL7YHNjJQXXl4EzliIiIrItNjdWUnMBP17jhoiIyLbY3FgJJxQTERHZB5sbK6io0qGovAoAmxsiIiJbY3NjBTWTiT3cXODr7mrnaoiIiJwLmxsrqHs3cIlEYudqiIiInAubGysoMM634TVuiIiIbI3NjRUYL+DH+TZEREQ2x+bGCvKLrl/jhsvAiYiIbI7NjRUYYykfNjdERES2xubGCupOKCYiIiLbYnNjBTVLwXmNGyIiIttjc2Nher3AxWKeuSEiIrIXNjcWdqVMiyqdgEQCBPlwKTgREZGtsbmxsJrJxK295HBz4eElIiKyNf76WpjxGjd+PGtDRERkD2xuLMx4jRsuAyciIrILNjcWVrMMnBfwIyIisg/estrCCop46wUiosYSQqC6uho6nc7epZAduLm5wcXFpcnvw+bGwvJ5XykiokbRarXIy8tDWVmZvUshO5FIJFAoFPD29m7S+7C5sbACxlJERGbT6/XIzMyEi4sLwsLCIJPJIJFI7F0W2ZAQApcuXUJ2djY6derUpDM4bG4sjHcEJyIyn1arhV6vh1KphKenp73LITtp06YNzp8/j6qqqiY1N5xQbEEVVTpcLasCwOaGiKgxpFL+LDkzS52t458iC7qoMSwDl7tK4evBk2JERET2wObGgureDZxZMRERkX2wubEg4zVuGEkRERHZDZsbC+I1boiInNP+/fvh4uKC+++//4bXdu/eDYlEgmvXrt3wWkREBBYvXmwy9tNPP2HkyJFo3bo1PD090b17d/zzn/9ETk6OlaoHKioqMGXKFLRu3Rre3t6Ii4tDQUHBLfeRSCT1Pt57770btq2srERkZCQkEgnS0tKs9C1qsbmxoLqxFBEROY/ly5dj2rRp+OWXX5Cbm9vo9/nvf/+LmJgYhISEIDk5GSdPnsSyZctQVFSEDz74wIIVm5oxYwa+++47rFu3Dj///DNyc3MxduzYW+6Tl5dn8lixYgUkEgni4uJu2Pbll19GWFiYtcq/AWe9WlABYykiIosRQqC8yvZXKvZwczFr3mRJSQnWrl2LQ4cOIT8/HytXrsSrr75q9udmZ2fjhRdewAsvvIAPP/zQOB4REYF77rmn3jM/llBUVITly5dj9erVGDp0KADgiy++QLdu3fDbb7/hL3/5S737hYSEmDzftGkThgwZgvbt25uMb9u2DT/88AOSk5Oxbds2q3yHP2NzY0G8xg0RkeWUV+nQfc4Om3/uyTdj4Slr+M/jt99+i65du6JLly544okn8I9//AOzZs0ye2HJunXroNVq8fLLL9f7ur+//033HTFiBPbs2XPT1++44w6cOHGi3tfUajWqqqoQExNjHOvatSvatm2L/fv337S5qaugoABbtmzBl19+ecP45MmTsXHjRptev6hZxFJLly5FREQE3N3dER0djQMHDtxy+3Xr1qFr165wd3dHr169sHXrVhtVemu1E4rldq6EiIhsZfny5XjiiScAAMOHD0dRURF+/vlns9/n999/h6+vL0JDQ83e9/PPP0daWtpNH7f6nczPz4dMJruheQoODkZ+fn6DPv/LL7+Ej4+PSZQlhMCTTz6JZ599Fv369TP7OzWF3c/crF27FomJiVi2bBmio6OxePFixMbGIj09HUFBQTds/+uvv+Kxxx7DwoUL8cADD2D16tUYM2YMUlNT0bNnTzt8AwMhBAquX+eGsRQRUdN5uLng5JuxdvnchkpPT8eBAwewYcMGAICrqyvGjRuH5cuX469//atZnyuEaPRlRMLDwxu1n6WsWLEC48ePh7t77e/fxx9/jOLiYsyaNcvm9dj9zM2iRYswefJkTJo0Cd27d8eyZcvg6emJFStW1Lv9Rx99hOHDh+Oll15Ct27d8NZbbyEqKgpLliyxceWmrpZVQVutB8DmhojIEiQSCTxlrjZ/mNNgLF++HNXV1QgLC4OrqytcXV3xySefIDk5GUVFRQAAX19fADA+r+vatWvw8/MDAHTu3BlFRUXIy8sz+1iNGDEC3t7eN3306NHjpvuGhIRAq9XeMKenoKDghnk19dmzZw/S09Px9NNPm4zv2rUL+/fvh1wuh6urKzp27AgA6NevHxISEsz+juawa3Oj1WqhVqtNcj6pVIqYmBjs37+/3n32799vsj0AxMbG3nT7yspKaDQak4c15F9fBt7aSwaZq917RiIisrLq6mqsWrUKH3zwgUkEdOTIEYSFheGbb74BAHTq1AlSqRRqtdpk/4yMDBQVFaFz584AgPj4eMhkMrz77rv1ft6tJhQ3JZZSqVRwc3NDSkqKcSw9PR0XLlzAgAEDbnscli9fDpVKhT59+piM//vf/8aRI0duqGHt2rWYP3/+bd+3KewaSxUWFkKn0yE4ONhkPDg4GKdPn653n/z8/Hq3v1kuuHDhQsybN88yBd9CcUUVfN1dedaGiMhJfP/997h69Sqeeuop49mXGnFxcVi+fDmeffZZ+Pj44Omnn8Y///lPuLq6olevXsjKysIrr7yCv/zlLxg4cCAAQKlU4sMPP8TUqVOh0WgwceJEREREIDs7G6tWrYK3t/dNl4M3JZby8/PDU089hcTERAQEBMDX1xfTpk3DgAEDTCYTd+3aFQsXLsRDDz1kHNNoNFi3bl29dbVt29bkube3NwCgQ4cOUCgUja63IRz+FMOsWbNQVFRkfGRlZVnlc6Lbt8bRN2Kxccogq7w/ERE1L8uXL0dMTMwNjQ1gaG4OHTqEo0ePAjBMqUhISMArr7yCHj164Mknn0Tv3r3x3XffmcRgzz//PH744Qfk5OTgoYceQteuXfH000/D19cXL774otW+y4cffogHHngAcXFxuOeeexASEoL169ebbJOenn5DtLZmzRoIIfDYY49ZrbbGkAghhL0+XKvVwtPTE0lJSRgzZoxxPCEhAdeuXcOmTZtu2Kdt27ZITEzEP/7xD+PY3LlzsXHjRhw5cuS2n6nRaODn54eioiJjDkpERPZVUVGBzMxMtGvXzmRSKjmXW/05MOf3265nbmQyGVQqlUnOp9frkZKSctOcb8CAASbbA8CPP/7YoFyQiIiIHJ/dl4InJiYiISEB/fr1Q//+/bF48WKUlpZi0qRJAICJEyciPDwcCxcuBABMnz4dgwcPxgcffID7778fa9aswaFDh/Dpp5/a82sQERFRM2H35mbcuHG4dOkS5syZg/z8fERGRmL79u3GScMXLlyAVFp7gmngwIFYvXo1Xn/9dbz66qvo1KkTNm7caNdr3BAREVHzYdc5N/bAOTdERM0P59wQ4CBzboiIiOpysv/fpj+x1L9/NjdERGR3bm5uAICysjI7V0L2pNVqAQAuLg2/BUZ97D7nhoiIyMXFBf7+/rh48SIAwNPTs9H3WaKWSa/X49KlS/D09ISra9PaEzY3RETULNTcx6imwSHnI5VK0bZt2yY3tmxuiIioWZBIJAgNDUVQUBCqqqrsXQ7ZgUwmM1kh3VhsboiIqFlxcXFp8pwLcm6cUExEREQOhc0NERERORQ2N0RERORQnG7OTc0FgjQajZ0rISIiooaq+d1uyIX+nK65KS4uBgAolUo7V0JERETmKi4uhp+f3y23cbp7S+n1euTm5sLHx8fiF4jSaDRQKpXIysrifausiMfZNnicbYPH2XZ4rG3DWsdZCIHi4mKEhYXddrm40525kUqlUCgUVv0MX19f/sWxAR5n2+Bxtg0eZ9vhsbYNaxzn252xqcEJxURERORQ2NwQERGRQ2FzY0FyuRxz586FXC63dykOjcfZNnicbYPH2XZ4rG2jORxnp5tQTERERI6NZ26IiIjIobC5ISIiIofC5oaIiIgcCpsbIiIicihsbsy0dOlSREREwN3dHdHR0Thw4MAtt1+3bh26du0Kd3d39OrVC1u3brVRpS2bOcf5s88+w913341WrVqhVatWiImJue2/FzIw989zjTVr1kAikWDMmDHWLdBBmHucr127hilTpiA0NBRyuRydO3fmfzsawNzjvHjxYnTp0gUeHh5QKpWYMWMGKioqbFRty/TLL79g1KhRCAsLg0QiwcaNG2+7z+7duxEVFQW5XI6OHTti5cqVVq8TghpszZo1QiaTiRUrVogTJ06IyZMnC39/f1FQUFDv9vv27RMuLi7i3XffFSdPnhSvv/66cHNzE8eOHbNx5S2Lucf58ccfF0uXLhWHDx8Wp06dEk8++aTw8/MT2dnZNq68ZTH3ONfIzMwU4eHh4u677xajR4+2TbEtmLnHubKyUvTr10+MHDlS7N27V2RmZordu3eLtLQ0G1fesph7nL/++mshl8vF119/LTIzM8WOHTtEaGiomDFjho0rb1m2bt0qXnvtNbF+/XoBQGzYsOGW22dkZAhPT0+RmJgoTp48KT7++GPh4uIitm/fbtU62dyYoX///mLKlCnG5zqdToSFhYmFCxfWu/0jjzwi7r//fpOx6Oho8fe//92qdbZ05h7nP6uurhY+Pj7iyy+/tFaJDqExx7m6uloMHDhQfP755yIhIYHNTQOYe5w/+eQT0b59e6HVam1VokMw9zhPmTJFDB061GQsMTFRDBo0yKp1OpKGNDcvv/yy6NGjh8nYuHHjRGxsrBUrE4KxVANptVqo1WrExMQYx6RSKWJiYrB///5699m/f7/J9gAQGxt70+2pccf5z8rKylBVVYWAgABrldniNfY4v/nmmwgKCsJTTz1lizJbvMYc582bN2PAgAGYMmUKgoOD0bNnTyxYsAA6nc5WZbc4jTnOAwcOhFqtNkZXGRkZ2Lp1K0aOHGmTmp2FvX4Hne7GmY1VWFgInU6H4OBgk/Hg4GCcPn263n3y8/Pr3T4/P99qdbZ0jTnOf/bKK68gLCzshr9QVKsxx3nv3r1Yvnw50tLSbFChY2jMcc7IyMCuXbswfvx4bN26FWfPnsXzzz+PqqoqzJ071xZltziNOc6PP/44CgsLcdddd0EIgerqajz77LN49dVXbVGy07jZ76BGo0F5eTk8PDys8rk8c0MO5Z133sGaNWuwYcMGuLu727sch1FcXIwJEybgs88+Q2BgoL3LcWh6vR5BQUH49NNPoVKpMG7cOLz22mtYtmyZvUtzKLt378aCBQvwn//8B6mpqVi/fj22bNmCt956y96lkQXwzE0DBQYGwsXFBQUFBSbjBQUFCAkJqXefkJAQs7anxh3nGu+//z7eeecd7Ny5E71797ZmmS2eucf53LlzOH/+PEaNGmUc0+v1AABXV1ekp6ejQ4cO1i26BWrMn+fQ0FC4ubnBxcXFONatWzfk5+dDq9VCJpNZteaWqDHHefbs2ZgwYQKefvppAECvXr1QWlqKZ555Bq+99hqkUv6/vyXc7HfQ19fXamdtAJ65aTCZTAaVSoWUlBTjmF6vR0pKCgYMGFDvPgMGDDDZHgB+/PHHm25PjTvOAPDuu+/irbfewvbt29GvXz9blNqimXucu3btimPHjiEtLc34ePDBBzFkyBCkpaVBqVTasvwWozF/ngcNGoSzZ88am0cAOHPmDEJDQ9nY3ERjjnNZWdkNDUxNQyl4y0WLsdvvoFWnKzuYNWvWCLlcLlauXClOnjwpnnnmGeHv7y/y8/OFEEJMmDBBzJw507j9vn37hKurq3j//ffFqVOnxNy5c7kUvAHMPc7vvPOOkMlkIikpSeTl5RkfxcXF9voKLYK5x/nPuFqqYcw9zhcuXBA+Pj5i6tSpIj09XXz//fciKChIvP322/b6Ci2Cucd57ty5wsfHR3zzzTciIyND/PDDD6JDhw7ikUcesddXaBGKi4vF4cOHxeHDhwUAsWjRInH48GHxxx9/CCGEmDlzppgwYYJx+5ql4C+99JI4deqUWLp0KZeCN0cff/yxaNu2rZDJZKJ///7it99+M742ePBgkZCQYLL9t99+Kzp37ixkMpno0aOH2LJli40rbpnMOc533HGHAHDDY+7cubYvvIUx989zXWxuGs7c4/zrr7+K6OhoIZfLRfv27cX8+fNFdXW1jatuecw5zlVVVeKNN94QHTp0EO7u7kKpVIrnn39eXL161faFtyA//fRTvf+9rTm2CQkJYvDgwTfsExkZKWQymWjfvr344osvrF6nRAiefyMiIiLHwTk3RERE5FDY3BAREZFDYXNDREREDoXNDRERETkUNjdERETkUNjcEBERkUNhc0NEREQOhc0NERERORQ2N0RkYuXKlfD397d3GY0mkUiwcePGW27z5JNPYsyYMTaph4hsj80NkQN68sknIZFIbnicPXvW3qVh5cqVxnqkUikUCgUmTZqEixcvWuT98/LyMGLECADA+fPnIZFIkJaWZrLNRx99hJUrV1rk827mjTfeMH5PFxcXKJVKPPPMM7hy5YpZ78NGjMh8rvYugIisY/jw4fjiiy9Mxtq0aWOnakz5+voiPT0der0eR44cwaRJk5Cbm4sdO3Y0+b1DQkJuu42fn1+TP6chevTogZ07d0Kn0+HUqVP429/+hqKiIqxdu9Ymn0/krHjmhshByeVyhISEmDxcXFywaNEi9OrVC15eXlAqlXj++edRUlJy0/c5cuQIhgwZAh8fH/j6+kKlUuHQoUPG1/fu3Yu7774bHh4eUCqVeOGFF1BaWnrL2iQSCUJCQhAWFoYRI0bghRdewM6dO1FeXg69Xo8333wTCoUCcrkckZGR2L59u3FfrVaLqVOnIjQ0FO7u7rjjjjuwcOFCk/euiaXatWsHAOjbty8kEgn++te/AjA9G/Lpp58iLCwMer3epMbRo0fjb3/7m/H5pk2bEBUVBXd3d7Rv3x7z5s1DdXX1Lb+nq6srQkJCEB4ejpiYGDz88MP48ccfja/rdDo89dRTaNeuHTw8PNClSxd89NFHxtffeOMNfPnll9i0aZPxLNDu3bsBAFlZWXjkkUfg7++PgIAAjB49GufPn79lPUTOgs0NkZORSqX497//jRMnTuDLL7/Erl278PLLL990+/Hjx0OhUODgwYNQq9WYOXMm3NzcAADnzp3D8OHDERcXh6NHj2Lt2rXYu3cvpk6dalZNHh4e0Ov1qK6uxkcffYQPPvgA77//Po4ePYrY2Fg8+OCD+P333wEA//73v7F582Z8++23SE9Px9dff42IiIh63/fAgQMAgJ07dyIvLw/r16+/YZuHH34Yly9fxk8//WQcu3LlCrZv347x48cDAPbs2YOJEydi+vTpOHnyJP773/9i5cqVmD9/foO/4/nz57Fjxw7IZDLjmF6vh0KhwLp163Dy5EnMmTMHr776Kr799lsAwIsvvohHHnkEw4cPR15eHvLy8jBw4EBUVVUhNjYWPj4+2LNnD/bt2wdvb28MHz4cWq22wTUROSyr33eciGwuISFBuLi4CC8vL+MjPj6+3m3XrVsnWrdubXz+xRdfCD8/P+NzHx8fsXLlynr3feqpp8QzzzxjMrZnzx4hlUpFeXl5vfv8+f3PnDkjOnfuLPr16yeEECIsLEzMnz/fZJ8777xTPP/880IIIaZNmyaGDh0q9Hp9ve8PQGzYsEEIIURmZqYAIA4fPmyyTUJCghg9erTx+ejRo8Xf/vY34/P//ve/IiwsTOh0OiGEEMOGDRMLFiwweY+vvvpKhIaG1luDEELMnTtXSKVS4eXlJdzd3QUAAUAsWrTopvsIIcSUKVNEXFzcTWut+ewuXbqYHIPKykrh4eEhduzYccv3J3IGnHND5KCGDBmCTz75xPjcy8sLgOEsxsKFC3H69GloNBpUV1ejoqICZWVl8PT0vOF9EhMT8fTTT+Orr74yRisdOnQAYIisjh49iq+//tq4vRACer0emZmZ6NatW721FRUVwdvbG3q9HhUVFbjrrrvw+eefQ6PRIDc3F4MGDTLZftCgQThy5AgAQ6R07733okuXLhg+fDgeeOAB3HfffU06VuPHj8fkyZPxn//8B3K5HF9//TUeffRRSKVS4/fct2+fyZkanU53y+MGAF26dMHmzZtRUVGB//u//0NaWhqmTZtmss3SpUuxYsUKXLhwAeXl5dBqtYiMjLxlvUeOHMHZs2fh4+NjMl5RUYFz58414ggQORY2N0QOysvLCx07djQZO3/+PB544AE899xzmD9/PgICArB371489dRT0Gq19f5Iv/HGG3j88cexZcsWbNu2DXPnzsWaNWvw0EMPoaSkBH//+9/xwgsv3LBf27Ztb1qbj48PUlNTIZVKERoaCg8PDwCARqO57feKiopCZmYmtm3bhp07d+KRRx5BTEwMkpKSbrvvzYwaNQpCCGzZsgV33nkn9uzZgw8//ND4eklJCebNm4exY8fesK+7u/tN31cmkxn/Hbzzzju4//77MW/ePLz11lsAgDVr1uDFF1/EBx98gAEDBsDHxwfvvfce/ve//92y3pKSEqhUKpOmskZzmTROZE9sboiciFqthl6vxwcffGA8K1Ezv+NWOnfujM6dO2PGjBl47LHH8MUXX+Chhx5CVFQUTp48eUMTdTtSqbTefXx9fREWFoZ9+/Zh8ODBxvF9+/ahf//+JtuNGzcO48aNQ3x8PIYPH44rV64gICDA5P1q5rfodLpb1uPu7o6xY8fi66+/xtmzZ9GlSxdERUUZX4+KikJ6errZ3/PPXn/9dQwdOhTPPfec8XsOHDgQzz//vHGbP595kclkN9QfFRWFtWvXIigoCL6+vk2qicgRcUIxkRPp2LEjqqqq8PHHHyMjIwNfffUVli1bdtPty8vLMXXqVOzevRt//PEH9u3bh4MHDxrjpldeeQW//vorpk6dirS0NPz+++/YtGmT2ROK63rppZfwr3/9C2vXrkV6ejpmzpyJtLQ0TJ8+HQCwaNEifPPNNzh9+jTOnDmDdevWISQkpN4LDwYFBcHDwwPbt29HQUEBioqKbvq548ePx5YtW7BixQrjROIac+bMwapVqzBv3jycOHECp06dwpo1a/D666+b9d0GDBiA3r17Y8GCBQCATp064dChQ9ixYwfOnDmD2bNn4+DBgyb7RERE4OjRo0hPT0dhYSGqqqowfvx4BAYGYvTo0dizZw8yMzOxe/duvPDCC8jOzjarJiKHZO9JP0RkefVNQq2xaNEiERoaKjw8PERsbKxYtWqVACCuXr0qhDCd8FtZWSkeffRRoVQqhUwmE2FhYWLq1Kkmk4UPHDgg7r33XuHt7S28vLxE7969b5gQXNefJxT/mU6nE2+88YYIDw8Xbm5uok+fPmLbtm3G1z/99FMRGRkpvLy8hK+vrxg2bJhITU01vo46E4qFEOKzzz4TSqVSSKVSMXjw4JseH51OJ0JDQwUAce7cuRvq2r59uxg4cKDw8PAQvr6+on///uLTTz+96feYO3eu6NOnzw3j33zzjZDL5eLChQuioqJCPPnkk8LPz0/4+/uL5557TsycOdNkv4sXLxqPLwDx008/CSGEyMvLExMnThSBgYFCLpeL9u3bi8mTJ4uioqKb1kTkLCRCCGHf9oqIiIjIchhLERERkUNhc0NEREQOhc0NERERORQ2N0RERORQ2NwQERGRQ2FzQ0RERA6FzQ0RERE5FDY3RERE5FDY3BAREZFDYXNDREREDoXNDRERETmU/wesxtYadOpUZAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "create_roc_auc_plot(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "09f76fd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgwAAAGwCAYAAADFZj2cAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8FElEQVR4nO3de1xUdf7H8feAchEY8AZIomKWSpqWtkYXy42kdCtX29aijUxzMy0vmdmvJC+VpVuWZVK5hbZa2kUrLYvVzUuSJal5pSxLUwFbFATlNnN+fxhjs2gDnuF6Xs/H4zxqzvmeM59DxHzm870cm2EYhgAAAH6HT20HAAAA6j4SBgAA4BEJAwAA8IiEAQAAeETCAAAAPCJhAAAAHpEwAAAAjxrVdgBmOJ1OHTx4UCEhIbLZbLUdDgCgigzD0LFjxxQVFSUfn+r7DltUVKSSkhLT1/Hz81NAQIAXIqp/6nXCcPDgQUVHR9d2GAAAk/bv36/WrVtXy7WLiooU0zZYWTkO09eKjIzU3r17LZk01OuEISQkRJL009ftZA+mdwUN05/P71rbIQDVpkylWq+PXH/Pq0NJSYmychz6KaOd7CFn/1mRf8yptj1+VElJCQlDfVPeDWEP9jH1SwDUZY1sjWs7BKD6/PpwgproVg4OsSk45Ozfxylrd33X64QBAIDKchhOOUw8PclhOL0XTD1EwgAAsASnDDl19hmDmXMbAur4AADAIyoMAABLcMopM50K5s6u/0gYAACW4DAMOYyz71Ywc25DQJcEAADwiAoDAMASGPRoDgkDAMASnDLkIGE4a3RJAAAAj6gwAAAsgS4Jc0gYAACWwCwJc+iSAAAAHlFhAABYgvPXzcz5VkaFAQBgCY5fZ0mY2api7dq1uuGGGxQVFSWbzaZly5a5HTcMQ8nJyWrVqpUCAwMVHx+v7777zq1Nbm6uEhMTZbfbFRYWpqFDh6qgoMCtzTfffKMrr7xSAQEBio6O1owZMyrE8vbbb6tTp04KCAhQ165d9dFHH1XpXiQSBgCARTgM81tVFBYWqlu3bpozZ85pj8+YMUOzZ89WSkqKNm7cqKCgICUkJKioqMjVJjExUTt27FBaWpqWL1+utWvXavjw4a7j+fn56tu3r9q2bauMjAzNnDlTkydP1iuvvOJqs2HDBt16660aOnSoNm/erAEDBmjAgAHavn17le7HZhj1dxRHfn6+QkNDdeTb9rKHkPugYUqI6l7bIQDVpswo1Wd6X3l5ebLb7dXyHuWfFd/sDFeIic+KY8ecujA2R/v373eL1d/fX/7+/r97rs1m09KlSzVgwABJJ6sLUVFReuCBBzR+/HhJUl5eniIiIpSamqrBgwdr165dio2N1VdffaWePXtKklauXKl+/frp559/VlRUlObOnatHHnlEWVlZ8vPzkyRNnDhRy5Yt0+7duyVJf/3rX1VYWKjly5e74rn00kvVvXt3paSkVPr++ZQFAFiC0wubJEVHRys0NNS1TZ8+vcqx7N27V1lZWYqPj3ftCw0NVa9evZSeni5JSk9PV1hYmCtZkKT4+Hj5+Pho48aNrja9e/d2JQuSlJCQoMzMTB05csTV5rfvU96m/H0qi0GPAABLcMomh2ymzpd02gpDVWVlZUmSIiIi3PZHRES4jmVlZSk8PNzteKNGjdSsWTO3NjExMRWuUX6sadOmysrK+t33qSwSBgAAqsBut1db90ldRpcEAMASnIb5zVsiIyMlSdnZ2W77s7OzXcciIyOVk5PjdrysrEy5ublubU53jd++x5nalB+vLBIGAIAlOH7tkjCzeUtMTIwiIyO1atUq1778/Hxt3LhRcXFxkqS4uDgdPXpUGRkZrjarV6+W0+lUr169XG3Wrl2r0tJSV5u0tDR17NhRTZs2dbX57fuUtyl/n8oiYQAAoBoUFBRoy5Yt2rJli6STAx23bNmiffv2yWazacyYMXr88cf1wQcfaNu2bbrjjjsUFRXlmknRuXNnXXfddbr77rv15Zdf6vPPP9eoUaM0ePBgRUVFSZJuu+02+fn5aejQodqxY4cWL16s559/XuPGjXPFMXr0aK1cuVLPPPOMdu/ercmTJ2vTpk0aNWpUle6HMQwAAEswWyWo6rmbNm1Snz59XK/LP8STkpKUmpqqCRMmqLCwUMOHD9fRo0d1xRVXaOXKlQoICHCds3DhQo0aNUrXXHONfHx8NGjQIM2ePdt1PDQ0VJ9++qlGjhypHj16qEWLFkpOTnZbq+Gyyy7TokWL9Oijj+r//u//dN5552nZsmXq0qVLle6HdRiAOo51GNCQ1eQ6DOu3RynYxGdFwTGnruhysFpjrcv4lAUAAB7RJQEAsISa7pJoaEgYAACW4JCPHCYK6w4vxlIfkTAAACzBMGxyGmdfJTBMnNsQMIYBAAB4RIUBAGAJjGEwh4QBAGAJDsNHDsPEGIZ6uwiBd9AlAQAAPKLCAACwBKdscpr4nuyUtUsMJAwAAEtgDIM5dEkAAACPqDAAACzB/KBHuiQAAGjwTo5hOPtuBTPnNgR0SQAAAI+oMAAALMFp8lkSzJIAAMACGMNgDgkDAMASnPJhHQYTGMMAAAA8osIAALAEh2GTw8Qjqs2c2xCQMAAALMFhctCjgy4JAACA30eFAQBgCU7DR04TsySczJIAAKDho0vCHLokAACAR1QYAACW4JS5mQ5O74VSL5EwAAAswfzCTdYuylv77gEAQKVQYQAAWIL5Z0lY+zs2CQMAwBKcsskpM2MYWOkRAIAGjwqDOda+ewAAUClUGAAAlmB+4SZrf8cmYQAAWILTsMlpZh0Giz+t0trpEgAAqBQqDAAAS3Ca7JKw+sJNJAwAAEsw/7RKaycM1r57AABQKVQYAACW4JBNDhOLL5k5tyEgYQAAWAJdEuZY++4BAEClUGEAAFiCQ+a6FRzeC6VeImEAAFgCXRLmkDAAACyBh0+ZY+27BwAAlUKFAQBgCYZscpoYw2AwrRIAgIaPLglzrH33AACgUqgwAAAsgcdbm0PCAACwBIfJp1WaObchsPbdAwCASqHCAACwBLokzCFhAABYglM+cpoorJs5tyGw9t0DAIBKocIAALAEh2GTw0S3gplzGwISBgCAJTCGwRwSBgCAJRgmn1ZpsNIjAADA76PCAACwBIdscph4gJSZcxsCEgYAgCU4DXPjEJyGF4Oph+iSAAAAHpEwNHDbvghS8h0xuvWiC5QQ1V0bPg51O24Y0vwZkbq1+wW6of2FeuiWc3XgBz+3Nnf8IVYJUd3dtsUvhLuOb90QrMfujNGt3S/Qjed21Yj4jlr9XlO3a/yYGaCpw9q5rvXeqy2r76aB33HLqGx9cnCr7ply4DRHDT3+rx/0ycGtirsu77TnhzQt07827dQnB7cqyO6o3mDhVc5fBz2a2ayMLokGrui4j9pfcEIJt+Zq6tCYCseXzAnX+6+11PjnflJkmxLNn9FK/3fbuXr1s93yCzhVf7vjwUO6PvG/rtdNgp2uf9+5qYnax57QLSOz1bRlmTb+266Z97dRkxCHLr02X5JUfMKmVm1K1PtPR/Xy5HOq8Y6BMzu/23H1vz1XP+wIOO3xP9/9iwwPZedxz+zX3l0BahlVWg0Rojo5ZZPTxDgEM+c2BHUiXZozZ47atWungIAA9erVS19++WVth9RgXPLHY7rzoSxdfn3Fb0uGIS2b11K3js7SZdflq31skSbM/kn/zW6sDSvdKxGBwU41Cy9zbQFNTiUMt96fo6QJWbrgkuOKaleiPw/7RT375Ovzj05do2P3E7o7+aCuHnBUjf0s3hGIWhHQxKGHXvxJzz3YWsfyfCscb3/BCQ36+2E9Oy76jNf40x2/KMju0DspVMjgmcPh0KRJkxQTE6PAwECde+65mjZtmozfZKWGYSg5OVmtWrVSYGCg4uPj9d1337ldJzc3V4mJibLb7QoLC9PQoUNVUFDg1uabb77RlVdeqYCAAEVHR2vGjBlev59aTxgWL16scePG6bHHHtPXX3+tbt26KSEhQTk5ObUdWoOXtc9PuTmNdfGVp37xguxOdbrouHZlBLm1XfJiuG6+oIvuvfZ8vf1SSznKfv/ahfm+CgmjXIu6Y9STB/TlKrs2rwupcMw/0KmJc37SnEfO0ZHDjU97fpvzinTb2GzNHN1GhtPa3zTrq/KVHs1sVfH0009r7ty5evHFF7Vr1y49/fTTmjFjhl544QVXmxkzZmj27NlKSUnRxo0bFRQUpISEBBUVFbnaJCYmaseOHUpLS9Py5cu1du1aDR8+3HU8Pz9fffv2Vdu2bZWRkaGZM2dq8uTJeuWVV8z/0H6j1rsknn32Wd19990aMmSIJCklJUUrVqzQa6+9pokTJ9ZydA1bbs7J//xhLd1Lq2EtS13HJOmmoYfVoesJhYSVaeemIL0+vZVycxrr75MPnva6az4I07dbm+j+GT9XX/BAFVx10xF16HpC9/U777TH/z75gHZuClL6J6GnPd7Yz6mHX/pJ86ZF6fABP7VqU1Kd4aKamB2HUNVzN2zYoJtuukn9+/eXJLVr105vvvmmq4puGIaee+45Pfroo7rpppskSQsWLFBERISWLVumwYMHa9euXVq5cqW++uor9ezZU5L0wgsvqF+/fvrHP/6hqKgoLVy4UCUlJXrttdfk5+enCy64QFu2bNGzzz7rlliYVasVhpKSEmVkZCg+Pt61z8fHR/Hx8UpPT6/Qvri4WPn5+W4bqt+gvx9Wt8sK1D62SH+6478annxQ77/WUiXFFbPtLZ8H65mx0Ro9c7/adSw6zdWAmtUyqkQjph7U06PaqLS44p+8S/vmqfvlBUpJjjrjNYY8fEj79gRUGMwLa/rfz6Hi4uLTtrvsssu0atUqffvtt5KkrVu3av369br++uslSXv37lVWVpbbZ2BoaKh69erl+gxMT09XWFiYK1mQpPj4ePn4+Gjjxo2uNr1795af36kB6wkJCcrMzNSRI0e8dt+1WmH45Zdf5HA4FBER4bY/IiJCu3fvrtB++vTpmjJlSk2F1+A1Cz/Zr3D0cGM1jzjVx3D0cGOde8GJM57X8eLjcpTZlL3fT9EdTv2P8k16kB5LitE9Uw7q2r9475cUMKPDhSfUtGWZ5nzyrWufbyOp66WFunHIL1q+oLlatSvRe7u3u5036dUftX1jkCbc3EHdryhQu05FurL/0ZMHf82V396+XW/OjtAb/4isobuBGU6ZfJbEr//ho6Pdx7k89thjmjx5coX2EydOVH5+vjp16iRfX185HA498cQTSkxMlCRlZWVJ0mk/A8uPZWVlKTw83O14o0aN1KxZM7c2MTExFa5RfqxpU+8kurXeJVEVDz/8sMaNG+d6nZ+fX+E/HCovsk2JmoWXavP6YJ3b5WSCUHjMR7s3N9Gf7vjljOf9sCNQPj6GwlqcSjK2bghW8h0xGvrIIfW7/b9nPBeoaVvWBWt4n/Pd9j0wa7/27wnQkjktlZ/bSCveaO52/JX/fKuXJ0fpi0/tkqRpw9rJL+DUQN+O3U/ogVn79cCfO+jgj+7TkFF3GSZnSRi/nrt//37Z7XbXfn9//9O2X7JkiRYuXKhFixa5ugnGjBmjqKgoJSUlnXUctaVWE4YWLVrI19dX2dnZbvuzs7MVGVkxY/f39z/jfxic3olCHx3ce+pnlrXfT99vD1RIWJnCW5dqwLDDevP5CJ0TU+yaVtk8olSX/ToHfeemJtq9OUjdLjumJsFO7coIUspjUfrjoCOuQY1bPj+ZLAwY9ouu6H/UNf6hUWND9qYn25SW2LTv25NT2UpLbfrvocb6fnugAoIcOieG/mBUnxOFvvopM9BtX9FxHx07cmr/6QY65hzwU/b+k//vHPrJ/e9OaLOTv9f7vgtQYX7FGReom7z1tEq73e6WMJzJgw8+qIkTJ2rw4MGSpK5du+qnn37S9OnTlZSU5Pqcy87OVqtWrVznZWdnq3v37pKkyMjICpMAysrKlJub6zo/MjLytJ+j5ce8pVYTBj8/P/Xo0UOrVq3SgAEDJElOp1OrVq3SqFGjajO0BuPbrU004eYOrtflayBce0uuxj+3T7eMzFHRcR89PyFaBfm+uuCSQj2x8AfXGgyN/QyteT9M/3omUqUlNkVGl2jg8MMaOPyw65r/fruZik/4avELEVr8wqnS2oVxBZr57h5J0n+zG+vevh1dx95JCdc7KeFubQCgITl+/Lh8fNzHzfj6+srpPFmtiomJUWRkpFatWuVKEPLz87Vx40aNGDFCkhQXF6ejR48qIyNDPXr0kCStXr1aTqdTvXr1crV55JFHVFpaqsaNTya/aWlp6tixo9e6IyTJZhielimpXosXL1ZSUpJefvll/eEPf9Bzzz2nJUuWaPfu3RX6df5Xfn6+QkNDdeTb9rKH1PoMUaBaJER1r+0QgGpTZpTqM72vvLy8Sn1rPxvlnxV/ThuixkFn34VUWliipde+XulY77zzTv373//Wyy+/rAsuuECbN2/W8OHDddddd+npp5+WdHLq5VNPPaX58+crJiZGkyZN0jfffKOdO3cqIOBkVfb6669Xdna2UlJSVFpaqiFDhqhnz55atGiRJCkvL08dO3ZU37599dBDD2n79u266667NGvWLK/Okqj1MQx//etfdfjwYSUnJysrK0vdu3fXypUrPSYLAABUhbe6JCrrhRde0KRJk3TvvfcqJydHUVFR+vvf/67k5GRXmwkTJqiwsFDDhw/X0aNHdcUVV2jlypWuZEGSFi5cqFGjRumaa66Rj4+PBg0apNmzZ7uOh4aG6tNPP9XIkSPVo0cPtWjRQsnJyV5NFqQ6UGEwgwoDrIAKAxqymqww3PTpXaYrDO/3fa1aY63Lar3CAABATeBZEuaQMAAALKGmuyQaGur4AADAIyoMAABLoMJgDgkDAMASSBjMoUsCAAB4RIUBAGAJVBjMIWEAAFiCIXNTI+vtokVeQsIAALAEKgzmMIYBAAB4RIUBAGAJVBjMIWEAAFgCCYM5dEkAAACPqDAAACyBCoM5JAwAAEswDJsMEx/6Zs5tCOiSAAAAHlFhAABYglM2Uws3mTm3ISBhAABYAmMYzKFLAgAAeESFAQBgCQx6NIeEAQBgCXRJmEPCAACwBCoM5jCGAQAAeESFAQBgCYbJLgmrVxhIGAAAlmBIMgxz51sZXRIAAMAjKgwAAEtwyiYbKz2eNRIGAIAlMEvCHLokAACAR1QYAACW4DRssrFw01kjYQAAWIJhmJwlYfFpEnRJAAAAj6gwAAAsgUGP5pAwAAAsgYTBHBIGAIAlMOjRHMYwAAAAj6gwAAAsgVkS5pAwAAAs4WTCYGYMgxeDqYfokgAAAB5RYQAAWAKzJMwhYQAAWILx62bmfCujSwIAAHhEhQEAYAl0SZhDwgAAsAb6JEwhYQAAWIPJCoMsXmFgDAMAAPCICgMAwBJY6dEcEgYAgCUw6NEcuiQAAIBHVBgAANZg2MwNXLR4hYGEAQBgCYxhMIcuCQAA4BEVBgCANbBwkykkDAAAS2CWhDmVShg++OCDSl/wxhtvPOtgAABA3VSphGHAgAGVupjNZpPD4TATDwAA1cfi3QpmVCphcDqd1R0HAADVii4Jc0zNkigqKvJWHAAAVC/DC5uFVTlhcDgcmjZtms455xwFBwfrhx9+kCRNmjRJ//znP70eIAAAqH1VThieeOIJpaamasaMGfLz83Pt79Kli+bNm+fV4AAA8B6bFzbrqnLCsGDBAr3yyitKTEyUr6+va3+3bt20e/durwYHAIDX0CVhSpUThgMHDqhDhw4V9judTpWWlnolKAAAULdUOWGIjY3VunXrKux/5513dNFFF3klKAAAvK4WKgwHDhzQ7bffrubNmyswMFBdu3bVpk2bToVkGEpOTlarVq0UGBio+Ph4fffdd27XyM3NVWJioux2u8LCwjR06FAVFBS4tfnmm2905ZVXKiAgQNHR0ZoxY0bVg/Wgyis9JicnKykpSQcOHJDT6dR7772nzMxMLViwQMuXL/d6gAAAeEUNP63yyJEjuvzyy9WnTx99/PHHatmypb777js1bdrU1WbGjBmaPXu25s+fr5iYGE2aNEkJCQnauXOnAgICJEmJiYk6dOiQ0tLSVFpaqiFDhmj48OFatGiRJCk/P199+/ZVfHy8UlJStG3bNt11110KCwvT8OHDz/5+/4fNMKr+/K1169Zp6tSp2rp1qwoKCnTxxRcrOTlZffv29VpglZGfn6/Q0FAd+ba97CE8RwsNU0JU99oOAag2ZUapPtP7ysvLk91ur5b3KP+siJ4zRT6BAWd9HeeJIu0f+VilY504caI+//zz01blpZPVhaioKD3wwAMaP368JCkvL08RERFKTU3V4MGDtWvXLsXGxuqrr75Sz549JUkrV65Uv3799PPPPysqKkpz587VI488oqysLNdkhIkTJ2rZsmVeHVt4Vp+yV155pdLS0pSTk6Pjx49r/fr1NZ4sAABQFeWPtzazSScTkN9uxcXFp32/Dz74QD179tRf/vIXhYeH66KLLtKrr77qOr53715lZWUpPj7etS80NFS9evVSenq6JCk9PV1hYWGuZEGS4uPj5ePjo40bN7ra9O7d223mYkJCgjIzM3XkyBGv/fzO+mv5pk2b9MYbb+iNN95QRkaG1wICAKBaeGkMQ3R0tEJDQ13b9OnTT/t2P/zwg+bOnavzzjtPn3zyiUaMGKH7779f8+fPlyRlZWVJkiIiItzOi4iIcB3LyspSeHi42/FGjRqpWbNmbm1Od43fvoc3VHkMw88//6xbb71Vn3/+ucLCwiRJR48e1WWXXaa33npLrVu39lpwAADUNfv373frkvD39z9tO6fTqZ49e+rJJ5+UJF100UXavn27UlJSlJSUVCOxelOVKwzDhg1TaWmpdu3apdzcXOXm5mrXrl1yOp0aNmxYdcQIAIB55YMezWyS7Ha723amhKFVq1aKjY1129e5c2ft27dPkhQZGSlJys7OdmuTnZ3tOhYZGamcnBy342VlZcrNzXVrc7pr/PY9vKHKCcOaNWs0d+5cdezY0bWvY8eOeuGFF7R27VqvBQYAgDfZDPNbVVx++eXKzMx02/ftt9+qbdu2kqSYmBhFRkZq1apVruP5+fnauHGj4uLiJElxcXE6evSoW9f/6tWr5XQ61atXL1ebtWvXuq2FlJaWpo4dO7rNyDCryglDdHT0aRdocjgcioqK8kpQAAB4XQ2vwzB27Fh98cUXevLJJ7Vnzx4tWrRIr7zyikaOHClJstlsGjNmjB5//HF98MEH2rZtm+644w5FRUVpwIABkk5WJK677jrdfffd+vLLL/X5559r1KhRGjx4sOsz97bbbpOfn5+GDh2qHTt2aPHixXr++ec1btw4Mz+tCqqcMMycOVP33Xef28ITmzZt0ujRo/WPf/zDq8EBAFBfXXLJJVq6dKnefPNNdenSRdOmTdNzzz2nxMREV5sJEybovvvu0/Dhw3XJJZeooKBAK1eudK3BIEkLFy5Up06ddM0116hfv3664oor9Morr7iOh4aG6tNPP9XevXvVo0cPPfDAA0pOTvbqGgxSJddhaNq0qWy2UwtWFBYWqqysTI0anRwzWf7vQUFBys3N9WqAv4d1GGAFrMOAhqxG12GYNc38OgxjJ1VrrHVZpWZJPPfcc9UcBgAA1czsA6Qs/vCpSiUM9XH6BwAA8J4qr8PwW0VFRSopKXHbZ8UyDQCgHqDCYEqVO/4LCws1atQohYeHKygoSE2bNnXbAACok2rhaZUNSZUThgkTJmj16tWaO3eu/P39NW/ePE2ZMkVRUVFasGBBdcQIAABqWZW7JD788EMtWLBAV199tYYMGaIrr7xSHTp0UNu2bbVw4UK36SIAANQZNfx464amyhWG3NxctW/fXtLJ8Qrl0yivuOIKVnoEANRZNb3SY0NT5YShffv22rt3rySpU6dOWrJkiaSTlYfyh1EBAICGpcoJw5AhQ7R161ZJ0sSJEzVnzhwFBARo7NixevDBB70eIAAAXsGgR1OqPIZh7Nixrn+Pj4/X7t27lZGRoQ4dOujCCy/0anAAAKBuMLUOgyS1bdvW9eQtAADqKpvMjUOw9pDHSiYMs2fPrvQF77///rMOBgAA1E2VShhmzZpVqYvZbLZaSRgGdu2pRrbGNf6+QE3waeJb2yEA1cbHKJGO19CbMa3SlEolDOWzIgAAqLdYGtoUngkNAAA8Mj3oEQCAeoEKgykkDAAASzC7WiMrPQIAAHhAhQEAYA10SZhyVhWGdevW6fbbb1dcXJwOHDggSXrjjTe0fv16rwYHAIDXsDS0KVVOGN59910lJCQoMDBQmzdvVnFxsSQpLy9PTz75pNcDBAAAta/KCcPjjz+ulJQUvfrqq2rc+NRiSZdffrm+/vprrwYHAIC38Hhrc6o8hiEzM1O9e/eusD80NFRHjx71RkwAAHgfKz2aUuUKQ2RkpPbs2VNh//r169W+fXuvBAUAgNcxhsGUKicMd999t0aPHq2NGzfKZrPp4MGDWrhwocaPH68RI0ZUR4wAAKCWVblLYuLEiXI6nbrmmmt0/Phx9e7dW/7+/ho/frzuu+++6ogRAADTWLjJnConDDabTY888ogefPBB7dmzRwUFBYqNjVVwcHB1xAcAgHewDoMpZ71wk5+fn2JjY70ZCwAAqKOqnDD06dNHNtuZR4quXr3aVEAAAFQLs1MjqTBUTffu3d1el5aWasuWLdq+fbuSkpK8FRcAAN5Fl4QpVU4YZs2addr9kydPVkFBgemAAABA3eO1p1Xefvvteu2117x1OQAAvIt1GEzx2tMq09PTFRAQ4K3LAQDgVUyrNKfKCcPAgQPdXhuGoUOHDmnTpk2aNGmS1wIDAAB1R5UThtDQULfXPj4+6tixo6ZOnaq+fft6LTAAAFB3VClhcDgcGjJkiLp27aqmTZtWV0wAAHgfsyRMqdKgR19fX/Xt25enUgIA6h0eb21OlWdJdOnSRT/88EN1xAIAAOqoKicMjz/+uMaPH6/ly5fr0KFDys/Pd9sAAKizmFJ51io9hmHq1Kl64IEH1K9fP0nSjTfe6LZEtGEYstlscjgc3o8SAACzGMNgSqUThilTpuiee+7Rf/7zn+qMBwAA1EGVThgM42RqddVVV1VbMAAAVBcWbjKnStMqf+8plQAA1Gl0SZhSpYTh/PPP95g05ObmmgoIAADUPVVKGKZMmVJhpUcAAOoDuiTMqVLCMHjwYIWHh1dXLAAAVB+6JEyp9DoMjF8AAMC6qjxLAgCAeokKgymVThicTmd1xgEAQLViDIM5VX68NQAA9RIVBlOq/CwJAABgPVQYAADWQIXBFBIGAIAlMIbBHLokAACAR1QYAADWQJeEKSQMAABLoEvCHLokAACAR1QYAADWQJeEKSQMAABrIGEwhS4JAADgERUGAIAl2H7dzJxvZSQMAABroEvCFBIGAIAlMK3SHMYwAABQzZ566inZbDaNGTPGta+oqEgjR45U8+bNFRwcrEGDBik7O9vtvH379ql///5q0qSJwsPD9eCDD6qsrMytzWeffaaLL75Y/v7+6tChg1JTU6vlHkgYAADWYHhhOwtfffWVXn75ZV144YVu+8eOHasPP/xQb7/9ttasWaODBw9q4MCBruMOh0P9+/dXSUmJNmzYoPnz5ys1NVXJycmuNnv37lX//v3Vp08fbdmyRWPGjNGwYcP0ySefnF2wv4OEAQBgHV5IFvLz89224uLiM75dQUGBEhMT9eqrr6pp06au/Xl5efrnP/+pZ599Vn/84x/Vo0cPvf7669qwYYO++OILSdKnn36qnTt36l//+pe6d++u66+/XtOmTdOcOXNUUlIiSUpJSVFMTIyeeeYZde7cWaNGjdLNN9+sWbNmeesn5kLCAABAFURHRys0NNS1TZ8+/YxtR44cqf79+ys+Pt5tf0ZGhkpLS932d+rUSW3atFF6erokKT09XV27dlVERISrTUJCgvLz87Vjxw5Xm/+9dkJCgusa3sSgRwCAJXhr0OP+/ftlt9td+/39/U/b/q233tLXX3+tr776qsKxrKws+fn5KSwszG1/RESEsrKyXG1+myyUHy8/9ntt8vPzdeLECQUGBlb+Bj0gYQAAWIOXplXa7Xa3hOF09u/fr9GjRystLU0BAQEm3rTuoEsCAAAvy8jIUE5Oji6++GI1atRIjRo10po1azR79mw1atRIERERKikp0dGjR93Oy87OVmRkpCQpMjKywqyJ8tee2tjtdq9WFyQSBgCARZR3SZjZKuuaa67Rtm3btGXLFtfWs2dPJSYmuv69cePGWrVqleuczMxM7du3T3FxcZKkuLg4bdu2TTk5Oa42aWlpstvtio2NdbX57TXK25Rfw5vokgAAWEMNrvQYEhKiLl26uO0LCgpS8+bNXfuHDh2qcePGqVmzZrLb7brvvvsUFxenSy+9VJLUt29fxcbG6m9/+5tmzJihrKwsPfrooxo5cqRr3MQ999yjF198URMmTNBdd92l1atXa8mSJVqxYoWJGz09EgYAAGrBrFmz5OPjo0GDBqm4uFgJCQl66aWXXMd9fX21fPlyjRgxQnFxcQoKClJSUpKmTp3qahMTE6MVK1Zo7Nixev7559W6dWvNmzdPCQkJXo/XZhhGvV3sMj8/X6Ghoerjf4sa2RrXdjhAtbD5+tZ2CEC1KTNKtPr4W8rLy/M4kPBslX9WXHjXk/L1O/sBiI6SIn3z2v9Va6x1GRUGAIA18PApU0gYAADWQMJgCrMkAACAR1QYAACWwOOtzSFhAABYA10SptAlAQAAPKLCAACwBJthyGZiJQEz5zYEJAwAAGugS8IUuiQAAIBHVBgAAJbALAlzSBgAANZAl4QpdEkAAACPqDAAACyBLglzSBgAANZAl4QpJAwAAEugwmAOYxgAAIBHVBgAANZAl4QpJAwAAMuwereCGXRJAAAAj6gwAACswTBObmbOtzASBgCAJTBLwhy6JAAAgEdUGAAA1sAsCVNIGAAAlmBzntzMnG9ldEkAAACPqDBAzSNKNHTifvW86qj8A506+GOAnp0Qo++2BUuSVu798rTnzZserXdeaSVJGjzyoP7Q56jaxx5XWalNN3frUWPxA78n9bOvFdG6uML+D/8VoQWzovW30T/r4iuOqmVUsfJyGys9rZkWzIrW8YJTfx7vmbRXsT2Oqd35x7VvT6BG3ditJm8B3kKXhCkkDBYXbC/Ts+/s1NZ0ux4d0lF5/22sc2KKVJB36lfj1ku6u53T8+o8jX16r9Z/3NS1r1Fjp9Z91Ey7Ngcr4ZbDNRU+4NHogV3l43PqL33b849r+oJdWvdxczUPL1Wz8BLNe6qt9u1povCoYo2a9oOaR5ToiVEd3a7z6Tvh6tjtmGI6Hq/pW4CXMEvCnFpNGNauXauZM2cqIyNDhw4d0tKlSzVgwIDaDMly/nLPIR0+5KdnJ7R37cv+2d+tzZFf/Nxex117RFvT7craH+Da96/nWkuSrh1EsoC6JS+3sdvrW/5+QAd/8te2jXZJNrfE4NC+AM1/to0mPPOdfHwNOR02SVLKtBhJUmizUhKG+ox1GEyp1TEMhYWF6tatm+bMmVObYVjapfFH9O03QXpkznd666uv9eLy7bpucM4Z24e1KNUf+uTpkyUtajBKwDsaNXaqz02/6NN3wiXZTtsmKKRMxwt8XckCgJNqtcJw/fXX6/rrr690++LiYhUXn+qLzM/Pr46wLKVVm2L96fYcvTcvUm/NidL53Qo14rGfVFZi07/fa1mhffygX3Si0Eefr2xWC9EC5sRdm6tge5nS3g0/7XF701LdOvJnffxWRA1HhppAl4Q59WqWxPTp0xUaGuraoqOjazukes9mk/ZsD1LqP6L1/c4gffxmuFa+Fa7+iaevMiT85bBWv99cpSX16lcHkCQl/CVHm9Y2VW6OX4VjTYLLNOXV3dq3p4n+Nbt1LUSHamd4YbOwevVX/+GHH1ZeXp5r279/f22HVO/lHm6sfXsC3fbt2xOgllElFdpecMkxRZ9bpJWLT//tDKjLwqOK1f2yPK1cUvH3NzDIoWmv7dKJQl9NG9FRjrJ69acRqBH1apaEv7+//P39PTdEpe3cFKzW7U+47Tsnpkg5Byr+nK+75bC+/aaJ9u5qUlPhAV5z7c05yvtvY335n6Zu+5sEl+nx13eptMRHU/7ekepZA0aXhDn8n2FxS1+LVKfuhfrrvQfVqm2Rrr7xF/W79bA+fMP9W1iTYIeu7Jd7xupCy6hite9cqJZRJfLxMdS+c6Hady5UQBNHTdwG8LtsNkPXDsrRv5e2dBvM2CS4TE+k7lJAoFPPPXyumgQ71LRFiZq2KHGbitmq7Qm171yopi1K5R/gdP1+N2ps8aX/6pvyWRJmNgurVxUGeN+33wRr6j0dNOTBn5V4/wFl7fdXyrQ2+s/77rMgrrrhv5JN+uzD0w92vGPsAV178y+u1y99tEOSNGFwJ32z0V59NwBUwkWX5yninBJ9+rZ7wnvuBYXq1L1AkvTa6s1ux5Kuukg5B05OHR7z5A+6sNepQdZzPvymQhugobMZRu2lTAUFBdqzZ48k6aKLLtKzzz6rPn36qFmzZmrTpo3H8/Pz8xUaGqo+/reoka2xx/ZAfWTz9a3tEIBqU2aUaPXxt5SXlye7vXq+XJR/VsRdP1WNGp99gldWWqT0j5OrNda6rFYrDJs2bVKfPn1cr8eNGydJSkpKUmpqai1FBQBokFga2pRaTRiuvvpq1WKBAwAAVBJjGAAAlsAsCXNIGAAA1uA0Tm5mzrcwEgYAgDUwhsEU1mEAAAAeUWEAAFiCTSbHMHgtkvqJhAEAYA1mV2u0+Kw+uiQAAIBHVBgAAJbAtEpzSBgAANbALAlT6JIAAAAeUWEAAFiCzTBkMzFw0cy5DQEJAwDAGpy/bmbOtzC6JAAAgEdUGAAAlkCXhDkkDAAAa2CWhCkkDAAAa2ClR1MYwwAAADyiwgAAsARWejSHhAEAYA10SZhClwQAAPCICgMAwBJszpObmfOtjIQBAGANdEmYQpcEAADwiAoDAMAaWLjJFCoMAABLKF8a2sxWFdOnT9cll1yikJAQhYeHa8CAAcrMzHRrU1RUpJEjR6p58+YKDg7WoEGDlJ2d7dZm37596t+/v5o0aaLw8HA9+OCDKisrc2vz2Wef6eKLL5a/v786dOig1NTUs/oZ/R4SBgAAqsGaNWs0cuRIffHFF0pLS1Npaan69u2rwsJCV5uxY8fqww8/1Ntvv601a9bo4MGDGjhwoOu4w+FQ//79VVJSog0bNmj+/PlKTU1VcnKyq83evXvVv39/9enTR1u2bNGYMWM0bNgwffLJJ169H5th1N9RHPn5+QoNDVUf/1vUyNa4tsMBqoXN17e2QwCqTZlRotXH31JeXp7sdnu1vIfrs6LHw2rUKOCsr1NWVqT/ZEw/61gPHz6s8PBwrVmzRr1791ZeXp5atmypRYsW6eabb5Yk7d69W507d1Z6erouvfRSffzxx/rTn/6kgwcPKiIiQpKUkpKihx56SIcPH5afn58eeughrVixQtu3b3e91+DBg3X06FGtXLnyrO/3f1FhAABYgyHJaWL79et1fn6+21ZcXFypt8/Ly5MkNWvWTJKUkZGh0tJSxcfHu9p06tRJbdq0UXp6uiQpPT1dXbt2dSULkpSQkKD8/Hzt2LHD1ea31yhvU34NbyFhAABYgrfGMERHRys0NNS1TZ8+3eN7O51OjRkzRpdffrm6dOkiScrKypKfn5/CwsLc2kZERCgrK8vV5rfJQvnx8mO/1yY/P18nTpyo+g/qDJglAQBAFezfv9+tS8Lf39/jOSNHjtT27du1fv366gytWpEwAACswZDJhZtO/sNut1dpDMOoUaO0fPlyrV27Vq1bt3btj4yMVElJiY4ePepWZcjOzlZkZKSrzZdfful2vfJZFL9t878zK7Kzs2W32xUYGFjpOD2hSwIAYA3lKz2a2ar0doZGjRqlpUuXavXq1YqJiXE73qNHDzVu3FirVq1y7cvMzNS+ffsUFxcnSYqLi9O2bduUk5PjapOWlia73a7Y2FhXm99eo7xN+TW8hQoDAADVYOTIkVq0aJHef/99hYSEuMYchIaGKjAwUKGhoRo6dKjGjRunZs2ayW6367777lNcXJwuvfRSSVLfvn0VGxurv/3tb5oxY4aysrL06KOPauTIka6ukHvuuUcvvviiJkyYoLvuukurV6/WkiVLtGLFCq/eDwkDAMAanJJsJs+vgrlz50qSrr76arf9r7/+uu68805J0qxZs+Tj46NBgwapuLhYCQkJeumll1xtfX19tXz5co0YMUJxcXEKCgpSUlKSpk6d6moTExOjFStWaOzYsXr++efVunVrzZs3TwkJCWd1m2fCOgxAHcc6DGjIanIdhmu6TFAjX88DFM+kzFGsVdtnVGusdRljGAAAgEd0SQAArIHHW5tCwgAAsAYSBlPokgAAAB5RYQAAWAMVBlNIGAAA1lDD0yobGhIGAIAl/PYBUmd7vpUxhgEAAHhEhQEAYA2MYTCFhAEAYA1OQ7KZ+NB3WjthoEsCAAB4RIUBAGANdEmYQsIAALAIkwmDrJ0w0CUBAAA8osIAALAGuiRMIWEAAFiD05CpbgVmSQAAAPw+KgwAAGswnCc3M+dbGAkDAMAaGMNgCgkDAMAaGMNgCmMYAACAR1QYAADWQJeEKSQMAABrMGQyYfBaJPUSXRIAAMAjKgwAAGugS8IUEgYAgDU4nZJMrKXgtPY6DHRJAAAAj6gwAACsgS4JU0gYAADWQMJgCl0SAADAIyoMAABrYGloU0gYAACWYBhOGSaeOGnm3IaAhAEAYA2GYa5KwBgGAACA30eFAQBgDYbJMQwWrzCQMAAArMHplGwmxiFYfAwDXRIAAMAjKgwAAGugS8IUEgYAgCUYTqcME10SVp9WSZcEAADwiAoDAMAa6JIwhYQBAGANTkOykTCcLbokAACAR1QYAADWYBiSzKzDYO0KAwkDAMASDKchw0SXhEHCAACABRhOmaswMK0SAADgd1FhAABYAl0S5pAwAACsgS4JU+p1wlCe7ZUZpbUcCVB9bIajtkMAqk353++a+PZeplJT6zaVydqfNfU6YTh27JgkaV3J0lqOBABgxrFjxxQaGlot1/bz81NkZKTWZ31k+lqRkZHy8/PzQlT1j82ox50yTqdTBw8eVEhIiGw2W22HYwn5+fmKjo7W/v37ZbfbazscwKv4/a55hmHo2LFjioqKko9P9Y3DLyoqUklJienr+Pn5KSAgwAsR1T/1usLg4+Oj1q1b13YYlmS32/mDigaL3++aVV2Vhd8KCAiw7Ae9tzCtEgAAeETCAAAAPCJhQJX4+/vrsccek7+/f22HAngdv9/AmdXrQY8AAKBmUGEAAAAekTAAAACPSBgAAIBHJAwAAMAjEgZU2pw5c9SuXTsFBASoV69e+vLLL2s7JMAr1q5dqxtuuEFRUVGy2WxatmxZbYcE1DkkDKiUxYsXa9y4cXrsscf09ddfq1u3bkpISFBOTk5thwaYVlhYqG7dumnOnDm1HQpQZzGtEpXSq1cvXXLJJXrxxRclnXyOR3R0tO677z5NnDixlqMDvMdms2np0qUaMGBAbYcC1ClUGOBRSUmJMjIyFB8f79rn4+Oj+Ph4paen12JkAICaQsIAj3755Rc5HA5FRES47Y+IiFBWVlYtRQUAqEkkDAAAwCMSBnjUokUL+fr6Kjs7221/dna2IiMjaykqAEBNImGAR35+furRo4dWrVrl2ud0OrVq1SrFxcXVYmQAgJrSqLYDQP0wbtw4JSUlqWfPnvrDH/6g5557ToWFhRoyZEhthwaYVlBQoD179rhe7927V1u2bFGzZs3Upk2bWowMqDuYVolKe/HFFzVz5kxlZWWpe/fumj17tnr16lXbYQGmffbZZ+rTp0+F/UlJSUpNTa35gIA6iIQBAAB4xBgGAADgEQkDAADwiIQBAAB4RMIAAAA8ImEAAAAekTAAAACPSBgAAIBHJAwAAMAjEgbApDvvvFMDBgxwvb766qs1ZsyYGo/js88+k81m09GjR8/YxmazadmyZZW+5uTJk9W9e3dTcf3444+y2WzasmWLqesAqF0kDGiQ7rzzTtlsNtlsNvn5+alDhw6aOnWqysrKqv2933vvPU2bNq1SbSvzIQ8AdQEPn0KDdd111+n1119XcXGxPvroI40cOVKNGzfWww8/XKFtSUmJ/Pz8vPK+zZo188p1AKAuocKABsvf31+RkZFq27atRowYofj4eH3wwQeSTnUjPPHEE4qKilLHjh0lSfv379ctt9yisLAwNWvWTDfddJN+/PFH1zUdDofGjRunsLAwNW/eXBMmTND/Po7lf7skiouL9dBDDyk6Olr+/v7q0KGD/vnPf+rHH390PfCoadOmstlsuvPOOyWdfHz49OnTFRMTo8DAQHXr1k3vvPOO2/t89NFHOv/88xUYGKg+ffq4xVlZDz30kM4//3w1adJE7du316RJk1RaWlqh3csvv6zo6Gg1adJEt9xyi/Ly8tyOz5s3T507d1ZAQIA6deqkl156qcqxAKjbSBhgGYGBgSopKXG9XrVqlTIzM5WWlqbly5ertLRUCQkJCgkJ0bp16/T5558rODhY1113neu8Z555RqmpqXrttde0fv165ebmaunSpb/7vnfccYfefPNNzZ49W7t27dLLL7+s4OBgRUdH691335UkZWZm6tChQ3r++eclSdOnT9eCBQuUkpKiHTt2aOzYsbr99tu1Zs0aSScTm4EDB+qGG27Qli1bNGzYME2cOLHKP5OQkBClpqZq586dev755/Xqq69q1qxZbm327NmjJUuW6MMPP9TKlSu1efNm3Xvvva7jCxcuVHJysp544gnt2rVLTz75pCZNmqT58+dXOR4AdZgBNEBJSUnGTTfdZBiGYTidTiMtLc3w9/c3xo8f7zoeERFhFBcXu8554403jI4dOxpOp9O1r7i42AgMDDQ++eQTwzAMo1WrVsaMGTNcx0tLS43WrVu73sswDOOqq64yRo8ebRiGYWRmZhqSjLS0tNPG+Z///MeQZBw5csS1r6ioyGjSpImxYcMGt7ZDhw41br31VsMwDOPhhx82YmNj3Y4/9NBDFa71vyQZS5cuPePxmTNnGj169HC9fuyxxwxfX1/j559/du37+OOPDR8fH+PQoUOGYRjGueeeayxatMjtOtOmTTPi4uIMwzCMvXv3GpKMzZs3n/F9AdR9jGFAg7V8+XIFBwertLRUTqdTt912myZPnuw63rVrV7dxC1u3btWePXsUEhLidp2ioiJ9//33ysvL06FDh9SrVy/XsUaNGqlnz54VuiXKbdmyRb6+vrrqqqsqHfeePXt0/PhxXXvttW77S0pKdNFFF0mSdu3a5RaHJMXFxVX6PcotXrxYs2fP1vfff6+CggKVlZXJbre7tWnTpo3OOecct/dxOp3KzMxUSEiIvv/+ew0dOlR33323q01ZWZlCQ0OrHA+AuouEAQ1Wnz59NHfuXPn5+SkqKkqNGrn/ugcFBbm9LigoUI8ePbRw4cIK12rZsuVZxRAYGFjlcwoKCiRJK1ascPuglk6Oy/CW9PR0JSYmasqUKUpISFBoaKjeeustPfPMM1WO9dVXX62QwPj6+notVgC1j4QBDVZQUJA6dOhQ6fYXX3yxFi9erPDw8Arfssu1atVKGzduVO/evSWd/CadkZGhiy+++LTtu3btKqfTqTVr1ig+Pr7C8fIKh8PhcO2LjY2Vv7+/9u3bd8bKROfOnV0DOMt98cUXnm/yNzZs2KC2bdvqkUcece376aefKrTbt2+fDh48qKioKNf7+Pj4qGPHjoqIiFBUVJR++OEHJSYmVun9AdQvDHoEfpWYmKgWLVropptu0rp167R371599tlnuv/++/Xzzz9LkkaPHq2nnnpKy5Yt0+7du3Xvvff+7hoK7dq1U1JSku666y4tW7bMdc0lS5ZIktq2bSubzably5fr8OHDKigoUEhIiMaPH6+xY8dq/vz5+v777/X111/rhRdecA0kvOeee/Tdd9/pwQcfVGZmphYtWqTU1NQq3e95552nffv26a233tL333+v2bNnn3YAZ0BAgJKSkrR161atW7dO999/v2655RZFRkZKkqZMmaLp06dr9uzZ+vbbb7Vt2za9/vrrevbZZ6sUD4C6jYQB+FWTJk20du1atWnTRgMHDlTnzp01dOhQFRUVuSoODzzwgP72t78pKSlJcXFxCgkJ0Z///Offve7cuXN18803695771WnTp109913q7CwUJJ0zjnnaMqUKZo4caIiIiI0atQoSdK0adM0adIkTZ8+XZ07d9Z1112nFStWKCYmRtLJcQXvvvuuli1bpm7duiklJUVPPvlkle73xhtv1NixYzVq1Ch1795dGzZs0KRJkyq069ChgwYOHKh+/fqpb9++uvDCC92mTQ4bNkzz5s3T66+/rq5du+qqq65SamqqK1YADYPNONNoLQAAgF9RYQAAAB6RMAAAAI9IGAAAgEckDAAAwCMSBgAA4BEJAwAA8IiEAQAAeETCAAAAPCJhAAAAHpEwAAAAj0gYAACAR/8PI7TAEbP64VMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "create_confusion_matrix_plot(model, y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccbc69ab",
   "metadata": {},
   "source": [
    "### MLFlow work Starts from here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "34fe5fb0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'accuracy': 0.91, 'precision': 0.62, 'recall': 0.52, 'entropy': 0.2}\n"
     ]
    }
   ],
   "source": [
    "experiment_name = \"basic_classifier\" ##basic classifier\n",
    "run_name=\"term_deposit\"\n",
    "run_metrics = get_metrics(y_test, y_pred, y_pred_prob)\n",
    "print(run_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e6223bed",
   "metadata": {},
   "outputs": [],
   "source": [
    "create_experiment(experiment_name,run_name,run_metrics,model,'confusion_matrix.png', 'roc_auc_curve.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa32199c",
   "metadata": {},
   "source": [
    "### Function to create an experiment in MLFlow and log parameters, metrics and artifacts files like images etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4424c524",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_experiment(experiment_name,run_name, run_metrics,model, confusion_matrix_path = None, \n",
    "                      roc_auc_plot_path = None, run_params=None):\n",
    "    import mlflow\n",
    "    mlflow.set_tracking_uri(\"http://localhost:5000\") #uncomment this line if you want to use any database like sqlite as backend storage for model\n",
    "    mlflow.set_experiment(experiment_name)\n",
    "    \n",
    "    with mlflow.start_run():\n",
    "        \n",
    "        if not run_params == None:\n",
    "            for param in run_params:\n",
    "                mlflow.log_param(param, run_params[param])\n",
    "            \n",
    "        for metric in run_metrics:\n",
    "            mlflow.log_metric(metric, run_metrics[metric])\n",
    "        \n",
    "        mlflow.sklearn.log_model(model, \"model\")\n",
    "        \n",
    "        if not confusion_matrix_path == None:\n",
    "            mlflow.log_artifact(confusion_matrix_path, 'confusion_materix')\n",
    "            \n",
    "        if not roc_auc_plot_path == None:\n",
    "            mlflow.log_artifact(roc_auc_plot_path, \"roc_auc_plot\")\n",
    "        \n",
    "        mlflow.set_tag(\"tag1\", \"Random Forest\")\n",
    "        mlflow.set_tags({\"tag2\":\"Randomized Search CV\", \"tag3\":\"Production\"})\n",
    "            \n",
    "    print('Run - %s is logged to Experiment - %s' %(run_name, experiment_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed79d9af",
   "metadata": {},
   "source": [
    "### Create another experiment after tuning hyperparameters and log the best set of parameters for which model gives the optimal performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "1ebd900c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 100 candidates, totalling 500 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Ashutosh Tripathi\\anaconda3\\envs\\mlops\\lib\\site-packages\\sklearn\\ensemble\\_forest.py:427: FutureWarning: `max_features='auto'` has been deprecated in 1.1 and will be removed in 1.3. To keep the past behaviour, explicitly set `max_features='sqrt'` or remove this parameter as it is also the default value for RandomForestClassifiers and ExtraTreesClassifiers.\n",
      "  warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random grid:  {'n_estimators': [5, 21, 51, 101], 'max_features': ['auto', 'sqrt'], 'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120], 'min_samples_split': [2, 6, 10], 'min_samples_leaf': [1, 3, 4], 'bootstrap': [True, False]} \n",
      "\n",
      "Best Parameters:  {'n_estimators': 101, 'min_samples_split': 10, 'min_samples_leaf': 4, 'max_features': 'auto', 'max_depth': 50, 'bootstrap': True}  \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Ashutosh Tripathi\\anaconda3\\envs\\mlops\\lib\\site-packages\\sklearn\\ensemble\\_forest.py:427: FutureWarning: `max_features='auto'` has been deprecated in 1.1 and will be removed in 1.3. To keep the past behaviour, explicitly set `max_features='sqrt'` or remove this parameter as it is also the default value for RandomForestClassifiers and ExtraTreesClassifiers.\n",
      "  warn(\n"
     ]
    }
   ],
   "source": [
    "import mlflow\n",
    "experiment_name = \"optimized model\"\n",
    "run_name=\"Random_Search_CV_Tuned_Model\"\n",
    "model_tuned,best_params = hyper_parameter_tuning(X_train, y_train)\n",
    "run_params = best_params\n",
    "\n",
    "y_pred = predict_on_test_data(model_tuned,X_test) #will return the predicted class\n",
    "y_pred_prob = predict_prob_on_test_data(model_tuned,X_test) #model.predict_proba(X_test)\n",
    "run_metrics = get_metrics(y_test, y_pred, y_pred_prob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "ef84e6c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.91, 'precision': 0.61, 'recall': 0.57, 'entropy': 0.2}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "c6d4067c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_estimators 101\n",
      "min_samples_split 10\n",
      "min_samples_leaf 4\n",
      "max_features auto\n",
      "max_depth 50\n",
      "bootstrap True\n"
     ]
    }
   ],
   "source": [
    "for param in run_params:\n",
    "    print(param, run_params[param])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "cce6bbb8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run - Random_Search_CV_Tuned_Model is logged to Experiment - optimized model\n"
     ]
    }
   ],
   "source": [
    "create_experiment(experiment_name,run_name,run_metrics,model_tuned,'confusion_matrix.png', 'roc_auc_curve.png',run_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef8bf23f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "34ab99c2",
   "metadata": {},
   "source": [
    "\n",
    "### if want to use the model registry feature, we need a database.\n",
    "\n",
    "#### _If you have MySQL installed then you can use the below command:_\n",
    "\n",
    "1. Create a database to use as an MLflow backend tracking server.\n",
    "\n",
    "`CREATE DATABASE mlflow_tracking_database;`\n",
    "\n",
    "2. Start MLflow tracking server using MySQL as a backend tracking store.\n",
    "` mlflow server \\\n",
    "   --backend-store-uri  mysql+pymysql://root@localhost/mlflow_tracking_database \\ \n",
    "   --default-artifact-root  file:/./mlruns \\\n",
    "   -h 0.0.0.0 -p 5000`\n",
    "\n",
    "\n",
    "3. Set the MLflow tracking uri (within code section).\n",
    "\n",
    "  mlflow.set_tracking_uri(\"http://localhost:5000\")\n",
    "\n",
    "#### _If you have sqlite installed then you can use the below command:_\n",
    "\n",
    "1. Start MLflow tracking server using sqlite as a backend tracking store.\n",
    "\n",
    "`mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./artifacts --host 0.0.0.0 --port 5000`\n",
    "\n",
    "\n",
    "2. Set the MLflow tracking uri (within code section).\n",
    "    \n",
    "    mlflow.set_tracking_uri(\"http://localhost:5000\")\n",
    "\n",
    "\n",
    "You can also follow the official documentation for more information on backend database for model registry\n",
    "\n",
    "https://www.mlflow.org/docs/latest/model-registry.html#model-registry-workflows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e90e5365",
   "metadata": {},
   "source": [
    "## Next steps:\n",
    "\n",
    "Using MLFlow\n",
    "* How deploy models from model registry\n",
    "* Model serving both batch serving and online serving\n",
    "* MLFlow pipelines\n",
    "* Model Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f323251",
   "metadata": {},
   "source": [
    "# Thank You "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1fbd185",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}