Customer_Credit_Risk_Analysis.ipynb

{"cells":[{"cell_type":"markdown","source":["# **Customer Credit Risk Analysis**"],"metadata":{"id":"6Q9NBSuPpkqZ"}},{"cell_type":"markdown","metadata":{"id":"yHlGQE4ZLhyk"},"source":["# Project Goals\n","\n","We have to prepare a report for credit division of a bank. We will find out the effect of a customer's marital status and the number of children he has on the probability of default in repayment of the loan. The bank already has some data regarding the creditworthiness of customers.\n","\n","Our report will be considered when making **credit assessment** for prospective customers. **Credit scoring** is used to evaluate the ability of potential borrowers to repay their loans.\n","\n","The objective of this project is to prepare credit assessment reports on prospective customers for a bank's credit division. Credit scoring is used to evaluate the ability of potential borrowers to repay their loans. The task that will be tested is to find out the correlation of a customer's marital status and the number of children he has on the probability of default in repaying a loan."]},{"cell_type":"markdown","metadata":{"id":"ZFCiSQAzLhyl"},"source":["## Open the data *file* and read the general information.\n"]},{"cell_type":"code","execution_count":111,"metadata":{"trusted":true,"id":"WSQjw2RSLhym","executionInfo":{"status":"ok","timestamp":1689650771637,"user_tz":-420,"elapsed":439,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# Load all *library*\n","import pandas as pd\n","import numpy as py\n","import seaborn as sns\n","import warnings\n","warnings.filterwarnings('ignore')\n","\n","#for importing files to google collab\n","from google.colab import files"]},{"cell_type":"code","source":["#upload datasets\n","#uploaded = files.upload()"],"metadata":{"id":"MWGtQLnRm3vB","executionInfo":{"status":"ok","timestamp":1689650772695,"user_tz":-420,"elapsed":43,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"execution_count":112,"outputs":[]},{"cell_type":"code","source":["# Load the data\n","df = pd.read_csv('/content/credit_scoring_eng.csv')"],"metadata":{"id":"wwvKjmYxmgEO","executionInfo":{"status":"ok","timestamp":1689650772696,"user_tz":-420,"elapsed":43,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"execution_count":113,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"rEfcJHZKLhyn"},"source":["## **1. Data Exploration**\n","\n","**Data Description**\n","- `children` - total number of children in a family\n","- `days_employed` - customer job experiences in day(s)\n","- `dob_years` - age of a customer in year(s)\n","- `education` - customer education level\n","- `education_id` - identifier for customer education level\n","- `family_status` - customer family status\n","- `family_status_id` - identifier for customer family status\n","- `gender` - customer gender\n","- `income_type` - customer income type\n","- `debt` - does the customer have loan payment debts?\n","- `total_income` - customer total income (monthly)\n","- `purpose` - customer purpose of getting a loan\n"]},{"cell_type":"code","execution_count":114,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"bT0rOXURLhyn","executionInfo":{"status":"ok","timestamp":1689650772697,"user_tz":-420,"elapsed":44,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"6ee57534-32dd-4396-c04e-4cb73a456d68"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(21525, 12)"]},"metadata":{},"execution_count":114}],"source":["# check how many data rows and columns in datasets\n","df.shape"]},{"cell_type":"code","execution_count":115,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":206},"id":"yfxkmBrSLhyo","executionInfo":{"status":"ok","timestamp":1689650772697,"user_tz":-420,"elapsed":41,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"91831129-7fee-449d-8c62-12475b95293b"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   children  days_employed  dob_years            education  education_id  \\\n","0         1   -8437.673028         42    bachelor's degree             0   \n","1         1   -4024.803754         36  secondary education             1   \n","2         0   -5623.422610         33  Secondary Education             1   \n","3         3   -4124.747207         32  secondary education             1   \n","4         0  340266.072047         53  secondary education             1   \n","\n","       family_status  family_status_id gender income_type  debt  total_income  \\\n","0            married                 0      F    employee     0     40620.102   \n","1            married                 0      F    employee     0     17932.802   \n","2            married                 0      M    employee     0     23341.752   \n","3            married                 0      M    employee     0     42820.568   \n","4  civil partnership                 1      F     retiree     0     25378.572   \n","\n","                   purpose  \n","0    purchase of the house  \n","1             car purchase  \n","2    purchase of the house  \n","3  supplementary education  \n","4        to have a wedding  "],"text/html":["\n","\n","  <div id=\"df-505cffa5-1c3c-4f6c-a556-dd710e85678b\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>-8437.673028</td>\n","      <td>42</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>-4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>-5623.422610</td>\n","      <td>33</td>\n","      <td>Secondary Education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23341.752</td>\n","      <td>purchase of the house</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>-4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>340266.072047</td>\n","      <td>53</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>25378.572</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-505cffa5-1c3c-4f6c-a556-dd710e85678b')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-b468438a-4cc2-4c7e-a46d-b23b15e59318\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b468438a-4cc2-4c7e-a46d-b23b15e59318')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-b468438a-4cc2-4c7e-a46d-b23b15e59318 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-505cffa5-1c3c-4f6c-a556-dd710e85678b button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-505cffa5-1c3c-4f6c-a556-dd710e85678b');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":115}],"source":["# Show first 10 datarows\n","df.head()"]},{"cell_type":"markdown","metadata":{"id":"VzTlXLpfLhyo"},"source":["**Conclusions**\n","\n","- There are missing values in the `days_employe` and '`total_income` columns. The total value does not match with another columns.\n","- There is a negative value in the `days_employed` column. The value must be changed to positive.\n","- There are duplicate values in the `education` column (there are uppercase and lowercase letters). The values must be adjusted to lowercase"]},{"cell_type":"code","execution_count":116,"metadata":{"scrolled":false,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"FQ_gZP3PLhyp","executionInfo":{"status":"ok","timestamp":1689650772698,"user_tz":-420,"elapsed":40,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"8062d1c4-4ee1-43b3-912d-26a8e8cb2213"},"outputs":[{"output_type":"stream","name":"stdout","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 21525 entries, 0 to 21524\n","Data columns (total 12 columns):\n"," #   Column            Non-Null Count  Dtype  \n","---  ------            --------------  -----  \n"," 0   children          21525 non-null  int64  \n"," 1   days_employed     19351 non-null  float64\n"," 2   dob_years         21525 non-null  int64  \n"," 3   education         21525 non-null  object \n"," 4   education_id      21525 non-null  int64  \n"," 5   family_status     21525 non-null  object \n"," 6   family_status_id  21525 non-null  int64  \n"," 7   gender            21525 non-null  object \n"," 8   income_type       21525 non-null  object \n"," 9   debt              21525 non-null  int64  \n"," 10  total_income      19351 non-null  float64\n"," 11  purpose           21525 non-null  object \n","dtypes: float64(2), int64(5), object(5)\n","memory usage: 2.0+ MB\n"]}],"source":["# Show data information\n","df.info()"]},{"cell_type":"markdown","metadata":{"id":"QjovzD55Lhyp"},"source":["**Conclusions**\n","\n","There are missing values in the `days_employed` and `total_income` columns. The total value does not match with another columns."]},{"cell_type":"code","execution_count":117,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"dUec38l-Lhyp","executionInfo":{"status":"ok","timestamp":1689650772698,"user_tz":-420,"elapsed":37,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"1f55187a-87a0-40c2-cd7c-86a31de3ea8a"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed  dob_years            education  education_id  \\\n","12            0            NaN         65  secondary education             1   \n","26            0            NaN         41  secondary education             1   \n","29            0            NaN         63  secondary education             1   \n","41            0            NaN         50  secondary education             1   \n","55            0            NaN         54  secondary education             1   \n","...         ...            ...        ...                  ...           ...   \n","21489         2            NaN         47  Secondary Education             1   \n","21495         1            NaN         50  secondary education             1   \n","21497         0            NaN         48    BACHELOR'S DEGREE             0   \n","21502         1            NaN         42  secondary education             1   \n","21510         2            NaN         28  secondary education             1   \n","\n","           family_status  family_status_id gender    income_type  debt  \\\n","12     civil partnership                 1      M        retiree     0   \n","26               married                 0      M  civil servant     0   \n","29             unmarried                 4      F        retiree     0   \n","41               married                 0      F  civil servant     0   \n","55     civil partnership                 1      F        retiree     1   \n","...                  ...               ...    ...            ...   ...   \n","21489            married                 0      M       business     0   \n","21495  civil partnership                 1      F       employee     0   \n","21497            married                 0      F       business     0   \n","21502            married                 0      F       employee     0   \n","21510            married                 0      F       employee     0   \n","\n","       total_income                   purpose  \n","12              NaN         to have a wedding  \n","26              NaN                 education  \n","29              NaN    building a real estate  \n","41              NaN  second-hand car purchase  \n","55              NaN         to have a wedding  \n","...             ...                       ...  \n","21489           NaN         purchase of a car  \n","21495           NaN          wedding ceremony  \n","21497           NaN       building a property  \n","21502           NaN    building a real estate  \n","21510           NaN              car purchase  \n","\n","[2174 rows x 12 columns]"],"text/html":["\n","\n","  <div id=\"df-c5d4a0f1-7603-4fab-a41b-e6aad819817e\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>12</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>65</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","    <tr>\n","      <th>26</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>41</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>education</td>\n","    </tr>\n","    <tr>\n","      <th>29</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>63</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>unmarried</td>\n","      <td>4</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>building a real estate</td>\n","    </tr>\n","    <tr>\n","      <th>41</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>second-hand car purchase</td>\n","    </tr>\n","    <tr>\n","      <th>55</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>54</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>21489</th>\n","      <td>2</td>\n","      <td>NaN</td>\n","      <td>47</td>\n","      <td>Secondary Education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>purchase of a car</td>\n","    </tr>\n","    <tr>\n","      <th>21495</th>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>wedding ceremony</td>\n","    </tr>\n","    <tr>\n","      <th>21497</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>48</td>\n","      <td>BACHELOR'S DEGREE</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>building a property</td>\n","    </tr>\n","    <tr>\n","      <th>21502</th>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>42</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>building a real estate</td>\n","    </tr>\n","    <tr>\n","      <th>21510</th>\n","      <td>2</td>\n","      <td>NaN</td>\n","      <td>28</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>car purchase</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>2174 rows × 12 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c5d4a0f1-7603-4fab-a41b-e6aad819817e')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-c5a3ce3e-07ae-4d86-8a57-5ac58a71c2fe\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c5a3ce3e-07ae-4d86-8a57-5ac58a71c2fe')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-c5a3ce3e-07ae-4d86-8a57-5ac58a71c2fe button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-c5d4a0f1-7603-4fab-a41b-e6aad819817e button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-c5d4a0f1-7603-4fab-a41b-e6aad819817e');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":117}],"source":["# Show filtered table with missing values on days_employed column\n","df[df['days_employed'].isnull()]"]},{"cell_type":"markdown","metadata":{"id":"IhjIkjrnLhyq"},"source":["**Conclusions**\n","\n","The total missing values are **2174** datarows."]},{"cell_type":"code","execution_count":118,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"1nbRWVQxLhyq","executionInfo":{"status":"ok","timestamp":1689650772698,"user_tz":-420,"elapsed":36,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"3e585d25-3ffc-4c32-d691-c319d81f8915"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["children               0\n","days_employed       2174\n","dob_years              0\n","education              0\n","education_id           0\n","family_status          0\n","family_status_id       0\n","gender                 0\n","income_type            0\n","debt                   0\n","total_income        2174\n","purpose                0\n","dtype: int64"]},"metadata":{},"execution_count":118}],"source":["# Check total missing value from datasets\n","df.isnull().sum()"]},{"cell_type":"code","execution_count":119,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"MaKlPbTGLhyq","executionInfo":{"status":"ok","timestamp":1689650772698,"user_tz":-420,"elapsed":34,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"8e97954f-c367-4bd7-ef49-56b419bebc5a"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed  dob_years            education  education_id  \\\n","12            0            NaN         65  secondary education             1   \n","26            0            NaN         41  secondary education             1   \n","29            0            NaN         63  secondary education             1   \n","41            0            NaN         50  secondary education             1   \n","55            0            NaN         54  secondary education             1   \n","...         ...            ...        ...                  ...           ...   \n","21489         2            NaN         47  Secondary Education             1   \n","21495         1            NaN         50  secondary education             1   \n","21497         0            NaN         48    BACHELOR'S DEGREE             0   \n","21502         1            NaN         42  secondary education             1   \n","21510         2            NaN         28  secondary education             1   \n","\n","           family_status  family_status_id gender    income_type  debt  \\\n","12     civil partnership                 1      M        retiree     0   \n","26               married                 0      M  civil servant     0   \n","29             unmarried                 4      F        retiree     0   \n","41               married                 0      F  civil servant     0   \n","55     civil partnership                 1      F        retiree     1   \n","...                  ...               ...    ...            ...   ...   \n","21489            married                 0      M       business     0   \n","21495  civil partnership                 1      F       employee     0   \n","21497            married                 0      F       business     0   \n","21502            married                 0      F       employee     0   \n","21510            married                 0      F       employee     0   \n","\n","       total_income                   purpose  \n","12              NaN         to have a wedding  \n","26              NaN                 education  \n","29              NaN    building a real estate  \n","41              NaN  second-hand car purchase  \n","55              NaN         to have a wedding  \n","...             ...                       ...  \n","21489           NaN         purchase of a car  \n","21495           NaN          wedding ceremony  \n","21497           NaN       building a property  \n","21502           NaN    building a real estate  \n","21510           NaN              car purchase  \n","\n","[2174 rows x 12 columns]"],"text/html":["\n","\n","  <div id=\"df-641c862f-7b3e-4e58-ad80-4d32cb4c0b53\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>12</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>65</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","    <tr>\n","      <th>26</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>41</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>education</td>\n","    </tr>\n","    <tr>\n","      <th>29</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>63</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>unmarried</td>\n","      <td>4</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>building a real estate</td>\n","    </tr>\n","    <tr>\n","      <th>41</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>second-hand car purchase</td>\n","    </tr>\n","    <tr>\n","      <th>55</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>54</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>21489</th>\n","      <td>2</td>\n","      <td>NaN</td>\n","      <td>47</td>\n","      <td>Secondary Education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>purchase of a car</td>\n","    </tr>\n","    <tr>\n","      <th>21495</th>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>wedding ceremony</td>\n","    </tr>\n","    <tr>\n","      <th>21497</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>48</td>\n","      <td>BACHELOR'S DEGREE</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>building a property</td>\n","    </tr>\n","    <tr>\n","      <th>21502</th>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>42</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>building a real estate</td>\n","    </tr>\n","    <tr>\n","      <th>21510</th>\n","      <td>2</td>\n","      <td>NaN</td>\n","      <td>28</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>car purchase</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>2174 rows × 12 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-641c862f-7b3e-4e58-ad80-4d32cb4c0b53')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-a709362d-7f0a-48c3-b82c-2cd277d6d419\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a709362d-7f0a-48c3-b82c-2cd277d6d419')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-a709362d-7f0a-48c3-b82c-2cd277d6d419 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-641c862f-7b3e-4e58-ad80-4d32cb4c0b53 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-641c862f-7b3e-4e58-ad80-4d32cb4c0b53');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":119}],"source":["df[ (df['days_employed'].isnull()) & (df['total_income'].isnull())]"]},{"cell_type":"code","execution_count":120,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"71nMC9FzLhyr","executionInfo":{"status":"ok","timestamp":1689650772699,"user_tz":-420,"elapsed":34,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"f9b3a4cd-fb27-4227-888c-86990b068ba8"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["children               0\n","days_employed       2174\n","dob_years              0\n","education              0\n","education_id           0\n","family_status          0\n","family_status_id       0\n","gender                 0\n","income_type            0\n","debt                   0\n","total_income        2174\n","purpose                0\n","dtype: int64"]},"metadata":{},"execution_count":120}],"source":["df[ (df['days_employed'].isnull()) & (df['total_income'].isnull())].isnull().sum()"]},{"cell_type":"code","execution_count":121,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"Pl4jH_nWLhyr","executionInfo":{"status":"ok","timestamp":1689650772699,"user_tz":-420,"elapsed":32,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"ce519e1a-d86c-4f83-9611-94443ec88fba"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["children             0.000000\n","days_employed       10.099884\n","dob_years            0.000000\n","education            0.000000\n","education_id         0.000000\n","family_status        0.000000\n","family_status_id     0.000000\n","gender               0.000000\n","income_type          0.000000\n","debt                 0.000000\n","total_income        10.099884\n","purpose              0.000000\n","dtype: float64"]},"metadata":{},"execution_count":121}],"source":["# Sum up the missing values in percentage\n","df.isnull().sum() / len(df) * 100"]},{"cell_type":"markdown","metadata":{"id":"Daq2OBpcLhyr"},"source":["**Conclusions**\n","\n","The number of rows in the table that has been filtered has matched with the number of missing values 21525 - 2174 = **19351**. When the `days_employed` column has a null value, the `total_income` column also has a null value.\n","\n","The total percentage of missing values is 10%. This value is quite small if we compared with the complete datasets.\n","\n","In the next step, we have to find out the possibilities that cause missing values."]},{"cell_type":"code","execution_count":122,"metadata":{"trusted":true,"id":"LjHUvKZfLhyr","executionInfo":{"status":"ok","timestamp":1689650772700,"user_tz":-420,"elapsed":31,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# Check customers who have no data on identified characteristics and columns with missing values\n","df_null = df[ (df['days_employed'].isnull()) & (df['total_income'].isnull())].reset_index(drop=True)"]},{"cell_type":"code","execution_count":123,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"M9JlFo8xLhys","executionInfo":{"status":"ok","timestamp":1689650772700,"user_tz":-420,"elapsed":31,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"97ad8a6c-9084-4414-afac-b587e0f90e25"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(2174, 12)"]},"metadata":{},"execution_count":123}],"source":["df_null.shape"]},{"cell_type":"code","execution_count":124,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":206},"id":"ShQljZO0Lhys","executionInfo":{"status":"ok","timestamp":1689650772700,"user_tz":-420,"elapsed":29,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"5d9178ea-a139-43cc-9307-1e103a0d2903"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   children  days_employed  dob_years            education  education_id  \\\n","0         0            NaN         65  secondary education             1   \n","1         0            NaN         41  secondary education             1   \n","2         0            NaN         63  secondary education             1   \n","3         0            NaN         50  secondary education             1   \n","4         0            NaN         54  secondary education             1   \n","\n","       family_status  family_status_id gender    income_type  debt  \\\n","0  civil partnership                 1      M        retiree     0   \n","1            married                 0      M  civil servant     0   \n","2          unmarried                 4      F        retiree     0   \n","3            married                 0      F  civil servant     0   \n","4  civil partnership                 1      F        retiree     1   \n","\n","   total_income                   purpose  \n","0           NaN         to have a wedding  \n","1           NaN                 education  \n","2           NaN    building a real estate  \n","3           NaN  second-hand car purchase  \n","4           NaN         to have a wedding  "],"text/html":["\n","\n","  <div id=\"df-b1be2c51-623b-4fa1-a6ba-4342fabbce01\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>65</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>41</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>education</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>63</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>unmarried</td>\n","      <td>4</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>building a real estate</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>second-hand car purchase</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>54</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b1be2c51-623b-4fa1-a6ba-4342fabbce01')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-da2799c5-97b0-45e9-b548-2cc28347a775\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-da2799c5-97b0-45e9-b548-2cc28347a775')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-da2799c5-97b0-45e9-b548-2cc28347a775 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-b1be2c51-623b-4fa1-a6ba-4342fabbce01 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-b1be2c51-623b-4fa1-a6ba-4342fabbce01');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":124}],"source":["df_null.head()"]},{"cell_type":"code","execution_count":125,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"eDRbmKbSLhys","executionInfo":{"status":"ok","timestamp":1689650772701,"user_tz":-420,"elapsed":29,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"ec9c7d47-7653-4961-ebfd-f7c3235182ea"},"outputs":[{"output_type":"execute_result","data":{"text/plain":[" 0     0.661914\n"," 1     0.218491\n"," 2     0.093836\n"," 3     0.016559\n"," 20    0.004140\n"," 4     0.003220\n","-1     0.001380\n"," 5     0.000460\n","Name: children, dtype: float64"]},"metadata":{},"execution_count":125}],"source":["# check distribution from `children` column\n","df_null['children'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"hU9yXxiDLhys"},"source":["**Conclusions**\n","\n","The missing values were mostly dominated by **66%** customers who don't have children."]},{"cell_type":"code","execution_count":126,"metadata":{"scrolled":false,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"Lz5rkb0QLhyt","executionInfo":{"status":"ok","timestamp":1689650772701,"user_tz":-420,"elapsed":27,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"8a25003a-891a-4253-ccc2-2466d5d433a8"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["secondary education    0.647654\n","bachelor's degree      0.228151\n","SECONDARY EDUCATION    0.030819\n","Secondary Education    0.029899\n","some college           0.025299\n","Bachelor's Degree      0.011500\n","BACHELOR'S DEGREE      0.010580\n","primary education      0.008740\n","Some College           0.003220\n","SOME COLLEGE           0.003220\n","Primary Education      0.000460\n","PRIMARY EDUCATION      0.000460\n","Name: education, dtype: float64"]},"metadata":{},"execution_count":126}],"source":["df_null['education'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"x6c9NTxcLhyt"},"source":["**Conclusions**\n","\n","The majority of missing values are dominated by **64%** by individuals with secondary education backgrounds."]},{"cell_type":"code","execution_count":127,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"rx7UL8jTLhyt","executionInfo":{"status":"ok","timestamp":1689650772701,"user_tz":-420,"elapsed":25,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"0bdc4282-99b9-4fd4-9fa4-c2fbcb685369"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["1    0.708372\n","0    0.250230\n","2    0.031739\n","3    0.009660\n","Name: education_id, dtype: float64"]},"metadata":{},"execution_count":127}],"source":["df_null['education_id'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"_0qOtW5FLhyt"},"source":["**Conclusions**\n","\n","The missing values were mostly dominated by **70%** `education_id` with value equals to 1."]},{"cell_type":"code","execution_count":128,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"fvEdBD2KLhyt","executionInfo":{"status":"ok","timestamp":1689650772701,"user_tz":-420,"elapsed":23,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"e26245ae-0f84-46fc-cc02-b3ccc4261368"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["F    0.682613\n","M    0.317387\n","Name: gender, dtype: float64"]},"metadata":{},"execution_count":128}],"source":["df_null['gender'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"t-T0GnecLhyt"},"source":["**Conclusions**\n","\n","The missing values were mostly dominated by **68%** customers with F (Female) gender."]},{"cell_type":"code","execution_count":129,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"aJoYAifHLhyt","executionInfo":{"status":"ok","timestamp":1689650772702,"user_tz":-420,"elapsed":22,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"6e738375-340b-4ab7-a569-3713fed40078"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["married              0.568997\n","civil partnership    0.203312\n","unmarried            0.132475\n","divorced             0.051518\n","widow / widower      0.043698\n","Name: family_status, dtype: float64"]},"metadata":{},"execution_count":129}],"source":["df_null['family_status'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"VBk0qkUuLhyu"},"source":["**Conclusions**\n","\n","The missing values were mostly dominated by **56%** customers that already married or `family_status = 'married'`."]},{"cell_type":"code","execution_count":130,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"aD5q42meLhyu","executionInfo":{"status":"ok","timestamp":1689650773318,"user_tz":-420,"elapsed":636,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"63a2b127-fae1-4491-93f7-e2900e908208"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["employee         0.508280\n","business         0.233671\n","retiree          0.189972\n","civil servant    0.067617\n","entrepreneur     0.000460\n","Name: income_type, dtype: float64"]},"metadata":{},"execution_count":130}],"source":["df_null['income_type'].value_counts(normalize=True)"]},{"cell_type":"code","execution_count":131,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"b-B59YAPLhyu","executionInfo":{"status":"ok","timestamp":1689650773319,"user_tz":-420,"elapsed":38,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"f09fe783-5136-433f-e60f-30b9cc3386ab"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["F    0.682613\n","M    0.317387\n","Name: gender, dtype: float64"]},"metadata":{},"execution_count":131}],"source":["df[ (df['days_employed'].isnull()) & (df['total_income'].isnull())]['gender'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"1RuXDGMyLhyu"},"source":["**Conclusions**\n","\n","The missing values were mostly dominated by **50%** customers with job type as an employee or `income_type = 'employee'`, and this number were dominated by **68%** from female gender."]},{"cell_type":"code","execution_count":132,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"sorMoo5LLhyu","executionInfo":{"status":"ok","timestamp":1689650773319,"user_tz":-420,"elapsed":37,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"44b64780-f0d0-44b8-8be8-8aeb4baf4bd5"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    0.921803\n","1    0.078197\n","Name: debt, dtype: float64"]},"metadata":{},"execution_count":132}],"source":["df_null['debt'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"H6yLJSnbLhyv"},"source":["**Conclusions**\n","\n","The missing values were mostly dominated by **92%** customers who pay on time (with no debts) or `debt = 0`."]},{"cell_type":"markdown","metadata":{"id":"itLfwHu3Lhyv"},"source":["**Conclusions**\n","\n","The missing values are mostly dominated by the following columns:\n","- 66% by customers who don't have children\n","- The majority of missing values are dominated by 64% by individuals with secondary education backgrounds.\n","- 68% by individuals with gender F (female)\n","- 56% by individuals who are married or with `family_status = 'married'`\n","- 50% by individuals with job type as an employee or with `income_type = 'employee'`\n","- 92% by individuals who pay on time (have no debt) or with debt = 0\n","\n","**Possibilities that caused missing values in datasets**\n","\n","It needs to be reviewed furthermore whether there is duplicate data or not, then whether there is a more specific pattern to be able to determine the cause of the missing data.\n","\n","Let's try to check whether the missing values has random pattern"]},{"cell_type":"code","execution_count":133,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"KN4IQ_c6Lhyv","executionInfo":{"status":"ok","timestamp":1689650773319,"user_tz":-420,"elapsed":34,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"9f9c2058-b96a-4a5d-a366-cbecbbff6cec"},"outputs":[{"output_type":"execute_result","data":{"text/plain":[" 0     0.657329\n"," 1     0.223833\n"," 2     0.095470\n"," 3     0.015331\n"," 20    0.003531\n","-1     0.002184\n"," 4     0.001905\n"," 5     0.000418\n","Name: children, dtype: float64"]},"metadata":{},"execution_count":133}],"source":["# check distribution in datasets\n","df['children'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"BXZlxJboLhyv"},"source":["- On `children` column, the values are slightly different from the data that has been filtered"]},{"cell_type":"code","execution_count":134,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"aO4rdrKyLhyv","executionInfo":{"status":"ok","timestamp":1689650773320,"user_tz":-420,"elapsed":34,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"c8efeb46-76e6-4cdc-db51-c1c9f718c3d0"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["secondary education    0.638792\n","bachelor's degree      0.219187\n","SECONDARY EDUCATION    0.035865\n","Secondary Education    0.033031\n","some college           0.031034\n","BACHELOR'S DEGREE      0.012729\n","Bachelor's Degree      0.012451\n","primary education      0.011614\n","Some College           0.002184\n","SOME COLLEGE           0.001347\n","PRIMARY EDUCATION      0.000790\n","Primary Education      0.000697\n","graduate degree        0.000186\n","Graduate Degree        0.000046\n","GRADUATE DEGREE        0.000046\n","Name: education, dtype: float64"]},"metadata":{},"execution_count":134}],"source":["df['education'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"UsbHkUaZLhyv"},"source":["- `education` column has data duplication"]},{"cell_type":"code","execution_count":135,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"bbfvhqriLhyv","executionInfo":{"status":"ok","timestamp":1689650773320,"user_tz":-420,"elapsed":31,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"b6adbc1c-1b46-4bd6-f6a5-cad575fbab8a"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["1    0.707689\n","0    0.244367\n","2    0.034564\n","3    0.013101\n","4    0.000279\n","Name: education_id, dtype: float64"]},"metadata":{},"execution_count":135}],"source":["df['education_id'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"a2_6FXnkLhyw"},"source":["- `education_id` column has differences, there is a value with number 4. Because there is no data in `df_null`"]},{"cell_type":"code","execution_count":136,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"BqRjXTWkLhyw","executionInfo":{"status":"ok","timestamp":1689650773320,"user_tz":-420,"elapsed":31,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"60886b5d-c55a-49d7-80e9-280ff8cd14dd"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["F      0.661370\n","M      0.338583\n","XNA    0.000046\n","Name: gender, dtype: float64"]},"metadata":{},"execution_count":136}],"source":["df['gender'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"g_USG5Q-Lhyw"},"source":[" - Kolom 'gender' terdapat gender XNA, data ini tidak masuk dengan data yg sudah di filter"]},{"cell_type":"code","execution_count":137,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"nMJzgptaLhyw","executionInfo":{"status":"ok","timestamp":1689650773320,"user_tz":-420,"elapsed":29,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"df652934-bb1f-402e-a4b3-88ff7f0e8a47"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["employee                       0.516562\n","business                       0.236237\n","retiree                        0.179141\n","civil servant                  0.067782\n","unemployed                     0.000093\n","entrepreneur                   0.000093\n","student                        0.000046\n","paternity / maternity leave    0.000046\n","Name: income_type, dtype: float64"]},"metadata":{},"execution_count":137}],"source":["df['income_type'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"VDw6LRL0Lhyw"},"source":["- `income_type` column has some values (`unemployed`, `paternity` / `maternity leave`, and `student`) which are not found with filtered data"]},{"cell_type":"markdown","metadata":{"id":"v8LxL5m6Lhyw"},"source":["Cleansing data by excluding the following columns:\n","- XNA `gender`, the reason for this gender is unknown because there is only one data.\n","- children with a value less than 20, and greater than 0. Assuming there may be a typo when user input data"]},{"cell_type":"code","execution_count":138,"metadata":{"trusted":true,"id":"zkIUK-RkLhyx","executionInfo":{"status":"ok","timestamp":1689650773320,"user_tz":-420,"elapsed":27,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["df_clean = df[(df['gender'] != 'XNA') & (df['children'] < 20) & (df['children'] > 0)]"]},{"cell_type":"code","execution_count":139,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"YUIi9JDBLhyx","executionInfo":{"status":"ok","timestamp":1689650773321,"user_tz":-420,"elapsed":27,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"f154a023-ba9b-4b5f-9e01-4b77b7d787ff"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed  dob_years            education  education_id  \\\n","0             1   -8437.673028         42    bachelor's degree             0   \n","1             1   -4024.803754         36  secondary education             1   \n","3             3   -4124.747207         32  secondary education             1   \n","8             2   -6929.865299         35    BACHELOR'S DEGREE             0   \n","10            2   -4171.483647         36    bachelor's degree             0   \n","...         ...            ...        ...                  ...           ...   \n","21519         1   -2351.431934         37      graduate degree             4   \n","21520         1   -4529.316663         43  secondary education             1   \n","21522         1   -2113.346888         38  secondary education             1   \n","21523         3   -3112.481705         38  secondary education             1   \n","21524         2   -1984.507589         40  secondary education             1   \n","\n","           family_status  family_status_id gender income_type  debt  \\\n","0                married                 0      F    employee     0   \n","1                married                 0      F    employee     0   \n","3                married                 0      M    employee     0   \n","8      civil partnership                 1      F    employee     0   \n","10               married                 0      M    business     0   \n","...                  ...               ...    ...         ...   ...   \n","21519           divorced                 3      M    employee     0   \n","21520  civil partnership                 1      F    business     0   \n","21522  civil partnership                 1      M    employee     1   \n","21523            married                 0      M    employee     1   \n","21524            married                 0      F    employee     0   \n","\n","       total_income                     purpose  \n","0         40620.102       purchase of the house  \n","1         17932.802                car purchase  \n","3         42820.568     supplementary education  \n","8         15337.093            having a wedding  \n","10        18230.959             buy real estate  \n","...             ...                         ...  \n","21519     18551.846  buy commercial real estate  \n","21520     35966.698        housing transactions  \n","21522     14347.610                    property  \n","21523     39054.888           buying my own car  \n","21524     13127.587                to buy a car  \n","\n","[7253 rows x 12 columns]"],"text/html":["\n","\n","  <div id=\"df-2602c748-dce1-487c-9ae6-2b4e556721b6\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>-8437.673028</td>\n","      <td>42</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>-4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>-4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>2</td>\n","      <td>-6929.865299</td>\n","      <td>35</td>\n","      <td>BACHELOR'S DEGREE</td>\n","      <td>0</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>15337.093</td>\n","      <td>having a wedding</td>\n","    </tr>\n","    <tr>\n","      <th>10</th>\n","      <td>2</td>\n","      <td>-4171.483647</td>\n","      <td>36</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>18230.959</td>\n","      <td>buy real estate</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>21519</th>\n","      <td>1</td>\n","      <td>-2351.431934</td>\n","      <td>37</td>\n","      <td>graduate degree</td>\n","      <td>4</td>\n","      <td>divorced</td>\n","      <td>3</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>18551.846</td>\n","      <td>buy commercial real estate</td>\n","    </tr>\n","    <tr>\n","      <th>21520</th>\n","      <td>1</td>\n","      <td>-4529.316663</td>\n","      <td>43</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>35966.698</td>\n","      <td>housing transactions</td>\n","    </tr>\n","    <tr>\n","      <th>21522</th>\n","      <td>1</td>\n","      <td>-2113.346888</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>14347.610</td>\n","      <td>property</td>\n","    </tr>\n","    <tr>\n","      <th>21523</th>\n","      <td>3</td>\n","      <td>-3112.481705</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>39054.888</td>\n","      <td>buying my own car</td>\n","    </tr>\n","    <tr>\n","      <th>21524</th>\n","      <td>2</td>\n","      <td>-1984.507589</td>\n","      <td>40</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>13127.587</td>\n","      <td>to buy a car</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>7253 rows × 12 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2602c748-dce1-487c-9ae6-2b4e556721b6')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-92347815-fa11-496a-a1ef-62ce5ff71532\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-92347815-fa11-496a-a1ef-62ce5ff71532')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-92347815-fa11-496a-a1ef-62ce5ff71532 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-2602c748-dce1-487c-9ae6-2b4e556721b6 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-2602c748-dce1-487c-9ae6-2b4e556721b6');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":139}],"source":["df_clean"]},{"cell_type":"markdown","metadata":{"id":"i--x0YeGLhyx"},"source":["**Conclusions**\n","\n","There is some similarities from the original datasets in compared with filtered table. We've to check whether there is data duplication, if there is some data duplication, we've to do data cleansing against the duplicated data\n"]},{"cell_type":"code","execution_count":140,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"5nnWOnh8Lhyx","executionInfo":{"status":"ok","timestamp":1689650773321,"user_tz":-420,"elapsed":27,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"840dca42-9543-4d3b-d9c9-5efcac5a84c9"},"outputs":[{"output_type":"execute_result","data":{"text/plain":[" 0     0.589140\n"," 1     0.252489\n"," 2     0.123982\n"," 3     0.026244\n"," 4     0.003620\n"," 20    0.002715\n"," 5     0.000905\n","-1     0.000905\n","Name: children, dtype: float64"]},"metadata":{},"execution_count":140}],"source":["# Check pattern and another cause that will result in data missing values\n","df_null[df_null['income_type']=='employee']['children'].value_counts(normalize=True)"]},{"cell_type":"code","execution_count":141,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"ciHo-mYILhyx","executionInfo":{"status":"ok","timestamp":1689650773321,"user_tz":-420,"elapsed":26,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"c0eb7c12-6453-4888-b48d-09e6182b8b7f"},"outputs":[{"output_type":"execute_result","data":{"text/plain":[" 0     0.593579\n"," 1     0.259646\n"," 2     0.118446\n"," 3     0.018797\n"," 20    0.003867\n"," 4     0.002788\n","-1     0.002338\n"," 5     0.000540\n","Name: children, dtype: float64"]},"metadata":{},"execution_count":141}],"source":["df[df['income_type']=='employee']['children'].value_counts(normalize=True)"]},{"cell_type":"code","execution_count":142,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"Hfv_jp-uLhyx","executionInfo":{"status":"ok","timestamp":1689650773322,"user_tz":-420,"elapsed":24,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"d7ba17f6-e7ea-47f2-b39b-b9ec6916fa0e"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["secondary education    0.647654\n","bachelor's degree      0.228151\n","SECONDARY EDUCATION    0.030819\n","Secondary Education    0.029899\n","some college           0.025299\n","Bachelor's Degree      0.011500\n","BACHELOR'S DEGREE      0.010580\n","primary education      0.008740\n","Some College           0.003220\n","SOME COLLEGE           0.003220\n","Primary Education      0.000460\n","PRIMARY EDUCATION      0.000460\n","Name: education, dtype: float64"]},"metadata":{},"execution_count":142}],"source":["# Check another pattern\n","df_null['education'].value_counts(normalize=True)"]},{"cell_type":"code","execution_count":143,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"Vy0LSdXLLhyx","executionInfo":{"status":"ok","timestamp":1689650773322,"user_tz":-420,"elapsed":23,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"f0a1ed11-977b-4f21-8176-4cbeadd35506"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["secondary education    0.638792\n","bachelor's degree      0.219187\n","SECONDARY EDUCATION    0.035865\n","Secondary Education    0.033031\n","some college           0.031034\n","BACHELOR'S DEGREE      0.012729\n","Bachelor's Degree      0.012451\n","primary education      0.011614\n","Some College           0.002184\n","SOME COLLEGE           0.001347\n","PRIMARY EDUCATION      0.000790\n","Primary Education      0.000697\n","graduate degree        0.000186\n","Graduate Degree        0.000046\n","GRADUATE DEGREE        0.000046\n","Name: education, dtype: float64"]},"metadata":{},"execution_count":143}],"source":["df['education'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"vSymeb7FLhyy"},"source":["- There is a pattern on missing values"]},{"cell_type":"markdown","metadata":{"id":"VWM87nEfLhyy"},"source":["**Conclusions**\n","\n","There is no specific pattern from unfiltered data and filtered data\n","\n","The missing value may occur randomly or it may happen due to human error when user input data, therefore we must fill in the missing values with **mean** or **median**.\n","\n","We have to adjust the value of duplicate data. For abnormal data, or has no value, we must fill in the value with the **mean** or **median**"]},{"cell_type":"markdown","metadata":{"id":"fy25IZhqLhyy"},"source":["## Data Transformation\n","\n"]},{"cell_type":"code","execution_count":144,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"oEeEmpyRLhyy","executionInfo":{"status":"ok","timestamp":1689650773322,"user_tz":-420,"elapsed":21,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"b9a21026-5ec0-471f-8f29-5e9e605f6730"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["54"]},"metadata":{},"execution_count":144}],"source":["df.duplicated().sum()"]},{"cell_type":"code","execution_count":145,"metadata":{"trusted":true,"id":"VY147nnWLhyy","executionInfo":{"status":"ok","timestamp":1689650773322,"user_tz":-420,"elapsed":19,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["df_new = df.drop_duplicates().reset_index(drop=True)"]},{"cell_type":"code","execution_count":146,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"kG-MIORaLhyy","executionInfo":{"status":"ok","timestamp":1689650773322,"user_tz":-420,"elapsed":19,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"e5209a27-da6d-441d-deef-80a90ddb8534"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0"]},"metadata":{},"execution_count":146}],"source":["df_new.duplicated().sum()"]},{"cell_type":"code","execution_count":147,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"IMdzT_DDLhyz","executionInfo":{"status":"ok","timestamp":1689650773323,"user_tz":-420,"elapsed":18,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"8cd64920-4dbd-468c-bdea-d8f3b01c436f"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["secondary education    13705\n","bachelor's degree       4710\n","SECONDARY EDUCATION      772\n","Secondary Education      711\n","some college             668\n","BACHELOR'S DEGREE        273\n","Bachelor's Degree        268\n","primary education        250\n","Some College              47\n","SOME COLLEGE              29\n","PRIMARY EDUCATION         17\n","Primary Education         15\n","graduate degree            4\n","Graduate Degree            1\n","GRADUATE DEGREE            1\n","Name: education, dtype: int64"]},"metadata":{},"execution_count":147}],"source":["# check all values in `education` column to review spelling that must be fixed\n","df_new['education'].value_counts()"]},{"cell_type":"code","execution_count":148,"metadata":{"scrolled":true,"trusted":true,"id":"acf2QxE3Lhyz","executionInfo":{"status":"ok","timestamp":1689650773323,"user_tz":-420,"elapsed":17,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# Fix spelling if needed\n","df_new['education'] = df_new['education'].str.lower()"]},{"cell_type":"code","execution_count":149,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"R48BCYpJLhyz","executionInfo":{"status":"ok","timestamp":1689650773324,"user_tz":-420,"elapsed":17,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"b11b50aa-a273-44b6-b73c-f875f326f8ee"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["secondary education    15188\n","bachelor's degree       5251\n","some college             744\n","primary education        282\n","graduate degree            6\n","Name: education, dtype: int64"]},"metadata":{},"execution_count":149}],"source":["# check all column value to ensure that we have fixed it properly\n","df_new['education'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"L98B0rImLhyz"},"source":["check `children` column"]},{"cell_type":"code","execution_count":150,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"1WZ2P0cCLhyz","executionInfo":{"status":"ok","timestamp":1689650773324,"user_tz":-420,"elapsed":16,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"76935290-674d-437b-d52f-c091f9e3483f"},"outputs":[{"output_type":"execute_result","data":{"text/plain":[" 0     14107\n"," 1      4809\n"," 2      2052\n"," 3       330\n"," 20       76\n","-1        47\n"," 4        41\n"," 5         9\n","Name: children, dtype: int64"]},"metadata":{},"execution_count":150}],"source":["# check value distribution on `children` column\n","df_new['children'].value_counts()"]},{"cell_type":"code","execution_count":151,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":430},"id":"fweiZjIPLhy0","executionInfo":{"status":"ok","timestamp":1689650773797,"user_tz":-420,"elapsed":487,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"d26b5448-01c1-4152-a374-92509b49f98c"},"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAUxElEQVR4nO3dbYhUh7nA8WfUOmsad6xVd91m05i+JbRUwSRWmkpCJbrQUG0KqeSDCaGFoCnpNoQIrfYNpKUECbXmU5sGGhvyQUtTVtpuq1JqEmII3kIbYmqJqdmNEdxRccdcnfshZK6bqHV15zn78vvBgZ1zzs48ySLz55wzZ0r1er0eAABJJhU9AAAwsYgPACCV+AAAUokPACCV+AAAUokPACCV+AAAUokPACDVlKIHeK8zZ87EoUOHYvr06VEqlYoeBwC4CPV6PY4dOxYdHR0xadKFj22Muvg4dOhQdHZ2Fj0GAHAJDh48GFddddUF9xl18TF9+vSIeGf41tbWgqcBAC5GtVqNzs7Oxvv4hYy6+Hj3VEtra6v4AIAx5mIumXDBKQCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKlG3U3GgPHrlltuafy8c+fOwuYAijWsIx8bN26MG2+8MaZPnx5z5syJFStWxMsvvzxkn8HBwVizZk18+MMfjiuvvDLuuOOO6O/vH9GhgbHn7PA412Ng4hhWfOzatSvWrFkTzz77bPzxj3+Mt99+O2677bY4ceJEY59vfetb8bvf/S6efvrp2LVrVxw6dCi+8pWvjPjgAMDYVKrX6/VL/eXDhw/HnDlzYteuXbFkyZIYGBiI2bNnx5NPPhlf/epXIyLin//8Z1x//fWxZ8+e+NznPvdfn7NarUalUomBgQHf7QLjxIWOcjj9AuPDcN6/L+uC04GBgYiImDlzZkRE7N27N95+++1YunRpY5/rrrsurr766tizZ885n6NWq0W1Wh2yAOPHM888c1nbgfHnkuPjzJkz8cADD8TnP//5+MxnPhMREX19fTF16tSYMWPGkH3b2tqir6/vnM+zcePGqFQqjaWzs/NSRwJGoZ/+9KeXtR0Yfy45PtasWRN///vf4ze/+c1lDbBu3boYGBhoLAcPHrys5wNGlwcffPCytgPjzyXFx9q1a+OZZ56Jv/zlL3HVVVc11re3t8epU6fi6NGjQ/bv7++P9vb2cz5XuVyO1tbWIQswfnzpS1+6rO3A+DOs+KjX67F27drYtm1b/PnPf4558+YN2b5w4cL4wAc+EL29vY11L7/8crz22muxePHikZkYGHPOd1Gpi01hYhrWTcbWrFkTTz75ZPz2t7+N6dOnN67jqFQqMW3atKhUKnHvvfdGd3d3zJw5M1pbW+P++++PxYsXX9QnXQCA8W9YH7UtlUrnXP/LX/4y7r777oh45yZj3/72t2Pr1q1Rq9Vi2bJl8fOf//y8p13ey0dtYfxyh1MYv4bz/n1Z9/loBvEBAGNP2n0+AACGS3wAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQSnwAAKnEBwCQatjxsXv37rj99tujo6MjSqVSbN++fcj2u+++O0ql0pBl+fLlIzUvADDGDTs+Tpw4EfPnz4/Nmzefd5/ly5fHG2+80Vi2bt16WUMCAOPHlOH+QldXV3R1dV1wn3K5HO3t7Zc8FAAwfjXlmo+dO3fGnDlz4lOf+lTcd999ceTIkfPuW6vVolqtDlkAgPFrxONj+fLl8cQTT0Rvb2/8+Mc/jl27dkVXV1ecPn36nPtv3LgxKpVKY+ns7BzpkQCAUaRUr9frl/zLpVJs27YtVqxYcd59/vWvf8XHPvax+NOf/hRf/OIX37e9VqtFrVZrPK5Wq9HZ2RkDAwPR2tp6qaMBAImq1WpUKpWLev9u+kdtr7322pg1a1bs37//nNvL5XK0trYOWQCA8avp8fH666/HkSNHYu7cuc1+KQBgDBj2p12OHz8+5CjGgQMH4qWXXoqZM2fGzJkz4/vf/37ccccd0d7eHq+++mo89NBD8fGPfzyWLVs2ooMDAGPTsOPjhRdeiFtvvbXxuLu7OyIiVq9eHVu2bIl9+/bFr371qzh69Gh0dHTEbbfdFj/84Q+jXC6P3NQAwJh1WRecNsNwLlgBAEaHUXXBKQDA2cQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqcQHAJBKfAAAqYYdH7t3747bb789Ojo6olQqxfbt24dsr9frsX79+pg7d25MmzYtli5dGq+88spIzQsAjHHDjo8TJ07E/PnzY/Pmzefc/pOf/CQeffTReOyxx+K5556LD37wg7Fs2bIYHBy87GEBgLFvynB/oaurK7q6us65rV6vx6ZNm+I73/lOfPnLX46IiCeeeCLa2tpi+/bt8bWvfe3ypgUAxrwRvebjwIED0dfXF0uXLm2sq1QqsWjRotizZ885f6dWq0W1Wh2yAADj14jGR19fX0REtLW1DVnf1tbW2PZeGzdujEql0lg6OztHciQAYJQp/NMu69ati4GBgcZy8ODBokcCAJpoROOjvb09IiL6+/uHrO/v729se69yuRytra1DFgBg/BrR+Jg3b160t7dHb29vY121Wo3nnnsuFi9ePJIvBQCMUcP+tMvx48dj//79jccHDhyIl156KWbOnBlXX311PPDAA/GjH/0oPvGJT8S8efPiu9/9bnR0dMSKFStGcm4AYIwadny88MILceuttzYed3d3R0TE6tWr4/HHH4+HHnooTpw4Ed/4xjfi6NGjcfPNN8eOHTuipaVl5KYGAMasUr1erxc9xNmq1WpUKpUYGBhw/QcAjBHDef8u/NMuAMDEIj4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFRTih4AmDhuu+22OHXqVEydOjX+8Ic/FD0OUJARP/Lxve99L0ql0pDluuuuG+mXAcaY3bt3x6lTpyIi4tSpU7F79+6CJwKK0pTTLp/+9KfjjTfeaCx//etfm/EywBiyfv36Cz4GJo6mxMeUKVOivb29scyaNasZLwOMEd/85jeHtR4Y35oSH6+88kp0dHTEtddeG3fddVe89tpr5923VqtFtVodsgDjx8mTJ2Pfvn3n3LZv3744efJk8kRA0UY8PhYtWhSPP/547NixI7Zs2RIHDhyIL3zhC3Hs2LFz7r9x48aoVCqNpbOzc6RHAgr0345uOPoBE0+pXq/Xm/kCR48ejY9+9KPxyCOPxL333vu+7bVaLWq1WuNxtVqNzs7OGBgYiNbW1maOBiQ4efJkdHV1nXd7T09PTJs2LXEioBmq1WpUKpWLev9u+kdtZ8yYEZ/85Cdj//7959xeLpejXC43ewygINOmTYvPfvaz5zz1smDBAuEBE1DTbzJ2/PjxePXVV2Pu3LnNfilglHr00UfPuX7Tpk25gwCjwojHx4MPPhi7du2Kf//73/G3v/0tVq5cGZMnT45Vq1aN9EsBY8gPfvCDCz4GJo4RP+3y+uuvx6pVq+LIkSMxe/bsuPnmm+PZZ5+N2bNnj/RLAWPIkiVLYurUqY07nC5ZsqTokYCCNP2C0+EazgUrAMDoMJz3b18sBwCkEh8AQCrxAQCkEh8AQKqm32QM4F233HJL4+edO3cWNgdQLEc+gBR33XXXBR8DE4f4AFL85z//ueBjYOIQH0DTnX265WLWA+Ob+ACa6h//+MdlbQfGH/EBNNV99913WduB8Ud8AE21ZcuWy9oOjD/iA2iq66+//rK2A+OP+ACa7nz39HCvD5iYxAeQ4iMf+cgFHwMTh/gAUvz617++4GNg4nB7dSCN0yxAhCMfAEAy8QEApBIfAEAq8QEApHLBKZDm7C+Sc/EpTFyOfAAp3vsNtr7RFiYu8QEApBIfQNOd7yiHox8wMYkPoKn+W2AIEJh4xAcAkEp8AACpxAfQVP/tI7U+cgsTj/gAmu58gSE8YGISHwBAKvEBpHjvUQ5HPWDicnt1II3gACIc+QAAkokPACCV+AAAUokPACCV+AAAUokPACCV+AAAUrnPB+NavV6PwcHBoscg3vlb1Gq1iIgol8tRKpUKnoh3tbS0+HuQSnwwrg0ODkZXV1fRY8Co1tPTE9OmTSt6DCYQp10AgFSOfDCutbS0RE9PT9FjEO8chVq5cmVERGzbti1aWloKnoh3+VuQTXwwrpVKJYeTR6GWlhZ/F5jAnHYBAFKJDwAglfgAAFKJDwAglfgAAFKJDwAglfgAAFKJDwAglfgAAFKJDwAglfgAAFL5bpcmqNfrMTg4WPQYMKqc/W/Cvw84t5aWliiVSkWP0XTiowkGBwejq6ur6DFg1Hr3222BoXp6eibEly467QIApHLko8mOL1gV9Un+N0PU6xFn/vednydNiZgAh5bhYpTO/G9c+dLWosdI5V2xyeqTpkRM/kDRY8AoMbXoAWDUqRc9QAGcdgEAUokPACCV+AAAUokPACCV+AAAUokPACCV+AAAUokPACCV+AAAUjUtPjZv3hzXXHNNtLS0xKJFi+L5559v1ksBAGNIU+Ljqaeeiu7u7tiwYUO8+OKLMX/+/Fi2bFm8+eabzXg5AGAMacp3uzzyyCPx9a9/Pe65556IiHjsscfi97//ffziF7+Ihx9+uBkvOarU6/9/p/7S24NRP/12gdPAKFGP93yxXKHTwKhROnO68fPZ7x/j2YjHx6lTp2Lv3r2xbt26xrpJkybF0qVLY8+ePe/bv1arRa1WazyuVqsjPVK6s/97rvyfpwucBICxpFarxRVXXFH0GE034qdd3nrrrTh9+nS0tbUNWd/W1hZ9fX3v23/jxo1RqVQaS2dn50iPBACMIk057TIc69ati+7u7sbjarU65gOkUqnEtm3bih4DRpXBwcFYtWpVRERs3bo1WlpaCp4IRp9KpVL0CClGPD5mzZoVkydPjv7+/iHr+/v7o729/X37l8vlKJfLIz1GoSZNmhQf+tCHih4DRpWTJ082fp4xY0ZMmzatwGmAIo34aZepU6fGwoULo7e3t7HuzJkz0dvbG4sXLx7plwMAxpimnHbp7u6O1atXxw033BA33XRTbNq0KU6cONH49AsAMHE1JT7uvPPOOHz4cKxfvz76+vpiwYIFsWPHjvddhAoATDxNu+B07dq1sXbt2mY9PQAwRvluFwAglfgAAFKJDwAglfgAAFKJDwAglfgAAFKJDwAgVeFfLAfNVK/XY3BwsOgxiBjyd/A3GV1aWlqiVCoVPQYTiPhgXBscHIyurq6ix+A9Vq5cWfQInKWnp8cX/ZHKaRcAIJUjH4xrLS0t0dPTU/QYxDunwGq1WkRElMtlh/lHkZaWlqJHYIIRH4xrpVLJ4eRR5Iorrih6BGAUcNoFAEglPgCAVOIDAEglPgCAVOIDAEglPgCAVOIDAEglPgCAVOIDAEglPgCAVOIDAEglPgCAVOIDAEg16r7Vtl6vR0REtVoteBIA4GK9+7797vv4hYy6+Dh27FhERHR2dhY8CQAwXMeOHYtKpXLBfUr1i0mURGfOnIlDhw7F9OnTo1QqFT0OMIKq1Wp0dnbGwYMHo7W1tehxgBFUr9fj2LFj0dHREZMmXfiqjlEXH8D4Va1Wo1KpxMDAgPiACcwFpwBAKvEBAKQSH0CacrkcGzZsiHK5XPQoQIFc8wEApHLkAwBIJT4AgFTiAwBIJT4AgFTiA0izefPmuOaaa6KlpSUWLVoUzz//fNEjAQUQH0CKp556Krq7u2PDhg3x4osvxvz582PZsmXx5ptvFj0akMxHbYEUixYtihtvvDF+9rOfRcQ73+PU2dkZ999/fzz88MMFTwdkcuQDaLpTp07F3r17Y+nSpY11kyZNiqVLl8aePXsKnAwogvgAmu6tt96K06dPR1tb25D1bW1t0dfXV9BUQFHEBwCQSnwATTdr1qyYPHly9Pf3D1nf398f7e3tBU0FFEV8AE03derUWLhwYfT29jbWnTlzJnp7e2Px4sUFTgYUYUrRAwATQ3d3d6xevTpuuOGGuOmmm2LTpk1x4sSJuOeee4oeDUgmPoAUd955Zxw+fDjWr18ffX19sWDBgtixY8f7LkIFxj/3+QAAUrnmAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFTiAwBIJT4AgFT/B4iBsKYnNKCpAAAAAElFTkSuQmCC\n"},"metadata":{}}],"source":["sns.boxplot(df_new['children']);"]},{"cell_type":"markdown","metadata":{"id":"JDPia8jaLhy0"},"source":["\n","Yes, there are values 20 and -1. However, based on research, humans can have up to 15-20 children, so 20 is not an abnormal data.\n","\n","Source: https://www.businessinsider.com/how-many-children-you-can-have-lifetime-2018-11\n","\n"]},{"cell_type":"markdown","source":["**Conclusions**\n","\n","we will only replace the negative value to be absolute (positive)"],"metadata":{"id":"4B1wQGkCN7tS"}},{"cell_type":"code","execution_count":152,"metadata":{"trusted":true,"id":"0wM2zC33Lhy0","executionInfo":{"status":"ok","timestamp":1689650773797,"user_tz":-420,"elapsed":17,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# fix data based on research\n","df_new['children'] = df_new['children'].replace(-1, 1)"]},{"cell_type":"code","execution_count":153,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"A1xEQhpnLhy0","executionInfo":{"status":"ok","timestamp":1689650773798,"user_tz":-420,"elapsed":18,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"1494cdf3-11e9-4921-ad58-e4881e47c821"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0     14107\n","1      4856\n","2      2052\n","3       330\n","20       76\n","4        41\n","5         9\n","Name: children, dtype: int64"]},"metadata":{},"execution_count":153}],"source":["# re-check `children` column to ensure that everything has been fixed\n","df_new['children'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"VyAfi-ovLhy0"},"source":["Check data on `days_employed` column"]},{"cell_type":"code","execution_count":154,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"k2ZcQqa2Lhy0","executionInfo":{"status":"ok","timestamp":1689650773798,"user_tz":-420,"elapsed":13,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"e0b9ca97-0d23-43f5-9af7-efdbb2a8cf91"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["-8437.673028      1\n","-3507.818775      1\n"," 354500.415854    1\n","-769.717438       1\n","-3963.590317      1\n","                 ..\n","-1099.957609      1\n","-209.984794       1\n"," 398099.392433    1\n","-1271.038880      1\n","-1984.507589      1\n","Name: days_employed, Length: 19351, dtype: int64"]},"metadata":{},"execution_count":154}],"source":["# Find the problematic data in `days_employed`, if there is a problem, and calculate how much the percentage\n","df_new['days_employed'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"usXnK9a2Lhy1"},"source":["If the amount of problematic data is high, it may be caused by some technical problems. We probably need to propose the clearest reason why this could be and how the data should be corrected, please note that we can't delete these problematic rows."]},{"cell_type":"code","execution_count":155,"metadata":{"trusted":true,"id":"qTYxCog4Lhy1","executionInfo":{"status":"ok","timestamp":1689650773798,"user_tz":-420,"elapsed":11,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# Fix the problematic values, if exists\n","# The solution is to absolute its value\n","df_new['days_employed'] = df_new['days_employed'].abs()"]},{"cell_type":"code","execution_count":156,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"dyc6t47HLhy1","executionInfo":{"status":"ok","timestamp":1689650773798,"user_tz":-420,"elapsed":10,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"8c49357d-5235-4d07-9408-965167a7833e"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["8437.673028      1\n","3507.818775      1\n","354500.415854    1\n","769.717438       1\n","3963.590317      1\n","                ..\n","1099.957609      1\n","209.984794       1\n","398099.392433    1\n","1271.038880      1\n","1984.507589      1\n","Name: days_employed, Length: 19351, dtype: int64"]},"metadata":{},"execution_count":156}],"source":["# check the result, ensure the problem has already been fixed\n","df_new['days_employed'].value_counts()"]},{"cell_type":"code","execution_count":157,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":49},"id":"iooDbny-Lhy5","executionInfo":{"status":"ok","timestamp":1689650773800,"user_tz":-420,"elapsed":10,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"80d4f9ee-34ce-410c-9cfc-430c0fa219d0"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["Empty DataFrame\n","Columns: [children, days_employed, dob_years, education, education_id, family_status, family_status_id, gender, income_type, debt, total_income, purpose]\n","Index: []"],"text/html":["\n","\n","  <div id=\"df-c9fde798-6647-4983-bc25-fdb5159f5a03\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c9fde798-6647-4983-bc25-fdb5159f5a03')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-0539b258-bad3-4d56-9757-233d2d811bd3\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0539b258-bad3-4d56-9757-233d2d811bd3')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-0539b258-bad3-4d56-9757-233d2d811bd3 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-c9fde798-6647-4983-bc25-fdb5159f5a03 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-c9fde798-6647-4983-bc25-fdb5159f5a03');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":157}],"source":["# ensure there is no more negative value\n","df_new[df_new['days_employed'] < 0]"]},{"cell_type":"code","execution_count":158,"metadata":{"trusted":true,"id":"wgry1z8OLhy6","executionInfo":{"status":"ok","timestamp":1689650773800,"user_tz":-420,"elapsed":10,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["df_new['years_employed'] = df_new['days_employed'] / 365"]},{"cell_type":"code","execution_count":159,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":206},"id":"r7ZOsoU1Lhy6","executionInfo":{"status":"ok","timestamp":1689650774312,"user_tz":-420,"elapsed":522,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"974e1a25-42f4-4ea3-c203-3423949dd92f"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   children  days_employed  dob_years            education  education_id  \\\n","0         1    8437.673028         42    bachelor's degree             0   \n","1         1    4024.803754         36  secondary education             1   \n","2         0    5623.422610         33  secondary education             1   \n","3         3    4124.747207         32  secondary education             1   \n","4         0  340266.072047         53  secondary education             1   \n","\n","       family_status  family_status_id gender income_type  debt  total_income  \\\n","0            married                 0      F    employee     0     40620.102   \n","1            married                 0      F    employee     0     17932.802   \n","2            married                 0      M    employee     0     23341.752   \n","3            married                 0      M    employee     0     42820.568   \n","4  civil partnership                 1      F     retiree     0     25378.572   \n","\n","                   purpose  years_employed  \n","0    purchase of the house       23.116912  \n","1             car purchase       11.026860  \n","2    purchase of the house       15.406637  \n","3  supplementary education       11.300677  \n","4        to have a wedding      932.235814  "],"text/html":["\n","\n","  <div id=\"df-5a8db052-5d79-4b61-a9e0-875a094c5213\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","      <th>years_employed</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>8437.673028</td>\n","      <td>42</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","      <td>23.116912</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","      <td>11.026860</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>5623.422610</td>\n","      <td>33</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23341.752</td>\n","      <td>purchase of the house</td>\n","      <td>15.406637</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","      <td>11.300677</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>340266.072047</td>\n","      <td>53</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>25378.572</td>\n","      <td>to have a wedding</td>\n","      <td>932.235814</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5a8db052-5d79-4b61-a9e0-875a094c5213')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-7a3622f7-7d42-4c54-b639-1e3deab50c0b\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7a3622f7-7d42-4c54-b639-1e3deab50c0b')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-7a3622f7-7d42-4c54-b639-1e3deab50c0b button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-5a8db052-5d79-4b61-a9e0-875a094c5213 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-5a8db052-5d79-4b61-a9e0-875a094c5213');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":159}],"source":["df_new.head()"]},{"cell_type":"markdown","metadata":{"id":"K8B99c1dLhy6"},"source":["Now let's look at the customer's age and check if there is an issue there. Again, think about what possible abnormalities we could encounter in this column, such as age numbers that doesn't make sense."]},{"cell_type":"code","execution_count":160,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"tnttewUgLhy6","executionInfo":{"status":"ok","timestamp":1689650774313,"user_tz":-420,"elapsed":14,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"de53f4d0-1ffb-4348-b460-a07029b44193"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["[0,\n"," 19,\n"," 20,\n"," 21,\n"," 22,\n"," 23,\n"," 24,\n"," 25,\n"," 26,\n"," 27,\n"," 28,\n"," 29,\n"," 30,\n"," 31,\n"," 32,\n"," 33,\n"," 34,\n"," 35,\n"," 36,\n"," 37,\n"," 38,\n"," 39,\n"," 40,\n"," 41,\n"," 42,\n"," 43,\n"," 44,\n"," 45,\n"," 46,\n"," 47,\n"," 48,\n"," 49,\n"," 50,\n"," 51,\n"," 52,\n"," 53,\n"," 54,\n"," 55,\n"," 56,\n"," 57,\n"," 58,\n"," 59,\n"," 60,\n"," 61,\n"," 62,\n"," 63,\n"," 64,\n"," 65,\n"," 66,\n"," 67,\n"," 68,\n"," 69,\n"," 70,\n"," 71,\n"," 72,\n"," 73,\n"," 74,\n"," 75]"]},"metadata":{},"execution_count":160}],"source":["# check `dob_years` for suspected value and show how much the percentage\n","sorted(df_new['dob_years'].unique())"]},{"cell_type":"markdown","metadata":{"id":"9oxYDCZsLhy6"},"source":["Decide what you will do with the problematic value and explain why."]},{"cell_type":"code","execution_count":161,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"5SVhkpkjLhy7","executionInfo":{"status":"ok","timestamp":1689650774313,"user_tz":-420,"elapsed":11,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"40086785-9055-47f7-9f36-b27ca0c9807a"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(101, 13)"]},"metadata":{},"execution_count":161}],"source":["# Fix the problem on `dob_years` column, if the problem exists\n","df_new[df_new['dob_years']==0].shape"]},{"cell_type":"markdown","metadata":{"id":"zqKH1G94Lhy7"},"source":["- There are 101 rows with 0 value"]},{"cell_type":"code","execution_count":162,"metadata":{"trusted":true,"id":"VhYYH6WMLhy7","executionInfo":{"status":"ok","timestamp":1689650774313,"user_tz":-420,"elapsed":8,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["df_new_2 = df_new[df_new['dob_years']!=0].reset_index(drop=True)"]},{"cell_type":"code","execution_count":163,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"57C7jltTLhy7","executionInfo":{"status":"ok","timestamp":1689650774313,"user_tz":-420,"elapsed":8,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"b209d864-6c5c-4c43-eb62-0a218bfdd56e"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed  dob_years            education  education_id  \\\n","0             1    8437.673028         42    bachelor's degree             0   \n","1             1    4024.803754         36  secondary education             1   \n","2             0    5623.422610         33  secondary education             1   \n","3             3    4124.747207         32  secondary education             1   \n","4             0  340266.072047         53  secondary education             1   \n","...         ...            ...        ...                  ...           ...   \n","21365         1    4529.316663         43  secondary education             1   \n","21366         0  343937.404131         67  secondary education             1   \n","21367         1    2113.346888         38  secondary education             1   \n","21368         3    3112.481705         38  secondary education             1   \n","21369         2    1984.507589         40  secondary education             1   \n","\n","           family_status  family_status_id gender income_type  debt  \\\n","0                married                 0      F    employee     0   \n","1                married                 0      F    employee     0   \n","2                married                 0      M    employee     0   \n","3                married                 0      M    employee     0   \n","4      civil partnership                 1      F     retiree     0   \n","...                  ...               ...    ...         ...   ...   \n","21365  civil partnership                 1      F    business     0   \n","21366            married                 0      F     retiree     0   \n","21367  civil partnership                 1      M    employee     1   \n","21368            married                 0      M    employee     1   \n","21369            married                 0      F    employee     0   \n","\n","       total_income                  purpose  years_employed  \n","0         40620.102    purchase of the house       23.116912  \n","1         17932.802             car purchase       11.026860  \n","2         23341.752    purchase of the house       15.406637  \n","3         42820.568  supplementary education       11.300677  \n","4         25378.572        to have a wedding      932.235814  \n","...             ...                      ...             ...  \n","21365     35966.698     housing transactions       12.409087  \n","21366     24959.969        purchase of a car      942.294258  \n","21367     14347.610                 property        5.789991  \n","21368     39054.888        buying my own car        8.527347  \n","21369     13127.587             to buy a car        5.437007  \n","\n","[21370 rows x 13 columns]"],"text/html":["\n","\n","  <div id=\"df-b0ab66cf-aa98-4dfa-94ca-1de70d7e29ec\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","      <th>years_employed</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>8437.673028</td>\n","      <td>42</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","      <td>23.116912</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","      <td>11.026860</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>5623.422610</td>\n","      <td>33</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23341.752</td>\n","      <td>purchase of the house</td>\n","      <td>15.406637</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","      <td>11.300677</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>340266.072047</td>\n","      <td>53</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>25378.572</td>\n","      <td>to have a wedding</td>\n","      <td>932.235814</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>21365</th>\n","      <td>1</td>\n","      <td>4529.316663</td>\n","      <td>43</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>35966.698</td>\n","      <td>housing transactions</td>\n","      <td>12.409087</td>\n","    </tr>\n","    <tr>\n","      <th>21366</th>\n","      <td>0</td>\n","      <td>343937.404131</td>\n","      <td>67</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>24959.969</td>\n","      <td>purchase of a car</td>\n","      <td>942.294258</td>\n","    </tr>\n","    <tr>\n","      <th>21367</th>\n","      <td>1</td>\n","      <td>2113.346888</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>14347.610</td>\n","      <td>property</td>\n","      <td>5.789991</td>\n","    </tr>\n","    <tr>\n","      <th>21368</th>\n","      <td>3</td>\n","      <td>3112.481705</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>39054.888</td>\n","      <td>buying my own car</td>\n","      <td>8.527347</td>\n","    </tr>\n","    <tr>\n","      <th>21369</th>\n","      <td>2</td>\n","      <td>1984.507589</td>\n","      <td>40</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>13127.587</td>\n","      <td>to buy a car</td>\n","      <td>5.437007</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>21370 rows × 13 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b0ab66cf-aa98-4dfa-94ca-1de70d7e29ec')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-442f0c58-fa4c-466c-b0ee-14e667711bd6\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-442f0c58-fa4c-466c-b0ee-14e667711bd6')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-442f0c58-fa4c-466c-b0ee-14e667711bd6 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-b0ab66cf-aa98-4dfa-94ca-1de70d7e29ec button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-b0ab66cf-aa98-4dfa-94ca-1de70d7e29ec');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":163}],"source":["df_new_2"]},{"cell_type":"markdown","metadata":{"id":"l4OtYYavLhy7"},"source":["- Filter rows with condition not equals to 0"]},{"cell_type":"code","execution_count":164,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"4OZJRn9JLhy7","executionInfo":{"status":"ok","timestamp":1689650774743,"user_tz":-420,"elapsed":437,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"b8584b33-4853-470d-c70f-c5001b6c5662"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["[19,\n"," 20,\n"," 21,\n"," 22,\n"," 23,\n"," 24,\n"," 25,\n"," 26,\n"," 27,\n"," 28,\n"," 29,\n"," 30,\n"," 31,\n"," 32,\n"," 33,\n"," 34,\n"," 35,\n"," 36,\n"," 37,\n"," 38,\n"," 39,\n"," 40,\n"," 41,\n"," 42,\n"," 43,\n"," 44,\n"," 45,\n"," 46,\n"," 47,\n"," 48,\n"," 49,\n"," 50,\n"," 51,\n"," 52,\n"," 53,\n"," 54,\n"," 55,\n"," 56,\n"," 57,\n"," 58,\n"," 59,\n"," 60,\n"," 61,\n"," 62,\n"," 63,\n"," 64,\n"," 65,\n"," 66,\n"," 67,\n"," 68,\n"," 69,\n"," 70,\n"," 71,\n"," 72,\n"," 73,\n"," 74,\n"," 75]"]},"metadata":{},"execution_count":164}],"source":["# check the result, ensure all the issue has been fixed\n","sorted(df_new_2['dob_years'].unique())"]},{"cell_type":"markdown","metadata":{"id":"C7GtcCZDLhy7"},"source":["- `dob_years` data with 0 value has been removed completely"]},{"cell_type":"markdown","metadata":{"id":"RyDqMoBJLhy7"},"source":["Now it's time to check the `family_status` column. Check what kind of values are contained in this column and what issues you might need to address."]},{"cell_type":"code","execution_count":165,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"WMyl-McOLhy8","executionInfo":{"status":"ok","timestamp":1689650774744,"user_tz":-420,"elapsed":28,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"e095b228-eaf5-4146-dfb0-45abadd362bf"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["married              12295\n","civil partnership     4142\n","unmarried             2794\n","divorced              1185\n","widow / widower        954\n","Name: family_status, dtype: int64"]},"metadata":{},"execution_count":165}],"source":["# check values on 'family_status' column\n","df_new_2['family_status'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"jgKsTRoSLhy8"},"source":["- There is no abnormal data in the `family_status` column, so, we don't need to do a data cleansing"]},{"cell_type":"markdown","metadata":{"id":"HUbWuc-jLhy8"},"source":["Now it's time to check the `gender` column. Check what kind of values are contained in this column and what issues you might need to address"]},{"cell_type":"code","execution_count":166,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"b_ArRZhPLhy8","executionInfo":{"status":"ok","timestamp":1689650774744,"user_tz":-420,"elapsed":25,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"f03eb532-efa3-44dd-df12-6ea507498f59"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["F      14117\n","M       7252\n","XNA        1\n","Name: gender, dtype: int64"]},"metadata":{},"execution_count":166}],"source":["# check values on 'gender' column\n","df_new_2['gender'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"b48dVsJSLhy8"},"source":["- There is only 1 row data with XNA value"]},{"cell_type":"code","execution_count":167,"metadata":{"trusted":true,"id":"WqQYv-bfLhy8","executionInfo":{"status":"ok","timestamp":1689650774745,"user_tz":-420,"elapsed":25,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["drop_index = df_new_2[df_new_2['gender']=='XNA'].index"]},{"cell_type":"code","execution_count":168,"metadata":{"trusted":true,"id":"_OR0uKL7Lhy9","executionInfo":{"status":"ok","timestamp":1689650774745,"user_tz":-420,"elapsed":23,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# Fix the problematic values, if exists\n","df_new_3 = df_new_2.drop(drop_index).reset_index(drop=True)"]},{"cell_type":"code","execution_count":169,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"Z22sL1PTLhy9","executionInfo":{"status":"ok","timestamp":1689650774745,"user_tz":-420,"elapsed":23,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"5cf97dd3-7c11-4645-a9e0-d41a0407b5b8"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed  dob_years            education  education_id  \\\n","0             1    8437.673028         42    bachelor's degree             0   \n","1             1    4024.803754         36  secondary education             1   \n","2             0    5623.422610         33  secondary education             1   \n","3             3    4124.747207         32  secondary education             1   \n","4             0  340266.072047         53  secondary education             1   \n","...         ...            ...        ...                  ...           ...   \n","21364         1    4529.316663         43  secondary education             1   \n","21365         0  343937.404131         67  secondary education             1   \n","21366         1    2113.346888         38  secondary education             1   \n","21367         3    3112.481705         38  secondary education             1   \n","21368         2    1984.507589         40  secondary education             1   \n","\n","           family_status  family_status_id gender income_type  debt  \\\n","0                married                 0      F    employee     0   \n","1                married                 0      F    employee     0   \n","2                married                 0      M    employee     0   \n","3                married                 0      M    employee     0   \n","4      civil partnership                 1      F     retiree     0   \n","...                  ...               ...    ...         ...   ...   \n","21364  civil partnership                 1      F    business     0   \n","21365            married                 0      F     retiree     0   \n","21366  civil partnership                 1      M    employee     1   \n","21367            married                 0      M    employee     1   \n","21368            married                 0      F    employee     0   \n","\n","       total_income                  purpose  years_employed  \n","0         40620.102    purchase of the house       23.116912  \n","1         17932.802             car purchase       11.026860  \n","2         23341.752    purchase of the house       15.406637  \n","3         42820.568  supplementary education       11.300677  \n","4         25378.572        to have a wedding      932.235814  \n","...             ...                      ...             ...  \n","21364     35966.698     housing transactions       12.409087  \n","21365     24959.969        purchase of a car      942.294258  \n","21366     14347.610                 property        5.789991  \n","21367     39054.888        buying my own car        8.527347  \n","21368     13127.587             to buy a car        5.437007  \n","\n","[21369 rows x 13 columns]"],"text/html":["\n","\n","  <div id=\"df-9019eeba-e021-41de-a422-afa543db6330\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","      <th>years_employed</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>8437.673028</td>\n","      <td>42</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","      <td>23.116912</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","      <td>11.026860</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>5623.422610</td>\n","      <td>33</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23341.752</td>\n","      <td>purchase of the house</td>\n","      <td>15.406637</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","      <td>11.300677</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>340266.072047</td>\n","      <td>53</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>25378.572</td>\n","      <td>to have a wedding</td>\n","      <td>932.235814</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>21364</th>\n","      <td>1</td>\n","      <td>4529.316663</td>\n","      <td>43</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>35966.698</td>\n","      <td>housing transactions</td>\n","      <td>12.409087</td>\n","    </tr>\n","    <tr>\n","      <th>21365</th>\n","      <td>0</td>\n","      <td>343937.404131</td>\n","      <td>67</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>24959.969</td>\n","      <td>purchase of a car</td>\n","      <td>942.294258</td>\n","    </tr>\n","    <tr>\n","      <th>21366</th>\n","      <td>1</td>\n","      <td>2113.346888</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>14347.610</td>\n","      <td>property</td>\n","      <td>5.789991</td>\n","    </tr>\n","    <tr>\n","      <th>21367</th>\n","      <td>3</td>\n","      <td>3112.481705</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>39054.888</td>\n","      <td>buying my own car</td>\n","      <td>8.527347</td>\n","    </tr>\n","    <tr>\n","      <th>21368</th>\n","      <td>2</td>\n","      <td>1984.507589</td>\n","      <td>40</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>13127.587</td>\n","      <td>to buy a car</td>\n","      <td>5.437007</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>21369 rows × 13 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9019eeba-e021-41de-a422-afa543db6330')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-dd920812-9c2e-4c53-8eeb-9b3a58c48984\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-dd920812-9c2e-4c53-8eeb-9b3a58c48984')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-dd920812-9c2e-4c53-8eeb-9b3a58c48984 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-9019eeba-e021-41de-a422-afa543db6330 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-9019eeba-e021-41de-a422-afa543db6330');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":169}],"source":["df_new_3"]},{"cell_type":"code","execution_count":170,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"J6GBsiM6Lhy9","executionInfo":{"status":"ok","timestamp":1689650774746,"user_tz":-420,"elapsed":20,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"a96b4960-e25a-452f-f50c-1017fd5e4650"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["F    14116\n","M     7252\n","Name: gender, dtype: int64"]},"metadata":{},"execution_count":170}],"source":["# check the result, ensure that everything has been fixed\n","df_new_3.drop(drop_index)['gender'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"e5_dhRxHLhy9"},"source":["- Data with XNA value already removed"]},{"cell_type":"markdown","metadata":{"id":"w2BKEfx_Lhy9"},"source":["Now it's time to examine the `income_type` column. Check what kind of values are contained in this column and what issues you might need to address"]},{"cell_type":"code","execution_count":171,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"WhSKiF9ILhy9","executionInfo":{"status":"ok","timestamp":1689650774746,"user_tz":-420,"elapsed":18,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"2ae0e89f-859b-41a6-af98-1032fd219840"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["employee                       11036\n","business                        5059\n","retiree                         3817\n","civil servant                   1451\n","unemployed                         2\n","entrepreneur                       2\n","student                            1\n","paternity / maternity leave        1\n","Name: income_type, dtype: int64"]},"metadata":{},"execution_count":171}],"source":["# check values on 'income_type' column\n","df_new_3['income_type'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"xdJ5XuT9Lhy-"},"source":["- There is no abnormal data in the `income_type` column, so, we don't need to do a data cleansing"]},{"cell_type":"markdown","metadata":{"id":"MCxqUKi7Lhy-"},"source":["Now it's time to see if there are any duplicates in our data. If we find one, you must decide what you will do with the duplicate and explain why."]},{"cell_type":"markdown","metadata":{"id":"Yu8NuPBpLhy-"},"source":["- The duplicate part already been fixed on point no 2 on Data Exploration"]},{"cell_type":"code","execution_count":172,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"MCJZITOQLhy-","executionInfo":{"status":"ok","timestamp":1689650774746,"user_tz":-420,"elapsed":16,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"626674e0-3a69-4e42-9dc9-f4363517a1db"},"outputs":[{"output_type":"stream","name":"stdout","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 21369 entries, 0 to 21368\n","Data columns (total 13 columns):\n"," #   Column            Non-Null Count  Dtype  \n","---  ------            --------------  -----  \n"," 0   children          21369 non-null  int64  \n"," 1   days_employed     19259 non-null  float64\n"," 2   dob_years         21369 non-null  int64  \n"," 3   education         21369 non-null  object \n"," 4   education_id      21369 non-null  int64  \n"," 5   family_status     21369 non-null  object \n"," 6   family_status_id  21369 non-null  int64  \n"," 7   gender            21369 non-null  object \n"," 8   income_type       21369 non-null  object \n"," 9   debt              21369 non-null  int64  \n"," 10  total_income      19259 non-null  float64\n"," 11  purpose           21369 non-null  object \n"," 12  years_employed    19259 non-null  float64\n","dtypes: float64(3), int64(5), object(5)\n","memory usage: 2.1+ MB\n"]}],"source":["# check dataset size that we've got now after data manipulation that we've been through\n","df_new_3.info()"]},{"cell_type":"markdown","metadata":{"id":"LyTSIaxnLhy_"},"source":["- We have dropped data with abnormal values. But empty data still exists, then we have to fill in the empty data with fillna"]},{"cell_type":"markdown","metadata":{"id":"Z3Vake4bLhy_"},"source":["# Work with missing values"]},{"cell_type":"markdown","metadata":{"id":"UFmFM_svLhy_"},"source":["To speed things up with a large amount of data, you may want to use a *dictionary* for multiple values that have an ID. Explain why and what *dictionary* you will use."]},{"cell_type":"code","execution_count":173,"metadata":{"trusted":true,"id":"jbl9rKJALhy_","executionInfo":{"status":"ok","timestamp":1689650774746,"user_tz":-420,"elapsed":15,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# find the dictionary\n","# Note: find out inter-column mapping\n","dict_edu = df_new_3[['education_id', 'education']].drop_duplicates().reset_index(drop=True)"]},{"cell_type":"code","execution_count":174,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"N6A94bLPLhy_","executionInfo":{"status":"ok","timestamp":1689650774746,"user_tz":-420,"elapsed":14,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"2e143ac1-f3c6-4dfe-d8ce-8f645ea925e6"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["[{'education_id': 0, 'education': \"bachelor's degree\"},\n"," {'education_id': 1, 'education': 'secondary education'},\n"," {'education_id': 2, 'education': 'some college'},\n"," {'education_id': 3, 'education': 'primary education'},\n"," {'education_id': 4, 'education': 'graduate degree'}]"]},"metadata":{},"execution_count":174}],"source":["dict_edu.to_dict(orient='records')"]},{"cell_type":"code","execution_count":175,"metadata":{"trusted":true,"id":"l9giTNTnLhy_","executionInfo":{"status":"ok","timestamp":1689650774747,"user_tz":-420,"elapsed":12,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["dict_family = df_new_3[['family_status_id', 'family_status']].drop_duplicates().reset_index(drop=True)"]},{"cell_type":"code","execution_count":176,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"GcpKxxOVLhy_","executionInfo":{"status":"ok","timestamp":1689650774747,"user_tz":-420,"elapsed":12,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"cde05024-9e73-44e1-ba26-d6e7da457666"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["[{'family_status_id': 0, 'family_status': 'married'},\n"," {'family_status_id': 1, 'family_status': 'civil partnership'},\n"," {'family_status_id': 2, 'family_status': 'widow / widower'},\n"," {'family_status_id': 3, 'family_status': 'divorced'},\n"," {'family_status_id': 4, 'family_status': 'unmarried'}]"]},"metadata":{},"execution_count":176}],"source":["dict_family.to_dict(orient='records')"]},{"cell_type":"markdown","metadata":{"id":"myNuvW3KLhy_"},"source":["### Fix missing value in `total_income` column"]},{"cell_type":"markdown","metadata":{"id":"MzerYuD2Lhy_"},"source":["Briefly describe which column with missing values you need to work with. Describe how you would fix it.\n","\n","Start by working out the total lost income value. Create an age category for the customer. Create a new column that contains age category. This strategy can help to calculate the total revenue value.\n"]},{"cell_type":"code","execution_count":177,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"N8MRJVH0LhzA","executionInfo":{"status":"ok","timestamp":1689650775203,"user_tz":-420,"elapsed":467,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"7eab2d1a-16de-4160-d485-aff2b7441f23"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed  dob_years            education  education_id  \\\n","0             1   -8437.673028         42    bachelor's degree             0   \n","1             1   -4024.803754         36  secondary education             1   \n","2             0   -5623.422610         33  Secondary Education             1   \n","3             3   -4124.747207         32  secondary education             1   \n","4             0  340266.072047         53  secondary education             1   \n","...         ...            ...        ...                  ...           ...   \n","21520         1   -4529.316663         43  secondary education             1   \n","21521         0  343937.404131         67  secondary education             1   \n","21522         1   -2113.346888         38  secondary education             1   \n","21523         3   -3112.481705         38  secondary education             1   \n","21524         2   -1984.507589         40  secondary education             1   \n","\n","           family_status  family_status_id gender income_type  debt  \\\n","0                married                 0      F    employee     0   \n","1                married                 0      F    employee     0   \n","2                married                 0      M    employee     0   \n","3                married                 0      M    employee     0   \n","4      civil partnership                 1      F     retiree     0   \n","...                  ...               ...    ...         ...   ...   \n","21520  civil partnership                 1      F    business     0   \n","21521            married                 0      F     retiree     0   \n","21522  civil partnership                 1      M    employee     1   \n","21523            married                 0      M    employee     1   \n","21524            married                 0      F    employee     0   \n","\n","       total_income                  purpose  \n","0         40620.102    purchase of the house  \n","1         17932.802             car purchase  \n","2         23341.752    purchase of the house  \n","3         42820.568  supplementary education  \n","4         25378.572        to have a wedding  \n","...             ...                      ...  \n","21520     35966.698     housing transactions  \n","21521     24959.969        purchase of a car  \n","21522     14347.610                 property  \n","21523     39054.888        buying my own car  \n","21524     13127.587             to buy a car  \n","\n","[21525 rows x 12 columns]"],"text/html":["\n","\n","  <div id=\"df-5959fb32-db32-4666-8591-7798673808c2\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>-8437.673028</td>\n","      <td>42</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>-4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>-5623.422610</td>\n","      <td>33</td>\n","      <td>Secondary Education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23341.752</td>\n","      <td>purchase of the house</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>-4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>340266.072047</td>\n","      <td>53</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>25378.572</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>21520</th>\n","      <td>1</td>\n","      <td>-4529.316663</td>\n","      <td>43</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>35966.698</td>\n","      <td>housing transactions</td>\n","    </tr>\n","    <tr>\n","      <th>21521</th>\n","      <td>0</td>\n","      <td>343937.404131</td>\n","      <td>67</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>24959.969</td>\n","      <td>purchase of a car</td>\n","    </tr>\n","    <tr>\n","      <th>21522</th>\n","      <td>1</td>\n","      <td>-2113.346888</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>14347.610</td>\n","      <td>property</td>\n","    </tr>\n","    <tr>\n","      <th>21523</th>\n","      <td>3</td>\n","      <td>-3112.481705</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>39054.888</td>\n","      <td>buying my own car</td>\n","    </tr>\n","    <tr>\n","      <th>21524</th>\n","      <td>2</td>\n","      <td>-1984.507589</td>\n","      <td>40</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>13127.587</td>\n","      <td>to buy a car</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>21525 rows × 12 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5959fb32-db32-4666-8591-7798673808c2')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-802fd298-4623-4277-a699-55f5cc4be332\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-802fd298-4623-4277-a699-55f5cc4be332')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-802fd298-4623-4277-a699-55f5cc4be332 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-5959fb32-db32-4666-8591-7798673808c2 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-5959fb32-db32-4666-8591-7798673808c2');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":177}],"source":["df"]},{"cell_type":"code","execution_count":178,"metadata":{"trusted":true,"id":"dEsgI9uSLhzA","executionInfo":{"status":"ok","timestamp":1689650775203,"user_tz":-420,"elapsed":21,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# write a function to calculate age category\n","def age_group(age):\n","    try:\n","        if age <= 17:\n","            return 'Teenager'\n","        elif 18 <= age <= 30:\n","            return 'Mature'\n","        elif 31 <= age <= 59:\n","            return 'Senior'\n","        else:\n","            return 'Old'\n","    except:\n","        return 'Not Specified'"]},{"cell_type":"code","execution_count":179,"metadata":{"trusted":true,"id":"wlbBSLCkLhzA","executionInfo":{"status":"ok","timestamp":1689650775203,"user_tz":-420,"elapsed":20,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# test the function, is it works or not\n","df_new_3['dob_years'] = df_new_3['dob_years'].replace(42, '42 Tahun')"]},{"cell_type":"code","execution_count":180,"metadata":{"trusted":true,"id":"sDdNPbAPLhzA","executionInfo":{"status":"ok","timestamp":1689650775204,"user_tz":-420,"elapsed":21,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# create a new column based on function\n","df_new_3['age_category'] = df_new_3['dob_years'].apply(age_group)"]},{"cell_type":"code","execution_count":181,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"W0E1-2fyLhzA","executionInfo":{"status":"ok","timestamp":1689650775204,"user_tz":-420,"elapsed":20,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"3862381c-1547-45fb-c2e4-3279312d7aa8"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["Senior           14549\n","Mature            3718\n","Old               2506\n","Not Specified      596\n","Name: age_category, dtype: int64"]},"metadata":{},"execution_count":181}],"source":["# review value in new column\n","df_new_3['age_category'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"-uyumK-ELhzA"},"source":["Think about everything that usually depends on income. Eventually, you'll find out whether you should use the mean or the median to replace the missing values. To make this decision, you may want to look at the distribution of the data that you already identify as impacted a person's income."]},{"cell_type":"markdown","metadata":{"id":"j6_KwdDbLhzA"},"source":["Create a table that contain only data with no missing values. This data will be used to fix missing values."]},{"cell_type":"code","execution_count":182,"metadata":{"trusted":true,"id":"y0kEKv_BLhzA","executionInfo":{"status":"ok","timestamp":1689650775204,"user_tz":-420,"elapsed":19,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# create table without missing value and show several datarows to ensure everything goes well\n","df_not_null = df[df['total_income'].notnull()].reset_index(drop=True)"]},{"cell_type":"code","execution_count":183,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"C4g_mn6cLhzB","executionInfo":{"status":"ok","timestamp":1689650775204,"user_tz":-420,"elapsed":19,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"2493f1bf-6ffa-438a-e62c-095fe5a305c6"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed  dob_years            education  education_id  \\\n","0             1   -8437.673028         42    bachelor's degree             0   \n","1             1   -4024.803754         36  secondary education             1   \n","2             0   -5623.422610         33  Secondary Education             1   \n","3             3   -4124.747207         32  secondary education             1   \n","4             0  340266.072047         53  secondary education             1   \n","...         ...            ...        ...                  ...           ...   \n","19346         1   -4529.316663         43  secondary education             1   \n","19347         0  343937.404131         67  secondary education             1   \n","19348         1   -2113.346888         38  secondary education             1   \n","19349         3   -3112.481705         38  secondary education             1   \n","19350         2   -1984.507589         40  secondary education             1   \n","\n","           family_status  family_status_id gender income_type  debt  \\\n","0                married                 0      F    employee     0   \n","1                married                 0      F    employee     0   \n","2                married                 0      M    employee     0   \n","3                married                 0      M    employee     0   \n","4      civil partnership                 1      F     retiree     0   \n","...                  ...               ...    ...         ...   ...   \n","19346  civil partnership                 1      F    business     0   \n","19347            married                 0      F     retiree     0   \n","19348  civil partnership                 1      M    employee     1   \n","19349            married                 0      M    employee     1   \n","19350            married                 0      F    employee     0   \n","\n","       total_income                  purpose  \n","0         40620.102    purchase of the house  \n","1         17932.802             car purchase  \n","2         23341.752    purchase of the house  \n","3         42820.568  supplementary education  \n","4         25378.572        to have a wedding  \n","...             ...                      ...  \n","19346     35966.698     housing transactions  \n","19347     24959.969        purchase of a car  \n","19348     14347.610                 property  \n","19349     39054.888        buying my own car  \n","19350     13127.587             to buy a car  \n","\n","[19351 rows x 12 columns]"],"text/html":["\n","\n","  <div id=\"df-5c25702b-06c6-4dac-99d6-70b5ce85bc9c\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>-8437.673028</td>\n","      <td>42</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>-4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>-5623.422610</td>\n","      <td>33</td>\n","      <td>Secondary Education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23341.752</td>\n","      <td>purchase of the house</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>-4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>340266.072047</td>\n","      <td>53</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>25378.572</td>\n","      <td>to have a wedding</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>19346</th>\n","      <td>1</td>\n","      <td>-4529.316663</td>\n","      <td>43</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>35966.698</td>\n","      <td>housing transactions</td>\n","    </tr>\n","    <tr>\n","      <th>19347</th>\n","      <td>0</td>\n","      <td>343937.404131</td>\n","      <td>67</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>24959.969</td>\n","      <td>purchase of a car</td>\n","    </tr>\n","    <tr>\n","      <th>19348</th>\n","      <td>1</td>\n","      <td>-2113.346888</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>14347.610</td>\n","      <td>property</td>\n","    </tr>\n","    <tr>\n","      <th>19349</th>\n","      <td>3</td>\n","      <td>-3112.481705</td>\n","      <td>38</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>1</td>\n","      <td>39054.888</td>\n","      <td>buying my own car</td>\n","    </tr>\n","    <tr>\n","      <th>19350</th>\n","      <td>2</td>\n","      <td>-1984.507589</td>\n","      <td>40</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>13127.587</td>\n","      <td>to buy a car</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>19351 rows × 12 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5c25702b-06c6-4dac-99d6-70b5ce85bc9c')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-37530417-a471-438d-bf9a-2b510edfb403\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-37530417-a471-438d-bf9a-2b510edfb403')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-37530417-a471-438d-bf9a-2b510edfb403 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-5c25702b-06c6-4dac-99d6-70b5ce85bc9c button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-5c25702b-06c6-4dac-99d6-70b5ce85bc9c');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":183}],"source":["df_not_null"]},{"cell_type":"code","execution_count":184,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"BeZzyj4ELhzB","executionInfo":{"status":"ok","timestamp":1689650775205,"user_tz":-420,"elapsed":19,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"3772503f-0acb-4b09-dd98-d3bf1b489e80"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["gender\n","F      24655.604757\n","M      30907.144369\n","XNA    32624.825000\n","Name: total_income, dtype: float64"]},"metadata":{},"execution_count":184}],"source":["# check the mean value for income based on asumption\n","df_not_null.groupby('gender')['total_income'].mean()"]},{"cell_type":"code","execution_count":185,"metadata":{"scrolled":false,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"_Tu6TlzGLhzB","executionInfo":{"status":"ok","timestamp":1689650775205,"user_tz":-420,"elapsed":17,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"be60f3fd-2919-4f80-feb9-0618cb3b4ada"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["gender\n","F      21464.845\n","M      26834.295\n","XNA    32624.825\n","Name: total_income, dtype: float64"]},"metadata":{},"execution_count":185}],"source":["# check the median value for income based on asumption\n","df_not_null.groupby('gender')['total_income'].median()"]},{"cell_type":"markdown","metadata":{"id":"AuVRV-X8LhzB"},"source":["- mean has fragile characteristic / can get easily infuenced by outliers\n","- meadian has robust characteristic / can't get easily infuenced by outliers"]},{"cell_type":"code","execution_count":186,"metadata":{"scrolled":false,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":430},"id":"_w0-LrbVLhzB","executionInfo":{"status":"ok","timestamp":1689650775205,"user_tz":-420,"elapsed":15,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"92b864ec-e295-4cac-db05-f3fc7073ad69"},"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAkIAAAGdCAYAAAD+JxxnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA2v0lEQVR4nO3df3RU9Z3/8dckMpMgTgIGEmJ+ILJC+b1GEqc/RGsOkWZ7BO0RhW2pRVk0uEAsSHZbcN2ek4qnrFQRt9s9hq5QfuxZf0uUExBPmwAhNhURaK1ASGGCYDOjmF/MfL5/+M09DIQfMciF+Twf59xznPt+z8174onzcubez/UYY4wAAAAslOD2AAAAAG4hCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArHWF2wNcyqLRqA4dOqSrrrpKHo/H7XEAAMB5MMbo008/VWZmphISzv6ZD0HoLA4dOqTs7Gy3xwAAAF/CwYMHlZWVddYegtBZXHXVVZK++EX6/X6XpwEAAOcjHA4rOzvbeR8/G4LQWXR+Heb3+wlCAABcZs7ntBZOlgYAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAmCl6upqTZkyRdXV1W6PAsBFBCEA1mltbdXSpUvV1NSkpUuXqrW11e2RALiEIATAOqtWrdKxY8ckSceOHdPq1atdngiAWwhCAKzS2Nio1atXyxgjSTLGaPXq1WpsbHR5MgBuIAgBsIYxRsuWLTvj/s5wBMAeBCEA1mhoaFBtba0ikUjM/kgkotraWjU0NLg0GQC3EIQAWCMnJ0fjxo1TYmJizP7ExETl5+crJyfHpckAuIUgBMAaHo9Hc+bMOeN+j8fjwlQA3EQQAmCVrKwsTZ061Qk9Ho9HU6dO1TXXXOPyZADcQBACYJ1p06bp6quvliSlpaVp6tSpLk8EwC0EIQDWSUpK0sSJE5WQkKDbb79dSUlJbo8EwCUEIQDWaW1t1YYNGxSNRrVhwwZWlgYsRhACYB1WlgbQqVtBaMWKFRo9erT8fr/8fr8CgYA2bNjg1G+55RZ5PJ6YbdasWTHHaGhoUHFxsXr37q0BAwZo/vz5OnHiREzP22+/rRtuuEE+n09DhgxRRUXFabMsX75cgwYNUlJSkgoKCrR9+/aYemtrq0pKSnT11VerT58+uuuuu9TU1NSdlwsgDrGyNICTdSsIZWVl6ec//7nq6uq0Y8cOffvb39Ydd9yhXbt2OT0PPPCADh8+7GxLlixxapFIRMXFxWpvb1d1dbVWrlypiooKLVq0yOnZt2+fiouLdeutt6q+vl5z587V/fffrzfffNPpWbt2rUpLS7V48WK9++67GjNmjIqKinTkyBGnZ968eXr11Ve1fv16bdmyRYcOHdKdd975pX5JAOIDK0sDOJXH9PAvv1+/fnryySc1Y8YM3XLLLRo7dqyeeuqpLns3bNigf/iHf9ChQ4eUnp4uSXruuef06KOP6uOPP5bX69Wjjz6q119/Xe+//77zvHvuuUfNzc2qrKyUJBUUFGjcuHF65plnJEnRaFTZ2dl6+OGHtXDhQoVCIfXv31+rV6/W9773PUnSnj179LWvfU01NTW66aabzuu1hcNhpaSkKBQKye/3f9lfEYBLxIEDBzR9+vQz1leuXKnc3NyLOBGAr0J33r+/9DlCkUhEa9as0fHjxxUIBJz9q1atUlpamkaOHKmysjJ9/vnnTq2mpkajRo1yQpAkFRUVKRwOO58q1dTUqLCwMOZnFRUVqaamRpLU3t6uurq6mJ6EhAQVFhY6PXV1dero6IjpGTZsmHJycpyerrS1tSkcDsdsAOIHK0sDONUV3X3Czp07FQgE1Nraqj59+ujFF1/U8OHDJUlTp05Vbm6uMjMz9d577+nRRx/V3r179X//93+SpGAwGBOCJDmPg8HgWXvC4bBaWlr0t7/9TZFIpMuePXv2OMfwer1KTU09rafz53SlvLxc//Zv/9bN3wiAy0XnCtKnfirEytKAvbodhIYOHar6+nqFQiH97//+r6ZPn64tW7Zo+PDhmjlzptM3atQoDRw4ULfddpv+8pe/6Lrrrrugg38VysrKVFpa6jwOh8PKzs52cSIAF1rnytIvvPCCjDGsLA1YrttfjXm9Xg0ZMkR5eXkqLy/XmDFjujz5UPriXB5J+vDDDyVJGRkZp1251fk4IyPjrD1+v1/JyclKS0tTYmJilz0nH6O9vV3Nzc1n7OmKz+dzrojr3ADEH1aWBtCpx+sIRaNRtbW1dVmrr6+XJA0cOFCSFAgEtHPnzpiruzZu3Ci/3+98vRYIBFRVVRVznI0bNzrnIXm9XuXl5cX0RKNRVVVVOT15eXnq1atXTM/evXvV0NAQcz4TADslJSWptLRU6enpmjdvHitLAzYz3bBw4UKzZcsWs2/fPvPee++ZhQsXGo/HY9566y3z4Ycfmscff9zs2LHD7Nu3z7z88stm8ODB5uabb3aef+LECTNy5EgzYcIEU19fbyorK03//v1NWVmZ0/PRRx+Z3r17m/nz55vdu3eb5cuXm8TERFNZWen0rFmzxvh8PlNRUWE++OADM3PmTJOammqCwaDTM2vWLJOTk2M2bdpkduzYYQKBgAkEAt15uSYUChlJJhQKdet5AADAPd15/+5WEPrRj35kcnNzjdfrNf379ze33Xabeeutt4wxxjQ0NJibb77Z9OvXz/h8PjNkyBAzf/7804bYv3+/mThxoklOTjZpaWnmkUceMR0dHTE9mzdvNmPHjjVer9cMHjzYPP/886fN8vTTT5ucnBzj9XpNfn6+2bp1a0y9paXFPPTQQ6Zv376md+/eZvLkyebw4cPdebkEIQAALkPdef/u8TpC8Yx1hAAAuPxclHWEAAAALncEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYq1tBaMWKFRo9erT8fr/8fr8CgYA2bNjg1FtbW1VSUqKrr75affr00V133aWmpqaYYzQ0NKi4uFi9e/fWgAEDNH/+fJ04cSKm5+2339YNN9wgn8+nIUOGqKKi4rRZli9frkGDBikpKUkFBQXavn17TP18ZgEAAHbrVhDKysrSz3/+c9XV1WnHjh369re/rTvuuEO7du2SJM2bN0+vvvqq1q9fry1btujQoUO68847nedHIhEVFxervb1d1dXVWrlypSoqKrRo0SKnZ9++fSouLtatt96q+vp6zZ07V/fff7/efPNNp2ft2rUqLS3V4sWL9e6772rMmDEqKirSkSNHnJ5zzQIAACDTQ3379jW//vWvTXNzs+nVq5dZv369U9u9e7eRZGpqaowxxrzxxhsmISHBBINBp2fFihXG7/ebtrY2Y4wxCxYsMCNGjIj5GVOmTDFFRUXO4/z8fFNSUuI8jkQiJjMz05SXlxtjzHnNcj5CoZCRZEKh0Hk/BwAAuKs7799f+hyhSCSiNWvW6Pjx4woEAqqrq1NHR4cKCwudnmHDhiknJ0c1NTWSpJqaGo0aNUrp6elOT1FRkcLhsPOpUk1NTcwxOns6j9He3q66urqYnoSEBBUWFjo95zNLV9ra2hQOh2M2AAAQv7odhHbu3Kk+ffrI5/Np1qxZevHFFzV8+HAFg0F5vV6lpqbG9KenpysYDEqSgsFgTAjqrHfWztYTDofV0tKio0ePKhKJdNlz8jHONUtXysvLlZKS4mzZ2dnn90sBAACXpW4HoaFDh6q+vl7btm3Tgw8+qOnTp+uDDz74Kma76MrKyhQKhZzt4MGDbo8EAAC+Qld09wler1dDhgyRJOXl5am2tlbLli3TlClT1N7erubm5phPYpqampSRkSFJysjIOO3qrs4ruU7uOfXqrqamJvn9fiUnJysxMVGJiYld9px8jHPN0hWfzyefz9eN3wYAALic9XgdoWg0qra2NuXl5alXr16qqqpyanv37lVDQ4MCgYAkKRAIaOfOnTFXd23cuFF+v1/Dhw93ek4+RmdP5zG8Xq/y8vJieqLRqKqqqpye85kFAACgW1eNLVy40GzZssXs27fPvPfee2bhwoXG4/GYt956yxhjzKxZs0xOTo7ZtGmT2bFjhwkEAiYQCDjPP3HihBk5cqSZMGGCqa+vN5WVlaZ///6mrKzM6fnoo49M7969zfz5883u3bvN8uXLTWJioqmsrHR61qxZY3w+n6moqDAffPCBmTlzpklNTY25Gu1cs5wPrhoDAODy0533724FoR/96EcmNzfXeL1e079/f3Pbbbc5IcgYY1paWsxDDz1k+vbta3r37m0mT55sDh8+HHOM/fv3m4kTJ5rk5GSTlpZmHnnkEdPR0RHTs3nzZjN27Fjj9XrN4MGDzfPPP3/aLE8//bTJyckxXq/X5Ofnm61bt8bUz2eWcyEIAQBw+enO+7fHGGPc/Uzq0hUOh5WSkqJQKCS/3+/2OAAA4Dx05/2be40BAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgCsVF1drSlTpqi6utrtUQC4iCAEwDqtra1aunSpmpqatHTpUrW2tro9EgCXEIQAWGfVqlU6evSoJOno0aNavXq1yxMBcAtBCIBVGhsb9cILL8Tse+GFF9TY2OjSRADcRBACYA1jjJYtWyZjTMz+aDTa5X4A8Y8gBMAaDQ0Nqq2t7bJWW1urhoaGizwRALcRhABYIysrS4mJiV3WEhMTlZWVdZEnAuA2ghAAa2zfvl2RSKTLWiQS0fbt2y/yRADcRhACYI2CggL5/f4uaykpKSooKLjIEwFwG0EIgDUSEhK0aNGiLmuLFy9WQgL/SQRsw189AKvceOONGjVqVMy+0aNH64YbbnBpIgBuIggBsM6///u/O5/+JCQk6PHHH3d5IgBuIQgBsE5qaqqmTZumhIQETZs2TampqW6PBMAlHsMKYmcUDoeVkpKiUCh0xhMsAQDApaU77998IgQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQBWqq6u1pQpU1RdXe32KABcRBACYJ3W1lYtXbpUTU1NWrp0qVpbW90eCYBLCEIArLNq1SodO3ZMknTs2DGtXr3a5YkAuIUgBMAqjY2NWr16tTrXkjXGaPXq1WpsbHR5MgBuIAgBsIYxRsuWLTvjfhbaB+xDEAJgjYaGBtXW1ioSicTsj0Qiqq2tVUNDg0uTAXALQQiANXJycjRu3DglJibG7E9MTFR+fr5ycnJcmgyAW7oVhMrLyzVu3DhdddVVGjBggCZNmqS9e/fG9Nxyyy3yeDwx26xZs2J6GhoaVFxcrN69e2vAgAGaP3++Tpw4EdPz9ttv64YbbpDP59OQIUNUUVFx2jzLly/XoEGDlJSUpIKCAm3fvj2m3traqpKSEl199dXq06eP7rrrLjU1NXXnJQOIIx6PR3PmzDnjfo/H48JUANzUrSC0ZcsWlZSUaOvWrdq4caM6Ojo0YcIEHT9+PKbvgQce0OHDh51tyZIlTi0Siai4uFjt7e2qrq7WypUrVVFRoUWLFjk9+/btU3FxsW699VbV19dr7ty5uv/++/Xmm286PWvXrlVpaakWL16sd999V2PGjFFRUZGOHDni9MybN0+vvvqq1q9fry1btujQoUO68847u/1LAhA/srKydPfdd8fsu/vuu3XNNde4NBEAV5keOHLkiJFktmzZ4uwbP368mTNnzhmf88Ybb5iEhAQTDAadfStWrDB+v9+0tbUZY4xZsGCBGTFiRMzzpkyZYoqKipzH+fn5pqSkxHkciURMZmamKS8vN8YY09zcbHr16mXWr1/v9OzevdtIMjU1Nef1+kKhkJFkQqHQefUDuDw899xzZvz48c72n//5n26PBOAC6s77d4/OEQqFQpKkfv36xexftWqV0tLSNHLkSJWVlenzzz93ajU1NRo1apTS09OdfUVFRQqHw9q1a5fTU1hYGHPMoqIi1dTUSJLa29tVV1cX05OQkKDCwkKnp66uTh0dHTE9w4YNU05OjtNzqra2NoXD4ZgNQHxpbGzUunXrYvatW7eOy+cBS33pIBSNRjV37lx94xvf0MiRI539U6dO1QsvvKDNmzerrKxM//M//6N//Md/dOrBYDAmBElyHgeDwbP2hMNhtbS06OjRo4pEIl32nHwMr9er1NTUM/acqry8XCkpKc6WnZ3djd8IgEud4fJ5AKe44ss+saSkRO+//75+97vfxeyfOXOm88+jRo3SwIEDddttt+kvf/mLrrvuui8/6UVQVlam0tJS53E4HCYMAXGk8/L5U518+Xxubq4LkwFwy5f6RGj27Nl67bXXtHnzZmVlZZ21t6CgQJL04YcfSpIyMjJOu3Kr83FGRsZZe/x+v5KTk5WWlqbExMQue04+Rnt7u5qbm8/Ycyqfzye/3x+zAYgfXD4P4FTdCkLGGM2ePVsvvviiNm3apGuvvfacz6mvr5ckDRw4UJIUCAS0c+fOmKu7Nm7cKL/fr+HDhzs9VVVVMcfZuHGjAoGAJMnr9SovLy+mJxqNqqqqyunJy8tTr169Ynr27t2rhoYGpweAXbh8HsBpunMW9oMPPmhSUlLM22+/bQ4fPuxsn3/+uTHGmA8//NA8/vjjZseOHWbfvn3m5ZdfNoMHDzY333yzc4wTJ06YkSNHmgkTJpj6+npTWVlp+vfvb8rKypyejz76yPTu3dvMnz/f7N692yxfvtwkJiaayspKp2fNmjXG5/OZiooK88EHH5iZM2ea1NTUmKvRZs2aZXJycsymTZvMjh07TCAQMIFA4LxfL1eNAfHp17/+tbnlllvM+PHjzS233GL++7//2+2RAFxA3Xn/7lYQktTl9vzzzxtjjGloaDA333yz6devn/H5fGbIkCFm/vz5pw2yf/9+M3HiRJOcnGzS0tLMI488Yjo6OmJ6Nm/ebMaOHWu8Xq8ZPHiw8zNO9vTTT5ucnBzj9XpNfn6+2bp1a0y9paXFPPTQQ6Zv376md+/eZvLkyebw4cPn/XoJQkB8amlpMXfddZcZP368+d73vmdaWlrcHgnABdSd92+PMVwmcSbhcFgpKSkKhUKcLwTEmerqai1btkxz5szR17/+dbfHAXABdef9+0tfNQYAl7Ovf/3rBCAA3HQVAADYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAK1VXV2vKlCmqrq52exQALiIIAbBOa2urli5dqqamJi1dulStra1ujwTAJQQhANZZtWqVjh07Jkk6duyYVq9e7fJEANxCEAJglcbGRq1evVqdt1k0xmj16tVqbGx0eTIAbiAIAbCGMUbLli07437uQQ3YhyAEwBoNDQ2qra1VJBKJ2R+JRFRbW6uGhgaXJgPgFoIQAGvk5ORo3LhxSkxMjNmfmJio/Px85eTkuDQZALcQhABYw+PxaM6cOWfc7/F4XJgKgJsIQgCskpWVpalTpzqhx+PxaOrUqbrmmmtcngyAGwhCAKwzbdo0XX311ZKktLQ0TZ061eWJALiFIATAOklJSSotLVV6errmzZunpKQkt0cC4BKCEAAAsBZBCIB1uMUGgE4EIQDW4RYbADoRhABYhVtsADgZQQiANbjFBoBTEYQAWINbbAA4FUEIgDW4xQaAUxGEAFiDW2wAOBVBCIBVuMUGgJMRhABYh1tsAOhEEAJgnaSkJE2cOFEJCQm6/fbbucUGYDGCEADrtLa26uWXX1Y0GtXLL7/MytKAxQhCAKyzcuVKhcNhSVI4HNZvfvMblycC4BaCEACrNDY2as2aNTH71qxZw8rSgKUIQgCsYYzRE088cdoK0tFotMv9AOIfQQiANQ4cOKCdO3d2Wdu5c6cOHDhwkScC4LZuBaHy8nKNGzdOV111lQYMGKBJkyZp7969MT2tra0qKSnR1VdfrT59+uiuu+5SU1NTTE9DQ4OKi4vVu3dvDRgwQPPnz9eJEydiet5++23dcMMN8vl8GjJkiCoqKk6bZ/ny5Ro0aJCSkpJUUFCg7du3d3sWAABgr24FoS1btqikpERbt27Vxo0b1dHRoQkTJuj48eNOz7x58/Tqq69q/fr12rJliw4dOqQ777zTqUciERUXF6u9vV3V1dVauXKlKioqtGjRIqdn3759Ki4u1q233qr6+nrNnTtX999/v958802nZ+3atSotLdXixYv17rvvasyYMSoqKtKRI0fOexYAdsnNzdWoUaO6rI0ePVq5ubkXeSIArjM9cOTIESPJbNmyxRhjTHNzs+nVq5dZv36907N7924jydTU1BhjjHnjjTdMQkKCCQaDTs+KFSuM3+83bW1txhhjFixYYEaMGBHzs6ZMmWKKioqcx/n5+aakpMR5HIlETGZmpikvLz/vWc4lFAoZSSYUCp1XP4BL38GDB8348eNjtltuucU0Nja6PRqAC6Q77989OkcoFApJkvr16ydJqqurU0dHhwoLC52eYcOGKScnRzU1NZKkmpoajRo1Sunp6U5PUVGRwuGwdu3a5fScfIzOns5jtLe3q66uLqYnISFBhYWFTs/5zHKqtrY2hcPhmA1AfMnKytKIESNi9o0YMYJbbACW+tJBKBqNau7cufrGN76hkSNHSpKCwaC8Xq9SU1NjetPT0xUMBp2ek0NQZ72zdraecDislpYWHT16VJFIpMuek49xrllOVV5erpSUFGfLzs4+z98GgMtFY2Oj9uzZE7Nvz549XD4PWOpLB6GSkhK9//77p63HcTkrKytTKBRytoMHD7o9EoALyBijZcuWdXn5fFf7AcS/LxWEZs+erddee02bN29WVlaWsz8jI0Pt7e1qbm6O6W9qalJGRobTc+qVW52Pz9Xj9/uVnJystLQ0JSYmdtlz8jHONcupfD6f/H5/zAYgfjQ0NKi2tlbRaDRmfzQaVW1trRoaGlyaDIBbuhWEjDGaPXu2XnzxRW3atEnXXnttTD0vL0+9evVSVVWVs2/v3r1qaGhQIBCQJAUCAe3cuTPm6q6NGzfK7/dr+PDhTs/Jx+js6TyG1+tVXl5eTE80GlVVVZXTcz6zALBLTk7OWa8ay8nJucgTAXDbFd1pLikp0erVq/Xyyy/rqquucs61SUlJUXJyslJSUjRjxgyVlpaqX79+8vv9evjhhxUIBHTTTTdJkiZMmKDhw4fr+9//vpYsWaJgMKif/OQnKikpkc/nkyTNmjVLzzzzjBYsWKAf/ehH2rRpk9atW6fXX3/dmaW0tFTTp0/XjTfeqPz8fD311FM6fvy47rvvPmemc80CAJ34WgywVHcuR5PU5fb88887PS0tLeahhx4yffv2Nb179zaTJ082hw8fjjnO/v37zcSJE01ycrJJS0szjzzyiOno6Ijp2bx5sxk7dqzxer1m8ODBMT+j09NPP21ycnKM1+s1+fn5ZuvWrTH185nlbLh8Hogv+/fvP+3S+ZO3/fv3uz0igAugO+/fHmP436AzCYfDSklJUSgU4nwhIA4YY7RgwQLV1dXFnCeUmJiovLw8PfHEE/J4PC5OCOBC6M77N/caA2ANj8ejOXPmnBZ2zrQfQPwjCAGwSlZWlu6+++6YfXfffTcLKgKWIggBAABrEYQAWKWxsVHr1q2L2bdu3TpWlgYsRRACYA3z/1eWPtN+rh0B7EMQAmCNzpWlI5FIzP5IJMLK0oClCEIArJGTk6Nx48YpMTExZn9iYqLy8/NZWRqwEEEIgDU6L5M/034unwfsQxACYBUunwdwMoIQAACwFkEIgFW4fB7AyQhCAKzB5fMATkUQAmANLp8HcCqCEABr5OTkaNSoUV3WRo8ezeXzgIUIQgCs0tbW1uX+1tbWizwJgEsBQQiANQ4cOKA//elPXdb+9Kc/6cCBAxd5IgBuIwgBsEY0Gu1RHUD8IQgBsMbhw4d7VAcQfwhCAKyRmZnZozqA+EMQAmCNQYMG6frrr++yNnToUA0aNOjiDgTAdQQhANbweDyaOXNml7WZM2dy01XAQgQhANYwxmjt2rVd1tasWcPK0oCFCEIArNG5snRXWFkasBNBCIA1cnJyNG7cuC5r+fn5rCwNWIggBMAaHo9HU6ZM6bI2ZcoUzhECLEQQAmANY4xWrlzZZa2iooJzhAALEYQAWOPAgQPauXNnl7WdO3dyiw3AQgQhAABgLYIQAGvk5ubq2muv7bI2ePBg5ebmXuSJALiNIATAGsYYNTU1dVkLBoOcIwRYiCAEwBpbt27V559/3mXt888/19atWy/yRADcRhACYI2BAwf2qA4g/hCEAFjjbDddHTZsGDddBSxEEAJgDY/Ho0WLFnVZ++lPf8qCioCFCEIAIHGiNGApghAAaxhj9MQTT3RZe+KJJwhDgIW6HYTeeecdffe731VmZqY8Ho9eeumlmPoPf/hDeTyemO3222+P6fnkk080bdo0+f1+paamasaMGfrss89iet577z1961vfUlJSkrKzs7VkyZLTZlm/fr2GDRumpKQkjRo1Sm+88UZM3RijRYsWaeDAgUpOTlZhYaH+/Oc/d/clA4gTrCwN4FTdDkLHjx/XmDFjtHz58jP23H777Tp8+LCz/fa3v42pT5s2Tbt27dLGjRv12muv6Z133tHMmTOdejgc1oQJE5Sbm6u6ujo9+eSTeuyxx/SrX/3K6amurta9996rGTNm6A9/+IMmTZqkSZMm6f3333d6lixZol/+8pd67rnntG3bNl155ZUqKipSa2trd182AACIQx7Tg8+CPR6PXnzxRU2aNMnZ98Mf/lDNzc2nfVLUaffu3Ro+fLhqa2t14403SpIqKyv1ne98R42NjcrMzNSKFSv0r//6rwoGg/J6vZKkhQsX6qWXXtKePXskfXGn6OPHj+u1115zjn3TTTdp7Nixeu6552SMUWZmph555BH9+Mc/liSFQiGlp6eroqJC99xzzzlfXzgcVkpKikKhkPx+/5f5FQG4hBhj9M///M9dfio0evRoLVu2jBOmgTjQnffvr+QcobffflsDBgzQ0KFD9eCDD+rYsWNOraamRqmpqU4IkqTCwkIlJCRo27ZtTs/NN9/shCBJKioq0t69e/W3v/3N6SksLIz5uUVFRaqpqZEk7du3T8FgMKYnJSVFBQUFTs+p2traFA6HYzYA8cPj8ejRRx89LeycaT+A+HfBg9Dtt9+u3/zmN6qqqtITTzyhLVu2aOLEiYpEIpK+WMZ+wIABMc+54oor1K9fPwWDQacnPT09pqfz8bl6Tq6f/Lyuek5VXl6ulJQUZ8vOzu726wdwacvKytKwYcNi9n3ta1/TNddc49JEANx0xYU+4MlfOY0aNUqjR4/Wddddp7ffflu33Xbbhf5xF1RZWZlKS0udx+FwmDAExJnGxkbt3r07Zt/u3bvV2NiorKwsl6YC4Jav/PL5wYMHKy0tTR9++KEkKSMjQ0eOHInpOXHihD755BNlZGQ4PafeGLHz8bl6Tq6f/Lyuek7l8/nk9/tjNgDx40yXz3fu5/J5wD5feRBqbGzUsWPHnHv4BAIBNTc3q66uzunZtGmTotGoCgoKnJ533nlHHR0dTs/GjRs1dOhQ9e3b1+mpqqqK+VkbN25UIBCQJF177bXKyMiI6QmHw9q2bZvTA8AuXD4P4FTdDkKfffaZ6uvrVV9fL+mLk5Lr6+vV0NCgzz77TPPnz9fWrVu1f/9+VVVV6Y477tCQIUNUVFQk6Yvv4m+//XY98MAD2r59u37/+99r9uzZuueee5SZmSlJmjp1qrxer2bMmKFdu3Zp7dq1WrZsWczXVnPmzFFlZaV+8YtfaM+ePXrssce0Y8cOzZ49W9IXJz/OnTtXP/vZz/TKK69o586d+sEPfqDMzMyYq9wAAIDFTDdt3rzZSDptmz59uvn888/NhAkTTP/+/U2vXr1Mbm6ueeCBB0wwGIw5xrFjx8y9995r+vTpY/x+v7nvvvvMp59+GtPzxz/+0Xzzm980Pp/PXHPNNebnP//5abOsW7fOXH/99cbr9ZoRI0aY119/PaYejUbNT3/6U5Oenm58Pp+57bbbzN69e8/7tYZCISPJhEKhbvyGAFyqIpGIKS4uNuPHjz9tKy4uNpFIxO0RAVwA3Xn/7tE6QvGOdYSA+HLgwAFNnz79jPWVK1cqNzf3Ik4E4Kvg+jpCAHApysnJ0bhx47qs5efnKycn5yJPBMBtBCEA1vB4PJozZ06XtTlz5rCgImAhghAAq5x8P8KTnelqMgDxjSAEwBqRSERLlizpsrZkyRJnBXwA9iAIAbDGK6+8omg02mUtGo3qlVdeucgTAXAbQQiANdLS0npUBxB/CEIArHGue4lxrzHAPgQhANbIzc1VUlJSl7Xk5GTWEAIsRBACYI2Ghga1trZ2WWtpaVFDQ8NFngiA2whCAKxxroX0WWgfsA9BCIA1CEIATkUQAmCNw4cP96gOIP4QhABYIzMzs0d1APGHIATAGoMGDdL111/fZW3o0KEaNGjQxR0IgOsIQgCs4fF4NHPmzC5rM2fO5KargIUIQgCsYYzR2rVru6ytWbOGk6UBCxGEAFijoaFBtbW1XdZqa2tZRwiwEEEIgDWys7Pl9/u7rPn9fmVnZ1/kiQC4jSAEwBoHDx5UOBzushYOh3Xw4MGLPBEAtxGEAFgjKytLCQld/2cvISGBm64CFiIIAbDG1q1bFY1Gu6xFo1Ft3br1Ik8EwG0EIQAAYC2CEABrBAIBeb3eLmter1eBQOAiTwTAbQQhAFY52zlCAOzDXz4Aa2zdulWtra1d1lpbWzlHCLAQQQiANTIyMnpUBxB/CEIArHGuW2hwiw3APgQhANb44x//2KM6gPhDEAJgjX79+vWoDiD+EIQAWOPPf/5zj+oA4g9BCIA1znTD1fOtA4g/BCEA1mhubu5RHUD8IQgBsMaZ7jN2vnUA8YcgBMAa51o9mtWlAfvwVw/AGn379u1RHUD8IQgBsMbf/va3HtUBxJ9uB6F33nlH3/3ud5WZmSmPx6OXXnoppm6M0aJFizRw4EAlJyersLDwtEtSP/nkE02bNk1+v1+pqamaMWOGPvvss5ie9957T9/61reUlJSk7OxsLVmy5LRZ1q9fr2HDhikpKUmjRo3SG2+80e1ZANiDc4QAnKrbQej48eMaM2aMli9f3mV9yZIl+uUvf6nnnntO27Zt05VXXqmioqKYGx1OmzZNu3bt0saNG/Xaa6/pnXfe0cyZM516OBzWhAkTlJubq7q6Oj355JN67LHH9Ktf/crpqa6u1r333qsZM2boD3/4gyZNmqRJkybp/fff79YsAOzh8Xh6VAcQh0wPSDIvvvii8zgajZqMjAzz5JNPOvuam5uNz+czv/3tb40xxnzwwQdGkqmtrXV6NmzYYDwej/nrX/9qjDHm2WefNX379jVtbW1Oz6OPPmqGDh3qPL777rtNcXFxzDwFBQXmn/7pn857lnMJhUJGkgmFQufVD+DStmTJEjN+/PgzbkuWLHF7RAAXQHfevy/oOUL79u1TMBhUYWGhsy8lJUUFBQWqqamRJNXU1Cg1NVU33nij01NYWKiEhARt27bN6bn55pvl9XqdnqKiIu3du9f5Dr+mpibm53T2dP6c85nlVG1tbQqHwzEbgPhxxx139KgOIP5c0CAUDAYlSenp6TH709PTnVowGNSAAQNi6ldccYX69esX09PVMU7+GWfqObl+rllOVV5erpSUFGfLzs4+j1cN4HJx+PDhHtUBxB+uGjtJWVmZQqGQsx08eNDtkQBcQJs3b+5RHUD8uaBBKCMjQ5LU1NQUs7+pqcmpZWRk6MiRIzH1EydO6JNPPonp6eoYJ/+MM/WcXD/XLKfy+Xzy+/0xG4D4MXXq1B7VAcSfCxqErr32WmVkZKiqqsrZFw6HtW3bNgUCAUlSIBBQc3Oz6urqnJ5NmzYpGo2qoKDA6XnnnXfU0dHh9GzcuFFDhw51FjwLBAIxP6ezp/PnnM8sAOxypq/Fz7cOIP50Owh99tlnqq+vV319vaQvTkqur69XQ0ODPB6P5s6dq5/97Gd65ZVXtHPnTv3gBz9QZmamJk2aJEn62te+pttvv10PPPCAtm/frt///veaPXu27rnnHmVmZkr64v/KvF6vZsyYoV27dmnt2rVatmyZSktLnTnmzJmjyspK/eIXv9CePXv02GOPaceOHZo9e7YkndcsAOzywQcf9KgOIP5c0d0n7NixQ7feeqvzuDOcTJ8+XRUVFVqwYIGOHz+umTNnqrm5Wd/85jdVWVmppKQk5zmrVq3S7NmzddtttykhIUF33XWXfvnLXzr1lJQUvfXWWyopKVFeXp7S0tK0aNGimLWGvv71r2v16tX6yU9+on/5l3/R3/3d3+mll17SyJEjnZ7zmQWAPc513h/nBQL28RhjjNtDXKrC4bBSUlIUCoU4XwiIA88++6zWrVt3xvrdd9+thx566CJOBOCr0J33b64aA2CNQ4cO9agOIP4QhABYo3fv3j2qA4g/BCEA1uBeYwBORRACYI2//vWvPaoDiD8EIQDWaGlp6VEdQPwhCAGwBkEIwKkIQgCsca4bKXOjZcA+BCEA1rjmmmt6VAcQfwhCAKxx6v0Ju1sHEH8IQgCsceWVV/aoDiD+EIQAWOPjjz/uUR1A/CEIAbBGQsLZ/5N3rjqA+MNfPQBrDB48uEd1APGHIATAGrt37+5RHUD8IQgBsEZiYmKP6gDiD0EIgDUikUiP6gDiD0EIgDVSUlJ6VAcQfwhCAKwRCoV6VAcQfwhCAADAWgQhANbweDw9qgOIPwQhANYwxvSoDiD+EIQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYK0r3B4AsIUxRq2trW6PgXNoaWlxewSrJSUlcfNbXFQEIeAiaW1t1cSJE90eA+fAvyN3bdiwQcnJyW6PAYvw1RgAALAWnwgBF0lSUpI2bNjg9hhWM8boO9/5zhnrb7zxBl/LuCwpKcntEWAbc4EtXrzYSIrZhg4d6tRbWlrMQw89ZPr162euvPJKc+edd5pgMBhzjAMHDpjvfOc7Jjk52fTv39/8+Mc/Nh0dHTE9mzdvNn//939vvF6vue6668zzzz9/2izPPPOMyc3NNT6fz+Tn55tt27Z167WEQiEjyYRCoW49D8Cl6+DBg2b8+PGnbY2NjW6PBuAC6c7791fy1diIESN0+PBhZ/vd737n1ObNm6dXX31V69ev15YtW3To0CHdeeedTj0Siai4uFjt7e2qrq7WypUrVVFRoUWLFjk9+/btU3FxsW699VbV19dr7ty5uv/++/Xmm286PWvXrlVpaakWL16sd999V2PGjFFRUZGOHDnyVbxkAJeJrKys085BufLKK3XNNde4NBEAV13oFLZ48WIzZsyYLmvNzc2mV69eZv369c6+3bt3G0mmpqbGGGPMG2+8YRISEmI+JVqxYoXx+/2mra3NGGPMggULzIgRI2KOPWXKFFNUVOQ8zs/PNyUlJc7jSCRiMjMzTXl5+Xm/Fj4RAuLTJ598EvNpUEtLi9sjAbiAXP9E6M9//rMyMzM1ePBgTZs2TQ0NDZKkuro6dXR0qLCw0OkdNmyYcnJyVFNTI0mqqanRqFGjlJ6e7vQUFRUpHA5r165dTs/Jx+js6TxGe3u76urqYnoSEhJUWFjo9HSlra1N4XA4ZgMQf04+D2Xx4sWclwJY7IIHoYKCAlVUVKiyslIrVqzQvn379K1vfUuffvqpgsGgvF6vUlNTY56Tnp6uYDAoSQoGgzEhqLPeWTtbTzgcVktLi44ePapIJNJlT+cxulJeXq6UlBRny87O/lK/AwCXj5tuusntEQC46IJfNXbyGhyjR49WQUGBcnNztW7dukt+bYiysjKVlpY6j8PhMGEIAIA49pWvI5Samqrrr79eH374oTIyMtTe3q7m5uaYnqamJmVkZEiSMjIy1NTUdFq9s3a2Hr/fr+TkZKWlpSkxMbHLns5jdMXn88nv98dsAAAgfn3lQeizzz7TX/7yFw0cOFB5eXnq1auXqqqqnPrevXvV0NCgQCAgSQoEAtq5c2fM1V0bN26U3+/X8OHDnZ6Tj9HZ03kMr9ervLy8mJ5oNKqqqiqnBwAA4IIHoR//+MfasmWL9u/fr+rqak2ePFmJiYm69957lZKSohkzZqi0tFSbN29WXV2d7rvvPgUCAed7+gkTJmj48OH6/ve/rz/+8Y9688039ZOf/EQlJSXy+XySpFmzZumjjz7SggULtGfPHj377LNat26d5s2b58xRWlqq//qv/9LKlSu1e/duPfjggzp+/Ljuu+++C/2SAQDAZeqCnyPU2Nioe++9V8eOHVP//v31zW9+U1u3blX//v0lSf/xH/+hhIQE3XXXXWpra1NRUZGeffZZ5/mJiYl67bXX9OCDDyoQCOjKK6/U9OnT9fjjjzs91157rV5//XXNmzdPy5YtU1ZWln7961+rqKjI6ZkyZYo+/vhjLVq0SMFgUGPHjlVlZeVpJ1ADAAB7eYwxxu0hLlXhcFgpKSkKhUKcLwTEkZaWFufCDm7yCcSf7rx/c9NVAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1LviCiri0GGPU2trq9hjAJeXkvwn+PoCuJSUlyePxuD3GV44gFOdaW1udheMAnG7y5MlujwBckmxZbJSvxgAAgLX4RMgin429VyaBf+WAjJGiJ77454QrJAs+/gfOhyd6Qn3qf+v2GBcV74oWMQlXSIm93B4DuER43R4AuOTYePNRvhoDAADWIggBAABrEYQAAIC1CEIAAMBanCwd54w56dS3SId7gwAALn0nvU/EvH/EMYJQnGtra3P++ao/rnFxEgDA5aStrU29e/d2e4yvHF+NAQAAa/GJUJzz+XzOP3865h7WEQIAnFmkw/n24OT3j3hGEIpzMTfMS+xFEAIAnBcbbrgq8dUYAACwGJ8IWcQTPWHl8unAabjXGNAlT+ffhUUIQhax7UZ6AACcC1+NAQAAa/GJUJxLSkrShg0b3B4DuKS0trZq8uTJkqQXX3xRSUlJLk8EXHps+bsgCMU5j8ej5ORkt8cALllJSUn8jQAW46sxAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1WEcIuEiMMWptbXV7DEgx/x74d3JpSUpKsuau57g0WBGEli9frieffFLBYFBjxozR008/rfz8fLfHgmVaW1s1ceJEt8fAKTpXmMalYcOGDSxwiYsq7r8aW7t2rUpLS7V48WK9++67GjNmjIqKinTkyBG3RwMAAC7zGGOM20N8lQoKCjRu3Dg988wzkqRoNKrs7Gw9/PDDWrhw4VmfGw6HlZKSolAoJL/ffzHGRRzjq7FLhzFGbW1tkiSfz8dXMZcQvhrDhdCd9++4/mqsvb1ddXV1Kisrc/YlJCSosLBQNTU1Lk4GG3Hft0tL79693R4BwCUgroPQ0aNHFYlElJ6eHrM/PT1de/bsOa2/ra3N+b9E6YtECQAA4lfcnyPUHeXl5UpJSXG27Oxst0cCAABfobgOQmlpaUpMTFRTU1PM/qamJmVkZJzWX1ZWplAo5GwHDx68WKMCAAAXxHUQ8nq9ysvLU1VVlbMvGo2qqqpKgUDgtH6fzye/3x+zAQCA+BXX5whJUmlpqaZPn64bb7xR+fn5euqpp3T8+HHdd999bo8GAABcFvdBaMqUKfr444+1aNEiBYNBjR07VpWVlaedQA0AAOwT9+sI9QTrCAEAcPnpzvt3XJ8jBAAAcDYEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1or7dYR6onNlAW6+CgDA5aPzfft8VggiCJ3Fp59+KkncfBUAgMvQp59+qpSUlLP2sKDiWUSjUR06dEhXXXWVPB6P2+MAuIDC4bCys7N18OBBFkwF4owxRp9++qkyMzOVkHD2s4AIQgCsxMrxACROlgYAABYjCAEAAGsRhABYyefzafHixfL5fG6PAsBFnCMEAACsxSdCAADAWgQhAABgLYIQAACwFkEIAABYiyAEwErLly/XoEGDlJSUpIKCAm3fvt3tkQC4gCAEwDpr165VaWmpFi9erHfffVdjxoxRUVGRjhw54vZoAC4yLp8HYJ2CggKNGzdOzzzzjKQv7iuYnZ2thx9+WAsXLnR5OgAXE58IAbBKe3u76urqVFhY6OxLSEhQYWGhampqXJwMgBsIQgCscvToUUUiEaWnp8fsT09PVzAYdGkqAG4hCAEAAGsRhABYJS0tTYmJiWpqaorZ39TUpIyMDJemAuAWghAAq3i9XuXl5amqqsrZF41GVVVVpUAg4OJkANxwhdsDAMDFVlpaqunTp+vGG29Ufn6+nnrqKR0/flz33Xef26MBuMgIQgCsM2XKFH388cdatGiRgsGgxo4dq8rKytNOoAYQ/1hHCAAAWItzhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACw1v8DmgYCm48LIEYAAAAASUVORK5CYII=\n"},"metadata":{}}],"source":["sns.boxplot(df_new['total_income']); #check outliers on total_income column"]},{"cell_type":"markdown","metadata":{"id":"c9_nZsxvLhzB"},"source":["- If there are too many outliers like above sample, you can use the median.\n","- If there are only a few outliers, you can use the mean."]},{"cell_type":"markdown","metadata":{"id":"ATQbLBGxLhzC"},"source":["Repeat the comparison for several things. Make sure you consider various aspects and explain your thought process."]},{"cell_type":"markdown","metadata":{"id":"FgFbxWlBLhzC"},"source":["Make a decision about the characteristics that has the most determine an income and whether to use the median or the average. Explain why you made this decision\n"]},{"cell_type":"code","execution_count":187,"metadata":{"trusted":true,"id":"4DtKWGrXLhzC","executionInfo":{"status":"ok","timestamp":1689650775206,"user_tz":-420,"elapsed":15,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# Write the function that can be used to fill in the missing values\n","def fill_missing_value (dataframe, agg_column, value_column):\n","    grouped_values = dataframe.groupby(agg_column)[value_column].median().reset_index()\n","    size = len(grouped_values)\n","    for i in range(size):\n","        group = grouped_values[agg_column][i]\n","        value = grouped_values[value_column][i]\n","        dataframe.loc[(dataframe[agg_column]==group) & (dataframe[value_column].isna()), value_column] = value\n","    return dataframe"]},{"cell_type":"code","execution_count":188,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"h8a4JsSJLhzC","executionInfo":{"status":"ok","timestamp":1689650775206,"user_tz":-420,"elapsed":15,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"66805499-185d-4cbe-a398-211233cced7e"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["children               0\n","days_employed       2110\n","dob_years              0\n","education              0\n","education_id           0\n","family_status          0\n","family_status_id       0\n","gender                 0\n","income_type            0\n","debt                   0\n","total_income        2110\n","purpose                0\n","years_employed      2110\n","age_category           0\n","dtype: int64"]},"metadata":{},"execution_count":188}],"source":["df_new_3.isnull().sum()"]},{"cell_type":"markdown","metadata":{"id":"XfPuZtT5LhzC"},"source":["- Check for null value in the recent dataframe"]},{"cell_type":"code","execution_count":189,"metadata":{"trusted":true,"id":"BxXIQ887LhzC","executionInfo":{"status":"ok","timestamp":1689650775206,"user_tz":-420,"elapsed":13,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# check values in the newly created column\n","df_new_4 = fill_missing_value (df_new_3, 'age_category', 'total_income')"]},{"cell_type":"code","execution_count":190,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"70x3kyiJLhzC","executionInfo":{"status":"ok","timestamp":1689650775207,"user_tz":-420,"elapsed":14,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"2c3d79ab-a79c-4a00-c165-91c207569fcc"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["children               0\n","days_employed       2110\n","dob_years              0\n","education              0\n","education_id           0\n","family_status          0\n","family_status_id       0\n","gender                 0\n","income_type            0\n","debt                   0\n","total_income           0\n","purpose                0\n","years_employed      2110\n","age_category           0\n","dtype: int64"]},"metadata":{},"execution_count":190}],"source":["# Apply the function to each row\n","df_new_4.isnull().sum()"]},{"cell_type":"markdown","metadata":{"id":"kdKMh28oLhzC"},"source":["- The total income is no longer has null value (already filled)"]},{"cell_type":"code","execution_count":191,"metadata":{"collapsed":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"3fe41kCaLhzC","executionInfo":{"status":"ok","timestamp":1689650775806,"user_tz":-420,"elapsed":611,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"85ee8d4c-e954-41bb-ab2d-76929cfa1380"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed dob_years            education  education_id  \\\n","12            0            NaN        65  secondary education             1   \n","26            0            NaN        41  secondary education             1   \n","29            0            NaN        63  secondary education             1   \n","41            0            NaN        50  secondary education             1   \n","55            0            NaN        54  secondary education             1   \n","...         ...            ...       ...                  ...           ...   \n","21333         2            NaN        47  secondary education             1   \n","21339         1            NaN        50  secondary education             1   \n","21341         0            NaN        48    bachelor's degree             0   \n","21346         1            NaN  42 Tahun  secondary education             1   \n","21354         2            NaN        28  secondary education             1   \n","\n","           family_status  family_status_id gender    income_type  debt  \\\n","12     civil partnership                 1      M        retiree     0   \n","26               married                 0      M  civil servant     0   \n","29             unmarried                 4      F        retiree     0   \n","41               married                 0      F  civil servant     0   \n","55     civil partnership                 1      F        retiree     1   \n","...                  ...               ...    ...            ...   ...   \n","21333            married                 0      M       business     0   \n","21339  civil partnership                 1      F       employee     0   \n","21341            married                 0      F       business     0   \n","21346            married                 0      F       employee     0   \n","21354            married                 0      F       employee     0   \n","\n","       total_income                   purpose  years_employed   age_category  \n","12        19761.425         to have a wedding             NaN            Old  \n","26        23857.908                 education             NaN         Senior  \n","29        19761.425    building a real estate             NaN            Old  \n","41        23857.908  second-hand car purchase             NaN         Senior  \n","55        23857.908         to have a wedding             NaN         Senior  \n","...             ...                       ...             ...            ...  \n","21333     23857.908         purchase of a car             NaN         Senior  \n","21339     23857.908          wedding ceremony             NaN         Senior  \n","21341     23857.908       building a property             NaN         Senior  \n","21346     25466.879    building a real estate             NaN  Not Specified  \n","21354     22955.474              car purchase             NaN         Mature  \n","\n","[2110 rows x 14 columns]"],"text/html":["\n","\n","  <div id=\"df-a4c83ff2-ad9d-403f-bcfe-014c5f9f287f\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","      <th>years_employed</th>\n","      <th>age_category</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>12</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>65</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>19761.425</td>\n","      <td>to have a wedding</td>\n","      <td>NaN</td>\n","      <td>Old</td>\n","    </tr>\n","    <tr>\n","      <th>26</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>41</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>education</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>29</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>63</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>unmarried</td>\n","      <td>4</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>19761.425</td>\n","      <td>building a real estate</td>\n","      <td>NaN</td>\n","      <td>Old</td>\n","    </tr>\n","    <tr>\n","      <th>41</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>second-hand car purchase</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>55</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>54</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>1</td>\n","      <td>23857.908</td>\n","      <td>to have a wedding</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>21333</th>\n","      <td>2</td>\n","      <td>NaN</td>\n","      <td>47</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>purchase of a car</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>21339</th>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>wedding ceremony</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>21341</th>\n","      <td>0</td>\n","      <td>NaN</td>\n","      <td>48</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>building a property</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>21346</th>\n","      <td>1</td>\n","      <td>NaN</td>\n","      <td>42 Tahun</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>25466.879</td>\n","      <td>building a real estate</td>\n","      <td>NaN</td>\n","      <td>Not Specified</td>\n","    </tr>\n","    <tr>\n","      <th>21354</th>\n","      <td>2</td>\n","      <td>NaN</td>\n","      <td>28</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>22955.474</td>\n","      <td>car purchase</td>\n","      <td>NaN</td>\n","      <td>Mature</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>2110 rows × 14 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a4c83ff2-ad9d-403f-bcfe-014c5f9f287f')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-18b57f4d-6ecf-45dd-963f-5110735e5172\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-18b57f4d-6ecf-45dd-963f-5110735e5172')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-18b57f4d-6ecf-45dd-963f-5110735e5172 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-a4c83ff2-ad9d-403f-bcfe-014c5f9f287f button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-a4c83ff2-ad9d-403f-bcfe-014c5f9f287f');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":191}],"source":["# check if we found an error\n","df_new_4[df_new_4['days_employed'].isnull()]"]},{"cell_type":"markdown","metadata":{"id":"R5IPsj71LhzD"},"source":["- All rows in `total_income` already filled with median value"]},{"cell_type":"markdown","metadata":{"id":"nh9u07aFLhzD"},"source":["Once we are done with `total income`, check if the sum of the values in this column matches with the sum of the values in other columns."]},{"cell_type":"code","execution_count":192,"metadata":{"collapsed":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"qKamf3ZlLhzD","executionInfo":{"status":"ok","timestamp":1689650775807,"user_tz":-420,"elapsed":14,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"fed509ab-cfa1-44ed-cf12-5ac9bc043fa7"},"outputs":[{"output_type":"stream","name":"stdout","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 21369 entries, 0 to 21368\n","Data columns (total 14 columns):\n"," #   Column            Non-Null Count  Dtype  \n","---  ------            --------------  -----  \n"," 0   children          21369 non-null  int64  \n"," 1   days_employed     19259 non-null  float64\n"," 2   dob_years         21369 non-null  object \n"," 3   education         21369 non-null  object \n"," 4   education_id      21369 non-null  int64  \n"," 5   family_status     21369 non-null  object \n"," 6   family_status_id  21369 non-null  int64  \n"," 7   gender            21369 non-null  object \n"," 8   income_type       21369 non-null  object \n"," 9   debt              21369 non-null  int64  \n"," 10  total_income      21369 non-null  float64\n"," 11  purpose           21369 non-null  object \n"," 12  years_employed    19259 non-null  float64\n"," 13  age_category      21369 non-null  object \n","dtypes: float64(3), int64(4), object(7)\n","memory usage: 2.3+ MB\n"]}],"source":["# check total entries in columns\n","df_new_4.info()"]},{"cell_type":"markdown","metadata":{"id":"iAsjp0AcLhzD"},"source":["- Yes, the total `income_column` values are the same as the other columns"]},{"cell_type":"markdown","metadata":{"id":"2iT0nC_FLhzD"},"source":["### Fixed the value in `days_employed`"]},{"cell_type":"markdown","metadata":{"id":"w2xxscJ9LhzD"},"source":["Think about parameters that can help us correct missing values in this column. Eventually, we'll find out whether we should use the mean or the median to replace the missing values. We may need to do the same research we did when correcting the data in the previous column."]},{"cell_type":"code","execution_count":193,"metadata":{"pycharm":{"name":"#%%\n"},"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"dk_GfcMhLhzD","executionInfo":{"status":"ok","timestamp":1689650775807,"user_tz":-420,"elapsed":12,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"23f72e11-a761-4a1a-a094-2e492d8f6f66"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["2197.320349646864"]},"metadata":{},"execution_count":193}],"source":["# The median distribution of `days_employed` based on the parameters we identified\n","df_new_4['days_employed'].median()"]},{"cell_type":"code","execution_count":194,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"XONvLSBCLhzD","executionInfo":{"status":"ok","timestamp":1689650775808,"user_tz":-420,"elapsed":10,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"271d58e3-5832-40bf-8177-3732530a1799"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["66905.78923678081"]},"metadata":{},"execution_count":194}],"source":["# The mean distribution of `days_employed` based on the parameters we identified\n","df_new_4['days_employed'].mean()"]},{"cell_type":"code","execution_count":195,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":430},"id":"ihosBmIWLhzD","executionInfo":{"status":"ok","timestamp":1689650775808,"user_tz":-420,"elapsed":9,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"8cf37ed9-2ee2-4d77-fe89-ef837263e8a0"},"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAkIAAAGdCAYAAAD+JxxnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAw0ElEQVR4nO3de3DU9b3/8VcSsptE3AQIJKQkXEoL5X4IJt1ptVByWDg5HVHOr6mlLaUoRaMjxAHNmRa0c2aCeE6r1YjtOMfQGRXMmdIebuHkBBKmZQEJpoSLmarYYGETb9lFzD2f3x9n8h0WImTlspLP8zHzneH7/bz2u++Nzuxrdve7G2OMMQIAALBQbLQHAAAAiBaKEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWoOiPcAXWU9Pj86cOaNbb71VMTEx0R4HAAD0gzFG586dU0ZGhmJjL/+aD0XoMs6cOaPMzMxojwEAAD6H06dPa9SoUZfNUIQu49Zbb5X0f39Ij8cT5WkAAEB/hEIhZWZmOs/jl0MRuozet8M8Hg9FCACAm0x/PtbCh6UBAIC1KEIAAMBaFCEAAGAtihAAALAWRQgAAFiLIgQAAKxFEQIAANaiCAEAAGtRhABYaf/+/SooKND+/fujPQqAKIoxxphoD/FFFQqFlJycrGAwyDdLAwNIW1ub5s+f7+xXVFQoISEhihMBuJYief7mFSEA1rmwBPW1D8AeV1WE1q9fr5iYGK1cudI51tbWpsLCQg0bNkyDBw/WokWL1NTUFHa7xsZG5efnKykpSSNGjNDq1avV1dUVlqmurtbMmTPldrs1fvx4lZWVXXL/paWlGjNmjBISEpSbm6tDhw6FrfdnFgB2ee+99yI6DmBg+9xF6PXXX9dvfvMbTZs2Lez4qlWrtG3bNpWXl6umpkZnzpzR3Xff7ax3d3crPz9fHR0d2r9/vzZt2qSysjKtXbvWyZw6dUr5+fmaM2eO6urqtHLlSt17773avXu3k9myZYuKioq0bt06HTlyRNOnT5fP51Nzc3O/ZwFgF2OMfvCDH/S59oMf/EB8UgCwkPkczp07Z77yla+YyspK861vfcs8/PDDxhhjWlpaTHx8vCkvL3eyJ0+eNJKM3+83xhizc+dOExsbawKBgJPZuHGj8Xg8pr293RhjzJo1a8zkyZPD7rOgoMD4fD5nPycnxxQWFjr73d3dJiMjw5SUlPR7lisJBoNGkgkGg/3KA/hi27x5s/nWt771mdvmzZujPSKAayCS5+/P9YpQYWGh8vPzlZeXF3a8trZWnZ2dYccnTpyorKws+f1+SZLf79fUqVOVlpbmZHw+n0KhkI4fP+5kLj63z+dzztHR0aHa2tqwTGxsrPLy8pxMf2a5WHt7u0KhUNgGYODYuHHjVa0DGHgGRXqDzZs368iRI3r99dcvWQsEAnK5XEpJSQk7npaWpkAg4GQuLEG9671rl8uEQiG1trbq448/Vnd3d5+ZN998s9+zXKykpERPPPHEZR49AAAYSCJ6Rej06dN6+OGH9fLLLw/IS02Li4sVDAad7fTp09EeCQAAXEcRFaHa2lo1Nzdr5syZGjRokAYNGqSamhr9+te/1qBBg5SWlqaOjg61tLSE3a6pqUnp6emSpPT09Euu3Ordv1LG4/EoMTFRqampiouL6zNz4TmuNMvF3G63PB5P2AYAAAauiIrQ3LlzVV9fr7q6OmebNWuWFi9e7Pw7Pj5eVVVVzm0aGhrU2Ngor9crSfJ6vaqvrw+7uquyslIej0eTJk1yMheeozfTew6Xy6Xs7OywTE9Pj6qqqpxMdnb2FWcBAAB2i+gzQrfeequmTJkSduyWW27RsGHDnOPLli1TUVGRhg4dKo/Ho4ceekher1df//rXJUnz5s3TpEmT9MMf/lAbNmxQIBDQz372MxUWFsrtdkuSVqxYoeeee05r1qzRT37yE+3Zs0evvfaaduzY4dxvUVGRlixZolmzZiknJ0dPP/20zp8/r6VLl0qSkpOTrzgLAACwW8Qflr6SX/3qV4qNjdWiRYvU3t4un8+n559/3lmPi4vT9u3bdf/998vr9eqWW27RkiVL9Itf/MLJjB07Vjt27NCqVav0zDPPaNSoUXrxxRfl8/mcTEFBgd5//32tXbtWgUBAM2bMUEVFRdgHqK80CwAAsBu/NXYZ/NYYMLDMnj37ipnq6urrPgeA64vfGgMAAOgHihAAALAWRQgAAFiLIgQAAKxFEQIAANaiCAEAAGtRhAAAgLUoQgAAwFoUIQAAYC2KEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrUYQAAIC1KEIAAMBaFCEAAGAtihAAALAWRQgAAFiLIgQAAKxFEQIAANaiCAEAAGtRhAAAgLUiKkIbN27UtGnT5PF45PF45PV6tWvXLmd99uzZiomJCdtWrFgRdo7Gxkbl5+crKSlJI0aM0OrVq9XV1RWWqa6u1syZM+V2uzV+/HiVlZVdMktpaanGjBmjhIQE5ebm6tChQ2HrbW1tKiws1LBhwzR48GAtWrRITU1NkTxcAAAwwEVUhEaNGqX169ertrZWhw8f1re//W3deeedOn78uJO57777dPbsWWfbsGGDs9bd3a38/Hx1dHRo//792rRpk8rKyrR27Vonc+rUKeXn52vOnDmqq6vTypUrde+992r37t1OZsuWLSoqKtK6det05MgRTZ8+XT6fT83NzU5m1apV2rZtm8rLy1VTU6MzZ87o7rvv/lx/JAAAMDDFGGPM1Zxg6NCheuqpp7Rs2TLNnj1bM2bM0NNPP91ndteuXfrnf/5nnTlzRmlpaZKkF154QY8++qjef/99uVwuPfroo9qxY4eOHTvm3O573/ueWlpaVFFRIUnKzc3Vbbfdpueee06S1NPTo8zMTD300EN67LHHFAwGNXz4cL3yyiv6l3/5F0nSm2++qa997Wvy+/36+te/3q/HFgqFlJycrGAwKI/H83n/RAC+IGbPnn3FTHV19XWfA8D1Fcnz9+f+jFB3d7c2b96s8+fPy+v1OsdffvllpaamasqUKSouLtann37qrPn9fk2dOtUpQZLk8/kUCoWcV5X8fr/y8vLC7svn88nv90uSOjo6VFtbG5aJjY1VXl6ek6mtrVVnZ2dYZuLEicrKynIyfWlvb1coFArbAADAwDUo0hvU19fL6/Wqra1NgwcP1tatWzVp0iRJ0ve//32NHj1aGRkZOnr0qB599FE1NDTo97//vSQpEAiElSBJzn4gELhsJhQKqbW1VR9//LG6u7v7zLz55pvOOVwul1JSUi7J9N5PX0pKSvTEE09E+BcBAAA3q4iL0IQJE1RXV6dgMKj/+q//0pIlS1RTU6NJkyZp+fLlTm7q1KkaOXKk5s6dq7fffltf/vKXr+ng10NxcbGKioqc/VAopMzMzChOBAAArqeI3xpzuVwaP368srOzVVJSounTp+uZZ57pM5ubmytJeuuttyRJ6enpl1y51bufnp5+2YzH41FiYqJSU1MVFxfXZ+bCc3R0dKilpeUzM31xu93OFXG9GwAAGLiu+nuEenp61N7e3udaXV2dJGnkyJGSJK/Xq/r6+rCruyorK+XxeJy317xer6qqqsLOU1lZ6XwOyeVyKTs7OyzT09OjqqoqJ5Odna34+PiwTENDgxobG8M+zwQAAOwW0VtjxcXFWrBggbKysnTu3Dm98sorqq6u1u7du/X222/rlVde0T/90z9p2LBhOnr0qFatWqU77rhD06ZNkyTNmzdPkyZN0g9/+ENt2LBBgUBAP/vZz1RYWCi32y1JWrFihZ577jmtWbNGP/nJT7Rnzx699tpr2rFjhzNHUVGRlixZolmzZiknJ0dPP/20zp8/r6VLl0qSkpOTtWzZMhUVFWno0KHyeDx66KGH5PV6+33FGAAAGPgiKkLNzc360Y9+pLNnzyo5OVnTpk3T7t279Y//+I86ffq0/vd//9cpJZmZmVq0aJF+9rOfObePi4vT9u3bdf/998vr9eqWW27RkiVL9Itf/MLJjB07Vjt27NCqVav0zDPPaNSoUXrxxRfl8/mcTEFBgd5//32tXbtWgUBAM2bMUEVFRdgHqH/1q18pNjZWixYtUnt7u3w+n55//vmr+VsBAIAB5qq/R2gg43uEgIGF7xEC7HBDvkcIAADgZkcRAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrUYQAWOM73/nOVa0DGHgoQgCssXLlyqtaBzDwUIQAWCMuLk6PPfZYn2v/+q//qri4uBs8EYBoowgBsMr8+fP7PD5v3rwbPAmALwKKEADrbNmy5bL7AOxBEQJgnbS0NM2ePVuSNHv2bKWlpUV3IABRE2OMMdEe4osqFAopOTlZwWBQHo8n2uMAAIB+iOT5m1eEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrUYQAAIC1KEIAAMBaFCEAAGAtihAAALBWREVo48aNmjZtmjwejzwej7xer3bt2uWst7W1qbCwUMOGDdPgwYO1aNEiNTU1hZ2jsbFR+fn5SkpK0ogRI7R69Wp1dXWFZaqrqzVz5ky53W6NHz9eZWVll8xSWlqqMWPGKCEhQbm5uTp06FDYen9mAQAAdouoCI0aNUrr169XbW2tDh8+rG9/+9u68847dfz4cUnSqlWrtG3bNpWXl6umpkZnzpzR3Xff7dy+u7tb+fn56ujo0P79+7Vp0yaVlZVp7dq1TubUqVPKz8/XnDlzVFdXp5UrV+ree+/V7t27ncyWLVtUVFSkdevW6ciRI5o+fbp8Pp+am5udzJVmAQAAkLlKQ4YMMS+++KJpaWkx8fHxpry83Fk7efKkkWT8fr8xxpidO3ea2NhYEwgEnMzGjRuNx+Mx7e3txhhj1qxZYyZPnhx2HwUFBcbn8zn7OTk5prCw0Nnv7u42GRkZpqSkxBhj+jVLfwSDQSPJBIPBft8GAABEVyTP35/7M0Ld3d3avHmzzp8/L6/Xq9raWnV2diovL8/JTJw4UVlZWfL7/ZIkv9+vqVOnKi0tzcn4fD6FQiHnVSW/3x92jt5M7zk6OjpUW1sblomNjVVeXp6T6c8sAAAAgyK9QX19vbxer9ra2jR48GBt3bpVkyZNUl1dnVwul1JSUsLyaWlpCgQCkqRAIBBWgnrXe9culwmFQmptbdXHH3+s7u7uPjNvvvmmc44rzdKX9vZ2tbe3O/uhUOgKfw0AAHAzi/gVoQkTJqiurk4HDx7U/fffryVLlujEiRPXY7YbrqSkRMnJyc6WmZkZ7ZEAAMB1FHERcrlcGj9+vLKzs1VSUqLp06frmWeeUXp6ujo6OtTS0hKWb2pqUnp6uiQpPT39kiu3evevlPF4PEpMTFRqaqri4uL6zFx4jivN0pfi4mIFg0FnO336dP/+KAAA4KZ01d8j1NPTo/b2dmVnZys+Pl5VVVXOWkNDgxobG+X1eiVJXq9X9fX1YVd3VVZWyuPxaNKkSU7mwnP0ZnrP4XK5lJ2dHZbp6elRVVWVk+nPLH1xu93OVwP0bgAAYACL5FPYjz32mKmpqTGnTp0yR48eNY899piJiYkx//M//2OMMWbFihUmKyvL7Nmzxxw+fNh4vV7j9Xqd23d1dZkpU6aYefPmmbq6OlNRUWGGDx9uiouLncw777xjkpKSzOrVq83JkydNaWmpiYuLMxUVFU5m8+bNxu12m7KyMnPixAmzfPlyk5KSEnY12pVm6Q+uGgMA4OYTyfN3REXoJz/5iRk9erRxuVxm+PDhZu7cuU4JMsaY1tZW88ADD5ghQ4aYpKQkc9ddd5mzZ8+GnePdd981CxYsMImJiSY1NdU88sgjprOzMyyzd+9eM2PGDONyucy4cePMSy+9dMkszz77rMnKyjIul8vk5OSYAwcOhK33Z5YroQgBAHDzieT5O8YYY6L7mtQXVygUUnJysoLBIG+TAQBwk4jk+ZvfGgMAANaiCAEAAGtRhAAAgLUoQgAAwFoUIQAAYC2KEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrUYQAAIC1KEIAAMBaFCEAAGAtihAAALAWRQgAAFiLIgQAAKxFEQIAANaiCAEAAGtRhAAAgLUoQgAAwFoUIQAAYC2KEAAAsBZFCAAAWCuiIlRSUqLbbrtNt956q0aMGKGFCxeqoaEhLDN79mzFxMSEbStWrAjLNDY2Kj8/X0lJSRoxYoRWr16trq6usEx1dbVmzpwpt9ut8ePHq6ys7JJ5SktLNWbMGCUkJCg3N1eHDh0KW29ra1NhYaGGDRumwYMHa9GiRWpqaorkIQMAgAEsoiJUU1OjwsJCHThwQJWVlers7NS8efN0/vz5sNx9992ns2fPOtuGDRucte7ubuXn56ujo0P79+/Xpk2bVFZWprVr1zqZU6dOKT8/X3PmzFFdXZ1Wrlype++9V7t373YyW7ZsUVFRkdatW6cjR45o+vTp8vl8am5udjKrVq3Stm3bVF5erpqaGp05c0Z33313xH8kAAAwQJmr0NzcbCSZmpoa59i3vvUt8/DDD3/mbXbu3GliY2NNIBBwjm3cuNF4PB7T3t5ujDFmzZo1ZvLkyWG3KygoMD6fz9nPyckxhYWFzn53d7fJyMgwJSUlxhhjWlpaTHx8vCkvL3cyJ0+eNJKM3+/v1+MLBoNGkgkGg/3KAwCA6Ivk+fuqPiMUDAYlSUOHDg07/vLLLys1NVVTpkxRcXGxPv30U2fN7/dr6tSpSktLc475fD6FQiEdP37cyeTl5YWd0+fzye/3S5I6OjpUW1sblomNjVVeXp6Tqa2tVWdnZ1hm4sSJysrKcjIXa29vVygUCtsAAMDANejz3rCnp0crV67UN77xDU2ZMsU5/v3vf1+jR49WRkaGjh49qkcffVQNDQ36/e9/L0kKBAJhJUiSsx8IBC6bCYVCam1t1ccff6zu7u4+M2+++aZzDpfLpZSUlEsyvfdzsZKSEj3xxBMR/iUAAMDN6nMXocLCQh07dkx/+tOfwo4vX77c+ffUqVM1cuRIzZ07V2+//ba+/OUvf/5Jb4Di4mIVFRU5+6FQSJmZmVGcCAAAXE+f662xBx98UNu3b9fevXs1atSoy2Zzc3MlSW+99ZYkKT09/ZIrt3r309PTL5vxeDxKTExUamqq4uLi+sxceI6Ojg61tLR8ZuZibrdbHo8nbAMAAANXREXIGKMHH3xQW7du1Z49ezR27Ngr3qaurk6SNHLkSEmS1+tVfX192NVdlZWV8ng8mjRpkpOpqqoKO09lZaW8Xq8kyeVyKTs7OyzT09OjqqoqJ5Odna34+PiwTENDgxobG50MAACwXCSfwr7//vtNcnKyqa6uNmfPnnW2Tz/91BhjzFtvvWV+8YtfmMOHD5tTp06ZP/7xj2bcuHHmjjvucM7R1dVlpkyZYubNm2fq6upMRUWFGT58uCkuLnYy77zzjklKSjKrV682J0+eNKWlpSYuLs5UVFQ4mc2bNxu3223KysrMiRMnzPLly01KSkrY1WgrVqwwWVlZZs+ePebw4cPG6/Uar9fb78fLVWMAANx8Inn+jqgISepze+mll4wxxjQ2Npo77rjDDB061LjdbjN+/HizevXqSwZ59913zYIFC0xiYqJJTU01jzzyiOns7AzL7N2718yYMcO4XC4zbtw45z4u9Oyzz5qsrCzjcrlMTk6OOXDgQNh6a2ureeCBB8yQIUNMUlKSueuuu8zZs2f7/XgpQgAA3Hwief6OMcaYaL0a9UUXCoWUnJysYDDI54UAALhJRPL8zW+NAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrUYQAAIC1KEIAAMBaFCEAAGAtihAAALAWRQgAAFiLIgQAAKxFEQIAANaiCAEAAGtRhAAAgLUoQgAAwFoUIQAAYC2KEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsFVERKikp0W233aZbb71VI0aM0MKFC9XQ0BCWaWtrU2FhoYYNG6bBgwdr0aJFampqCss0NjYqPz9fSUlJGjFihFavXq2urq6wTHV1tWbOnCm3263x48errKzsknlKS0s1ZswYJSQkKDc3V4cOHYp4FgAAYK+IilBNTY0KCwt14MABVVZWqrOzU/PmzdP58+edzKpVq7Rt2zaVl5erpqZGZ86c0d133+2sd3d3Kz8/Xx0dHdq/f782bdqksrIyrV271smcOnVK+fn5mjNnjurq6rRy5Urde++92r17t5PZsmWLioqKtG7dOh05ckTTp0+Xz+dTc3Nzv2cBAACWM1ehubnZSDI1NTXGGGNaWlpMfHy8KS8vdzInT540kozf7zfGGLNz504TGxtrAoGAk9m4caPxeDymvb3dGGPMmjVrzOTJk8Puq6CgwPh8Pmc/JyfHFBYWOvvd3d0mIyPDlJSU9HuWKwkGg0aSCQaD/coDAIDoi+T5+6o+IxQMBiVJQ4cOlSTV1taqs7NTeXl5TmbixInKysqS3++XJPn9fk2dOlVpaWlOxufzKRQK6fjx407mwnP0ZnrP0dHRodra2rBMbGys8vLynEx/ZrlYe3u7QqFQ2AYAAAauz12Eenp6tHLlSn3jG9/QlClTJEmBQEAul0spKSlh2bS0NAUCASdzYQnqXe9du1wmFAqptbVVH3zwgbq7u/vMXHiOK81ysZKSEiUnJztbZmZmP/8aAADgZvS5i1BhYaGOHTumzZs3X8t5oqq4uFjBYNDZTp8+He2RAADAdTTo89zowQcf1Pbt27Vv3z6NGjXKOZ6enq6Ojg61tLSEvRLT1NSk9PR0J3Px1V29V3JdmLn46q6mpiZ5PB4lJiYqLi5OcXFxfWYuPMeVZrmY2+2W2+2O4C8BAABuZhG9ImSM0YMPPqitW7dqz549Gjt2bNh6dna24uPjVVVV5RxraGhQY2OjvF6vJMnr9aq+vj7s6q7Kykp5PB5NmjTJyVx4jt5M7zlcLpeys7PDMj09PaqqqnIy/ZkFAABYLpJPYd9///0mOTnZVFdXm7Nnzzrbp59+6mRWrFhhsrKyzJ49e8zhw4eN1+s1Xq/XWe/q6jJTpkwx8+bNM3V1daaiosIMHz7cFBcXO5l33nnHJCUlmdWrV5uTJ0+a0tJSExcXZyoqKpzM5s2bjdvtNmVlZebEiRNm+fLlJiUlJexqtCvNciVcNQYAwM0nkufviIqQpD63l156ycm0traaBx54wAwZMsQkJSWZu+66y5w9ezbsPO+++65ZsGCBSUxMNKmpqeaRRx4xnZ2dYZm9e/eaGTNmGJfLZcaNGxd2H72effZZk5WVZVwul8nJyTEHDhwIW+/PLJdDEQIA4OYTyfN3jDHGROvVqC+6UCik5ORkBYNBeTyeaI8DAAD6IZLnb35rDAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrUYQAAIC1KEIAAMBaFCEAAGAtihAAALAWRQgAAFiLIgQAAKxFEQIAANaiCAEAAGtRhAAAgLUoQgAAwFoUIQAAYC2KEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgrYiL0L59+/Sd73xHGRkZiomJ0R/+8Iew9R//+MeKiYkJ2+bPnx+W+eijj7R48WJ5PB6lpKRo2bJl+uSTT8IyR48e1e23366EhARlZmZqw4YNl8xSXl6uiRMnKiEhQVOnTtXOnTvD1o0xWrt2rUaOHKnExETl5eXpr3/9a6QPGQAADFARF6Hz589r+vTpKi0t/czM/PnzdfbsWWd79dVXw9YXL16s48ePq7KyUtu3b9e+ffu0fPlyZz0UCmnevHkaPXq0amtr9dRTT+nxxx/Xb3/7Wyezf/9+3XPPPVq2bJneeOMNLVy4UAsXLtSxY8eczIYNG/TrX/9aL7zwgg4ePKhbbrlFPp9PbW1tkT5sAAAwEJmrIMls3bo17NiSJUvMnXfe+Zm3OXHihJFkXn/9defYrl27TExMjPn73/9ujDHm+eefN0OGDDHt7e1O5tFHHzUTJkxw9r/73e+a/Pz8sHPn5uaan/70p8YYY3p6ekx6erp56qmnnPWWlhbjdrvNq6++2q/HFwwGjSQTDAb7lQcAANEXyfP3dfmMUHV1tUaMGKEJEybo/vvv14cffuis+f1+paSkaNasWc6xvLw8xcbG6uDBg07mjjvukMvlcjI+n08NDQ36+OOPnUxeXl7Y/fp8Pvn9fknSqVOnFAgEwjLJycnKzc11Mhdrb29XKBQK2wAAwMB1zYvQ/Pnz9bvf/U5VVVV68sknVVNTowULFqi7u1uSFAgENGLEiLDbDBo0SEOHDlUgEHAyaWlpYZne/StlLly/8HZ9ZS5WUlKi5ORkZ8vMzIz48QMAgJvHoGt9wu9973vOv6dOnapp06bpy1/+sqqrqzV37txrfXfXVHFxsYqKipz9UChEGQIAYAC77pfPjxs3TqmpqXrrrbckSenp6Wpubg7LdHV16aOPPlJ6erqTaWpqCsv07l8pc+H6hbfrK3Mxt9stj8cTtgEAgIHruheh9957Tx9++KFGjhwpSfJ6vWppaVFtba2T2bNnj3p6epSbm+tk9u3bp87OTidTWVmpCRMmaMiQIU6mqqoq7L4qKyvl9XolSWPHjlV6enpYJhQK6eDBg04GAADYLeIi9Mknn6iurk51dXWS/u9DyXV1dWpsbNQnn3yi1atX68CBA3r33XdVVVWlO++8U+PHj5fP55Mkfe1rX9P8+fN133336dChQ/rzn/+sBx98UN/73veUkZEhSfr+978vl8ulZcuW6fjx49qyZYueeeaZsLetHn74YVVUVOg//uM/9Oabb+rxxx/X4cOH9eCDD0qSYmJitHLlSv3bv/2b/vu//1v19fX60Y9+pIyMDC1cuPAq/2wAAGBAiPSStL179xpJl2xLliwxn376qZk3b54ZPny4iY+PN6NHjzb33XefCQQCYef48MMPzT333GMGDx5sPB6PWbp0qTl37lxY5i9/+Yv55je/adxut/nSl75k1q9ff8ksr732mvnqV79qXC6XmTx5stmxY0fYek9Pj/n5z39u0tLSjNvtNnPnzjUNDQ39fqxcPg8AwM0nkufvGGOMiWIP+0ILhUJKTk5WMBjk80IAANwkInn+5rfGAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrUYQAAIC1KEIAAMBaFCEAAGAtihAAALAWRQgAAFiLIgQAAKxFEQIAANaiCAEAAGtRhAAAgLUoQgAAwFoUIQAAYC2KEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWirgI7du3T9/5zneUkZGhmJgY/eEPfwhbN8Zo7dq1GjlypBITE5WXl6e//vWvYZmPPvpIixcvlsfjUUpKipYtW6ZPPvkkLHP06FHdfvvtSkhIUGZmpjZs2HDJLOXl5Zo4caISEhI0depU7dy5M+JZAACAvSIuQufPn9f06dNVWlra5/qGDRv061//Wi+88IIOHjyoW265RT6fT21tbU5m8eLFOn78uCorK7V9+3bt27dPy5cvd9ZDoZDmzZun0aNHq7a2Vk899ZQef/xx/fa3v3Uy+/fv1z333KNly5bpjTfe0MKFC7Vw4UIdO3YsolkAAIDFzFWQZLZu3ers9/T0mPT0dPPUU085x1paWozb7TavvvqqMcaYEydOGEnm9ddfdzK7du0yMTEx5u9//7sxxpjnn3/eDBkyxLS3tzuZRx991EyYMMHZ/+53v2vy8/PD5snNzTU//elP+z3LlQSDQSPJBIPBfuUBAED0RfL8fU0/I3Tq1CkFAgHl5eU5x5KTk5Wbmyu/3y9J8vv9SklJ0axZs5xMXl6eYmNjdfDgQSdzxx13yOVyORmfz6eGhgZ9/PHHTubC++nN9N5Pf2a5WHt7u0KhUNgGAAAGrmtahAKBgCQpLS0t7HhaWpqzFggENGLEiLD1QYMGaejQoWGZvs5x4X18VubC9SvNcrGSkhIlJyc7W2ZmZj8eNQAAuFlx1dgFiouLFQwGne306dPRHgkAAFxH17QIpaenS5KamprCjjc1NTlr6enpam5uDlvv6urSRx99FJbp6xwX3sdnZS5cv9IsF3O73fJ4PGEbAAAYuK5pERo7dqzS09NVVVXlHAuFQjp48KC8Xq8kyev1qqWlRbW1tU5mz5496unpUW5urpPZt2+fOjs7nUxlZaUmTJigIUOGOJkL76c303s//ZkFAABYLtJPYp87d8688cYb5o033jCSzC9/+UvzxhtvmL/97W/GGGPWr19vUlJSzB//+Edz9OhRc+edd5qxY8ea1tZW5xzz5883//AP/2AOHjxo/vSnP5mvfOUr5p577nHWW1paTFpamvnhD39ojh07ZjZv3mySkpLMb37zGyfz5z//2QwaNMj8+7//uzl58qRZt26diY+PN/X19U6mP7NcDleNAQBw84nk+TviIrR3714j6ZJtyZIlxpj/u2z95z//uUlLSzNut9vMnTvXNDQ0hJ3jww8/NPfcc48ZPHiw8Xg8ZunSpebcuXNhmb/85S/mm9/8pnG73eZLX/qSWb9+/SWzvPbaa+arX/2qcblcZvLkyWbHjh1h6/2Z5XIoQgAA3Hwief6OMcaYaL0a9UUXCoWUnJysYDDI54UAALhJRPL8zVVjAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrUYQAAIC1KEIAAMBaFCEAAGAtihAAALAWRQgAAFiLIgQAAKxFEQIAANaiCAEAAGtRhAAAgLUoQgAAwFoUIQAAYC2KEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBAABrXfMi9PjjjysmJiZsmzhxorPe1tamwsJCDRs2TIMHD9aiRYvU1NQUdo7Gxkbl5+crKSlJI0aM0OrVq9XV1RWWqa6u1syZM+V2uzV+/HiVlZVdMktpaanGjBmjhIQE5ebm6tChQ9f64QIAgJvYdXlFaPLkyTp79qyz/elPf3LWVq1apW3btqm8vFw1NTU6c+aM7r77bme9u7tb+fn56ujo0P79+7Vp0yaVlZVp7dq1TubUqVPKz8/XnDlzVFdXp5UrV+ree+/V7t27ncyWLVtUVFSkdevW6ciRI5o+fbp8Pp+am5uvx0MGAAA3I3ONrVu3zkyfPr3PtZaWFhMfH2/Ky8udYydPnjSSjN/vN8YYs3PnThMbG2sCgYCT2bhxo/F4PKa9vd0YY8yaNWvM5MmTw85dUFBgfD6fs5+Tk2MKCwud/e7ubpORkWFKSkr6/ViCwaCRZILBYL9vAwAAoiuS5+/r8orQX//6V2VkZGjcuHFavHixGhsbJUm1tbXq7OxUXl6ek504caKysrLk9/slSX6/X1OnTlVaWpqT8fl8CoVCOn78uJO58By9md5zdHR0qLa2NiwTGxurvLw8J9OX9vZ2hUKhsA0AAAxc17wI5ebmqqysTBUVFdq4caNOnTql22+/XefOnVMgEJDL5VJKSkrYbdLS0hQIBCRJgUAgrAT1rveuXS4TCoXU2tqqDz74QN3d3X1mes/Rl5KSEiUnJztbZmbm5/obAACAm8Oga33CBQsWOP+eNm2acnNzNXr0aL322mtKTEy81nd3TRUXF6uoqMjZD4VClCEAAAaw6375fEpKir761a/qrbfeUnp6ujo6OtTS0hKWaWpqUnp6uiQpPT39kqvIevevlPF4PEpMTFRqaqri4uL6zPSeoy9ut1sejydsAwAAA9d1L0KffPKJ3n77bY0cOVLZ2dmKj49XVVWVs97Q0KDGxkZ5vV5JktfrVX19fdjVXZWVlfJ4PJo0aZKTufAcvZnec7hcLmVnZ4dlenp6VFVV5WQAAACu+VVjjzzyiKmurjanTp0yf/7zn01eXp5JTU01zc3NxhhjVqxYYbKyssyePXvM4cOHjdfrNV6v17l9V1eXmTJlipk3b56pq6szFRUVZvjw4aa4uNjJvPPOOyYpKcmsXr3anDx50pSWlpq4uDhTUVHhZDZv3mzcbrcpKyszJ06cMMuXLzcpKSlhV6NdCVeNAQBw84nk+fuaF6GCggIzcuRI43K5zJe+9CVTUFBg3nrrLWe9tbXVPPDAA2bIkCEmKSnJ3HXXXebs2bNh53j33XfNggULTGJioklNTTWPPPKI6ezsDMvs3bvXzJgxw7hcLjNu3Djz0ksvXTLLs88+a7KysozL5TI5OTnmwIEDET0WihAAADefSJ6/Y4wxJrqvSX1xhUIhJScnKxgM8nkhAABuEpE8f/NbYwAAwFoUIQAAYC2KEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADWoggBsNL+/ftVUFCg/fv3R3sUAFFEEQJgnba2Nq1fv15NTU1av3692traoj0SgCihCAGwzqZNmxQKhSRJoVBIv/vd76I8EYBooQgBsMp7772nV199NezYK6+8ovfeey9KEwGIJooQAGsYY/Tkk0/2ufbkk0/KGHODJwIQbRQhANb429/+pvr6+j7X6uvr9be//e0GTwQg2ihCAKzR3t5+VesABh6KEABrPPvss1e1DmDgGRTtAQBbGGO4TDvKGhoarrje2tp6g6ZBXxISEhQTExPtMWARihBwg7S1tWnBggXRHgOX0dnZyX+jKNu1a5cSExOjPQYswltjAADAWrwiBNwgCQkJ2rVrV7THsFpTU5N+/OMff+Z6WVmZ0tLSbtxAuERCQkK0R4BlKELADRITE8NL/lE2ZswYxcXFqbu7+5K1uLg4jRkz5sYPBSCqeGsMgFWqqqoiOg5gYOMVoQGOK5WAS91xxx3at2+fsz9nzhyuFgMuYssVfDGG75T/TKFQSMnJyQoGg/J4PNEe53NpbW3lKhgAQMRu5iv4Inn+tuKtsdLSUo0ZM0YJCQnKzc3VoUOHoj0SAAD4Ahjwb41t2bJFRUVFeuGFF5Sbm6unn35aPp9PDQ0NGjFiRLTHu+4ufMHvk6n/TyY2LorTAF8QRlJP1//9O3aQNPBf/Qf6JaanW4PryyXJmh8hHvBF6Je//KXuu+8+LV26VJL0wgsvaMeOHfrP//xPPfbYY1Ge7vq78LeTev/nBgDgStrb25WUlBTtMa67Af3WWEdHh2pra5WXl+cci42NVV5envx+/yX59vZ2hUKhsA0AAAxcA7oIffDBB+ru7r7kC9LS0tIUCAQuyZeUlCg5OdnZMjMzb9So143b7Y72CACAm5Atzx8D/q2xSBQXF6uoqMjZD4VCN30ZSkxM5NuMvyCMMWFvVQK4lNvttuKS7ZuBLd/yPaCLUGpqquLi4tTU1BR2vKmpSenp6Zfk3W73gGvAfJvxF4sN77cDwM1kQL815nK5lJ2dHfaNsT09PaqqqpLX643iZAAA4ItgQL8iJElFRUVasmSJZs2apZycHD399NM6f/68cxUZAACw14AvQgUFBXr//fe1du1aBQIBzZgxQxUVFfzCNAAA4Cc2Lmcg/MQGAAC24Sc2AAAA+oEiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYa8D/xMbV6P3S7VAoFOVJAABAf/U+b/fnxzMoQpdx7tw5SVJmZmaUJwEAAJE6d+6ckpOTL5vht8Yuo6enR2fOnNGtt96qmJiYaI8D4BoKhULKzMzU6dOn+S1BYIAxxujcuXPKyMhQbOzlPwVEEQJgJX5UGYDEh6UBAIDFKEIAAMBaFCEAVnK73Vq3bp3cbne0RwEQRXxGCAAAWItXhAAAgLUoQgAAwFoUIQAAYC2KEAAAsBZFCICVSktLNWbMGCUkJCg3N1eHDh2K9kgAooAiBMA6W7ZsUVFRkdatW6cjR45o+vTp8vl8am5ujvZoAG4wLp8HYJ3c3Fzddttteu655yT93+8KZmZm6qGHHtJjjz0W5ekA3Ei8IgTAKh0dHaqtrVVeXp5zLDY2Vnl5efL7/VGcDEA0UIQAWOWDDz5Qd3e30tLSwo6npaUpEAhEaSoA0UIRAgAA1qIIAbBKamqq4uLi1NTUFHa8qalJ6enpUZoKQLRQhABYxeVyKTs7W1VVVc6xnp4eVVVVyev1RnEyANEwKNoDAMCNVlRUpCVLlmjWrFnKycnR008/rfPnz2vp0qXRHg3ADUYRAmCdgoICvf/++1q7dq0CgYBmzJihioqKSz5ADWDg43uEAACAtfiMEAAAsBZFCAAAWIsiBAAArEURAgAA1qIIAQAAa1GEAACAtShCAADAWhQhAABgLYoQAACwFkUIAABYiyIEAACsRRECAADW+v/PAiP99AyuOwAAAABJRU5ErkJggg==\n"},"metadata":{}}],"source":["sns.boxplot(df_new['days_employed']); #check outliers in the days_employed column"]},{"cell_type":"markdown","metadata":{"id":"UPbXJNcMLhzE"},"source":["[Tentukan apa yang akan Anda gunakan: rata-rata atau median. Jelaskan mengapa.]"]},{"cell_type":"markdown","metadata":{"id":"ex6nxaPtLhzE"},"source":["- I used median values, because there are too many outliers"]},{"cell_type":"code","execution_count":196,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"8qeSodi2LhzE","executionInfo":{"status":"ok","timestamp":1689650776297,"user_tz":-420,"elapsed":497,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"52e343c0-bd7c-481c-c125-a6263db87308"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["children               0\n","days_employed          0\n","dob_years              0\n","education              0\n","education_id           0\n","family_status          0\n","family_status_id       0\n","gender                 0\n","income_type            0\n","debt                   0\n","total_income           0\n","purpose                0\n","years_employed      2110\n","age_category           0\n","dtype: int64"]},"metadata":{},"execution_count":196}],"source":["# write a function that calculates the mean or median based on the parameters we identified\n","df_new_5 = fill_missing_value (df_new_4, 'age_category', 'days_employed')\n","df_new_5.isnull().sum()"]},{"cell_type":"code","execution_count":197,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":424},"id":"pyPqPF1GLhzE","executionInfo":{"status":"ok","timestamp":1689650776298,"user_tz":-420,"elapsed":30,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"7def4292-5ef3-4dc1-e0da-0d5acd9e1f7f"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["       children  days_employed dob_years            education  education_id  \\\n","12            0  355229.618218        65  secondary education             1   \n","26            0    2288.041199        41  secondary education             1   \n","29            0  355229.618218        63  secondary education             1   \n","41            0    2288.041199        50  secondary education             1   \n","55            0    2288.041199        54  secondary education             1   \n","...         ...            ...       ...                  ...           ...   \n","21333         2    2288.041199        47  secondary education             1   \n","21339         1    2288.041199        50  secondary education             1   \n","21341         0    2288.041199        48    bachelor's degree             0   \n","21346         1    2253.981630  42 Tahun  secondary education             1   \n","21354         2    1045.752744        28  secondary education             1   \n","\n","           family_status  family_status_id gender    income_type  debt  \\\n","12     civil partnership                 1      M        retiree     0   \n","26               married                 0      M  civil servant     0   \n","29             unmarried                 4      F        retiree     0   \n","41               married                 0      F  civil servant     0   \n","55     civil partnership                 1      F        retiree     1   \n","...                  ...               ...    ...            ...   ...   \n","21333            married                 0      M       business     0   \n","21339  civil partnership                 1      F       employee     0   \n","21341            married                 0      F       business     0   \n","21346            married                 0      F       employee     0   \n","21354            married                 0      F       employee     0   \n","\n","       total_income                   purpose  years_employed   age_category  \n","12        19761.425         to have a wedding             NaN            Old  \n","26        23857.908                 education             NaN         Senior  \n","29        19761.425    building a real estate             NaN            Old  \n","41        23857.908  second-hand car purchase             NaN         Senior  \n","55        23857.908         to have a wedding             NaN         Senior  \n","...             ...                       ...             ...            ...  \n","21333     23857.908         purchase of a car             NaN         Senior  \n","21339     23857.908          wedding ceremony             NaN         Senior  \n","21341     23857.908       building a property             NaN         Senior  \n","21346     25466.879    building a real estate             NaN  Not Specified  \n","21354     22955.474              car purchase             NaN         Mature  \n","\n","[2110 rows x 14 columns]"],"text/html":["\n","\n","  <div id=\"df-d5d1f9b6-39b7-4f18-be02-8655c3ced604\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","      <th>years_employed</th>\n","      <th>age_category</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>12</th>\n","      <td>0</td>\n","      <td>355229.618218</td>\n","      <td>65</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>M</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>19761.425</td>\n","      <td>to have a wedding</td>\n","      <td>NaN</td>\n","      <td>Old</td>\n","    </tr>\n","    <tr>\n","      <th>26</th>\n","      <td>0</td>\n","      <td>2288.041199</td>\n","      <td>41</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>education</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>29</th>\n","      <td>0</td>\n","      <td>355229.618218</td>\n","      <td>63</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>unmarried</td>\n","      <td>4</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>19761.425</td>\n","      <td>building a real estate</td>\n","      <td>NaN</td>\n","      <td>Old</td>\n","    </tr>\n","    <tr>\n","      <th>41</th>\n","      <td>0</td>\n","      <td>2288.041199</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>civil servant</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>second-hand car purchase</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>55</th>\n","      <td>0</td>\n","      <td>2288.041199</td>\n","      <td>54</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>1</td>\n","      <td>23857.908</td>\n","      <td>to have a wedding</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>21333</th>\n","      <td>2</td>\n","      <td>2288.041199</td>\n","      <td>47</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>purchase of a car</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>21339</th>\n","      <td>1</td>\n","      <td>2288.041199</td>\n","      <td>50</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>wedding ceremony</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>21341</th>\n","      <td>0</td>\n","      <td>2288.041199</td>\n","      <td>48</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>business</td>\n","      <td>0</td>\n","      <td>23857.908</td>\n","      <td>building a property</td>\n","      <td>NaN</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>21346</th>\n","      <td>1</td>\n","      <td>2253.981630</td>\n","      <td>42 Tahun</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>25466.879</td>\n","      <td>building a real estate</td>\n","      <td>NaN</td>\n","      <td>Not Specified</td>\n","    </tr>\n","    <tr>\n","      <th>21354</th>\n","      <td>2</td>\n","      <td>1045.752744</td>\n","      <td>28</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>22955.474</td>\n","      <td>car purchase</td>\n","      <td>NaN</td>\n","      <td>Mature</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>2110 rows × 14 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d5d1f9b6-39b7-4f18-be02-8655c3ced604')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-cf5777ba-7356-4f42-8a97-77584645352f\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-cf5777ba-7356-4f42-8a97-77584645352f')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-cf5777ba-7356-4f42-8a97-77584645352f button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-d5d1f9b6-39b7-4f18-be02-8655c3ced604 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-d5d1f9b6-39b7-4f18-be02-8655c3ced604');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":197}],"source":["# check the function to ensure it works well\n","df_new_5[df_new_5['years_employed'].isnull()]"]},{"cell_type":"markdown","metadata":{"id":"jyEUxFtjLhzE"},"source":["- Yes, the values for the days_employed column are the same as for the other columns"]},{"cell_type":"markdown","metadata":{"id":"KerG5pc5LhzE"},"source":["- There is no missing value in `income_type`, so there is no need to fill in a new value in the column"]},{"cell_type":"markdown","metadata":{"id":"xetTpPLiLhzF"},"source":["Once you are done with `total income`, check if the sum of the values in this column matches with the sum of the values in the other columns."]},{"cell_type":"code","execution_count":198,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"CwPNldOqLhzF","executionInfo":{"status":"ok","timestamp":1689650776298,"user_tz":-420,"elapsed":29,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"be20f329-12db-4016-a5a1-f8efe7e7dc85"},"outputs":[{"output_type":"stream","name":"stdout","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 21369 entries, 0 to 21368\n","Data columns (total 14 columns):\n"," #   Column            Non-Null Count  Dtype  \n","---  ------            --------------  -----  \n"," 0   children          21369 non-null  int64  \n"," 1   days_employed     21369 non-null  float64\n"," 2   dob_years         21369 non-null  object \n"," 3   education         21369 non-null  object \n"," 4   education_id      21369 non-null  int64  \n"," 5   family_status     21369 non-null  object \n"," 6   family_status_id  21369 non-null  int64  \n"," 7   gender            21369 non-null  object \n"," 8   income_type       21369 non-null  object \n"," 9   debt              21369 non-null  int64  \n"," 10  total_income      21369 non-null  float64\n"," 11  purpose           21369 non-null  object \n"," 12  years_employed    19259 non-null  float64\n"," 13  age_category      21369 non-null  object \n","dtypes: float64(3), int64(4), object(7)\n","memory usage: 2.3+ MB\n"]}],"source":["# Check entries in all columns - make sure we fix all missing values\n","df_new_5.info()"]},{"cell_type":"markdown","metadata":{"id":"E9-MJTWYLhzF"},"source":["- All column values have been filled in. `years_employed` is a new field, so it can be skipped"]},{"cell_type":"markdown","metadata":{"id":"KBN6Bf8XLhzF"},"source":["## Categorized Data\n","\n","To answer questions and test hypothesis, we will work with data that has been categorized. Take a look at the questions that will be asked and that we had to answer. Think about which data needs to be categorized to answer these questions. On below, we'll find templates we can use to categorize data. The first process includes text data; the second addresses numerical data that needs to be categorized. We can use both suggested hints or neither - we're free to choose.\n","\n","Regardless of our decision to overcome categorization, make sure that we clearly explain why we made that decision. Remember: this is our job and we are the one who has the right to make all the decisions.\n"]},{"cell_type":"code","execution_count":199,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":206},"id":"03K70YNZLhzF","executionInfo":{"status":"ok","timestamp":1689650776299,"user_tz":-420,"elapsed":27,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"082adccc-5514-44bf-d358-8c84b81c0e01"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   children  days_employed dob_years            education  education_id  \\\n","0         1    8437.673028  42 Tahun    bachelor's degree             0   \n","1         1    4024.803754        36  secondary education             1   \n","2         0    5623.422610        33  secondary education             1   \n","3         3    4124.747207        32  secondary education             1   \n","4         0  340266.072047        53  secondary education             1   \n","\n","       family_status  family_status_id gender income_type  debt  total_income  \\\n","0            married                 0      F    employee     0     40620.102   \n","1            married                 0      F    employee     0     17932.802   \n","2            married                 0      M    employee     0     23341.752   \n","3            married                 0      M    employee     0     42820.568   \n","4  civil partnership                 1      F     retiree     0     25378.572   \n","\n","                   purpose  years_employed   age_category  \n","0    purchase of the house       23.116912  Not Specified  \n","1             car purchase       11.026860         Senior  \n","2    purchase of the house       15.406637         Senior  \n","3  supplementary education       11.300677         Senior  \n","4        to have a wedding      932.235814         Senior  "],"text/html":["\n","\n","  <div id=\"df-75157ab1-0e15-46cb-ac76-649f5c1c0010\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","      <th>years_employed</th>\n","      <th>age_category</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>8437.673028</td>\n","      <td>42 Tahun</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","      <td>23.116912</td>\n","      <td>Not Specified</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","      <td>11.026860</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>5623.422610</td>\n","      <td>33</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23341.752</td>\n","      <td>purchase of the house</td>\n","      <td>15.406637</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","      <td>11.300677</td>\n","      <td>Senior</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>340266.072047</td>\n","      <td>53</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>25378.572</td>\n","      <td>to have a wedding</td>\n","      <td>932.235814</td>\n","      <td>Senior</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-75157ab1-0e15-46cb-ac76-649f5c1c0010')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-daee4e67-20f8-4de3-b7d3-484a8094bc07\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-daee4e67-20f8-4de3-b7d3-484a8094bc07')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-daee4e67-20f8-4de3-b7d3-484a8094bc07 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-75157ab1-0e15-46cb-ac76-649f5c1c0010 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-75157ab1-0e15-46cb-ac76-649f5c1c0010');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":199}],"source":["# show data values that we already chosen for categorization\n","# categories can be divided based on total_income, purpose (no more than 5 types of categories)\n","df_new_5.head()"]},{"cell_type":"markdown","metadata":{"id":"s5wL9MV5LhzF"},"source":["check unique values"]},{"cell_type":"code","execution_count":200,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"8f1ONYQNLhzF","executionInfo":{"status":"ok","timestamp":1689650776299,"user_tz":-420,"elapsed":26,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"f052cfcc-3de4-41e8-b5cd-56f8d2ca0ab4"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['building a property',\n"," 'building a real estate',\n"," 'buy commercial real estate',\n"," 'buy real estate',\n"," 'buy residential real estate',\n"," 'buying a second-hand car',\n"," 'buying my own car',\n"," 'buying property for renting out',\n"," 'car',\n"," 'car purchase',\n"," 'cars',\n"," 'construction of own property',\n"," 'education',\n"," 'getting an education',\n"," 'getting higher education',\n"," 'going to university',\n"," 'having a wedding',\n"," 'housing',\n"," 'housing renovation',\n"," 'housing transactions',\n"," 'profile education',\n"," 'property',\n"," 'purchase of a car',\n"," 'purchase of my own house',\n"," 'purchase of the house',\n"," 'purchase of the house for my family',\n"," 'real estate transactions',\n"," 'second-hand car purchase',\n"," 'supplementary education',\n"," 'to become educated',\n"," 'to buy a car',\n"," 'to get a supplementary education',\n"," 'to have a wedding',\n"," 'to own a car',\n"," 'transactions with commercial real estate',\n"," 'transactions with my real estate',\n"," 'university education',\n"," 'wedding ceremony']"]},"metadata":{},"execution_count":200}],"source":["# check unique values\n","sorted(df_new['purpose'].unique())"]},{"cell_type":"markdown","metadata":{"id":"ECRgLPeyLhzF"},"source":["Which primary group can we identify based on their unique value?\n","\n","- it's `purpose` group\n","\n","We will categorize our data based on these topics.\n"]},{"cell_type":"code","execution_count":218,"metadata":{"trusted":true,"id":"1o_LHYmLLhzF","executionInfo":{"status":"ok","timestamp":1689651608313,"user_tz":-420,"elapsed":383,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# write a function to categorize data by common topics\n","def categorize_purpose(row):\n","\tif 'car' in row:\n","\t\treturn 'car'\n","\telif 'hous' in row or 'prop' in row or 'real est' in row:\n","\t\treturn 'real estate'\n","\telif 'wedd' in row:\n","\t\treturn 'wedding'\n","\telif 'educ' in row or 'uni' in row:\n","\t\treturn 'education'\n","\telse:\n","\t\treturn 'Unidentified'"]},{"cell_type":"code","execution_count":202,"metadata":{"trusted":true,"id":"-Vn3-8q_LhzG","executionInfo":{"status":"ok","timestamp":1689650776300,"user_tz":-420,"elapsed":24,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# Create a column that contains categories and calculate the values\n","df_new_5['categorize_purpose'] = df_new_5['purpose'].apply(categorize_purpose)"]},{"cell_type":"markdown","metadata":{"id":"YBEPcS6qLhzG"},"source":["If we decide to categorize numeric data, we also have to create categories for the data."]},{"cell_type":"code","execution_count":203,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/","height":250},"id":"FXm7x_yJLhzG","executionInfo":{"status":"ok","timestamp":1689650776300,"user_tz":-420,"elapsed":24,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"e040c9bb-d271-4707-8b96-fdd195167577"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["   children  days_employed dob_years            education  education_id  \\\n","0         1    8437.673028  42 Tahun    bachelor's degree             0   \n","1         1    4024.803754        36  secondary education             1   \n","2         0    5623.422610        33  secondary education             1   \n","3         3    4124.747207        32  secondary education             1   \n","4         0  340266.072047        53  secondary education             1   \n","\n","       family_status  family_status_id gender income_type  debt  total_income  \\\n","0            married                 0      F    employee     0     40620.102   \n","1            married                 0      F    employee     0     17932.802   \n","2            married                 0      M    employee     0     23341.752   \n","3            married                 0      M    employee     0     42820.568   \n","4  civil partnership                 1      F     retiree     0     25378.572   \n","\n","                   purpose  years_employed   age_category categorize_purpose  \n","0    purchase of the house       23.116912  Not Specified        real estate  \n","1             car purchase       11.026860         Senior                car  \n","2    purchase of the house       15.406637         Senior        real estate  \n","3  supplementary education       11.300677         Senior          education  \n","4        to have a wedding      932.235814         Senior            wedding  "],"text/html":["\n","\n","  <div id=\"df-a782d3eb-8a19-4c4a-9e28-df1b86e445b4\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>children</th>\n","      <th>days_employed</th>\n","      <th>dob_years</th>\n","      <th>education</th>\n","      <th>education_id</th>\n","      <th>family_status</th>\n","      <th>family_status_id</th>\n","      <th>gender</th>\n","      <th>income_type</th>\n","      <th>debt</th>\n","      <th>total_income</th>\n","      <th>purpose</th>\n","      <th>years_employed</th>\n","      <th>age_category</th>\n","      <th>categorize_purpose</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>8437.673028</td>\n","      <td>42 Tahun</td>\n","      <td>bachelor's degree</td>\n","      <td>0</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>40620.102</td>\n","      <td>purchase of the house</td>\n","      <td>23.116912</td>\n","      <td>Not Specified</td>\n","      <td>real estate</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>4024.803754</td>\n","      <td>36</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>F</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>17932.802</td>\n","      <td>car purchase</td>\n","      <td>11.026860</td>\n","      <td>Senior</td>\n","      <td>car</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0</td>\n","      <td>5623.422610</td>\n","      <td>33</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>23341.752</td>\n","      <td>purchase of the house</td>\n","      <td>15.406637</td>\n","      <td>Senior</td>\n","      <td>real estate</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>4124.747207</td>\n","      <td>32</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>married</td>\n","      <td>0</td>\n","      <td>M</td>\n","      <td>employee</td>\n","      <td>0</td>\n","      <td>42820.568</td>\n","      <td>supplementary education</td>\n","      <td>11.300677</td>\n","      <td>Senior</td>\n","      <td>education</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0</td>\n","      <td>340266.072047</td>\n","      <td>53</td>\n","      <td>secondary education</td>\n","      <td>1</td>\n","      <td>civil partnership</td>\n","      <td>1</td>\n","      <td>F</td>\n","      <td>retiree</td>\n","      <td>0</td>\n","      <td>25378.572</td>\n","      <td>to have a wedding</td>\n","      <td>932.235814</td>\n","      <td>Senior</td>\n","      <td>wedding</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a782d3eb-8a19-4c4a-9e28-df1b86e445b4')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-6cfe515c-040f-4e93-a83c-53df5fd5b852\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6cfe515c-040f-4e93-a83c-53df5fd5b852')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-6cfe515c-040f-4e93-a83c-53df5fd5b852 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-a782d3eb-8a19-4c4a-9e28-df1b86e445b4 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-a782d3eb-8a19-4c4a-9e28-df1b86e445b4');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":203}],"source":["# check all numeric data in the column that we've selected for categorization\n","df_new_5.head()"]},{"cell_type":"code","execution_count":204,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"kmnXTOoYLhzG","executionInfo":{"status":"ok","timestamp":1689650776300,"user_tz":-420,"elapsed":23,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"9bd60db0-7eb7-4a7b-a242-26e223bf9b01"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["real estate    10766\n","car             4286\n","education       3996\n","wedding         2321\n","Name: categorize_purpose, dtype: int64"]},"metadata":{},"execution_count":204}],"source":["# make a statistical conclusions for the column\n","df_new_5['categorize_purpose'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"DO3G77PxLhzG"},"source":["Decide what range we will use for clustering and explain why."]},{"cell_type":"code","execution_count":205,"metadata":{"trusted":true,"id":"NdNCSRWQLhzG","executionInfo":{"status":"ok","timestamp":1689650776301,"user_tz":-420,"elapsed":21,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# Create a function that performs categorizing data into different numeric groups based on range\n","def income_level(income):\n","\tif (income > 5000) and (income <= 10000):\n","\t\treturn 'small'\n","\tif (income > 10000) and (income <= 15000):\n","\t\treturn 'average'\n","\tif (income > 20000) and (income <= 25000):\n","\t\treturn 'above average'\n","\tif (income > 25000) and (income <= 50000):\n","\t\treturn 'high'\n","\tif income > 50000:\n","\t\treturn 'very high'"]},{"cell_type":"code","execution_count":206,"metadata":{"trusted":true,"id":"NSm1H_-cLhzH","executionInfo":{"status":"ok","timestamp":1689650776301,"user_tz":-420,"elapsed":21,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}}},"outputs":[],"source":["# create a new column to load category\n","df_new_5['income_level'] = df_new_5['total_income'].apply(income_level)"]},{"cell_type":"code","execution_count":207,"metadata":{"scrolled":false,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"lYgWR6nFLhzH","executionInfo":{"status":"ok","timestamp":1689650776301,"user_tz":-420,"elapsed":20,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"a68d4702-068f-4a8f-a0f1-56a8357d5ca1"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["high             7305\n","above average    5163\n","average          2802\n","very high        1318\n","small             895\n","Name: income_level, dtype: int64"]},"metadata":{},"execution_count":207}],"source":["# Count each category value to see its distribution\n","df_new_5['income_level'].value_counts()"]},{"cell_type":"markdown","metadata":{"id":"R2FvKp36LhzH"},"source":["## Check the hypothesis\n"]},{"cell_type":"markdown","metadata":{"id":"tOgJVhoBLhzH"},"source":["**Is there a correlation between having children and paying off on time?**"]},{"cell_type":"code","execution_count":208,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"HGFG-o58LhzH","executionInfo":{"status":"ok","timestamp":1689650776302,"user_tz":-420,"elapsed":20,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"01fc7342-01da-4a2c-c651-812fa1bf7c52"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(0     12979\n"," 1      4398\n"," 2      1845\n"," 3       301\n"," 20       67\n"," 4        37\n"," 5         9\n"," Name: children, dtype: int64,\n"," 0     1058\n"," 1      442\n"," 2      194\n"," 3       27\n"," 20       8\n"," 4        4\n"," Name: children, dtype: int64)"]},"metadata":{},"execution_count":208}],"source":["# Check the child data and repayment on time data\n","ch_debt_0 = df_new_5.loc[(df_new_5['debt'] == 0), 'children'].value_counts()\n","# count customer that failed to pay based on total child\n","ch_debt_1 = df_new_5.loc[(df_new_5['debt'] == 1), 'children'].value_counts()\n","ch_debt_0, ch_debt_1"]},{"cell_type":"code","execution_count":209,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"MrwbFbgTLhzH","executionInfo":{"status":"ok","timestamp":1689650776302,"user_tz":-420,"elapsed":18,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"ea4e0697-fb5a-4c06-f569-229eb7f97773"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["debt  children\n","0     0           0.660980\n","      1           0.223976\n","      2           0.093960\n","      3           0.015329\n","      20          0.003412\n","      4           0.001884\n","      5           0.000458\n","1     0           0.610502\n","      1           0.255049\n","      2           0.111945\n","      3           0.015580\n","      20          0.004616\n","      4           0.002308\n","Name: children, dtype: float64"]},"metadata":{},"execution_count":209}],"source":["df_new_5.groupby('debt')['children'].value_counts(normalize=True)"]},{"cell_type":"code","execution_count":210,"metadata":{"scrolled":true,"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"4K8xeMkcLhzH","executionInfo":{"status":"ok","timestamp":1689650776303,"user_tz":-420,"elapsed":17,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"25103420-a561-4d5f-bcd6-7c01b00b4b95"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    0.918901\n","1    0.081099\n","Name: debt, dtype: float64"]},"metadata":{},"execution_count":210}],"source":["df_new_5['debt'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"56eJOLcLLhzI"},"source":["**Conclusion**\n","\n","- In total, there are around 8% of individuals have failed to pay. And, the highest number of failed to pay is dominated by 61% by individuals who do not have children"]},{"cell_type":"markdown","metadata":{"id":"zOWhTJnvLhzI"},"source":["**Is there a correlation between family status and repayment on time?**"]},{"cell_type":"code","execution_count":211,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"hOruZptBLhzI","executionInfo":{"status":"ok","timestamp":1689650776303,"user_tz":-420,"elapsed":16,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"5460c9ca-23cc-4e9b-e0d2-9ef273f4dc81"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(married              11368\n"," civil partnership     3755\n"," unmarried             2521\n"," divorced              1100\n"," widow / widower        892\n"," Name: family_status, dtype: int64,\n"," married              927\n"," civil partnership    386\n"," unmarried            273\n"," divorced              85\n"," widow / widower       62\n"," Name: family_status, dtype: int64)"]},"metadata":{},"execution_count":211}],"source":["# check data from `family_status` and repayment on time\n","fam_debt_0 = df_new_5.loc[(df_new_5['debt'] == 0), 'family_status'].value_counts()\n","# counts failed to pay based on family status\n","fam_debt_1 = df_new_5.loc[(df_new_5['debt'] == 1), 'family_status'].value_counts()\n","fam_debt_0, fam_debt_1"]},{"cell_type":"code","execution_count":212,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"L7-_4W3lLhzI","executionInfo":{"status":"ok","timestamp":1689650776738,"user_tz":-420,"elapsed":14,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"c505dd76-79c4-4e0b-b69e-fb9843097bc7"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["debt  family_status    \n","0     married              0.578937\n","      civil partnership    0.191230\n","      unmarried            0.128387\n","      divorced             0.056020\n","      widow / widower      0.045427\n","1     married              0.534911\n","      civil partnership    0.222735\n","      unmarried            0.157530\n","      divorced             0.049048\n","      widow / widower      0.035776\n","Name: family_status, dtype: float64"]},"metadata":{},"execution_count":212}],"source":["df_new_5.groupby('debt')['family_status'].value_counts(normalize=True)"]},{"cell_type":"code","execution_count":213,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"Gue2iW0hLhzI","executionInfo":{"status":"ok","timestamp":1689650776739,"user_tz":-420,"elapsed":13,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"b423f7be-b7a0-4f48-ec2b-3932b97f517e"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    0.918901\n","1    0.081099\n","Name: debt, dtype: float64"]},"metadata":{},"execution_count":213}],"source":["df_new_5['debt'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"4TvIfd6ILhzI"},"source":["**Conclusion**\n","\n","- In total, there is around 8% of individuals that have failed to pay. And, the highest number of this were dominated by 53% from individuals who already been married or family status 'married'"]},{"cell_type":"markdown","metadata":{"id":"Pkt-d699LhzI"},"source":["**Is there a correlation between income levels and repayment on time?**"]},{"cell_type":"code","execution_count":214,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"48br5sRPLhzI","executionInfo":{"status":"ok","timestamp":1689650776739,"user_tz":-420,"elapsed":11,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"00ad0d65-a745-498f-8fe9-ca8764e61087"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(employee         9981\n"," business         4684\n"," retiree          3602\n"," civil servant    1365\n"," entrepreneur        2\n"," student             1\n"," unemployed          1\n"," Name: income_type, dtype: int64,\n"," employee                       1055\n"," business                        375\n"," retiree                         215\n"," civil servant                    86\n"," unemployed                        1\n"," paternity / maternity leave       1\n"," Name: income_type, dtype: int64)"]},"metadata":{},"execution_count":214}],"source":["# Check on income rate data and repayment on time\n","inc_debt_0 = df_new_5.loc[(df_new_5['debt'] == 0), 'income_type'].value_counts()\n","# counts on failed to pay data based on income type\n","inc_debt_1 = df_new_5.loc[(df_new_5['debt'] == 1), 'income_type'].value_counts()\n","inc_debt_0, inc_debt_1"]},{"cell_type":"code","execution_count":215,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"f7_tfGynLhzI","executionInfo":{"status":"ok","timestamp":1689650776739,"user_tz":-420,"elapsed":9,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"756952cc-a161-4dc8-ae67-deedb6399b2e"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["debt  income_type                \n","0     employee                       0.508301\n","      business                       0.238541\n","      retiree                        0.183439\n","      civil servant                  0.069515\n","      entrepreneur                   0.000102\n","      student                        0.000051\n","      unemployed                     0.000051\n","1     employee                       0.608771\n","      business                       0.216388\n","      retiree                        0.124062\n","      civil servant                  0.049625\n","      paternity / maternity leave    0.000577\n","      unemployed                     0.000577\n","Name: income_type, dtype: float64"]},"metadata":{},"execution_count":215}],"source":["df_new_5.groupby('debt')['income_type'].value_counts(normalize=True)"]},{"cell_type":"code","execution_count":216,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"MvDw0i5YLhzI","executionInfo":{"status":"ok","timestamp":1689650776740,"user_tz":-420,"elapsed":8,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"489ff975-fab9-42eb-b224-2d918a6b9eb3"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    0.918901\n","1    0.081099\n","Name: debt, dtype: float64"]},"metadata":{},"execution_count":216}],"source":["df_new_5['debt'].value_counts(normalize=True)"]},{"cell_type":"markdown","metadata":{"id":"alilmG0rLhzI"},"source":["**Conclusion**\n","\n","- In total, there is around 8% of individuals have been failed to pay. And, the largest number of this were dominated by 60% from individuals with a profession as employees"]},{"cell_type":"markdown","metadata":{"id":"rAbCVKaJLhzJ"},"source":["**How do credit goals affect failed to pay rates?**"]},{"cell_type":"code","execution_count":217,"metadata":{"trusted":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"g03LsIZNLhzJ","executionInfo":{"status":"ok","timestamp":1689650776740,"user_tz":-420,"elapsed":6,"user":{"displayName":"Guntur Wibawa","userId":"10578009354531389415"}},"outputId":"e484fe1e-853e-46d6-8929-9a479eb599c4"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["real estate    0.449510\n","car            0.230814\n","education      0.213503\n","wedding        0.106174\n","Name: categorize_purpose, dtype: float64"]},"metadata":{},"execution_count":217}],"source":["# Check the failed to pay rate percentage for each credit purpose and perform analysis\n","pur_debt_1 = df_new_5.loc[(df_new_5['debt'] == 1), 'categorize_purpose'].value_counts(normalize=True)\n","pur_debt_1"]},{"cell_type":"markdown","metadata":{"id":"VyYPnITJLhzJ"},"source":["**Conclusion**\n","\n","- In total, there is around 44% default cases for credit purposes in the real estate category. It is possible that it was triggered by long-term credit which led to over credit to third parties, or an increase in the benchmark interest rate which resulted in an increasing credit interest on real estate."]},{"cell_type":"markdown","metadata":{"id":"jLYWvo3OLhzJ"},"source":["# **Final Conclusions**\n","\n","- Based on the datasets we used, there are around 21525 rows of data, where 10% of the data has missing values in the `days_employed` and `total_income` columns. Then, duplicate data was also found in the `education` column. It is possible that this could be triggered due to a human error or due to certain conditions. As a solution, I did a boxplot analysis to find out the number of outliers in the data. Because the missing values has a large outliers, I replaced the data using median value. And for duplicate data, I convert it to lowercase.\n","\n","- Pertanyaan yg diajukan sebenarnya sudah cukup umum ketika kita mendapatkan suatu data, dan untuk mengolah data tsb menjadi suatu data yg bermanfaat sehingga memiliki informasi penting didalamnya.\n","\n","- The questions that were asked are actually quite general when we receive the datasets, and to process the data into a useful data that has important information in it.\n"]}],"metadata":{"ExecuteTimeLog":[{"duration":2741,"start_time":"2021-10-21T01:08:49.480Z"},{"duration":52,"start_time":"2021-10-21T01:08:53.315Z"},{"duration":49,"start_time":"2021-10-21T01:08:57.432Z"},{"duration":70,"start_time":"2021-10-21T01:09:07.706Z"}],"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.5"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":true,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{},"toc_section_display":true,"toc_window_display":false},"vscode":{"interpreter":{"hash":"ef1728426330606e01c1da091626d7dbd7604032247e9790b44969d857c4f729"}},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":0}