workshop_1_car_price_prediction(exercise).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Car Price Prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem Statement\n",
    "Given data of US cars. We need to create model which predicts prices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Business Goal \n",
    "\n",
    "We also need to find top features that have higher correlation to price value. We aren't looking for complex dependencies. We need to consider all features separately with the target. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to remove some user warnings from pandas and numpy\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# for numerical computations\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# for plotting and visualizations\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# for training models, evaluations and feature processing\n",
    "from sklearn.metrics import mean_squared_error, make_scorer, r2_score\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split, cross_validate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "c4422c44-97d7-44d9-a4ae-793e6e2da4c4",
    "_uuid": "354f82983a107141a40f8952a451123bb30e4a7a"
   },
   "outputs": [],
   "source": [
    "def hist_plot(dataframe, features, rows, cols):\n",
    "    fig = plt.figure(figsize=(20, 20))\n",
    "    for i, feature in enumerate(features):\n",
    "        ax = fig.add_subplot(rows, cols, i+1)\n",
    "        dataframe[feature].hist(bins=20, ax=ax, facecolor='midnightblue')\n",
    "        ax.set_title(feature+\" Distribution\", color='DarkRed')\n",
    "\n",
    "    fig.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pie_plot(series, figsize=(7, 7)):\n",
    "    ax_ = plt.figure(figsize=figsize).add_subplot()\n",
    "    ax_.pie(series.value_counts(), labels=series.value_counts().index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def corr_plot(dataframe, figsize=(10, 10)):\n",
    "    axes_ = plt.figure(figsize=figsize).add_subplot()\n",
    "    corr = dataframe.corr()\n",
    "    ax = sns.heatmap(\n",
    "        corr,\n",
    "        vmin=-1, vmax=1, center=0,\n",
    "        cmap=sns.diverging_palette(20, 220, n=200),\n",
    "        square=True\n",
    "    )\n",
    "    ax.set_xticklabels(\n",
    "        ax.get_xticklabels(),\n",
    "        rotation=45,\n",
    "        horizontalalignment='right'\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def barh_plot(series, figsize=(10, 10)):\n",
    "    ax_ = plt.figure(figsize=figsize).add_subplot()\n",
    "    ax_.barh(series.index, series.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_top_unique_values(dataframe, features=None, top_n=10):\n",
    "    if not features:\n",
    "        features = dataframe.columns\n",
    "    \n",
    "    data = {}\n",
    "    for feature in features:\n",
    "        if feature in dataframe.columns:\n",
    "            data[feature] = dataframe[feature].value_counts().index.values[:top_n]\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading and Understanding the Data\n",
    "\n",
    "Let's start with the following steps:\n",
    "\n",
    "1. Importing data using the pandas library\n",
    "2. Understanding the structure of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extra code for colab\n",
    "# donwload one file from github -> https://raw.githubusercontent.com/user/repository/branch/filename\n",
    "# our case -> https://raw.githubusercontent.com/MaxinAI/school-of-ai/master/data/ml_intro/Advertising.csv\n",
    "\n",
    "\n",
    "try:\n",
    "    import google.colab\n",
    "    IN_COLAB = True\n",
    "except:\n",
    "    IN_COLAB = False\n",
    "\n",
    "if IN_COLAB:\n",
    "    print('colab')\n",
    "    import os\n",
    "    os.system(\"wget https://raw.githubusercontent.com/MaxinAI/school-of-ai/master/data/workshop_1/CarPrice_Assignment.csv\")\n",
    "    # read the data\n",
    "    cars = pd.read_csv('./CarPrice_Assignment.csv')\n",
    "else:\n",
    "    # just the old code\n",
    "    cars = pd.read_csv('data/workshop_1/CarPrice_Assignment.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>car_ID</th>\n",
       "      <th>symboling</th>\n",
       "      <th>CarName</th>\n",
       "      <th>fueltype</th>\n",
       "      <th>aspiration</th>\n",
       "      <th>doornumber</th>\n",
       "      <th>carbody</th>\n",
       "      <th>drivewheel</th>\n",
       "      <th>enginelocation</th>\n",
       "      <th>wheelbase</th>\n",
       "      <th>...</th>\n",
       "      <th>enginesize</th>\n",
       "      <th>fuelsystem</th>\n",
       "      <th>boreratio</th>\n",
       "      <th>stroke</th>\n",
       "      <th>compressionratio</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>peakrpm</th>\n",
       "      <th>citympg</th>\n",
       "      <th>highwaympg</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>alfa-romero giulia</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>convertible</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>88.6</td>\n",
       "      <td>...</td>\n",
       "      <td>130</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.47</td>\n",
       "      <td>2.68</td>\n",
       "      <td>9.0</td>\n",
       "      <td>111</td>\n",
       "      <td>5000</td>\n",
       "      <td>21</td>\n",
       "      <td>27</td>\n",
       "      <td>13495.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>alfa-romero stelvio</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>convertible</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>88.6</td>\n",
       "      <td>...</td>\n",
       "      <td>130</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.47</td>\n",
       "      <td>2.68</td>\n",
       "      <td>9.0</td>\n",
       "      <td>111</td>\n",
       "      <td>5000</td>\n",
       "      <td>21</td>\n",
       "      <td>27</td>\n",
       "      <td>16500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>alfa-romero Quadrifoglio</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>hatchback</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>94.5</td>\n",
       "      <td>...</td>\n",
       "      <td>152</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>2.68</td>\n",
       "      <td>3.47</td>\n",
       "      <td>9.0</td>\n",
       "      <td>154</td>\n",
       "      <td>5000</td>\n",
       "      <td>19</td>\n",
       "      <td>26</td>\n",
       "      <td>16500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>audi 100 ls</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.8</td>\n",
       "      <td>...</td>\n",
       "      <td>109</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>10.0</td>\n",
       "      <td>102</td>\n",
       "      <td>5500</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>13950.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>audi 100ls</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>4wd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.4</td>\n",
       "      <td>...</td>\n",
       "      <td>136</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>8.0</td>\n",
       "      <td>115</td>\n",
       "      <td>5500</td>\n",
       "      <td>18</td>\n",
       "      <td>22</td>\n",
       "      <td>17450.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   car_ID  symboling                   CarName fueltype aspiration doornumber  \\\n",
       "0       1          3        alfa-romero giulia      gas        std        two   \n",
       "1       2          3       alfa-romero stelvio      gas        std        two   \n",
       "2       3          1  alfa-romero Quadrifoglio      gas        std        two   \n",
       "3       4          2               audi 100 ls      gas        std       four   \n",
       "4       5          2                audi 100ls      gas        std       four   \n",
       "\n",
       "       carbody drivewheel enginelocation  wheelbase  ...  enginesize  \\\n",
       "0  convertible        rwd          front       88.6  ...         130   \n",
       "1  convertible        rwd          front       88.6  ...         130   \n",
       "2    hatchback        rwd          front       94.5  ...         152   \n",
       "3        sedan        fwd          front       99.8  ...         109   \n",
       "4        sedan        4wd          front       99.4  ...         136   \n",
       "\n",
       "   fuelsystem  boreratio  stroke compressionratio horsepower  peakrpm citympg  \\\n",
       "0        mpfi       3.47    2.68              9.0        111     5000      21   \n",
       "1        mpfi       3.47    2.68              9.0        111     5000      21   \n",
       "2        mpfi       2.68    3.47              9.0        154     5000      19   \n",
       "3        mpfi       3.19    3.40             10.0        102     5500      24   \n",
       "4        mpfi       3.19    3.40              8.0        115     5500      18   \n",
       "\n",
       "   highwaympg    price  \n",
       "0          27  13495.0  \n",
       "1          27  16500.0  \n",
       "2          26  16500.0  \n",
       "3          30  13950.0  \n",
       "4          22  17450.0  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## number of records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have very low amount of records our model won't be as good as it should be in general. Anyway it's a example for demonstrating your skills on model selection, hyperparameter tuning and model selection/scoring "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cars.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Names of these features are easy to understand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cars.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## See top frequent values from each column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get quick overview on values for each feature we can list top frequent values for each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_top_unique_values(cars, top_n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List Numerical Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you check data types for each feature you can select numerical ones. It may happen that numerical values are in \"Object\" format if numbers are represented as strings, so please double check to make sure you got them correctly!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find numerical features by checking their data type not equals to \"object\" \n",
    "numerical_features = \n",
    "numerical_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We have not empty (NAN) values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# count number of nan's in total in data\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Take only numerical features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only use numerical features for simplicity now. Usually all features are used considering their importance and correlation with target. Feature engineering will be taught later in separate lecture. You can get some examples from [here](https://jakevdp.github.io/PythonDataScienceHandbook/05.04-feature-engineering.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take only chosen numerical features and assign back to dataframe\n",
    "cars = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing the data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show target distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,8))\n",
    "\n",
    "plt.subplot(1,2,1)\n",
    "plt.title('Car Price Distribution Plot')\n",
    "sns.distplot(cars.price, bins=20)\n",
    "\n",
    "plt.subplot(1,2,2)\n",
    "plt.title('Car Price Spread')\n",
    "sns.boxplot(y=cars.price)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show some percentiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cars.price.describe(percentiles = [0.25,0.50,0.75,0.85,0.90,1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inference :\n",
    "\n",
    "1. The plot seemed to be right-skewed, meaning that the most prices in the dataset are low(Below 15,000).\n",
    "2. There is a significant difference between the mean and the median of the price distribution.\n",
    "3. The data points are far spread out from the mean, which indicates a high variance in the car prices.(85% of the prices are below 18,500, whereas the remaining 15% are between 18,500 and 45,400.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualising numerical data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### show histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_plot(dataframe=cars, features=cars.columns, rows=6, cols=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Look at main statistical properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cars.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show Feature PairWise Scatter Plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.plotting.scatter_matrix(cars, figsize=(20, 20));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show feature correlations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_plot(cars, figsize=(20,20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show Features Correlations with Target sorted with absolute value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "barh_plot(cars.corr()['price'].abs().sort_values(ascending=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cars.corr()['price'].abs().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take top correlated features with correlation more than 0.5\n",
    "top_correlated_features = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Take top correlated features and target only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take only high correlated features + target and assign all of them to cars dataframe (overwrite)\n",
    "cars = \n",
    "cars.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show Pair Plot of top correlated features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(cars)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train-Test Split and feature scaling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "RANDOM_SEED = 42\n",
    "TEST_SIZE = 0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split data into training and test parts using random seed, test size (here we don't use stratify since price is not binary)\n",
    "df_train, df_test = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scale Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Min-Max Scaler does the scaling of each feature into \\[0,1\\] range. That is the basic scaling function which could be used. You can explore variety of scalers [here](https://scikit-learn.org/stable/modules/preprocessing.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initialize scaler object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find MinMaxScaler [here](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creta min-max scaler object from scikit learn with default parameters.\n",
    "scaler = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "list numerical features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list numerical features again (including price since it's not binary)\n",
    "numerical_features = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "fit scaler on training data and then transform it. Will will transform test data using fitted scaler. It's usually done like that to simulate unseen data processing where."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit and transform numerical features from training data using scaler object created above \n",
    "df_train[numerical_features] = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transform numerical features from test data using scaler object trained on training data \n",
    "df_test[numerical_features] = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check how correlations changed using pair plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you compare that to the original one (without transformations) you will see that correlations are changed. We are doing feature scaling to speedup process of convergence on training. However, using good feature engineering and scaling you can increase the performance of model. We aren't doing that for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(pd.concat([df_train, df_test], axis=0))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dividing data into X and y variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split all columns in training and test data into input features and target_feature \n",
    "input_features = \n",
    "target_feature = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate input features and target for training data\n",
    "y_train = \n",
    "X_train = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate input features and target for test data\n",
    "y_test = \n",
    "X_test = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "d791f658-cfbb-496b-93d1-b2d0ef639b7e",
    "_uuid": "a14fae661d7902e8c58a9b57b6f6873b6aeb0a79"
   },
   "source": [
    "# Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "17d7af59-bd19-4448-b26c-f2c52b6c5381",
    "_uuid": "2244062f695fd7f76c0b86af5b49d6a597679c4d"
   },
   "source": [
    "Linear regression is a type of regression analysis in statistics used for prediction of outcome of a numerical dependent variable from a set of predictor or independent variables. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Linear Regression Documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Linear Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Init Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create linear regression model\n",
    "model = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Scoring for cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create scoring parameter with mean squared error measure named as \"mse\"\n",
    "SCORING = {\n",
    "    \n",
    "}\n",
    "CV = 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross validate and get scores and models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate cross validation scores of linear regression model on training data using scoring and cross validation.\n",
    "# use return_estimator=True to get also all estimator models tried\n",
    "scores = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "since we don't have Hyperparameters to tune for linear regression all the models are same and scores actually describe the how hard is each split to fit by linear hyperplane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(scores.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show Scores and models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take out \"estimator\" and \"test_mse\" from scores data \n",
    "all_models = \n",
    "all_scores = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_models, all_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Scores Histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.Series(all_scores).hist(bins=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we said we don't have any chance to find best model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose one model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get \"Best\" model from all models with lowest mse score\n",
    "best_model = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Do prediction on training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict on training data using best model\n",
    "y_train_pred = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### show error terms distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's check how error terms are distributed. If it's close to normal distribution, it can mean that hyperplane is well fitted to our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "sns.distplot((y_train - y_train_pred), bins = 20)\n",
    "fig.suptitle('Error Terms', fontsize = 20) \n",
    "plt.xlabel('Errors', fontsize = 18); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Error terms seem to be approximately normally distributed, so the assumption on the linear modeling seems to be fulfilled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prediction and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## predict on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict on test data using best model\n",
    "y_test_pred = best_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MSE and R2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate and print \"mse\" and \"r2\" scores for test data predictions (hint: use imported metrics from sklearn.metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting y_test and y_test_pred to understand the spread"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.scatter(y_test, y_test_pred)\n",
    "fig.suptitle('y_test vs y_test_pred', fontsize=20) \n",
    "plt.xlabel('y_test', fontsize=18)\n",
    "plt.ylabel('y_test_pred', fontsize=16); "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "336px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}