COMP9318_Project-specs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deadline + Late Penalty\n",
    "\n",
    "**Note :** It will take you quite some time to complete this project, therefore, we earnestly recommend that you start working as early as possible.\n",
    "\n",
    "\n",
    "* Submission deadline for the Project is **20:59:59 on 23rd Apr, 2021 (Sydney Time)**.\n",
    "* **LATE PENALTY: Late Penalty: 10-% on day-1 and 20% on each subsequent day.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "1. This note book contains instructions for **COMP9318-Project**.\n",
    "\n",
    "2. You are required to complete your implementation in a file `submission.py` provided along with this notebook.\n",
    "\n",
    "3. You are not allowed to print out unnecessary stuff. We will not consider any output printed out on the screen. All results should be returned in appropriate data structures via corresponding functions.\n",
    "\n",
    "4. You can submit your implementation for the **Project** via following link: http://kg.cse.unsw.edu.au/submit/\n",
    "\n",
    "5. For each part, we have provided you with detailed instructions. In case of any problem, you can post your query @ Ed.\n",
    "\n",
    "6. You are allowed to add other functions (you may have to for this project), but you are not allowed to define global variables. **Only functions are allowed** in `submission.py`.\n",
    "\n",
    "7. You are allowed to import other modules, but only from the following modules/libraries.\n",
    " * **Scikit-Learn 0.24.1**\n",
    " * **Numpy 1.19.5**\n",
    " * **Pandas 1.1.5**\n",
    " * **Python 3.6.5**\n",
    "\n",
    " Importing other modules will lead to errors.\n",
    "\n",
    "8. For some parts of the project, i.e., **Project-Part1**, we will provide immediate feedback on your submission **based on the dataset provided with the specs (1st April 2021 onwards)**. You can view the feedback using the online submission portal on the same day.\n",
    "\n",
    "9. You are allowed a limited number of Feedback Attempts **(15 Attempts for each Student)**, we will use your **LAST** submission for Final Evaluation. Please **DO NOT** forget to submit **Report.pdf** along with your last submission."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project-Part1: Predict COVID-19 Confirmed Cases (45 Points)\n",
    "\n",
    "Given the fact that the world is exposed to COVID-19, in this project, we aim to analyze the time series of COVID-19 cases as a function of past COVID-19 cases and the weather conditions.\n",
    "\n",
    "In this question, you are required to formulate a model that can predict confirmed COVID-19 cases for a state $X$ by analyzing the time series data of the COVID-19 cases along with the weather conditions. Specifically, you are required to complete the function `predict_COVID_part1()` in the file `submission.py`. The inputs and the outputs of the function are explained as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input and Output formats\n",
    "\n",
    "### Inputs:\n",
    "\n",
    "1. `svm_model`, Scikit-learn's Support Vector Regression model with hyper-parameters initialized. **Note** that for part1 of the project, you are not required to change the model and its hyper-parameters, we recommend using the hyper-parameters settings provided as model input. \n",
    "\n",
    "* `train_df`, pandas dataframe corresponding the csv file: `COVID_train_data.csv`. The format of all fields of the data set is explained below. This dataset is intended for model training.\n",
    "\n",
    "* `train_labels_df`, pandas dataframe corresponding to the csv file: `COVID_train_labels.csv`. It comprises the number of COVID-19 confirmed cases for each single day. This dataset is intended for model training.\n",
    "\n",
    "* `past_cases_interval`, an integer value representing number of past days of COVID-19 cases to consider for the model training.\n",
    "\n",
    "* `past_weather_interval`, an integer value representing the number of past days of weather conditions to consider for the model training.\n",
    "\n",
    "* `test_feature`, A feature vector encompassing a subset of features from the file `test_features.csv` used for predicting the COVID-19 cases of the future. We provide details of the file: `test_features.csv` in the following section.\n",
    "\n",
    "### Outputs:\n",
    "\n",
    "Based on the feature space used for the model training (i.e., features constructed from the `train_data`), you are required to select the corresponding subset of features from the test features `test_features.csv` and predict the probable cases of COVID-19 cases for each single day.\n",
    "\n",
    "**NOTE: You should use math.floor(x) to convert the prediction result to an integer.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Format Explained\n",
    "\n",
    "#### 1. `train_df:`\n",
    "\n",
    "The `train_df` dataframe encompasses time series data of weather conditions and COVID-19 cases for the state $X$ in the increasing order of time. The contents of the dataframe are explained below:\n",
    "\n",
    "1. **day**: The day number of each observation in the file `train_df`. The day numbers are increasing in order of time.\n",
    "\n",
    "* **temp**: Temperature of state $X$ in $^{\\circ}F$. We provide $maximum$, $average$ and $minimum$ temperature of state $X$ in the fields `max_temp`, `avg_temp` and `min_temp` respectively.\n",
    "\n",
    "* **dew**: Dew point of state $X$ in $^{\\circ}F$. We provide $maximum$, $average$ and $minimum$ dew point of state $X$ in the fields `max_dew`, `avg_dew` and `min_dew` respectively.\n",
    "\n",
    "* **humid**: % Humidity of state $X$. We provide $maximum$, $average$ and $minimum$ humidity of state $X$ in the fields `max_humid`, `avg_humid` and `min_humid` respectively.\n",
    "\n",
    "* **wind_speed**: Wind speed in state $X$ measured in mph. We provide $maximum$, $average$ and $minimum$ wind speed  of state $X$ in the fields `max_wind_speed`, `avg_wind_speed` and `min_wind_speed` respectively.\n",
    "\n",
    "* **pressure**: Sea level pressure of state $X$ measured in $Hg$. We provide $maximum$, $average$ and $minimum$ sea level pressure of state $X$ in the fields `max_pressure`, `avg_pressure` and `min_pressure` respectively.\n",
    "\n",
    "* **precipitation**: Total dailly Precipitation of state $X$ measured in inches, represented as variable `precipitation`.\n",
    "\n",
    "* **dailly_cases**: Total number of confirmed COVID-19 cases for state $X$ reported each single day.\n",
    "\n",
    "\n",
    "#### 2. `train_labels_df:`\n",
    "\n",
    "The `train_labels_df` encompasses the number of COVID-19 confirmed cases of state $X$ being reported every single day. The contents of this file are shown below.\n",
    "\n",
    "1. **day**: The day number of each observation in the file `train_labels_df`. The day numbers are increasing in order of time.\n",
    "\n",
    "* **dailly_cases**: Total number of confirmed COVID-19 cases for state $X$ reported each single day.\n",
    "\n",
    "\n",
    "#### 3. `test_features.csv:`\n",
    "\n",
    "The `test_features.csv` encompasses all possible feature values of the test features, encompassing the weather conditions and the past COVID-19 cases for the past `N-days (N=30)`. For a given day, the corresponding day-id is added along with the feature name. For this project, you are allowed to use all the features and/or a subset of it as per your requirement.\n",
    "\n",
    "\n",
    "**Note:** \n",
    "\n",
    "* We restrict the maximum allowed values of the features from the past (weather and COVID-19 cases) to `N=30` days.\n",
    "* The features in the file `test_features.csv` will follow the same sequential order unless specified otherwise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to construct feature matrix\n",
    "Given the fact that we restrict the value of the past instances of weather and past cases to be: `N=30`. If we construct a feature matrix of all the parameters involved in the training data, our initial feature matrix will be of the shape: `162 x 510`. In this project we require you to formulate the COVID-19 cases prediction as a regression problem. \n",
    "Specifically, you will be using the information for the `days: 1,...,t-1` to predict the cases for the `day = t`, as shown in a simple linear regression model below, for illustration purposes only:\n",
    "\n",
    "\n",
    "<center>$\\sum_{i=1}^{t-1}a_{i}*max\\_temp_{i} +...+ \\sum_{i=1}^{t-1}p_{i}*precipitation_{i} + \\sum_{i=1}^{t-1}q_{i}*Cases_{i}= Cases_{day = t}$</center>\n",
    "\n",
    "### Training Feature Matrix:\n",
    "\n",
    "For the project-part1, we require you to form a feature matrix (`x_train`) encompassing the maximum values of the following subset of weather information and the past cases, in the same order as mentioned below: \n",
    "\n",
    "`[max_temp, max_dew, max_humid, past_cases]`\n",
    "\n",
    "For the values of the input parameters: `past_weather_interval=10` and `past_cases_interval=10`, the resulting training matrix formed using the training data file: `train_df`, and the features mentioned above (i.e., `[max_temp, max_dew, max_humid, past_cases]`), the training matrix will be of the shape: `162x40`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to run your implementation Project-Part1 (Example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[945, 897, 832, 881, 907, 921, 1028, 819, 812, 809, 860, 845, 837, 898, 861, 811, 846, 839, 855, 892]\n"
     ]
    }
   ],
   "source": [
    "import submission\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.svm import SVR\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "## Parameters settings\n",
    "past_cases_interval = 10\n",
    "past_weather_interval = 10\n",
    "\n",
    "\n",
    "## Read training data\n",
    "train_file = './data/COVID_train_data.csv'\n",
    "train_df = pd.read_csv(train_file)\n",
    "\n",
    "## Read Training labels\n",
    "train_label_file = './data/COVID_train_labels.csv'\n",
    "train_labels_df = pd.read_csv(train_label_file)\n",
    "\n",
    "\n",
    "## Read testing Features\n",
    "test_fea_file = './data/test_features.csv'\n",
    "test_features = pd.read_csv(test_fea_file)\n",
    "\n",
    "\n",
    "## Set hyper-parameters for the SVM Model\n",
    "svm_model = SVR()\n",
    "svm_model.set_params(**{'kernel': 'rbf', 'degree': 1, 'C': 5000,\n",
    "                        'gamma': 'scale', 'coef0': 0.0, 'tol': 0.001, 'epsilon': 10})\n",
    "\n",
    "\n",
    "## Generate Prediction Results\n",
    "predicted_cases_part1 = []\n",
    "for idx in range(len(test_features)):\n",
    "    test_feature = test_features.loc[idx]\n",
    "    prediction = submission.predict_COVID_part1(svm_model, train_df, train_labels_df, \n",
    "                                                past_cases_interval, past_weather_interval, test_feature)\n",
    "    predicted_cases_part1.append(prediction)\n",
    "\n",
    "\n",
    "print(predicted_cases_part1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project-Part2: (45 Points)\n",
    "\n",
    "In this part, you are required to formulate a model that can improve the performance of the model proposed in the Project-Part1 by a significant margin in terms of `Mean Absolute Error(MAE)`, explained below:\n",
    "\n",
    "<br>\n",
    "\n",
    "<center> $MAE = (\\frac{1}{test\\_interval})\\sum_{i=1}^{test\\_interval}\\left | prediction_{i} - ground\\_truth_{i} \\right |$ </center>\n",
    "\n",
    "<br>\n",
    "\n",
    "For part2, in order to boost the model performance, unlike part1, you are allowed to design any approach and/or propose new model.\n",
    "\n",
    "Specifically, you are required to complete the method `predict_COVID_part2()` in the file `submission.py`. The inputs and outputs alongwith their formats are defined below: \n",
    "\n",
    "\n",
    "**Note:** For part-2, you are only allowed to use the Python libraries already explained above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input and Output formats\n",
    "\n",
    "### Inputs:\n",
    "\n",
    "1. `train_df`, pandas dataframe corresponding the csv file `COVID_train_data.csv`. The format of all fields of the data set is explained above. This dataset is intended for model training.\n",
    "\n",
    "* `train_labels_df`, pandas dataframe corresponding the csv file `COVID_train_labels.csv`. It comprises the number of COVID-19 confirmed cases for each single day.\n",
    "\n",
    "* `test_feature`, A feature vector constructed using a subset of features from the file `test_features.csv` used for predicting the COVID-19 cases of the future.\n",
    "\n",
    "\n",
    "\n",
    "### Outputs:\n",
    "\n",
    "Based on the feature space of the model (i.e., features constructed from the `train_data`), you are required to select the corresponding subset of features from the file `test_features.csv` and predict the probable cases of COVID-19 cases for each single day.\n",
    "\n",
    "**NOTE: You should use math.floor(x) to convert the prediction result to nearest integer.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to run your implementation Project-Part2 (Example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import submission\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.svm import SVR\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "\n",
    "\n",
    "## Read training data\n",
    "train_file = './data/COVID_train_data.csv'\n",
    "train_df = pd.read_csv(train_file)\n",
    "\n",
    "## Read Training labels\n",
    "train_label_file = './data/COVID_train_labels.csv'\n",
    "train_labels_df = pd.read_csv(train_label_file)\n",
    "\n",
    "\n",
    "## Read testing Features\n",
    "test_fea_file = './data/test_features.csv'\n",
    "test_features = pd.read_csv(test_fea_file)\n",
    "\n",
    "\n",
    "## Generate Prediction Results\n",
    "predicted_cases_part2 = []\n",
    "for idx in range(len(test_features)):\n",
    "    test_feature = test_features.loc[idx]\n",
    "    prediction = submission.predict_COVID_part2(train_df, train_labels_df, test_feature)\n",
    "    predicted_cases_part2.append(prediction)\n",
    "\n",
    "print(predicted_cases_part2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error Computation\n",
    "We compare the prediction results for each day against the ground truth values to compute the absolute error of each day. Later, we compute the mean over all the absolute error terms corresponding to the `test_interval` to compute the `MAE`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MeanAbsError =  95.1\n"
     ]
    }
   ],
   "source": [
    "## MeanAbsoluteError Computation...!\n",
    "\n",
    "test_label_file ='./data/COVID_test_labels.csv'\n",
    "test_labels_df = pd.read_csv(test_label_file)\n",
    "ground_truth = test_labels_df['dailly_cases'].to_list()\n",
    "\n",
    "\n",
    "MeanAbsError = mean_absolute_error(predicted_cases_part1, ground_truth)\n",
    "print('MeanAbsError = ', MeanAbsError)\n",
    "MeanAbsError_2 = mean_absolute_error(predicted_cases_part2, ground_truth)\n",
    "print('MeanAbsError = ', MeanAbsError_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "\n",
    "Your implementation will be tested using multiple different training and test data sets. \n",
    "\n",
    "1. For the `project-part1`, we test the correctness of implementation. For a given set of input parameters, you are required to correctly compute the feature vectors and generate the results for the predicted COVID-19 cases.\n",
    "\n",
    "\n",
    "<center>$\n",
    "score_{part1} =  \\begin{cases}\n",
    "    \\sum_{i=1}^{3}15; & \\text{if}\\;\\; \\text{Correctly implemented}\\\\\n",
    "    0,              & \\text{otherwise}\n",
    "\\end{cases} $</center>\n",
    "\n",
    "* For the `project-part2`, we will test your implementation in terms of performance improvement compared to the project-part1. We will be using the following linear function to assign scores:\n",
    "\n",
    "\n",
    "<center>$\n",
    "score_{part2} = \\begin{cases}\n",
    "    \\text{math.floor}(-1.32 * MAE_{avg} + 125.52) &; \\text{if} \\;\\; 61.0 \\leq MAE_{avg} \\leq 95.0\\\\\n",
    "    45 &; \\text{if} \\;\\; MAE_{avg} < 61.0\n",
    "\\end{cases} $</center>\n",
    "\n",
    "where $MAE_{avg}$ is the average of the mean-absolute values over $N$ different test data sets, as shown below.\n",
    "\n",
    "<center>$MAE_{avg} = \\frac{1}{N}\\sum_{i=1}^{N}MAE_{i}$</center>\n",
    "\n",
    "**NOTE: For the project part-2, we will be using the same training data as provided along with the specs (i.e., `train_df`)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BONUS Points \n",
    "\n",
    "We will be awarding BONUS scores to top-20 students with best scores for the part-2. The bonus scores will be awarded in decreasing order of the performance.\n",
    "\n",
    "* The best performing student will be awarded `10 points`.\n",
    "* Second best performing student will be awarded `9.5 points`.\n",
    "* Third best performing student will be awarded `9 points` and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project Submission and Feedback\n",
    "\n",
    "For project submission and feedback, you are required to submit the following files:\n",
    "\n",
    "1. Your implementation in a python file `submission.py`.\n",
    "\n",
    "2. A report `Project.pdf` (**10 points**). You need to write a concise and simple report illustrating:\n",
    "    - Implementation details of part 1.\n",
    "    - Implementation details of part 2. Especially, it should include:\n",
    "        * Comprehensive feature analysis, i.e., which features were used to boost the performance of part 2 compared with part 1, why?\n",
    "        * What additional techniques were used to augment the performance of the model compared to part1?\n",
    "\n",
    "\n",
    "**Note:** \n",
    "1. Every student will be entitled to **15 Feedback Attempts** (use them wisely), we will use the last submission for final evaluation.\n",
    "* We will not provide any feedback for the project-part2.\n",
    "* It is mandatory for the students to submit the report along with the last submission. \n",
    "* **The students failing to submit the report will be penalized by 10 points.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "9318_21t1",
   "language": "python",
   "name": "9318_21t1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "nteract": {
   "version": "0.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}