04-TrainTestSplit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train Test Split\n",
    "\n",
    "To make sure that the classifier doesn't simply \"remember\" all the data, we will put some data aside before we train the classifier. This way, we can ensure that the algorithm actually extracts useful rules and is not only remembering examples.\n",
    "\n",
    "We will do this by using the function [`sklearn.model_selection.train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html), which allows us to get well sampled training and test sets.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('train.csv')\n",
    "df = df.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "features = [\"Pclass\", \"Age\", \"SibSp\", \"Parch\", \"Fare\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us train the algorithm with the training set and and score it on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see the accuracy we can expect on unknown data is only 0.65%. Given that the probability to die is $1 - 0.38 = 0.62$ we could have achieved the same accuracy saying everyone will die. Not to good is it?\n",
    "\n",
    "This occurs if the algorithm learns noise. E.g. everyone of us would say learning both of the following rules will not generalize well: \n",
    "```\n",
    "if age > 29 -> dead\n",
    "if age > 31 -> survived\n",
    "```\n",
    "We would clearly see that there was simply one person of age 30 in the dataset that died, but there is (probably) no general knowledge extractable from this.\n",
    "```\n",
    "if age > 15 -> dead\n",
    "if age > 60 -> survived\n",
    "```\n",
    "Would be a completely different story though! Children are normally more likely to get a spot in a rescue boat (and maybe there grandmas and grandpas, too.\n",
    "\n",
    "\n",
    "This behaviour is called overfitting. This is the case if the training score is higher than the test score. On the otherhand there is underfitting. This is the case if the training score and the test score are nearly equal and both are low. In this case the algorithm is not complex enough to grasp all realtions in your data, or, there are simply no such connections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "So how can we restrict the algorithm to learn more generally applicable knowledge? Turns out each algorithm has a lot of parameters we can adjust. Let us look back to the documentation of the [`\"DecisionTreeClassifier\"`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier) and try to restrict it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well this makes it better, but not a lot. What else can we do?\n",
    "If you remember back, one of our first decisions was to discard all data which are not numeric. \n",
    "Let us have a look at [feature extraction and engineering](./05-FeatureEngineering.ipynb) to make use of the remaining columns."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}