Ex1.StdDev.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "42142d60",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float:right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "756ba942",
   "metadata": {},
   "source": [
    "# 1st exercise: <font color=\"#C70039\">Work with standard deviations for anomaly detection</font>\n",
    "* Course: AML\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date:   08.08.2023\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/8/8c/Standard_deviation_diagram.svg/1200px-Standard_deviation_diagram.svg.png\" style=\"float: center;\" width=\"450\">\n",
    "\n",
    "---------------------------------\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information on your tasks (e.g. regarding the set of certain paramaters or a specific computational trick), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation please use English language, too\n",
    "* This applies to all exercises throughout this course.  \n",
    "\n",
    "---------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "This notebook allows you for getting into standard deviations as a common technique to detect anomalies when the data is normally distributed. \n",
    "\n",
    "-------------------------------------------------------------------------------------------------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "The tasks that you need to work on within this notebook are always indicated below as bullet points. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done by using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab or use your local machine.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date. \n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully \n",
    "    * add comments whereever you feel it necessary for better understanding\n",
    "    * run the notebook for the first time.\n",
    "    * understand the output\n",
    "4. go and find three different data sets on the web \n",
    "    * kaggle.com might be a good source (they also offer an API for data download)\n",
    "    * make sure two of the three data sets are normally distributed\n",
    "    * download one data set that is not normally distributed\n",
    "5. visualize the data\n",
    "6. compute the anomalies\n",
    "7. visualize the anomalies\n",
    "8. does the 0,3% rule apply?\n",
    "9. what are differences between the normally distributed and the non-normally distributed data sets with respect to the outlier detection? \n",
    "10. which statement can be made and which cannot? \n",
    "-----------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c30b564",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from numpy.random import randn\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84567403",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Function to Detection Outlier on one-dimentional datasets.\n",
    "def find_anomalies(random_data):\n",
    "    #define a list to accumlate anomalies\n",
    "    anomalies = []\n",
    "    \n",
    "    # Set upper and lower limit to 3 standard deviation\n",
    "    random_data_std = np.std(random_data)\n",
    "    random_data_mean = np.mean(random_data)\n",
    "    anomaly_cut_off = random_data_std * 3\n",
    "    \n",
    "    lower_limit  = random_data_mean - anomaly_cut_off \n",
    "    upper_limit = random_data_mean + anomaly_cut_off\n",
    "    \n",
    "    print(\"lower limit=\", round(lower_limit,8))\n",
    "    print(\"upper limit=\", round(upper_limit,8))\n",
    "    \n",
    "    # Generate outliers list\n",
    "    for outlier in random_data:\n",
    "        if outlier > upper_limit or outlier < lower_limit:\n",
    "            anomalies.append(outlier)\n",
    "    \n",
    "    return anomalies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ea28b74",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# multiply and add by random numbers to get some real values\n",
    "# randn generates samples from the normal distribution (important - see below)\n",
    "data = np.random.randn(50000)  * 20 + 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5cbe6cc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "anomalies = find_anomalies(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f84c895a",
   "metadata": {},
   "source": [
    "## Result\n",
    "These anomalies are exceeding the lower and upper 3rd scatter range.\n",
    "Thus, statistically spoken, they do belong to a population size of less than 0,3% of the entire data set! \n",
    "For sure, the above conclusion is true if and only if the data is normally distributed!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17cf3b07",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(anomalies)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}