lecture_10_naive_bayes.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "center",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Naive Baye's Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Naive Baye's classifier is based on Baye's theorem and is used for classification problems. For instance in NLP text classification: topic modeling, sentiment analysis, spam detection etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Bayes' theorem:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Partition of set $X$ on subsets $\\{A_i \\subset X|I \\in I\\}$ is $X = \\bigcup_{i \\in I}A_i$ and $A_i \\cap A_j = \\emptyset$ for every pair $i,j \\in I$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For each subset $A \\subset X$ we have partition $X = A \\cup A^c$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Total probability theorem:\n",
    "For every partition $A_1, A_2 \\dots, A_k$ of $\\Omega$ and event $B \\subset \\Omega$:\n",
    "$$P(B) = \\sum_{i=1}^{k}P(B|A_i)P(A_i)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Theorem (Bayes' theorem):\n",
    "Let $A_1, A_2 \\dots, A_k$ be a partition of $\\Omega$ such that $P(A_i) > 0$ for each $i \\in \\{1, 2, \\dots, k\\}$. Then for $B \\subset \\Omega$ event, such that $P(B) > 0$, for each $i \\in \\{1, 2, \\dots, k\\}$:\n",
    "$$\n",
    "P(A_i|B) = \\frac{P(B|A_i)P(A_i)}{\\sum_{j=1}^{k}P(B|A_j)P(A_j)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Note:\n",
    "We call $P(A_i)$ the prior probability and $P(A_i|B)$ the posterior probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the events $A$ and $B$ such that $P(B) \\gt 0$ we have:\n",
    "$$\n",
    "P(A|B) = \\frac{P(B|A)P(A)}{P(B)}\n",
    "$$\n",
    "<br>\n",
    "We can consider the partition of $\\Omega$ on $A$ and $A^c$, the from Byes' theorem we have:\n",
    "$$\n",
    "P(A|B) = \\frac{P(B|A)P(A)}{P(B|A)P(A) + P(B|A^c)P(A^c)} = \\text{(by the total probability) }\\frac{P(B|A)P(A)}{P(B)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Example:\n",
    "Divide emails $A_1 = \\text{\"spam\"}$, $A_2 = \\text{\"low priority\"}$ and $A_3 = \\text{\"high priority\"}$ and let: $P(A_1) = 0.7$, $P(A_2) = 0.2$ and $P(A_3) = 0.1$. ($P(A_1) + P(A_2) + P(A_3) = 0.7 + 0.2 + 0.1 = 1$)\n",
    "<br>\n",
    "Let $B$ be the event that email contains the word \"free\" and we know from previous experience that: $P(B|A_1) = 0.9$, $P(B|A_2) = 0.01$ and $P(B|A_3) = 0.01$.\n",
    "<br>\n",
    "If we receive the email with word \"free\" in it, what is the probability, that this email is spam?\n",
    "From Bayes' theorem:\n",
    "$$\n",
    "P(A_1|B) = \\frac{P(B|A_1)P(A_1)}{P(B|A_1)P(A_1) + P(B|A_2)P(A_2) + P(B|A_3)P(A_3)} = \\frac{0.9 \\cdot 0.7}{0.9 \\cdot 0.7 + 0.01 \\cdot 0.2 + 0.01 \\cdot .01} = 0.995\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Multi-dimensional case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Example Golf and Weather"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "path = Path('data')\n",
    "nb = path / 'naive_bayes'\n",
    "golf_csv = nb / 'golf.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def strip_txt(txt:str) -> str:\n",
    "    return txt.replace(\"'\", '').strip() if txt else txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(golf_csv, converters={'outlook':strip_txt,\n",
    "                                       'temp': strip_txt,\n",
    "                                       'humidity': strip_txt,\n",
    "                                       'wind': strip_txt,\n",
    "                                       'label': strip_txt})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's calculate each Label probabilities\n",
    "<br>\n",
    "$P(\\text{\"Yes\"}) = 9/14$ and $P(\\text{\"No\"}) = 5/14$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let't conditional probability of outlook 'Sunny' feature with respect of labels\n",
    "<br>\n",
    "$P(\\text{\"Yes\"}|\\text{\"Sunny\"}) = 2/9$ and $P(\\text{\"No\"}|\\text{\"Sunny\"}) = 3/5$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let't conditional probability of outlook 'Overcast' feature with respect of labels\n",
    "<br>\n",
    "$𝑃(\"Yes\"|\"Overcast\")=3/9$  and  $𝑃(\"No\"|\"Overcast\")=0/5$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df.outlook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "y_vals = df[df.label.str.contains('Yes')].count()[0]\n",
    "n_vals = df[df.label.str.contains('No')].count()[0]\n",
    "f_vals = df.count()[0]\n",
    "y_vals, n_vals, f_vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df.outlook.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "sunny_y = df[df.outlook.str.contains('Sunny') & df.label.str.contains('Yes')].count()[0] \n",
    "sunny_n = df[df.outlook.str.contains('Sunny') & df.label.str.contains('No')].count()[0] \n",
    "overcast_y = df[df.outlook.str.contains('Overcast') & df.label.str.contains('Yes')].count()[0] \n",
    "overcast_n = df[df.outlook.str.contains('Overcast') & df.label.str.contains('No')].count()[0] \n",
    "rain_y = df[df.outlook.str.contains('Rain') & df.label.str.contains('Yes')].count()[0]\n",
    "rain_n = df[df.outlook.str.contains('Rain') & df.label.str.contains('No')].count()[0]\n",
    "print(f'sunny_y = {sunny_y}/{y_vals}, sunny_n = {sunny_n}/{n_vals}')\n",
    "print(f'overcast_y = {overcast_y}/{y_vals}, overcast_n = {overcast_n}/{n_vals}')\n",
    "print(f'rain_y = {rain_y}/{y_vals}, rain_y = {rain_y}/{n_vals}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def count_feat(col_name:str, col_val:str) -> int:\n",
    "    return df[df[col_name].str.contains(col_val)].count()[0]\n",
    "\n",
    "def count_cond(col_name:str, col_val:str, lab:str) -> int:\n",
    "    return df[df[col_name].str.contains(col_val) & df.label.str.contains(lab)].count()[0]\n",
    "\n",
    "def count_probs(col_name:str) -> int:\n",
    "    col_vals = df[col_name].unique()\n",
    "    for col_val in col_vals:\n",
    "        val_y = count_cond(col_name, col_val, 'Yes')\n",
    "        val_n = count_cond(col_name, col_val, 'No')\n",
    "        val_f = count_feat(col_name, col_val)\n",
    "        yield val_y, val_n, val_f, col_val\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "col_vals = df.temp.unique()\n",
    "temp_vals = [(ys, ns, fs, vls) for (ys, ns, fs, vls) in count_probs('temp')]\n",
    "temp_vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "col_vals = [(col_name, [(ys, ns, fs, vls) for (ys, ns, fs, vls) in count_probs(col_name)]) \n",
    "            for col_name in df.columns]\n",
    "col_vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "lns = ''\n",
    "for col_val in col_vals:\n",
    "    ln = f'{col_val[0]}: \\n' + '\\n'.join(f'P({nm}) = {f_v}, P({nm}|Yes) = {y_v}/{y_vals}, P({nm}|No) = {n_v}/{n_vals}'\n",
    "                               for y_v, n_v, f_v, nm in col_val[1]) + '\\n'\n",
    "    lns += ln\n",
    "    lns += '===============\\n'\n",
    "print(lns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "model_vals = {col_name: {vls: (ys, ns, fs) for (ys, ns, fs, vls) in count_probs(col_name)} \n",
    "            for col_name in df.columns if col_name != 'label'}\n",
    "model_vals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Outlook: \"Sunny\",\n",
    "Temperature: \"Cool\",\n",
    "Humidity: \"High\",\n",
    "Wind: \"Strong\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "out_v = model_vals['outlook']['Sunny']\n",
    "tmp_v = model_vals['temp']['Cool']\n",
    "hum_v = model_vals['humidity']['High']\n",
    "wnd_v = model_vals['wind']['Strong']\n",
    "yes_raw = (out_v[0] / y_vals) * (tmp_v[0] / y_vals) * (hum_v[0] / y_vals) * (wnd_v[0] / y_vals) * (y_vals / f_vals) \n",
    "no_raw = (out_v[1] / n_vals) * (tmp_v[1] / n_vals) * (hum_v[1] / n_vals) * (wnd_v[1] / n_vals) * (n_vals / f_vals)\n",
    "yes_raw, no_raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "p_x = (out_v[2] / f_vals) * (tmp_v[2] / f_vals) * (hum_v[2] / f_vals) * (wnd_v[2] / f_vals)\n",
    "p_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "yes_pred = yes_raw / p_x\n",
    "no_pred = no_raw / p_x\n",
    "print(f'yes_pred = {yes_pred}, no_pred = {no_pred}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def predict(x:tuple) -> tuple:\n",
    "    out_v = model_vals['outlook'][x[0].capitalize()]\n",
    "    tmp_v = model_vals['temp'][x[1].capitalize()]\n",
    "    hum_v = model_vals['humidity'][x[2].capitalize()]\n",
    "    wnd_v = model_vals['wind'][x[3].capitalize()]\n",
    "    yes_raw = (out_v[0] / y_vals) * (tmp_v[0] / y_vals) * (hum_v[0] / y_vals) * (wnd_v[0] / y_vals) * (y_vals / f_vals) \n",
    "    no_raw = (out_v[1] / n_vals) * (tmp_v[1] / n_vals) * (hum_v[1] / n_vals) * (wnd_v[1] / n_vals) * (n_vals / f_vals)\n",
    "    p_x = (out_v[2] / f_vals) * (tmp_v[2] / f_vals) * (hum_v[2] / f_vals) * (wnd_v[2] / f_vals)\n",
    "    yes_pred = yes_raw / p_x\n",
    "    no_pred = no_raw / p_x\n",
    "    \n",
    "    return yes_pred, no_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "x_vec = ('Sunny', 'Cool', 'High', 'Strong')\n",
    "y_pr, n_pr = predict(x_vec)\n",
    "print(f'yes_pred = {y_pr}, no_pred = {n_pr}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Analysis of algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Learning a Naive Bayes classifier is just a matter of counting how many times each attribute co-occurs with each class\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As you observed algorithm fits more for categorical variables rather than numerical. Also if sample distribution is representative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Advantages\n",
    "- It performs better on multi-class classification. \n",
    "- It performs better on smaller datasets. \n",
    "- If independence is hold, Naive Baye's performs better than Logistic regression on smaller datasets.\n",
    "- It performs better for categorical variables, for numerical variables Gaussian distribution is preffered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Disadvantages\n",
    "- If categorical variable is unseen during the training then we get zero frequency problem and smoothing technique is used (Laplasian Smoothing)\n",
    "- It is known as bad estimator, probabilities should not be tacken in consideration\n",
    "- In real life features are not independent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Main question is how can we deal with zero frequencies. Laplassian smoothing is a little trick:\n",
    "$$P(x_i|C_p) = \\frac{count(x_i, C_p) + 1}{\\sum_{j=1}^{n}(count(x_j, C_p) + 1)}$$\n",
    "<br>\n",
    "or\n",
    "$$P(x_i|C_p) = \\frac{count(x_i, C_p) + 1}{\\sum_{j=1}^{n}count(x_j, C_p) + n}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Theoretical Part"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall conditional probability\n",
    "$$p(C_k \\mid x_1, \\dots, x_n)$$\n",
    "and Baye's theorem\n",
    "$$p(C_k \\mid \\mathbf{x}) = \\frac{p(C_k) \\ p(\\mathbf{x} \\mid C_k)}{p(\\mathbf{x})}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall that for given $C$ the probability $$P(A \\mid C)$$ can be considered as usual probability mesure:\n",
    "- $$P(\\Omega \\mid C) = 1$$\n",
    "- $$P(A \\mid C) \\le 1$$\n",
    "etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The evens $A$ and $B$ are independent if $P(A\\cap B) = P(A)P(B)$ or $P(A, B) = P(A)P(B)$ we can use this formula for conditional independence (implies from ebove property of conditional probability)\n",
    "$$P(A \\cap B | C) = P(A \\mid C)P(B \\mid C)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "By the definition of conditional probability $P(A\\cap B)= P(A \\mid B)P(B)$\n",
    "<br>\n",
    "$$\\begin{align}\n",
    "P(A \\mid B \\cap C) = \\frac{P(B \\cap C \\cap A)}{P(B \\cap C)}= \\frac{P(B \\cap A \\cap C)}{P(B \\cap C)} = \\\\\n",
    "\\frac{P(A \\cap B \\mid C)P(C)}{P(B \\cap C)} = \\\\\n",
    "\\frac{P(A \\mid C)P(B \\mid C)P(C)}{P(B \\cap C)} = \\\\\n",
    "\\frac{P(A \\mid C)P(B \\mid C)P(C)}{P(B \\mid C)P(C)} = P(A \\mid C)\n",
    "\\end{align}$$\n",
    "<br>\n",
    "$$P(A| B \\cap C) = P(A \\mid B, C) = P(A\\mid C)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$\\begin{align}\n",
    "P(C_k, x_1, \\dots, x_n) & = P(x_1, \\dots, x_n, C_k) \\\\\n",
    "                        & = P(x_1 \\mid x_2, \\dots, x_n, C_k) \\ P(x_2, \\dots, x_n, C_k) \\\\\n",
    "                        & = P(x_1 \\mid x_2, \\dots, x_n, C_k) \\ P(x_2 \\mid x_3, \\dots, x_n, C_k) \\ P(x_3, \\dots, x_n, C_k) \\\\\n",
    "                        & = \\dots \\\\\n",
    "                        & = P(x_1 \\mid x_2, \\dots, x_n, C_k) \\ P(x_2 \\mid x_3, \\dots, x_n, C_k) \\dots   P(x_{n-1} \\mid x_n, C_k) \\ p(x_n \\mid C_k) \\ P(C_k) \\\\\n",
    "\\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In general\n",
    "$$P(x_i \\mid x_{i+1}, \\dots ,x_{n}, C_k ) = P(x_i \\mid C_k)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have formula:\n",
    "$$\\begin{align}\n",
    "P(C_k, x_1, \\dots, x_n) & = \\frac{P(x_1 \\mid C_k)P(x_2 \\mid C_k) \\dots P(x_n \\mid C_k)P(C_k)}{P(x)} \\\\\n",
    "& = \\frac{P(x_1 \\mid C_k)P(x_2 \\mid C_k) \\dots P(x_n \\mid C_k)P(C_k)}{P(x_1, x_2, \\dots, x_n)} \\\\\n",
    "&= \\frac{P(x_1 \\mid C_k)P(x_2 \\mid C_k) \\dots P(x_n \\mid C_k)P(C_k)}{P(x_1)P(x_2) \\dots P(x_n)} \\\\\n",
    "\\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Denominator is the same for all classes so classifier will decide by the formula:\n",
    "$$\\hat{y} = \\underset{k \\in \\{1, \\dots, K\\}}{\\operatorname{argmax}} \\ P(C_k) \\displaystyle\\prod_{i=1}^n P(x_i \\mid C_k)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Different types of Naive Baye's"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Multinomial naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let event occures with $(p_1, \\dots, p_n)$ probabilities, then we have:\n",
    "<br>\n",
    "$$P(\\mathbf{x} \\mid C_k) = \\frac{(\\sum_i x_i)!}{\\prod_i x_i !} \\prod_i {p_{ki}}^{x_i}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then our estimator with $\\log$ becomes linear classifier:\n",
    "$$\n",
    "\\begin{align}\n",
    "\\log p(C_k \\mid \\mathbf{x}) & \\varpropto \\log \\left( p(C_k) \\prod_{i=1}^n {p_{ki}}^{x_i} \\right) \\\\\n",
    "                       & = \\log p(C_k) + \\sum_{i=1}^n x_i \\cdot \\log p_{ki}                 \\\\\n",
    "                       & = b + \\mathbf{w}_k^\\top \\mathbf{x}\n",
    "\\end{align}\n",
    "$$\n",
    "<br>\n",
    "where $b = \\log p(C_k)$ and $w_{ki} = \\log p_{ki}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bernoulli naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we have occurrence instead of frequency:\n",
    "$$P(\\mathbf{x} \\mid C_k) = \\prod_{i=1}^n p_{ki}^{x_i} (1 - p_{ki})^{(1-x_i)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Gaussian naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let for some $x$ and class $C_k$ data is distrubuted with normal (Gaussian) distribution.\n",
    "Compute mean and variance of $x$ for each class: $\\mu_k$ and $\\sigma_{k}^{2}$, then fop value $v$:\n",
    "$$p(x=v \\mid C_k)=\\frac{1}{\\sqrt{2\\pi\\sigma^2_k}}\\,e^{ -\\frac{(v-\\mu_k)^2}{2\\sigma^2_k} }$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Excercise:\n",
    "Try The approach for spam classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Excercise:\n",
    "Classify <a href=\"https://archive.ics.uci.edu/ml/datasets/iris\">Iris dataset</a> with Naive Baye's classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Excercise:\n",
    "Classify <a href=\"https://archive.ics.uci.edu/ml/datasets/Wine\">Wine dataset</a> with Naive Baye's classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Excercise:\n",
    "Classify <a href=\"https://archive.ics.uci.edu/ml/datasets/Adult\">Adult dataset</a> with Naive Baye's classifier"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}