notebooks/ce8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computational Exercise 8: Text Pre-Processing\n",
    "\n",
    "---\n",
    "In this exercise, we'll convert sentences from different sections of medical abstracts (e.g. background, methods, etc) into bag of words feature vectors. In a subsequent exercise, we'll then use these feature vectors to develop and test a predictive model.\n",
    "\n",
    "Goals are as follows:\n",
    "\n",
    "- Further improve your understanding of count-based text features\n",
    "- Learn how to convert text data into features that can be used to develop a predictive model\n",
    "\n",
    "We'll begin by importing the usual libraries in addition to `requests`, which will help us load the dataset from url. Later on, we'll also import a new one, the **natural language toolkit (nltk)**, which will help us preprocess our text data.\n",
    "\n",
    "- numpy for efficient math operations\n",
    "- pandas for data and dataframe manipulations\n",
    "- matplotlib for visualization/plotting\n",
    "- requests to load data from url\n",
    "- **nltk for text pre-processing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the dataset\n",
    "\n",
    "We'll be working with the \"PubMed 200k RCT dataset\" dataset developed by Franck Dernoncourt. This dataset contains sentences from different sections of Pubmed abstracts along with labels indicating which section they're from. The sections are:\n",
    "\n",
    "- OBJECTIVE\n",
    "- BACKGROUND\n",
    "- METHODS\n",
    "- RESULTS\n",
    "- CONCLUSIONS\n",
    "\n",
    "Over the next few exercises, our goal will be to develop a classifier that assigns sentences to the correct label. This is not a very useful classifier, but shows that natural language processing is effective even for text with complex terminology, including clinical notes. The training, validation, and test data are found at the following addresses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_url = 'https://github.com/Franck-Dernoncourt/pubmed-rct/raw/master/PubMed_20k_RCT/train.txt?raw=true'\n",
    "val_url = 'https://github.com/Franck-Dernoncourt/pubmed-rct/raw/master/PubMed_20k_RCT/dev.txt?raw=true'\n",
    "test_url = 'https://github.com/Franck-Dernoncourt/pubmed-rct/raw/master/PubMed_20k_RCT/test.txt?raw=true'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll begin by defining a function to read these data. Much like in previous exercises, **the details here are *not* important to our goals;** we just need the data. For now, we'll load only the training data (as `sentences` and `labels`), but in later exercises, we'll reuse these addresses and the function below to load the validation and test sets as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 180040 sentences in the training set\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "def read_pubmed_rct(url):\n",
    "\n",
    "    labels = []\n",
    "    sentences = []\n",
    "    \n",
    "    with requests.get(url) as r:\n",
    "        for line in r.iter_lines():\n",
    "            fields = line.decode('utf-8').strip().split('\\t')\n",
    "            if len(fields) == 2:\n",
    "                labels.append(fields[0])\n",
    "                sentences.append(fields[1])\n",
    "                \n",
    "    return sentences, labels\n",
    "\n",
    "sentences, labels = read_pubmed_rct(train_url)\n",
    "\n",
    "print('There are %i sentences in the training set' % len(sentences))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the dataset\n",
    "\n",
    "It's always a good idea to take a look at a dataset when working with it for the first time. Let's count the number of each label in our training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAAFqCAYAAAC9NCRuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvoElEQVR4nO3debwkVXnw8d8DI4tGkGVAZNBBwSDwxoURMRqWDMgYVFBAx4XFgCQEXLIpxERxwUBIJGIEwys6A2qA4AIRUXAAiXlxYDASFgVGQZiwzMiOOsDA8/5xTktPT3dP3zt9b13u/X0/n/p096mqU6eWrn761KlTkZlIkiRJas5aTRdAkiRJmuoMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJati0pgswEWy66aY5c+bMposhSZKkSe6aa675ZWZO70w3KAdmzpzJokWLmi6GJEmSJrmI+EW3dJuvSJIkSQ0zKJckSZIaZlAuSZIkNcygXJIkSWqYQbkkSZLUMINySZIkqWHjHpRHxHMi4ryI+GlE/CQiXh0RG0fEJRFxS33dqG36YyNicUTcFBF7t6XvFBHX1XGnRETU9HUj4pyavjAiZo73OkqSJEkj0URN+WeA72TmdsBLgZ8AxwALMnNbYEH9TERsD8wFdgDmAKdGxNo1n9OAI4Bt6zCnph8G3J+Z2wAnAyeOx0pJkiRJozWuQXlEbADsCpwBkJmPZeYDwL7A/DrZfGC/+n5f4OzMfDQzbwUWAztHxBbABpl5ZWYmcGbHPK28zgNmt2rRJUmSpIlovGvKXwgsA74UEf8dEV+IiGcBm2fmXQD1dbM6/ZbAHW3zL6lpW9b3nekrzZOZK4AHgU3GZnUkSZKkNTfeQfk04BXAaZn5cuBX1KYqPXSr4c4+6f3mWTnjiCMiYlFELFq2bFn/UkuSJEljaLyD8iXAksxcWD+fRwnS76lNUqivS9um36pt/hnAnTV9Rpf0leaJiGnAhsB9nQXJzNMzc1Zmzpo+ffoQVk2SJEkanXENyjPzbuCOiPjdmjQbuBG4ADikph0CnF/fXwDMrT2qbE25ofOq2sTl4YjYpbYXP7hjnlZeBwCX1nbnkiRJ0oQ0rYFlvhf4SkSsA/wceDflz8G5EXEYcDtwIEBm3hAR51IC9xXAUZn5RM3nSGAesD5wUR2g3ER6VkQsptSQzx2PlZKkQcw85sKmi6Aubjthn6aLIGmKG/egPDN/DMzqMmp2j+mPB47vkr4I2LFL+nJqUC9JkiQ9HfhET0mSJKlhBuWSJElSwwzKJUmSpIYZlEuSJEkNMyiXJEmSGtZEl4iSJElThl2hTjwTsRtUa8olSZKkhhmUS5IkSQ0zKJckSZIaZlAuSZIkNcygXJIkSWqYQbkkSZLUMINySZIkqWEG5ZIkSVLDDMolSZKkhhmUS5IkSQ0zKJckSZIaZlAuSZIkNcygXJIkSWqYQbkkSZLUMINySZIkqWEG5ZIkSVLDDMolSZKkhhmUS5IkSQ0zKJckSZIaZlAuSZIkNcygXJIkSWqYQbkkSZLUMINySZIkqWEG5ZIkSVLDDMolSZKkhhmUS5IkSQ0zKJckSZIaZlAuSZIkNcygXJIkSWqYQbkkSZLUMINySZIkqWEG5ZIkSVLDDMolSZKkhhmUS5IkSQ0zKJckSZIaZlAuSZIkNcygXJIkSWqYQbkkSZLUsHEPyiPitoi4LiJ+HBGLatrGEXFJRNxSXzdqm/7YiFgcETdFxN5t6TvVfBZHxCkRETV93Yg4p6YvjIiZ472OkiRJ0kg0VVO+R2a+LDNn1c/HAAsyc1tgQf1MRGwPzAV2AOYAp0bE2nWe04AjgG3rMKemHwbcn5nbACcDJ47D+kiSJEmjNlGar+wLzK/v5wP7taWfnZmPZuatwGJg54jYAtggM6/MzATO7Jinldd5wOxWLbokSZI0ETURlCdwcURcExFH1LTNM/MugPq6WU3fErijbd4lNW3L+r4zfaV5MnMF8CCwSWchIuKIiFgUEYuWLVs2lBWTJEmSRmNaA8t8TWbeGRGbAZdExE/7TNuthjv7pPebZ+WEzNOB0wFmzZq1ynhJkiRpvIx7TXlm3llflwLfAHYG7qlNUqivS+vkS4Ct2mafAdxZ02d0SV9pnoiYBmwI3DcW6yJJkiQNw7gG5RHxrIh4dus98DrgeuAC4JA62SHA+fX9BcDc2qPK1pQbOq+qTVwejohdanvxgzvmaeV1AHBpbXcuSZIkTUjj3Xxlc+Ab9b7LacBXM/M7EXE1cG5EHAbcDhwIkJk3RMS5wI3ACuCozHyi5nUkMA9YH7ioDgBnAGdFxGJKDfnc8VgxSZIkabTGNSjPzJ8DL+2Sfi8wu8c8xwPHd0lfBOzYJX05NaiXJEmSng4mSpeIkiRJ0pRlUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJalgjQXlErB0R/x0R36qfN46ISyLilvq6Udu0x0bE4oi4KSL2bkvfKSKuq+NOiYio6etGxDk1fWFEzBz3FZQkSZJGoKma8vcDP2n7fAywIDO3BRbUz0TE9sBcYAdgDnBqRKxd5zkNOALYtg5zavphwP2ZuQ1wMnDi2K6KJEmStGbGPSiPiBnAPsAX2pL3BebX9/OB/drSz87MRzPzVmAxsHNEbAFskJlXZmYCZ3bM08rrPGB2qxZdkiRJmogGCsoj4tKI2K7HuBdHxKUjWOY/Ax8EnmxL2zwz7wKor5vV9C2BO9qmW1LTtqzvO9NXmiczVwAPApt0KfcREbEoIhYtW7ZsBMWXJEmShmvQmvLdgQ16jHs2sNsgmUTEG4ClmXnNgMvtVsOdfdL7zbNyQubpmTkrM2dNnz59wOJIkiRJwzdtBNOuEthWLwIeGTCP1wBviog/AtYDNoiILwP3RMQWmXlXbZqytE6/BNiqbf4ZwJ01fUaX9PZ5lkTENGBD4L4ByydJkiSNu5415RHx7oi4IiKuoATkp7c+tw1XU9pv/+cgC8vMYzNzRmbOpNzAeWlmvgu4ADikTnYIcH59fwEwt/aosjXlhs6rahOXhyNil9pe/OCOeVp5HVCX0esPhSRJktS4fjXlTwJP1PfR8bnlXkovKGvaw8kJwLkRcRhwO3AgQGbeEBHnAjcCK4CjMrNVhiOBecD6wEV1ADgDOCsiFlNqyOeuYdkkSZKkMdUzKM/M+dReTCLiMuDIzPzpsBacmZcDl9f39wKze0x3PHB8l/RFwI5d0pdTg3pJkiTp6WCgNuWZucdYF0SSJEmaqga+0TMiNgD+CHg+5SbNdpmZnxhmwSRJkqSpYqCgPCJeA/wH8JwekyRgUC5JkiSNwqD9lP8zcBvwSmC9zFyrY1h7rAooSZIkTXaDNl95CfDWETz0R5IkSdKABq0pvx1YdywLIkmSJE1VgwblHwOOqTd7SpIkSRqiQZuvvAHYHLg1Iq5k1cfWZ2YesupskiRJklZn0KD8tZQeVh4Cdugy3sfYS5IkSaM06MODth7rgkiSJElT1aBtyiVJkiSNkUEfHvT81U2TmbeveXEkSZKkqWfQNuW3sfp24z5ASJIkSRqFQYPyP2bVoHwTYB/ghcAnhlkoSZIkaSoZ9EbPeT1GfToizqIE5pIkSZJGYRg3en6ZUpMuSZIkaRSGEZRvBqw3hHwkSZKkKWnQ3ld27ZK8DrAjcCzwn8MslCRJkjSVDHqj5+WseqNn1NfvA0cOq0CSJEnSVDNoUL5Hl7TlwC8y8+4hlkeSJEmacgbtfeX7Y10QSZIkaaoatKYcgIjYEdgN2Bi4F7giM68fi4JJkiRJU8WgN3pOA+YBb+eptuQAGRFfBQ7NzCeGXzxJkiRp8hu0S8SPAm8FPgJsDaxfXz8CvK2+SpIkSRqFQZuvvAv4RGYe35b2C+D4iFgbeDclcJckSZI0QoPWlD8PuLLHuP9Xx0uSJEkahUGD8juB1/QY9/t1vCRJkqRRGLT5yleAD0fEk/X9XcBzgbnAh4ETx6Z4kiRJ0uQ3aFB+HPBC4GP1fUsA/1bTJUmSJI3CoA8PWgG8IyKOB3al9FN+H/D9zLxxDMsnSZIkTXojenhQZt4A3DBGZZEkSZKmpJ43ekbEiyLimoh4U59p3lSnmTkmpZMkSZKmgH69r3wAeDIzL+g1QR23AnjfkMslSZIkTRn9gvK9gC8OkMcXgdcPpziSJEnS1NMvKH8BMMhNnD8FZg6lNJIkSdIU1C8ofxxYZ4A81qE0YZEkSZI0Cv2C8lvo/RTPdq8Fbh5OcSRJkqSpp19Qfh7w3oh4Ya8JIuJFwNHAvw+7YJIkSdJU0S8o/wxwN3BVRPx5RGwTEc+owzYR8efAD4E7gVPGo7CSJEnSZNTz4UGZ+euImA18Gfgn4B87JglgAXBQZv567IooSZIkTW59n+iZmXcDe0bEK4E9ga3qqDuA72Xm1WNcPkmSJGnS6xuUt9Tg2wBckiRJGgP92pRLkiRJGgfjGpRHxHoRcVVEXBsRN0TEx2r6xhFxSUTcUl83apvn2IhYHBE3RcTebek7RcR1ddwpERE1fd2IOKemL4yImeO5jpIkSdJIjXdN+aPAH2bmS4GXAXMiYhfgGGBBZm5LuXn0GICI2B6YC+wAzAFOjYi1a16nAUcA29ZhTk0/DLg/M7cBTgZOHIf1kiRJkkZtXIPyLB6pH59RhwT2BebX9PnAfvX9vsDZmfloZt4KLAZ2jogtgA0y88rMTODMjnlaeZ0HzG7VokuSJEkT0bi3KY+ItSPix8BS4JLMXAhsnpl3AdTXzerkW1J6emlZUtO2rO8701eaJzNXAA8Cm4zJykiSJElDMO5BeWY+kZkvA2ZQar137DN5txru7JPeb56VM444IiIWRcSiZcuWrabUkiRJ0tjp2SViRNxKl2C2h8zMF41kwZn5QERcTmkLfk9EbJGZd9WmKUvrZEt4qm90KIH8nTV9Rpf09nmWRMQ0YEPgvi7LPx04HWDWrFmDrqckSZI0dP36Kf8+gwflA4mI6cDjNSBfn/JAohOBC4BDgBPq6/l1lguAr0bEp4HnUW7ovCozn4iIh+tNoguBg4HPts1zCHAlcABwaW13LkmSJE1IPYPyzDx0DJa3BTC/9qCyFnBuZn4rIq4Ezo2Iw4DbgQNrGW6IiHOBG4EVwFGZ+UTN60hgHrA+cFEdAM4AzoqIxZQa8rljsB6SJEnS0Az0RM9hycz/AV7eJf1eYHaPeY4Hju+SvghYpT16Zi6nBvWSJEnS08HAN3pGxMsj4usR8cuIWBERr6jpn4qIOaubX5IkSVJ3AwXlEfFaShvt7YCvdsz3JPCnwy+aJEmSNDUMWlN+AvBdypM1/6Jj3I+AVwyzUJIkSdJUMmib8lcAb8nMjIjOnkx+CUwfbrEkSZKkqWPQmvLlwDN7jNuC8tRMSZIkSaMwaFD+A+ADtSvDllaN+WHApUMtlSRJkjSFDNp85e+A/wKuBc6jBOSH1If67AS8cmyKJ0mSJE1+A9WUZ+a1wK7APcCHgQCOrqN3y8ybxqZ4kiRJ0uQ38MODMvNHwOyIWA/YGHggM389ZiWTJEmSpoiBHx7UUp+Y+bgBuSRJkjQcA9eUR8RuwMeBnYF1IuIxYCHwkcy8YozKJ01KM4+5sOkiqIvbTtin6SJIkqaoQZ/oeSClh5XNgJOA9wH/CGwOXBoRB4xZCSVJkqRJbtCa8o8DFwL7ZeaTrcSI+ChwAfAJSq8skiRJkkZo0DblWwOntQfkAPXzqcDMIZdLkiRJmjIGDcpvAab3GDcdWDyc4kiSJElTz6BB+YeBj0XESg8JiohXAccBxw65XJIkSdKU0bNNeUR09qiyHvDDiLiD8hChzYGtgKXAXwPfGqtCSpIkSZNZvxs9nwSy7fNP69Byax0kSZIkrYGeQXlm7j6O5ZAkSZKmrBE/0VOSJEnScA38RE+AiNgI2JbSvnwlPtVTkiRJGp2BgvKIWA/4IvBWIHpMtvawCiVJkiRNJYM2X/k7YHfgEEpQfjRwOPAD4GfAG8aicJIkSdJUMGhQvj/wceDs+nlhZn4pM3cDrgXmjEXhJEmSpKlg0KD8+cANmfkE8DjwrLZxXwTeNuyCSZIkSVPFoEH5vcDv1Pd3AC9tG7cpsP4wCyVJkiRNJYP2vvJD4OXARcDXgE9ExLOBFcBfUtqWS5IkSRqFQYPyEylNWAA+CWxDaWO+NiVg/7PhF02SJEmaGgYKyjNzEbCovn8Y2D8i1gXWzcyHxrB8kiRJ0qQ36id6ZuajmflQROwZEf8zzEJJkiRJU8mog/I2GwI7DCEfSZIkaUoaRlAuSZIkaQ0YlEuSJEkNMyiXJEmSGtaz95WIeOGAeTx3SGWRJEmSpqR+XSIuBnKAPGLA6SRJkiR10S8of/e4lUKSJEmawnoG5Zk5fzwLIkmSJE1VAz3RU2Nn5jEXNl0EdbjthH2aLoIkSZpi7H1FkiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGjWtQHhFbRcRlEfGTiLghIt5f0zeOiEsi4pb6ulHbPMdGxOKIuCki9m5L3ykirqvjTomIqOnrRsQ5NX1hRMwcz3WUJEmSRmq8a8pXAH+ZmS8BdgGOiojtgWOABZm5LbCgfqaOmwvsAMwBTo2ItWtepwFHANvWYU5NPwy4PzO3AU4GThyPFZMkSZJGa1yD8sy8KzN/VN8/DPwE2BLYF2g9rGg+sF99vy9wdmY+mpm3AouBnSNiC2CDzLwyMxM4s2OeVl7nAbNbteiSJEnSRNRYm/LarOTlwEJg88y8C0rgDmxWJ9sSuKNttiU1bcv6vjN9pXkycwXwILBJl+UfERGLImLRsmXLhrRWkiRJ0sg1EpRHxO8AXwM+kJkP9Zu0S1r2Se83z8oJmadn5qzMnDV9+vTVFVmSJEkaM+MelEfEMygB+Vcy8+s1+Z7aJIX6urSmLwG2apt9BnBnTZ/RJX2leSJiGrAhcN/w10SSJEkajvHufSWAM4CfZOan20ZdABxS3x8CnN+WPrf2qLI15YbOq2oTl4cjYpea58Ed87TyOgC4tLY7lyRJkiakaeO8vNcABwHXRcSPa9rfACcA50bEYcDtwIEAmXlDRJwL3EjpueWozHyiznckMA9YH7ioDlCC/rMiYjGlhnzuGK+TJEmStEbGNSjPzB/Qvc03wOwe8xwPHN8lfRGwY5f05dSgXpIkSXo68ImekiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGTWu6AJIkTQUzj7mw6SKow20n7NN0EaTfsqZckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsMMyiVJkqSGGZRLkiRJDTMolyRJkhpmUC5JkiQ1bFyD8oj4YkQsjYjr29I2johLIuKW+rpR27hjI2JxRNwUEXu3pe8UEdfVcadERNT0dSPinJq+MCJmjuf6SZIkSaMx3jXl84A5HWnHAAsyc1tgQf1MRGwPzAV2qPOcGhFr13lOA44Atq1DK8/DgPszcxvgZODEMVsTSZIkaUjGNSjPzCuA+zqS9wXm1/fzgf3a0s/OzEcz81ZgMbBzRGwBbJCZV2ZmAmd2zNPK6zxgdqsWXZIkSZqoJkKb8s0z8y6A+rpZTd8SuKNtuiU1bcv6vjN9pXkycwXwILBJt4VGxBERsSgiFi1btmxIqyJJkiSN3EQIynvpVsOdfdL7zbNqYubpmTkrM2dNnz59lEWUJEmS1txECMrvqU1SqK9La/oSYKu26WYAd9b0GV3SV5onIqYBG7JqcxlJkiRpQpkIQfkFwCH1/SHA+W3pc2uPKltTbui8qjZxeTgidqntxQ/umKeV1wHApbXduSRJkjRhTRvPhUXEvwG7A5tGxBLgo8AJwLkRcRhwO3AgQGbeEBHnAjcCK4CjMvOJmtWRlJ5c1gcuqgPAGcBZEbGYUkM+dxxWS5IkSVoj4xqUZ+bbe4ya3WP644Hju6QvAnbskr6cGtRLkiRJTxcTofmKJEmSNKUZlEuSJEkNMyiXJEmSGmZQLkmSJDXMoFySJElqmEG5JEmS1DCDckmSJKlhBuWSJElSwwzKJUmSpIYZlEuSJEkNMyiXJEmSGmZQLkmSJDXMoFySJElqmEG5JEmS1DCDckmSJKlhBuWSJElSwwzKJUmSpIYZlEuSJEkNMyiXJEmSGmZQLkmSJDXMoFySJElqmEG5JEmS1DCDckmSJKlhBuWSJElSwwzKJUmSpIYZlEuSJEkNMyiXJEmSGmZQLkmSJDXMoFySJElqmEG5JEmS1DCDckmSJKlhBuWSJElSwwzKJUmSpIYZlEuSJEkNMyiXJEmSGmZQLkmSJDXMoFySJElqmEG5JEmS1DCDckmSJKlhBuWSJElSwwzKJUmSpIYZlEuSJEkNMyiXJEmSGjYpg/KImBMRN0XE4og4punySJIkSf1MuqA8ItYGPge8HtgeeHtEbN9sqSRJkqTeJl1QDuwMLM7Mn2fmY8DZwL4Nl0mSJEnqaTIG5VsCd7R9XlLTJEmSpAkpMrPpMgxVRBwI7J2Zh9fPBwE7Z+Z7O6Y7Ajiifvxd4KZxLejksynwy6YLoUa476cu9/3U5b6futz3a+4FmTm9M3FaEyUZY0uArdo+zwDu7JwoM08HTh+vQk12EbEoM2c1XQ6NP/f91OW+n7rc91OX+37sTMbmK1cD20bE1hGxDjAXuKDhMkmSJEk9Tbqa8sxcERFHA98F1ga+mJk3NFwsSZIkqadJF5QDZOa3gW83XY4pxqZAU5f7fupy309d7vupy30/RibdjZ6SJEnS081kbFMuSZIkPa0YlI+TiDg0IrJteCIi/jcizo2I3+0xzxfqtJ/uk29ExDsjYkFE3BsRj0fEkog4OyL2aJvuuJrXtI7594mI5RHx3YhYvy1904g4PiKui4hH6jQ/i4gzI2L3jjzmdazbryPiRxHxx33KPTcivh8RD9Tpr4uIv2kvQ51u95rnnj3yuS0ivtz2ubWdH4iIjTqmnVbHHdcl/9bwm7r9vh0Rh9ebhYciIl5d9/edEfFY3V+XRMQh9Um0releGRFfi4h7IuLRuo6nRsQq/e1HxOW13Gd2GXd4HTezI32Njpm28fMiYkmPcavst4jYICI+FhE3RsSvIuL+ut//NSI2W12+EbFRRPx9RNxUj8f76nG7d5dpR3Qc1PTdIuI7df8sr9vkOxHxzm7rOF5i5XPHi7uMbz+G96xpx8XKx3X78OM6Ta/x7cNtHfmtcixExDZ13KFdxr0uIi6qx9nyiLg5Ik7s3Cd12tvalruizrMwIk7oPIbr9GtFxLsj4qp6LPwqyjnq7IjYeeRbeuIZdPt1bLuMcs7+n4h4b0REx7QZEZ9s+9x5DuwcntMx//Mj4l8i4pZapkci4uqI+HBEbBir/h70GnaPiJn1fasL4/Prvly3x/Z4dt3P8+rnzt/V9uGBoeyECaTL+j5Wj/lPRcR6bdMNvE8j4mVRfm9uj/J7c1dEXBYR7+uS30h/h7fpU/Zew3F1+s0j4pR6zP8mIn4ZEddExGd6HR+TwaRsUz7BHUjptnFt4EXA3wELImKHzHywNVGU4PTA+vGdEfHBzFzRnlGUQO5s4M3AfOCzwH2ULiEPrPlu1J5vx/xvrvNfDByQmY/W9B0pN8oG8C/AIuBxSn/u7wIui4jnZuY9bdktA95U328OvB84IyIeyszzOpb7r8B7gHnAPwC/BnYFjgX2i4g9M/Ohvltx9TYEPgQcM+D076P03PMM4HnAXsDngKMjYq/MXLYmhYmIDwCfBi6t5foFsBHwOuA04AHg/Cj96n8J+AFlG94JvAT4IHBA3Tb/02UR74yIEzLzxtWUY42OmdGqy/0eMBM4Efgx8CxgR+DtlG2+tM/8WwGXARvU+a8BngMcBHwnIj6cmZ/qMutAx0FE7Ad8ndJT09GUbfICynHwR8BXBljNsfYwZX3/riP94Dru2V3meS3wREfar+rrqzvSvwFcCxzXlvboaAoKEBF/AxwPfBM4nLJNd6Lsj/0jYo/MvKNjtu/W5Qdl/76Ccq44OiIOysxvtE37j5Tv7T/XeR4HXkw5tl8FXDXask8Eo9h+rW0H5XvyBuAUYB3gnwZYZOsc2OnhtjLtSvmOLK15X085Z+4CHEXpv/oTwOfb5j8cOIxVj8UbgY07ljWf8jvyBuBrXcpyAPDMOl271u9quxVMXq31fTbleD+2vn9vx3R992lEvBL4T2Ah5Tfmbko30q+t+Z4yxDJfyMrnnFdQfmM7y7gkIjaoZXoSOAn4KeVYeRnwTuCjrMG5aULLTIdxGIBDgQS26Ujfs6a/viP9HTX9wvr6hi55/m0dt3+PZb4OeGZ9f1yddlr9PJfyI/Y14Blt8zwDuLkO03vk+w5g47bP84AlHdP8DnA/8J0e2+H9XfJ9JeWL9qW2tN3r9Hv2KMttwJe75P9dSvDx3LZx0+q44wbJn3ICWQ78xxru+10pJ5dTeox/EfB7lD89y4HzgLU6ptkEWFz3S/v+uhz4b+Ae4Gsd8xxe123mMI6ZLtOust97bVfgD+vnfXtMv1a/fOt63gts3WXek+v23X0NjoMrKIF+9CtbE0PbuswDbm0vI7A+8CDlj1z79u677wb5LnWM65kfsE0dd2hb2h51n5zcZfqtKQHmZYMsn3Iu+S/Kn/cZbev9aLf8J8I+G8I+H9H267PtfgAs7EhL4JNtn1f6rvYp00aUYPxK4Fldxj8L2GuEx87MOu7w+nkdykNpzu9RhssoFRqt++Fa341t+pV9sgy91he4pH4/1hrhPj2TEoiv22Vc+zm5b36dx98g+6VfnsAf13Ev7TIu6HKeniyDzVea16oRfkZH+iGUoPZQ4DeU2rDfitKs4i+BCzOzW40CmXlxZv66Mz0iDga+DJwLvC0zH28bvT+wLfCh7FE7nJlfzcz7+q1UZj5CCSCf3zHqQ8ANdPkHnplXA2cAB0XE8/rlP4DW5dkPjzaDzLySUov9hoh40RqU5RjKj+gHeyznZ1lqvz9AuYLy3sx8smOae4G/oeybt3Rk8SvgU8BbIqLnAx3W5JgZglaN2N09lvtkt3SAiHgVsBtwQmbe2mWSYynflQ91GTfocbAxsDTrWX/Qso2zsyi1969tS3sz5Zjpuj8b9EHKMX9s54i6D08Adq/7tq96LvkzSiD+JzX5WZQAbsTH09PEsLbfQ6z62zJa7wGmU85Pv+ocmZm/ysxL1mQBmfkY5Ure6yNi0/ZxEfF8ynngrG7f0ynuR5Tvx6arm7DDxsD9Wa+St2v4O9Tz9yKrcS7PuDEoH39rR2nTum5EvIQSTC2l1AQCUAPSPYFzamD8TeBNsXI7wlmUy7sjfTBSq9nImcBB2dEkBphNucT4nRHmu5LaXGEr4Gdtac8DtqPUPPf6Ul1ACTJ2W5PlA3dRmt4cEREvWIN8Wl1rvmY0M9ftsDtwcWYuX83ks4FFmXlXj/EXUmrP/rDLuM8Dt/NUENrNaI+ZYfgR5XLyv0bEm6NLm+I+ZtfXruWu2/USYNdoa5tfDXocXAW8LiI+GRG/F7FyO9wJ4heUGv2D2tIOpjQ7eaTHPK3zTfuwJuf9VfKjfF9/q6btBlzS55hv7ctux/IqMvNaSlOu19TPv6RcNfiriPjTGrBNCmuw/aJtv2xUK1/2As4ZcNFrdTlW2vftnsDdmbloJOszCvMpfyTe1pH+Lkot6Sr3zzD84/zpZiblitm9Hemr26dXAdtFxOcjYufocf9QA1pNz86OiL0j4lmNlmYcTaWDdqL4KaXZyHJKm7qXUJqmtLehPoiyb1onn/nAuqx8ktqqvv5ihMs/ldKG7LAe/4RnAMsy8zftiVFuqmr/Yq8StLSNex7wGUp73vYgsVXm2/qUrzVuqz7TDOpEylWGj65BHrfX1y1GOf+mlBqMQfbTVvTZNrV2ahldtk2t6fg4sHdt99krfwYsy1Bl5s+BIylNHb4O3BsRN0TESQNcFRn0uHkmpZlPp0GOg2MoTSQ+TGlX/UBEfDMi3rqaso23M4EDI2K9iNiCEih1C1JallPON+3DmrQT7ZbfTzum2YRyzN/WJ5/WuJF8z29n5e/hOyhXiU4DfhHlxvkz4ul/k+dot987eGqf3Ef53fgS5b6dQXyXVffttW3j+56fhqVeMb2RjqvDlN/FKzPz5i6ztX5X24fJ/CTv1p+QjaJ0qLA/8LeZ2Xn/yOr26UmUSr8/obThfigiLo6I93Sp4Bg3mXkF5Xz9GkoF4YMRsSjKDefPaapc42Gi/CuaSt5MuUEjKDe3HQ18OyJ2zcyf1GkOBm6pzSeg3CB3Z03/PGvmQsqNa4dSTtidetUQfhto7+XiPcAX2j5vSfnCtyTw9sxcOEDegyx/xDLzvoj4J+CjEXEibbX2I9Aqz0S5XNZv+8yjXPY+HviDcSnNCGTmFyLia5Tj7w8otYF/BRweEa/N3k/eXaPjZpDjIDOXUmraXwnModwouCewb73R9z0DlGE8/Dul5v+NlKYsdwMLKPctdLMLq97o2fOG2gF0y28Gpba+ZayuMgRt38PM/GGUnqv2oOyrV1Oa/R0aEe/OzH5/Viay0W6/i4CP1PfPBHaunx+n3IS5Okex6s2xv+k24Tg4EzghIl6cmTfXP1rbUf7Yd9P6XW33wBiWr2mdf4RPzcx/6TJd331aK9/eHBHbU87Lr6Z8n/aidCwwp6mmIpn58Yg4HdiH0mRvd0qgfnhE7JQrdzQxaRiUj7/rM3Nx60NEXAzcQbkZ5m01KNgeOLHjH+HXKT0QvLjWFLTuuh9p04y3UtqffiEiHs3Mr3aMvwPYMyLW76gtfy+l5nsLutdALKV8edai3Lj4SeCLEXFtZrZOIK0yz+xTvtb6tKZtNa/p9a99bfrfZX9yLfvHKXdtj1SrJqpXk5LVuZdyEhxkPy2hz7apl/A25alts5LMfCIiPkK55Pf6LpOM9pjpZQX990trmt/KzPspPZl8BSAi9qUc2x+j9KzQTftxc1OPaV5A2c6dl29bBjoOai3d1bVsz6HcdHt4RHwmM6/vNd94ycyHI+KblFrDmcBXMvPJPq1trunSRG1NrJJfrNr13C8p+2Jmn3xa47oeyz1sRUcwUq8QfacOROky8jJKT0dP16B8tNvvvo6mJVfUK5r/EBGfy9X0zATcvJqmKXcA/2c1eQzLlylNOw+m3Jx+MOXG3l5NcVb6XZ0CWn9CpgN/AfxZRCzs8kd0dfsUgHps3AgQpWvF/0tpLrQP8C3W/Hd4VDLzbsp9ZmfUsh1FqZT4a0qFzqRj85WG1cD355TeN6DU9EC5ae3+tuHomt66pLeIUhPwxhEu8jHKF/py4MyI6AyELqV8weZ0lPOW+uW+rke+j2fmosy8KjP/jfKvexptXXFl5v9Sgqo39mmz+yZKTdz36+dWrd4qTRxq+7fNKD2PdFVvEvt7ShdSL+s1XR/71Nf/GsW81ADmcmCvWH3fqguAWbVZQq+yrEXZR72cS+lu8JOsWuM22mOml6XAptG9L/fW/upbm5GZ51Mup27fZ7IF9fVN3UbWH5G9gO93uXzbWs6Ij4PMfICnmnr0K994O5NyLPwfJmDgWY/5KyjH/Ho9Jmvty37H8m9FxMsox9QPVrPsmymB2ybR1vf908mQt1/r6tMwgunvAc+NiJ2GkFdf9bfie8C76vnlbcAF9U+9yp+QRZl5EaX7yJuBk4bR9rrex3BS/dg6763R7/CwZObnKPHQRDofD5VBecMi4pmUmuVl9eQzl9K2a48uw48pPZNEvUv9nyg9g+zfI++9av4rqV+6NwL/D/hqRLQHaV+jXN4/MSKmj3a9MvMmSh+kf1Rr/1tOAnZg1f5UW32mHkap/buzJt9CqRHo7HEEysloHUrNWD+nAv9L/5sgVxERr6a0tftmbRM9WidQ2ome1G1kRGwdEb9HaYf/JPDZzpuUImJjSs3RYkrNclf1UuPfUvqA3b9j3KiPmR4uo/zx6hYs70+5unBTzXfTbgFG/RHZij5XIjLzh5T7II6JiK27TPL3lLv1u27fNj2Pgyj9oHezXX0d7ZWSsXAJ5c/X5/s0+WnaSZRjfpW+4+s+/BBwRUfztq4i4nco55JfA/9a054REZv3mGU7Sk3zUPvaH2fD2n6typ41es5C9QVKLf6/dAv+IuKZ0ePhMqM0n3IF7O8pVwgn3B/QiaBeLfprSmD8ZyOZNyJm9BjVed4bxu/wSMr13G43ndYKqw2ZWOfjobL5yvh7WZSunoLSFORoSkDxWcrBvQnwl5l5eeeMUR66cxqlbdVllJPVS4Fzojzh7D8oN/jMoARFb6H0LbuKzPx1ROxDeXDQv0fEvpn53cx8LCLeQrlB5McR8TnK5fzHgOfyVKD3cLd8O5wAHEFp1/jGutwzIuL3gX+OiJdS/gT8htLG+K8oD6N4f1s5MyKOBc6q7ZG/Sunmaxali8BLa1l7ysxHI+LjwOl9JntJRDxC+U5sQemv+yDKJb01ak+cmVdExF8An47S4848yk1rG1F6FjkceEdmnh8Rf0L58VsQEZ+nnHy2o7QVfw6lH+DHV13KSsu7MCL+i5XvAWgZzTHzlojovCn4LkpN1iXAvIjYjvJn8tmUP5b7Au9uu5l4d+C0usz/pNTYv4Dy52xjSnODfg6iHPM/jIh/oNT6P4dy5egtwEcys2+t4WqOg29HxFJKLetNlBvtdqV0IXklo7xSMhbq1YC3Dzj5qyKi8+rBE7WZzpjJzAW1KdXHozyN80xKDdcrKDfVPsjKvci0bBoRu1DOjxvy1MODplPuUWn9Wd8QuD0izqXc77KEcu6cC7we+Idu3bw9XYxy+7W2HZTj91U8dePyFZ2L6LLY1jmw03VZuju8r/6ZvwD4UUR8lqceHrQz8KeU5l7fG9HK9vYNyrn+zyk1tf16BGv9rnZaNOTmWxNSZl4QEVdTeiNqb1ved58Cn69/bs+i7Mu1Kc8L+SClcu4bNf/R/g7PiYjObg0fzNV3nfku4H0R8SXgh5Q/5C+mnI8fo/xJn5xyAnSWPhUGnupMv31YSjmY967TnE850J/ZI48NKQfnvLa0oD5lk3LSfpzyA/VvwB+0TXccXR7gUPNcVPPdoy19OiWAu76OW075ks4Hdu3IYx69HyLzqbrcl3ekv4PyQ/EQJSi/nlLD22vd30hp0vIw5Ut5S817/R7bufPhCtMol/iS7g8Pag3LKbWp36bU2q8zxGPg9yk36t3FUz0kXFz3X/uDGnahnAyX1XX9BeUG36265Hk58IMu6bu1rdPMjnEjPWa6Dd+q06xHqXm+mdLm82FK0L1vxzJn1ONpIeW4f7yu37eBPxzkeKIE7yfWZS2vZb+YjgdvjfI4eBul9vlnlOP9N5Q/ZJ8Cnj1Bzh0DP4hjNfvukR553MaQHh7UNm4O5cf6/np83EKpBd64x/JbZXyiznM15c/9CzqmXYdSO3hxPXYfo5xLrqRUBEyKh4sMuv06tl3rPHYTpeeV9ge9rV/H/22XY6fXMKtjWS+gtOv9WS3TI3U/HQNsMMJjZ2Ydd3iP9f9CHX/yar4bvYZNm96HQz4eep4LKBVJSfkTM9A+pVTczK/HysN1f/6s7t/NuyxjpL/D3YbrO467bg8PegnlPqD/ptwn9Djld/M84BVN74exHFpPxZIkSZNYbSZ3LeUZFV9uujySVmbzFUmSJrF6f9BOlCeEPsIaPhxO0tjwRk9Jkia3fShN56ZRmgv8suHySOrC5iuSJElSw6wplyRJkhpmUC5JkiQ1zKBckiRJaphBuSRJktQwg3JJkiSpYQblkiRJUsP+P1nnyV864HKaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lbl, count = np.unique(labels, return_counts=True)\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.bar(np.arange(5), count)\n",
    "plt.xticks(np.arange(5), lbl, fontsize=16)\n",
    "plt.ylabel('Label Count', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also take a look at the first few sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['To investigate the efficacy of 6 weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at 12 weeks in older adults with moderate to severe knee osteoarthritis ( OA ) .',\n",
       " 'A total of 125 patients with primary knee OA were randomized 1:1 ; 63 received 7.5 mg/day of prednisolone and 62 received placebo for 6 weeks .',\n",
       " 'Outcome measures included pain reduction and improvement in function scores and systemic inflammation markers .']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the Natural Language Toolkit (NLTK) for text processing\n",
    "\n",
    "We can now import NLTK. We'll first make sure it's installed, since it's not part of the Anaconda base environment. We'll also import:\n",
    "- `word_tokenize`, which splits a sentence into a list of *tokens* (e.g. words, numbers, punctuation)\n",
    "- `stopwords`, a list of commonly used words that we can safely ignore when processing our text\n",
    "- `PorterStemmer`, which will convert words into stems, as described in the lecture and shown in an example below\n",
    "\n",
    "We'll also download lists of punctuation ('punkt') and stopwords ('stopwords'), then create `sw`, a set containing all the stopwords, and `ps`, an instance of `PorterStemmer` that we can apply to our words. **Before moving on, take a look at** `sw` **and try out** `word_tokenize` **on a few different sentences.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in /Users/mme/miniforge3/lib/python3.9/site-packages (3.6.5)\r\n",
      "Requirement already satisfied: regex>=2021.8.3 in /Users/mme/miniforge3/lib/python3.9/site-packages (from nltk) (2021.10.23)\r\n",
      "Requirement already satisfied: joblib in /Users/mme/miniforge3/lib/python3.9/site-packages (from nltk) (1.0.1)\r\n",
      "Requirement already satisfied: tqdm in /Users/mme/miniforge3/lib/python3.9/site-packages (from nltk) (4.61.2)\r\n",
      "Requirement already satisfied: click in /Users/mme/miniforge3/lib/python3.9/site-packages (from nltk) (8.0.3)\r\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /Users/mme/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /Users/mme/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "!pip install nltk\n",
    "import nltk\n",
    "\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import PorterStemmer\n",
    "\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "\n",
    "sw = set(stopwords.words('english'))\n",
    "ps = PorterStemmer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other useful string methods\n",
    "\n",
    "We'll also need to convert tokens to lowercase using the string method `.lower()`, and we can check whether a given string contains only letters with the string method `.isalpha()`. However, since we don't want to discard contractions, we can apply `.isalpha()` after first removing at most one apostrophe, as shown in the example below. Note that we can create strings that contain apostrophes (i.e. single quotes) by enclosing them in double quotes (rather than single). To summarize, we can:\n",
    "- convert a string to lower case with `.lower()`\n",
    "- check if a string contains only letters with `.isalpha()`\n",
    "- check if string contains only letters and no more than one apostrophe with `.replace(\"'\", \"\", 1).isalpha()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"shouldn't\", False, True)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# String methods in action\n",
    "\n",
    "s = \"SHOULDN'T\"\n",
    "\n",
    "s.lower(), s.isalpha(), s.replace(\"'\", \"\", 1).isalpha()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also take a quick look at stemming. If we stem the words 'coding', 'coded', and 'code', we end up with the same stem, 'code', each time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('code', 'code', 'code')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ps.stem('coding'), ps.stem('coded'), ps.stem('code')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8.1: Process a single sentence\n",
    "\n",
    "We now have all the tools we need to convert a sentence into a list of stemmed words that we can use to build our feature vectors. In the code block below, you should add lines to the `tokenize` function that will:\n",
    "- remove stopwords\n",
    "- convert words to lowercase\n",
    "- step all words\n",
    "\n",
    "Each of these steps will require a list comprehension (or a `for` loop). Optionally, try completing *all* steps with a single list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(sentence):\n",
    "    \n",
    "    ### (done) SPLIT SENTENCE INTO WORDS USING word_tokenize ###\n",
    "    words = word_tokenize(sentence)    \n",
    "    \n",
    "    ### (done) REMOVE WORDS THAT CONTAIN CHARACTERS OTHER THAN LETTERS AND AN APOSTROPHE ###\n",
    "    words = [w for w in words if w.replace(\"'\", '', 1).isalpha()]    \n",
    "    \n",
    "    ### REMOVE STOPWORDS ###\n",
    "    \n",
    "    \n",
    "    ### CONVERT WORDS TO LOWERCASE ###\n",
    "    \n",
    "    \n",
    "    ### STEM ALL WORDS ###\n",
    "    \n",
    "    \n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['To',\n",
       " 'investigate',\n",
       " 'the',\n",
       " 'efficacy',\n",
       " 'of',\n",
       " 'weeks',\n",
       " 'of',\n",
       " 'daily',\n",
       " 'oral',\n",
       " 'prednisolone',\n",
       " 'in',\n",
       " 'improving',\n",
       " 'pain',\n",
       " 'mobility',\n",
       " 'and',\n",
       " 'systemic',\n",
       " 'inflammation',\n",
       " 'in',\n",
       " 'the',\n",
       " 'short',\n",
       " 'term',\n",
       " 'and',\n",
       " 'whether',\n",
       " 'the',\n",
       " 'effect',\n",
       " 'would',\n",
       " 'be',\n",
       " 'sustained',\n",
       " 'at',\n",
       " 'weeks',\n",
       " 'in',\n",
       " 'older',\n",
       " 'adults',\n",
       " 'with',\n",
       " 'moderate',\n",
       " 'to',\n",
       " 'severe',\n",
       " 'knee',\n",
       " 'osteoarthritis',\n",
       " 'OA']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenize(sentences[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8.2: Process all sentences\n",
    "\n",
    "You can now use a single list comprehension to apply `tokenize` to *all* of the sentences, resulting in a list of 180,040 stemmed, tokenized sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ADD YOUR CODE BELOW ###\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8.3: Create your vocabulary\n",
    "\n",
    "We're now ready to create our vocabulary using the approach described in the [bag of words lecture](https://github.com/mengelhard/bsrt_ml4h/blob/master/lectures/al10.pdf). You'll need to complete the following steps:\n",
    "- Put the stemmed tokens from *all* sentences together in a single list or array. This can be done with a list comprehension or `np.concatenate`.\n",
    "- Count the number of occurrences of each distinct token. This can be done with `np.unique` (use `return_counts=True`) or `pd.value_counts`.\n",
    "- Remove those that occur fewer than 50 times. This can be done using boolean indexing: if we have the arrays `words` and `word_counts`, for example, we can write `vocabulary = words[word_counts >= 50]`. Later on, we'll explore how making this number larger or smaller affects model performance.\n",
    "\n",
    "The resulting list (or array) is your vocabulary, which defines the features for our bag of words model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ADD YOUR CODE BELOW ###\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a function to create features\n",
    "\n",
    "Finally, we can use (a) the vocabulary, and (b) our list of stemmed, tokenized sentences to create numeric features corresponding to each sentence. The block below defines a function `create_features` and shows how it can be applied to a sample list of tokenized sentences along with a sample vocabulary. **You do not need to make changes to this block, but please take a look at the code and verify that it is creating feature vectors using the approach described in our lecture.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1., 1., 0., 0., 0., 0.],\n",
       "       [0., 1., 1., 0., 1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def create_features(tokenized_sentences, vocabulary):\n",
    "    \n",
    "    vocab_dict = {v:i for i, v in enumerate(vocabulary)}\n",
    "    \n",
    "    features = np.zeros((len(tokenized_sentences), len(vocabulary)))\n",
    "    \n",
    "    for i, tokenized_sentence in enumerate(tokenized_sentences):\n",
    "        for word in tokenized_sentence:\n",
    "            if word in vocabulary:\n",
    "                features[i, vocab_dict[word]] += 1\n",
    "            \n",
    "    return features\n",
    "\n",
    "\n",
    "sample_sentences = [\n",
    "    ['a', 'smart', 'science', 'fiction', 'film'],\n",
    "    ['unlike', 'anything', \"we've\", 'seen', 'in', 'science', 'fiction', 'before']\n",
    "]\n",
    "\n",
    "sample_vocab = ['smart', 'science', 'fiction', 'film', 'unlike', 'anything', 'seen', 'before']\n",
    "\n",
    "\n",
    "create_features(sample_sentences, sample_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8.4: Create the feature vectors\n",
    "\n",
    "*Your* list and vocabulary have been stemmed, so they'll look different than those in the example above. What's important is that the format of tokens in the vocabulary matches the format in the tokenized sentences, which should be the case if you've followed the steps outlined above.\n",
    "\n",
    "In the block below, apply `create_features` to your tokenized sentence list and vocabulary to create `x_train`, which we'll use to train a predictive model in our next computational exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Once you've completed these exercises, please turn in the assignment as follows:\n",
    "\n",
    "If you're using Anaconda on your local machine:\n",
    "- download your notebook as html (see File > Download as > HTML (.html))\n",
    "- submit the resulting file as your assignment\n",
    "- if .html files are not accepted, .zip the file (i.e. place it in a .zip archive) and submit the .zip file instead\n",
    "\n",
    "If you're using Google Colab:\n",
    "- download your notebook as .ipynb (see File > Download > Download .ipynb)\n",
    "- if you have nbconvert installed, convert it to .html; if not, leave is as .ipynb\n",
    "- submit the resulting file as your assignment\n",
    "- if this file is not accepted, .zip the file (i.e. place it in a .zip archive) and submit the .zip file instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}