notebooks/ce9.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computational Exercise 9: Bag of Words Models\n",
    "\n",
    "**Please note that (optionally) this exercise may be completed in groups of 2 students.**\n",
    "\n",
    "---\n",
    "In this exercise, we'll use these feature vectors we constructed in the previous exercise ([CE8](https://github.com/mengelhard/bsrt_ml4h/blob/master/notebooks/ce8.ipynb)) to develop and test a predictive model.\n",
    "\n",
    "Goals are as follows:\n",
    "\n",
    "- Fully implement a bag of words model\n",
    "- Explain the model's predictions\n",
    "- Continue to gain experience with the model development process\n",
    "- Explore how hyperparameter settings affect performance\n",
    "\n",
    "We'll begin by importing the usual libraries in addition to `requests`, which will help us load the dataset from url. Later on, we'll also import a new one, the **natural language toolkit (nltk)**, which will help us preprocess our text data.\n",
    "\n",
    "- numpy for efficient math operations\n",
    "- pandas for data and dataframe manipulations\n",
    "- matplotlib for visualization/plotting\n",
    "- requests to load data from url\n",
    "- **nltk for text pre-processing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in /Users/mme/miniforge3/lib/python3.9/site-packages (3.6.5)\r\n",
      "Requirement already satisfied: click in /Users/mme/miniforge3/lib/python3.9/site-packages (from nltk) (8.0.3)\r\n",
      "Requirement already satisfied: joblib in /Users/mme/miniforge3/lib/python3.9/site-packages (from nltk) (1.0.1)\r\n",
      "Requirement already satisfied: tqdm in /Users/mme/miniforge3/lib/python3.9/site-packages (from nltk) (4.61.2)\r\n",
      "Requirement already satisfied: regex>=2021.8.3 in /Users/mme/miniforge3/lib/python3.9/site-packages (from nltk) (2021.10.23)\r\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /Users/mme/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /Users/mme/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import requests\n",
    "\n",
    "!pip install nltk\n",
    "import nltk\n",
    "\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import PorterStemmer\n",
    "\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "\n",
    "sw = set(stopwords.words('english'))\n",
    "ps = PorterStemmer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and preprocess the dataset\n",
    "\n",
    "In the following block, we'll prepare the PubMed 200k RCT dataset for model development. Please review [CE8](https://github.com/mengelhard/bsrt_ml4h/blob/master/notebooks/ce8.ipynb) if/as needed to understand this process. **Please note that this block may take a few minutes to run.**\n",
    "\n",
    "Steps:\n",
    "1. Load and tokenize all sentences (train, val, test)\n",
    "2. Create the vocabulary (**note: you may want to revisit this part later on**)\n",
    "3. Create features based on the sentences + vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 180040 sentences in the training set\n",
      "There are 30212 sentences in the validation set\n",
      "There are 30135 sentences in the test set\n"
     ]
    }
   ],
   "source": [
    "# STEP ONE: LOAD AND TOKENIZE\n",
    "\n",
    "train_url = 'https://github.com/Franck-Dernoncourt/pubmed-rct/raw/master/PubMed_20k_RCT/train.txt?raw=true'\n",
    "val_url = 'https://github.com/Franck-Dernoncourt/pubmed-rct/raw/master/PubMed_20k_RCT/dev.txt?raw=true'\n",
    "test_url = 'https://github.com/Franck-Dernoncourt/pubmed-rct/raw/master/PubMed_20k_RCT/test.txt?raw=true'\n",
    "\n",
    "import requests\n",
    "\n",
    "def tokenize(sentence):\n",
    "    return [\n",
    "        ps.stem(w.lower())\n",
    "        for w in word_tokenize(sentence)\n",
    "        if w.replace(\"'\", \"\", 1).isalpha() and (w not in sw)\n",
    "    ]\n",
    "\n",
    "def read_and_tokenize_pubmed_rct(url):\n",
    "\n",
    "    labels = []\n",
    "    tokenized_sentences = []\n",
    "    \n",
    "    with requests.get(url) as r:\n",
    "        for line in r.iter_lines():\n",
    "            fields = line.decode('utf-8').strip().split('\\t')\n",
    "            if len(fields) == 2:\n",
    "                labels.append(fields[0])\n",
    "                tokenized_sentences.append(tokenize(fields[1]))\n",
    "                \n",
    "    return tokenized_sentences, labels\n",
    "\n",
    "s_train, y_train = read_and_tokenize_pubmed_rct(train_url)\n",
    "print('There are %i sentences in the training set' % len(s_train))\n",
    "\n",
    "s_val, y_val = read_and_tokenize_pubmed_rct(val_url)\n",
    "print('There are %i sentences in the validation set' % len(s_val))\n",
    "\n",
    "s_test, y_test = read_and_tokenize_pubmed_rct(test_url)\n",
    "print('There are %i sentences in the test set' % len(s_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "### STEP ONE AND A HALF: CONVERT THE LABELS TO INTEGERS\n",
    "\n",
    "sections = ['BACKGROUND', 'OBJECTIVE', 'METHODS', 'RESULTS', 'CONCLUSIONS']\n",
    "section_to_idx = {s: i for i, s in enumerate(sections)}\n",
    "\n",
    "y_train = [section_to_idx[l] for l in y_train]\n",
    "y_val = [section_to_idx[l] for l in y_val]\n",
    "y_test = [section_to_idx[l] for l in y_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 3986 words in our vocabulary\n"
     ]
    }
   ],
   "source": [
    "# STEP TWO: CREATE THE VOCABULARY\n",
    "\n",
    "MIN_COUNT = 50\n",
    "\n",
    "vcs = pd.value_counts([w for s in s_train for w in s])\n",
    "vocabulary = vcs.index.values[vcs >= MIN_COUNT]\n",
    "print('There are %i words in our vocabulary' % len(vocabulary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The training set has shape (180040, 3986)\n",
      "The validation set has shape (30212, 3986)\n",
      "The test set has shape (30135, 3986)\n"
     ]
    }
   ],
   "source": [
    "# STEP THREE: CREATE FEATURES\n",
    "\n",
    "def create_features(tokenized_sentences, vocabulary):\n",
    "    \n",
    "    vocab_dict = {v:i for i, v in enumerate(vocabulary)}\n",
    "    \n",
    "    features = np.zeros((len(tokenized_sentences), len(vocabulary)))\n",
    "    \n",
    "    for i, tokenized_sentence in enumerate(tokenized_sentences):\n",
    "        for word in tokenized_sentence:\n",
    "            if word in vocabulary:\n",
    "                features[i, vocab_dict[word]] += 1\n",
    "            \n",
    "    return features\n",
    "\n",
    "x_train = create_features(s_train, vocabulary)\n",
    "print('The training set has shape', x_train.shape)\n",
    "\n",
    "x_val = create_features(s_val, vocabulary)\n",
    "print('The validation set has shape', x_val.shape)\n",
    "\n",
    "x_test = create_features(s_test, vocabulary)\n",
    "print('The test set has shape', x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 9.1: A first bag of words model\n",
    "\n",
    "In this part of the exercise, you should create a logistic regression model that predicts the PubMed abstract section associated with a given sentence. Then, evaluate it on the **validation** set. We'll save the test set for later. This is going to take a while; you may want to either (a) limit the number of iterations, or (b) train on only a subset of the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "### CREATE AND TRAIN THE MODEL ###\n",
    "\n",
    "\n",
    "### EVALUATE ACCURACY ON THE VALIDATION SET ###\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 9.2: Important words\n",
    "\n",
    "Now, we can inspect the parameters of our trained model to determine which words increase the log-odds most for a each section. The parameters can be accessed via the `.coef_` attribute of the trained model. Similar to activity 10, we can use a `pandas` series to sort words in our vocabulary.\n",
    "\n",
    "The block below contains code to determine which words increase the log-odds of the 'BACKGROUND' section most. Note that you'll need to change `model` to the name of your model from the previous code block. In this block, you should extend the code to the remaining four sections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sort_arr_by_vals(arr, vals):\n",
    "    return pd.Series(vals, index=arr).sort_values(ascending=False)\n",
    "\n",
    "### DETERMINE WHICH WORDS ARE MOST PREDICTIVE OF BACKGROUND ###\n",
    "#sort_arr_by_vals(vocabulary, lr_model.coef_[0])\n",
    "\n",
    "### DETERMINE WHICH WORDS ARE MOST PREDICTIVE OF OBJECTIVE ###\n",
    "\n",
    "\n",
    "### DETERMINE WHICH WORDS ARE MOST PREDICTIVE OF METHODS ###\n",
    "\n",
    "\n",
    "### DETERMINE WHICH WORDS ARE MOST PREDICTIVE OF RESULTS ###\n",
    "\n",
    "\n",
    "### DETERMINE WHICH WORDS ARE MOST PREDICTIVE OF CONCLUSIONS ###\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 9.3: Tune the model and evaluate it on the test set\n",
    "\n",
    "We can probably build a better model. In the following block, you should:\n",
    "1. explore at least one modification to the previous model\n",
    "2. compare the performance of both/all models on the validation set\n",
    "3. choose the one that performs best on the validation set as your final model\n",
    "4. evaluate the accuracy of your final model on the test set\n",
    "\n",
    "Here are some modifications you might try:\n",
    "- Make the vocabulary larger or smaller by changing `MIN_COUNT`, then generating an updated set of features\n",
    "- Use tf-idf features instead of raw counts (see `sklearn.feature_extraction.text.TfidfTransformer`)\n",
    "- Increase or decrease the regularization penalty (via the `C` parameter) of your logistic regression model\n",
    "- Instead of logistic regression, use an `MLPClassifier` or other classification model\n",
    "- (challenge) include 2-grams in your vocabulary\n",
    "\n",
    "You don't need to try all of these or even most of them, but you do need to make at least one modification to the model and/or preprocessing that you believe is likely to improve performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "### YOUR CODE HERE ###\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 9.4: Plot and label the confusion matrix for your final model\n",
    "\n",
    "So far, we've been using accuracy as a crude measure of performance, but it'd be better to break down prediction performance between each of the five abstract sections. In this section, you should use the `confusion_matrix` function from `sklearn` (e.g. `confusion_matrix(y_test, y_test_pred)`) to create the confusion matrix, then plot it with `plt.matshow`.\n",
    "\n",
    "(optional) **challenge**: In a separate code block, plot the ROC curve for a single section (e.g. BACKGROUND vs all other sections)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "### CREATE THE CONFUSION MATRIX ###\n",
    "\n",
    "\n",
    "### PLOT IT USING plt.matshow ###\n",
    "\n",
    "\n",
    "### CHANGE THE TICKS FROM NUMBERS TO SECTION LABELS ###\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Once you've completed these exercises, please turn in the assignment as follows:\n",
    "\n",
    "If you're using Anaconda on your local machine:\n",
    "- download your notebook as html (see File > Download as > HTML (.html))\n",
    "- .zip the file (i.e. place it in a .zip archive)\n",
    "- submit the .zip file in Talent LMS\n",
    "\n",
    "If you're using Google Colab:\n",
    "- download your notebook as .ipynb (see File > Download > Download .ipynb)\n",
    "- if you have nbconvert installed, convert it to .html; if not, leave is as .ipynb\n",
    "- .zip the file (i.e. place it in a .zip archive)\n",
    "- submit the .zip file in Talent LMS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}