06_03_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/COGS108/Lectures-Sp23/blob/master/06_02_text.ipynb)\n",
    "\n",
    "# Text Analysis\n",
    "\n",
    "- **Sentiment Analysis**\n",
    "    - tokenization\n",
    "    - stop words\n",
    "    - stemming\n",
    "- **TF-IDF**\n",
    "    - Bag of Words\n",
    "    - term frequency\n",
    "    - inverse document frequency\n",
    "- Tools: `nltk`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# pandas and matplotlib setup\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = (17, 7)\n",
    "plt.rcParams.update({'font.size': 14})\n",
    "import seaborn as sns\n",
    "\n",
    "#improve resolution\n",
    "#comment this line if erroring on your machine/screen\n",
    "%config InlineBackend.figure_format ='retina'\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# import natural language toolkit\n",
    "# if you don't already have nltk installed, uncomment and run this\n",
    "#import sys\n",
    "#!{sys.executable} -m pip install nltk\n",
    "\n",
    "import nltk\n",
    "\n",
    "# download stopwords & punkt\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Reminder: **Natural Language Processing** is a whole field of study.\n",
    "\n",
    "Like most topics in this course, there are many courses solely focused on the appropriate analysis of text. We'll cover the general concepts in this course, but know you're missing lots of important details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Natural Language Toolkit (`nltk`)\n",
    "\n",
    "For more details on using the functionality within this package, check out the [NLTK Book](http://www.nltk.org/book/).\n",
    "\n",
    "0. Preface\n",
    "1. Language Processing and Python\n",
    "2. Accessing Text Corpora and Lexical Resources\n",
    "3. Processing Raw Text\n",
    "4. Writing Structured Programs\n",
    "5. Categorizing and Tagging Words \n",
    "6. Learning to Classify Text\n",
    "7. Extracting Information from Text\n",
    "8. Analyzing Sentence Structure\n",
    "9. Building Feature Based Grammars\n",
    "10. Analyzing the Meaning of Sentences \n",
    "11. Managing Linguistic Data\n",
    "12. Afterword: Facing the Language Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[VADER](https://github.com/cjhutto/vaderSentiment) is a particularly helpful tool/lexicon when working with sentiments expressed in social media (tweets, online reviews, etc.)\n",
    "\n",
    "Its functionality is available through `nltk`, so we'll download the vader lexicon for use later in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# get lexicon we'll be working with today\n",
    "nltk.download('vader_lexicon') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df_wi21 = pd.read_csv('https://raw.githubusercontent.com/shanellis/datasets/master/COGS108_feedback_Wi21.csv')\n",
    "df_wi21.head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# read in feedback dataset - Fall 2020\n",
    "df_fa20 = pd.read_csv('https://raw.githubusercontent.com/shanellis/datasets/master/COGS108_feedback_Fa20.csv')\n",
    "df_fa20.head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# read in feedback dataset - Spring 2020\n",
    "df_sp20 = pd.read_csv('https://raw.githubusercontent.com/shanellis/datasets/master/COGS108_feedback_Sp20.csv')\n",
    "df_sp20.head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# read in feedback dataset - Winter 2020\n",
    "df_wi20 = pd.read_csv('https://raw.githubusercontent.com/shanellis/datasets/master/COGS108_feedback_Wi20.csv')\n",
    "df_wi20.head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# read in feedback dataset - Spring 2019\n",
    "df_sp19 = pd.read_csv('https://raw.githubusercontent.com/shanellis/datasets/master/COGS108_feedback_Sp19.csv')\n",
    "df_sp19.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Describe & Explore\n",
    "\n",
    "We'll quickly describe and explore the data to see what information we have before moving on to Text Analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Data Considerations\n",
    "\n",
    "- duplicate responses?\n",
    "- PIDs for individuals in the class (typos?)\n",
    "- missingness?\n",
    "- reflect reality?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# information about rows and columns\n",
    "print(df_wi21.shape, df_fa20.shape, df_sp19.shape, df_wi20.shape, df_sp20.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note: Response Rates\n",
    "- Spring 2019: 384/826 (46%)\n",
    "- Winter 2020: 295/444 (66%)\n",
    "- Spring 2020: 397/475 (84%)\n",
    "- Fall 2020: 321/447 (72%)\n",
    "- Winter 2021: 314/438 (72%)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Missingness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# how many nonresponses\n",
    "df_wi21.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# how many nonresponses\n",
    "df_fa20.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# how many nonresponses\n",
    "df_sp19.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# how many nonresponses\n",
    "df_wi20.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# how many nonresponses\n",
    "df_sp20.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We see that there are more nonresponses in the `enjoyed_least` category than the `enjoyed_most` category. So, more people left what they enjoyed least blank than they did what they enjoyed most."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Previous Quarters\n",
    "\n",
    "Typically, there are a few people who have what they enjoy least but don't have an enjoy most. We don't have any last quarter...but often these students' feedback is of particular interest to me."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fall 2020\n",
    "check_least = df_fa20[df_fa20['enjoyed_most'].isnull() & df_fa20['enjoyed_least'].notnull()]\n",
    "list(check_least['enjoyed_least'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Spring 2020\n",
    "check_least = df_sp20[df_sp20['enjoyed_most'].isnull() & df_sp20['enjoyed_least'].notnull()]\n",
    "list(check_least['enjoyed_least'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Winter 2020\n",
    "check_least = df_wi20[df_wi20['enjoyed_most'].isnull() & df_wi20['enjoyed_least'].notnull()]\n",
    "list(check_least['enjoyed_least'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Spring 2019\n",
    "check_least = df_sp19[df_sp19['enjoyed_most'].isnull() & df_sp19['enjoyed_least'].notnull()]\n",
    "list(check_least['enjoyed_least'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Missing data causes a problem in `nltk`, so we either get rid of individuals who didn't respond to both, or we can replace their missing data with 'No response', knowing that this text will be included in the analysis now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def fill_no_response(df):\n",
    "    '''replace missing data in enjoyed_most/least series with string No response'''\n",
    "    \n",
    "    df['enjoyed_most'] = df['enjoyed_most'].fillna('No response')\n",
    "    df['enjoyed_least'] = df['enjoyed_least'].fillna('No response')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# fill NAs with string 'No response'\n",
    "fill_no_response(df_wi21)\n",
    "fill_no_response(df_fa20)\n",
    "fill_no_response(df_sp20)\n",
    "fill_no_response(df_sp19)\n",
    "fill_no_response(df_wi20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Exploratory Plots\n",
    "\n",
    "These can give us a quick idea of students' thoughts on the course by using a word cloud.  These kinds of viz show the size of the word in proportion to how common it is in the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install wordcloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "\n",
    "df = df_fa20\n",
    "\n",
    "most_liked = \" \".join( review for review in df.enjoyed_most)\n",
    "least_liked = \" \".join( review for review in df.enjoyed_least)\n",
    "\n",
    "wordcloud = WordCloud(max_font_size=50, max_words=100, background_color=\"white\").generate( most_liked )\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.title('Most Liked - Fa20')\n",
    "plt.show()\n",
    "\n",
    "\n",
    "\n",
    "wordcloud = WordCloud(max_font_size=50, max_words=100, background_color=\"white\").generate( least_liked )\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.title('Least Liked - Fa20')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_wi21\n",
    "\n",
    "most_liked = \" \".join( review for review in df.enjoyed_most)\n",
    "least_liked = \" \".join( review for review in df.enjoyed_least)\n",
    "\n",
    "wordcloud = WordCloud(max_font_size=50, max_words=100, background_color=\"white\").generate( most_liked )\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.title('Most Liked - Wi21')\n",
    "plt.show()\n",
    "\n",
    "\n",
    "\n",
    "wordcloud = WordCloud(max_font_size=50, max_words=100, background_color=\"white\").generate( least_liked )\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.title('Least Liked - Wi21')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets make an all inclusive dataset\n",
    "\n",
    "df_all = pd.concat( [df_sp19, df_wi20, df_sp20, df_fa20, df_wi21])\n",
    "\n",
    "\n",
    "df = df_all\n",
    "\n",
    "most_liked = \" \".join( review for review in df.enjoyed_most)\n",
    "least_liked = \" \".join( review for review in df.enjoyed_least)\n",
    "\n",
    "wordcloud = WordCloud(max_font_size=50, max_words=100, background_color=\"white\").generate( most_liked )\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.title('Most Liked - All quarters')\n",
    "plt.show()\n",
    "\n",
    "\n",
    "\n",
    "wordcloud = WordCloud(max_font_size=50, max_words=100, background_color=\"white\").generate( least_liked )\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.title('Least Liked - All quarters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Quick checks: Words of interest\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['enjoyed_least']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_word_freq(df, word):\n",
    "    \"\"\"checks for frequenccy of word specified in most and least enjoyed responses\"\"\"\n",
    "    \n",
    "    most = df['enjoyed_most'].str.lower().str.contains(word).sum()/df['enjoyed_most'].notnull().sum()\n",
    "    least = df['enjoyed_least'].str.lower().str.contains(word).sum()/df['enjoyed_least'].notnull().sum()\n",
    "    \n",
    "    out = pd.DataFrame({'most_freq': [most], 'least_freq': [least]})\n",
    "    return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "## check for assignment\n",
    "df = df_wi21\n",
    "check_word_freq(df, 'assignment')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "## check for project in free text\n",
    "check_word_freq(df, 'project')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "## check for group in free text\n",
    "check_word_freq(df, 'group')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "## check for individual in free text\n",
    "check_word_freq(df, 'individual')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Quizzes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_word_freq(df_wi21, 'quiz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "check_word_freq(df_fa20, 'quiz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_word_freq(df_sp20, 'quiz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "check_word_freq(df_wi20, 'quiz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "check_word_freq(df_sp19, 'quiz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Labs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_word_freq(df_wi21, 'lab')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_word_freq(df_fa20, 'lab')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sentiment Analysis\n",
    "\n",
    "We get a quick snapshot of what's going on in COGS 108, but we really want to understand the details. To do this, analyzing the sentiment of the text is a good next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Step 1: Tokenization\n",
    "\n",
    "Tokenization is the first step in analyzing text. \n",
    "\n",
    "1. Aquire text of interest\n",
    "2. Break text down (tokenize) into smaller chunks (i.e. words, bigrams, sentences, etc.)\n",
    "\n",
    "A **token** is a single entity - think of it as a building block of language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tokenization Example\n",
    "\n",
    "Here we demonstrate what a tokenized single response looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "## import word tokenizer\n",
    "from nltk.tokenize import word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# just focus on last quarter's responses\n",
    "df = df_wi21"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[25,'enjoyed_most']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tokenized_word = word_tokenize(df.loc[25,'enjoyed_most'])\n",
    "print(tokenized_word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[25,'enjoyed_most']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Tokenize COGS108 data\n",
    "\n",
    "Using that concept we'll tokenize the words in the enjoyed_most and `enjoyed_least` columns for the data in our COGS108 data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# tokenize most and least responses\n",
    "df['most_token'] = df['enjoyed_most'].apply(word_tokenize) \n",
    "df['least_token'] = df['enjoyed_least'].dropna().apply(word_tokenize) \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Step 2: Stop Words\n",
    "\n",
    "**Stop words** are words that are of less interest to your analysis. \n",
    "\n",
    "For example, you wouldn't expect the following words to be important: is, am, are, this, a, an, the, etc.\n",
    "\n",
    "By removing stopwords, you can lower the computational burden, focusing on only the words of interest.\n",
    "\n",
    "To do so in `nltk`, you need to create a list of stopwords and filter them from your tokens.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# import stop words\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "# look at stop words\n",
    "print(stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Stop Words Example\n",
    "\n",
    "Here we compare a sentence after tokenization to one that has been tokenized _and had stop words removed_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# example of removing stop words\n",
    "filtered_sent=[]\n",
    "for w in tokenized_word:\n",
    "    if w not in stop_words:\n",
    "        filtered_sent.append(w)\n",
    "print(\"Tokenized Sentence:\", tokenized_word)\n",
    "print(\"Filtered Sentence:\", filtered_sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Remove Stop Words: COGS108 data\n",
    "\n",
    "Using that idea, we can go ahead and remove stop words from our tokenized most and least liked tokenized data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['most_token']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# remove stop words\n",
    "df['most_stop'] = df['most_token'].apply(lambda x: [item for item in x if item not in stop_words])\n",
    "df['least_stop'] = df['least_token'].dropna().apply(lambda x: [item for item in x if item not in stop_words])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Step 3: Lexicon Normalization (**Stemming**)\n",
    "\n",
    "In language, many different words come from the same root word. \n",
    "\n",
    "For example, \"intersection\", \"intersecting\", \"intersects\", and \"intersected\" are all related to the common root word - \"intersect\".\n",
    "\n",
    "**Stemming** is how linguistic normalization occurs - it reduces words to their root words (and chops off additional things like 'ing') - all of the above words would be reduced to their common stem \"intersect.\"\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Stemming Example\n",
    "\n",
    "After tokenization and removing stop words, we can get the stem for all tokens (words) in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Stemming\n",
    "from nltk.stem import PorterStemmer\n",
    "\n",
    "ps = PorterStemmer()\n",
    "\n",
    "stemmed_words=[]\n",
    "for w in filtered_sent:\n",
    "    stemmed_words.append(ps.stem(w))\n",
    "\n",
    "print(\"Filtered Sentence:\", filtered_sent)\n",
    "print(\"Stemmed Sentence:\", stemmed_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Stemming: COGS108 data\n",
    "\n",
    "Here, we obtain the stem (root word) for all tokens in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df['most_stem'] = df['most_stop'].apply(lambda x: [ps.stem(y) for y in x])\n",
    "df['least_stem'] = df['least_stop'].dropna().apply(lambda x: [ps.stem(y) for y in x])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Step 4: Frequency Distribution\n",
    "\n",
    "It can be helpful to get a sense of which words are most frequent in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# get series of all most and least liked words after stemming\n",
    "# note that \"No Response\" is still being included in the analysis\n",
    "most = df['most_stem'].apply(pd.Series).stack()\n",
    "least = df['least_stem'].apply(pd.Series).stack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "most"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`FreqDist` calculates the frequency of each word in the text and we can plot the most frequent words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from nltk.probability import FreqDist\n",
    "import string\n",
    "\n",
    "# calculation word frequency\n",
    "fdist_most = FreqDist(most)\n",
    "fdist_least = FreqDist(least)\n",
    "\n",
    "# remove punctuation counts\n",
    "for punc in string.punctuation:\n",
    "    del fdist_most[punc]\n",
    "    del fdist_least[punc]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Frequency Distribution Plot - top 20\n",
    "# for words in what students like most\n",
    "fdist_most.plot(20, cumulative=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fdist_least.plot(20, cumulative=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Step 5: Sentiment Analysis!\n",
    "\n",
    "**Sentiment Analysis** quantifies the content, idea, beliefs and opinions conveyed in text. \n",
    "\n",
    "Two general approaches:\n",
    "\n",
    "1. **Lexicon-based** - count number of words in a text belonging to each sentiment (positive, negative, happy, angry, etc.)\n",
    "2. **Machine learning-based** - develop a classification model on pre-labeled data\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment Example\n",
    "\n",
    "To get a measure of overall sentiment in our text, we'll compare our text to the VADER lexicon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer \n",
    "analyser = SentimentIntensityAnalyzer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "VADER handles:\n",
    "\n",
    "- capitalization (great vs GREAT) & punctuation (exclamation makes more positive!)\n",
    "- emojis and emoticons\n",
    "- degree modifiers (extremely good vs. marginally good)\n",
    "- contractions and conjunctions (but signals shift)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`pos` + `neg` + `neu` = 1\n",
    "\n",
    "**`compound`** score - metric that calculates sum of all the lexicon ratings and normalizes between -1 (most extreme negative) and +1 (most extreme positive)\n",
    "- positive: `compound` >= 0.05 \n",
    "- neutral: -0.05 < `compound` < 0.05\n",
    "- negative : `compound` <= -0.05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "analyser.polarity_scores(\"The class is super cool.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "analyser.polarity_scores(\"The class is not super cool.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "analyser.polarity_scores(\"The class is NOT super cool!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyser.polarity_scores(\"The class is NOT awful!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Sentiment Analysis: COGS108 data\n",
    "\n",
    "Here, we will calculate the sentiment of each most liked and least liked student response from the survey."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# get list of the 'sentences' (responses) from each individual\n",
    "most_list = list(df['enjoyed_most'].values)\n",
    "least_list = list(df['enjoyed_least'].dropna().values)\n",
    "most_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# create function that will output dataframe \n",
    "# that stores sentiment information\n",
    "def get_sentiments(input_list):\n",
    "    \n",
    "    output = pd.DataFrame()\n",
    "\n",
    "    for sentence in input_list:\n",
    "        ss = analyser.polarity_scores(sentence)\n",
    "        ss['sentence'] = sentence\n",
    "        output = output.append(ss, ignore_index=True)\n",
    "\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# get sentiment measures\n",
    "least_sentiments = get_sentiments(least_list)\n",
    "most_sentiments = get_sentiments(most_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Sentiment Analysis: COGS108 data output\n",
    "\n",
    "After calculating the sentiment of each response, we can look at the output of each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# take a look at the output\n",
    "least_sentiments.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# take a look at the output\n",
    "most_sentiments.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Let's deal with those `No response` values\n",
    "\n",
    "We've left them in there long enough. Let's remove the 'No response' values before we look at any overall patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "most_sentiments = most_sentiments[most_sentiments['sentence'] != 'No response']\n",
    "least_sentiments = least_sentiments[least_sentiments['sentence'] != 'No response']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Sentiment Analysis: COGS108 data - `describe`\n",
    "\n",
    "To get an overall sense of the values stored in each of these dataframes, we can use `describe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "most_sentiments.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "least_sentiments.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Sentiment Analysis: COGS108 data - plotting\n",
    "\n",
    "We can compare the distribution of the `compound` metric between the two analyses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "most_sentiments['compound'].plot.density(label='most')\n",
    "least_sentiments['compound'].plot.density(label='least')\n",
    "plt.legend()\n",
    "plt.xlabel('Compound Sentiment Scores')\n",
    "plt.xlim(-1,1);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# include label for boxplot\n",
    "most_sentiments['which'] = 'most'\n",
    "least_sentiments['which'] = 'least'\n",
    "# concatenate data frames together\n",
    "compound_out = pd.concat([most_sentiments, least_sentiments])\n",
    "compound_out.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot compound by resonse type\n",
    "sns.boxplot(data=compound_out, x='which', y='compound')\n",
    "plt.xlabel('response');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Probably unsurprisingly, the overall sentiment of what students like tends to be more positive than what students like less. \n",
    "\n",
    "Probably not surprising given the data and question on the survey. But, let's dig deeper into these data moving beyond sentiment analysis..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## TF-IDF\n",
    "\n",
    "Term Frequency - Inverse Document Frequency (**TF-IDF**) sets out to identify the tokens most unique to your document of interest (relative to all documents in your corpus). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Term Frequency (TF)** - counts the number of times a given word (or token, term, etc) occurs in each document divided by the number of words in that document\n",
    "\n",
    "**Inverse Document Frequency (IDF)** - weights the word by their relative frequency across documents. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$IDF_{word} = log(\\frac{\\# documents}{\\# \\ documents\\_containing\\_word})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$TF-IDF = TF \\times IDF$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "words with a high TF-IDF are those with high frequency in one document & relatively low frequency in other documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For our purposes, our **corpus** will be students' responses to what they like most and least about COGS108.\n",
    "\n",
    "We'll treat this as **two separate documents**:\n",
    "1. What students like most\n",
    "2. What students like least"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag of Words (BoW) approach\n",
    "\n",
    "Converts the text into a co-occurrence matrix across documents within the corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To do this, let's get our text ready.\n",
    "\n",
    "We're going to make sure all our words are lower case, remove punctuation from each, and then provide the text (`corpus`) to `TfidfVectorizer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import string \n",
    "\n",
    "# lowercase text\n",
    "least = list(map(str.lower, least_list))\n",
    "most = list(map(str.lower, most_list))\n",
    "\n",
    "# remove punctuation\n",
    "for c in string.punctuation:\n",
    "    least = str(least).replace(c, \"\")\n",
    "    most = str(most).replace(c, \"\")\n",
    "\n",
    "# get list of two documents together\n",
    "corpus = [str(least), str(most)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Calculate TF-IDF\n",
    "\n",
    "With our text ready for analysis, it's time to calculate TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To start our TF-IDF analysis, we'll first **create a `TfidfVectorizer` object to transform our text data into vectors.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TfidfVectorizer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# create vectorizer\n",
    "tfidf = TfidfVectorizer(sublinear_tf=True,\n",
    "                        analyzer='word',\n",
    "                        max_features=2000,\n",
    "                        tokenizer=word_tokenize,\n",
    "                        stop_words=stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### TF-IDF: COGS108 data - calculation\n",
    "\n",
    "Here, we use our vectorizer to calculate TF-IDF across the words in our word matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# calculate TF-IDF\n",
    "cogs_tfidf = pd.DataFrame(tfidf.fit_transform(corpus).toarray())\n",
    "cogs_tfidf.columns = tfidf.get_feature_names()\n",
    "cogs_tfidf = cogs_tfidf.rename(index={0:'least', 1:'most'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cogs_tfidf.T.sort_values(by='most')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### TF-IDF: COGS108 data - output\n",
    "\n",
    "If we just want to look at the word most uniuqe in each document..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "most_unique = cogs_tfidf.idxmax(axis=1) \n",
    "most_unique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Alternatively, we can sort by the set or words most unique to each document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "cogs_tfidf.sort_values(by='most', axis=1, ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "cogs_tfidf.sort_values(by='least', axis=1, ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Sentiment Analysis** and **TF-IDF** are really helpful when analyzing documents and corpuses of text.\n",
    "\n",
    "But, what if, from the text itself we wanted to predict whether or not the text was likely a 'most' liked or a 'least' liked comment? We'll discuss how to do this in the coming **machine learning** lectures!"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "rise": {
   "scroll": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}