Practical_10_Sentiment_analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "orig_nbformat": 2,
    "deepnote_notebook_id": "7ee4fa74-cc74-419f-9569-536f7301ace1",
    "deepnote_execution_queue": [],
    "colab": {
      "name": "Practical_10_Sentiment_analysis.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00000-44bc9024-d3c4-4085-a229-841e61a637f4",
        "output_cleared": false,
        "id": "XwHUJ4oPLw0x"
      },
      "source": [
        "# Sentiment analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00001-528e434a-a113-4fb2-98dc-5c161f9a02c7",
        "output_cleared": false,
        "id": "4U30esPTLw00"
      },
      "source": [
        "** Sentiment analysis** (also known as **opinion mining** or **emotion AI**) -  using natural language processing and computational linguistics to systematically identify, extract, quantify, and study affective states and subjective information. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00002-98fa7b7d-9721-454c-a715-9604c88aee92",
        "output_cleared": false,
        "id": "sQB3hSXKLw01"
      },
      "source": [
        "Affective states?\n",
        "Subjective information?\n",
        "Basically it is about determining whether the writer's attitude towards a particular topic, product, etc. is **positive**, **negative**, or **neutral**.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00003-3920059b-9324-4e7b-87cf-ccc1f609cb53",
        "output_cleared": false,
        "id": "Kd5CV0VRLw01"
      },
      "source": [
        "**REFLECT AND WRITE (1)**:  What are the applications of sentiment analysis? Think about modern world and how opinion mining can be used (3-5 sentences). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00004-78a8549a-2719-4940-9ef2-a2a7b1a06898",
        "output_cleared": false,
        "id": "pO2j2hexLw01"
      },
      "source": [
        "**YOUR ANSWER**:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00004-b16594a2-f5f7-46c2-bea9-e5ca62bc0f2c",
        "output_cleared": false,
        "id": "tmqtDQbmLw02"
      },
      "source": [
        "There are different ways to do sentiment analysis. In this tutorial we will look at some:\n",
        "- Vader (lexicon-based)\n",
        "- Machine learning (Random Forest algorithm)\n",
        "- Neural Networks (RNN/LSTM)\n",
        "\n",
        "It is meant to be an introduction so if you want to learn more about some of these topics you may need to consult some other external resources."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00005-6774e96c-1e95-4a2d-b634-148e4a4a41d0",
        "output_cleared": false,
        "id": "cycwdFpPLw02"
      },
      "source": [
        "**IF YOU FANCY** (some more advanced stuff): If you want to continue learning about some cutting edge technique check out this tutorial (there is a Colab notebook too) https://curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00005-09c8d356-6df8-44ac-8847-43c92a5fd23e",
        "output_cleared": false,
        "id": "LYr1kVbyLw03"
      },
      "source": [
        "## METHOD 1: (Darth) VADER"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00006-bf74ad95-0b28-41d5-999e-c44f8df5eb34",
        "output_cleared": false,
        "id": "OSlOXoG3Lw03"
      },
      "source": [
        "**VADER (Valence Aware Dictionary and sEntiment Reasoner)** is a lexicon and rule-based sentiment analysis tool specifically attuned to sentiments expressed in social media. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00007-c4991497-599a-4910-ac82-554be5b4b5f7",
        "output_cleared": false,
        "id": "iDdwomtELw03"
      },
      "source": [
        "More info: https://github.com/cjhutto/vaderSentiment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00007-3b540562-df42-4923-8b61-b312ef0c4e33",
        "output_cleared": false,
        "id": "ZEQQ6ko5Lw09"
      },
      "source": [
        "**Lexical approach**: it uses words scored as positie and negative, with the scores based on a pre-trained model labeled as such by human reviewers.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00008-939f8e03-c7f8-41d3-8e49-e5205d928e1f",
        "output_cleared": false,
        "id": "TEsPUBVSLw09"
      },
      "source": [
        "From the authors' GitHub page:\n",
        "\"Empirically validated by **multiple independent human judges**, VADER incorporates a \"gold-standard\" sentiment lexicon that is especially attuned to microblog-like contexts\"\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00009-7fcd5b66-d359-4145-867a-436adb74123e",
        "output_cleared": false,
        "id": "O4AK0JrhLw0-"
      },
      "source": [
        "The VADER sentiment lexicon is sensitive both the **polarity** (positive/negative) and the **intensity** (how positive/how negative) of sentiments expressed in social media contexts, and is also generally applicable to sentiment analysis in other domains.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00010-3fbcb853-7280-4d9f-a531-f9d76dc4158f",
        "output_cleared": false,
        "source_hash": "1a5536b2",
        "execution_millis": 3026,
        "execution_start": 1605623339604,
        "id": "NxhM1r_cLw0-"
      },
      "source": [
        "!pip install vaderSentiment\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00011-cbef9c87-0ba2-4657-81db-d2452a600fd1",
        "output_cleared": false,
        "id": "uqbi2xnALw0_"
      },
      "source": [
        "Let's call the sentiment intensity analyser object:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00012-bd78641c-9692-4733-9738-f1885ef54116",
        "output_cleared": false,
        "source_hash": "55a14060",
        "execution_millis": 28,
        "execution_start": 1605623342665,
        "id": "UbEqcHkmLw0_"
      },
      "source": [
        "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
        "analyser = SentimentIntensityAnalyzer()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00013-65982835-0b16-4f20-b080-3a4d3fdb2d0f",
        "output_cleared": false,
        "source_hash": "cffe7330",
        "execution_millis": 2,
        "execution_start": 1605623342705,
        "id": "RAj3mXZmLw0_"
      },
      "source": [
        "def sentiment_analyzer_scores(sentence):\n",
        "    score = analyser.polarity_scores(sentence)\n",
        "    print(\"{:-<40} {}\".format(sentence, str(score)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00014-23ca0505-acc7-4bfb-bd7a-8c2238c03818",
        "output_cleared": false,
        "source_hash": "35a19943",
        "execution_millis": 6,
        "execution_start": 1605625577344,
        "id": "Tze-6x_NLw0_"
      },
      "source": [
        "sentiment_analyzer_scores(\"Very nice :(\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00015-8980256e-12ba-419b-b42c-fd610358d9b0",
        "output_cleared": false,
        "id": "T5ogj-NsLw1A"
      },
      "source": [
        "Let's analyse the result:\n",
        "**Positive, Negative and Neutral scores**: the proportion of text that falls in the corresponding categories. \n",
        "(our sentence is 67% Positive, 33% Neutral and 0% Negative  - these metrics should add up to 1). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00018-93e0a225-daca-4a51-be52-4f3d3ff699cb",
        "output_cleared": false,
        "id": "VUnDq68aLw1A"
      },
      "source": [
        "In addition to the words themselves carrying some positive or negative emotions, there are certain aspects of data that imply the change in the emotion magnitude. \n",
        "\n",
        "Some of them are: \n",
        "\n",
        "- exclamation marks\n",
        "- capitalisations (especialle ALL CAPS)\n",
        "- degree modifiers (\"very\", \"marginally\")\n",
        "- emoji\n",
        "- conjunction \"but\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00016-3890770d-963e-4e03-b90e-548f6664401a",
        "output_cleared": false,
        "source_hash": "445fd02e",
        "execution_millis": 10,
        "execution_start": 1605623342731,
        "id": "M5JT55fLLw1A"
      },
      "source": [
        "sentences = [\"VADER is smart, handsome, and funny.\",  #  positive sentence \n",
        "             \"VADER is smart, handsome, and funny!\",  # punctuation emphasis\n",
        "             \"VADER is very smart, handsome, and funny.\", # degree modifiers\n",
        "             \"VADER is VERY SMART, handsome, and FUNNY.\",  # ALLCAPS \n",
        "             \"VADER is VERY SMART, handsome, and FUNNY!!!\", # combination of signals\n",
        "             \"VADER is VERY SMART, uber handsome, and FRIGGIN FUNNY!!!\", # booster words & punctuation make this close to ceiling for score\n",
        "             \"VADER is not smart, handsome, nor funny.\",  # negation sentence \n",
        "             \"The book was good.\",  # positive sentence\n",
        "             \"At least it isn't a horrible book.\",  # negated negative sentence with contraction\n",
        "             \"The book was only kind of good.\", # qualified positive sentence \n",
        "             \"The plot was good, but the characters are uncompelling and the dialog is not great.\", # mixed negation sentence\n",
        "             \"Today SUX!\",  # negative slang with capitalization emphasis\n",
        "             \"Today only kinda sux! But I'll get by, lol\", # mixed sentiment example with slang and constrastive conjunction \"but\"\n",
        "             \"Make sure you :) or :D today!\",  # emoticons \n",
        "             \"Catch utf-8 emoji such as such as 💘 and 💋 and 😁\",  # emojis \n",
        "             \"Not bad at all\"  # Capitalized negation\n",
        "             ]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00020-e9f6b460-9a95-47fa-abe5-71a3d53d20a3",
        "output_cleared": false,
        "id": "tD6cec8yLw1B"
      },
      "source": [
        "Check how VADER handles all these sentences and adjusts sentiment intensity accordingly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00017-95b5fe7b-4d76-486f-a0ab-8e0a1d6c9ce8",
        "output_cleared": false,
        "source_hash": "bee43213",
        "execution_millis": 33,
        "execution_start": 1605623342748,
        "id": "I9tWc4hdLw1B"
      },
      "source": [
        "for i in range(len(sentences)):\n",
        "    vs = analyser.polarity_scores(sentences[i])\n",
        "    print(\"{:-<65} {}\".format(sentences[i],  str(vs)))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00020-3e996d42-0598-4e5f-b84a-6ccaea12a47a",
        "output_cleared": false,
        "id": "tBR6ptsVLw1C"
      },
      "source": [
        "\"The **compound score** is computed by summing the valence scores of each word in the lexicon, adjusted according to the rules, and then normalized to be between -1 (most extreme negative) and +1 (most extreme positive). This is the most useful metric if you want a single unidimensional measure of sentiment for a given sentence. Calling it a 'normalized, weighted composite score' is accurate\".\n",
        "\n",
        "It is also useful for researchers who would like to set standardized thresholds for classifying sentences as either positive, neutral, or negative. Typical threshold values (used in the literature cited on this page) are:\n",
        "\n",
        "- positive sentiment: compound score >= 0.05\n",
        "- neutral sentiment: (compound score > -0.05) and (compound score < 0.05)\n",
        "- negative sentiment: compound score <= -0.05\n",
        "\n",
        "The pos, neu, and neg scores are ratios for proportions of text that fall in each category (so these should all add up to be 1... or close to it with float operation). These are the most useful metrics if you want multidimensional measures of sentiment for a given sentence\"\n",
        "\n",
        "https://github.com/cjhutto/vaderSentiment#code-examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00019-12276e8a-ffd1-45d1-872e-ff810b0f3aa6",
        "output_cleared": false,
        "id": "Ui1HQh0lLw1C"
      },
      "source": [
        "**CODE IT (1)**: Write code to score your own 6 sentences with VADER. Try different types of sentences as in the example above (e.g. with degree modifiers, capitalisation, emoji etc.).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00025-90d81ffd-32e8-4354-8b98-daa7865e6f79",
        "output_cleared": false,
        "source_hash": "6317bc4",
        "execution_start": 1605623342787,
        "execution_millis": 14,
        "id": "ygaLRxEuLw1C"
      },
      "source": [
        "## YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00020-a3747306-2dc9-416a-aebf-ea4d5e0bc50b",
        "output_cleared": false,
        "id": "lf1kVy8gLw1D"
      },
      "source": [
        "# Method 2: Machine learning (sklearn) & Random Forest Classifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00004-be05c5c3-1773-4dc8-85c5-65d00c317af8",
        "output_cleared": false,
        "id": "dzKp5118Lw1D"
      },
      "source": [
        "Anohter way to do sentiment analysis is using **machine learning** algorithms.  \n",
        "This notebook is based on the following tutotial: https://stackabuse.com/python-for-nlp-sentiment-analysis-with-scikit-learn/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00024-c8155e7e-67f1-4099-9d40-3adfa0e4abfb",
        "output_cleared": false,
        "id": "mnLyrI8xLw1D"
      },
      "source": [
        "- **Data**:  tweets about six US airlines, \n",
        "- **Task**: predict whether a tweet contains positive, negative, or neutral sentiment about the airline. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00005-2dd0a0c7-ab08-49c1-9b6c-a7f1d37dc2e6",
        "output_cleared": false,
        "source_hash": "96307baf",
        "execution_millis": 2133,
        "execution_start": 1605623342806,
        "id": "Lx6UR5ShLw1E"
      },
      "source": [
        "!pip install nltk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00025-789cc6c9-a1f3-4e2b-a653-11abc6612fc7",
        "output_cleared": false,
        "id": "DJMfhd_ULw1E"
      },
      "source": [
        "Let's import packages we are going to need to explore and clean our data before feeding it to a machine learning algorithm."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00000-62ed62d2-385d-471a-bf7d-cef04fb95c03",
        "output_cleared": false,
        "source_hash": "75ac18c8",
        "execution_millis": 2371,
        "execution_start": 1605623344953,
        "id": "WHCIOKnWLw1F"
      },
      "source": [
        "import nltk \n",
        "import pandas as pd \n",
        "import re\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00027-11550802-922f-48ef-bb53-d858f976a496",
        "output_cleared": false,
        "id": "XOf0f-18Lw1F"
      },
      "source": [
        "Let's load the data from an online source:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00006-c93cddf0-fb10-4370-b72d-0b959c804a3a",
        "output_cleared": false,
        "source_hash": "26743fce",
        "execution_millis": 1732,
        "execution_start": 1605623347331,
        "id": "P-I_bLESLw1F"
      },
      "source": [
        "data_source_url = \"https://raw.githubusercontent.com/kolaveridi/kaggle-Twitter-US-Airline-Sentiment-/master/Tweets.csv\"\n",
        "airline_tweets = pd.read_csv(data_source_url)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00030-b338b96c-18a0-4b5c-9490-d003ee7f424f",
        "output_cleared": false,
        "id": "Vhtm8sNcLw1G"
      },
      "source": [
        "Pandas allows us to look at the first 5 rows to get an idea of what the data looks like. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00008-83d508e3-2b49-4bde-aee6-83b099b46d13",
        "output_cleared": false,
        "source_hash": "f2af428d",
        "execution_millis": 22,
        "execution_start": 1605623349127,
        "id": "zWdr2CgmLw1G"
      },
      "source": [
        "airline_tweets.head()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00032-0245f520-d005-4a1d-9f0c-e955778dbf34",
        "output_cleared": false,
        "id": "OmeCN62LLw1G"
      },
      "source": [
        "Every tweet is identified with the \"tweet_id\" number and assigned a sentiment (\"neutral\", \"positive\", or \"negative\"), as well as the airline it is related to, and the tweet text. There are other columns too (e.g. date and time, sentiment confidence, reason for a negative comment).  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00033-f92af794-9965-4d3a-a057-7e4e00df3e72",
        "output_cleared": false,
        "id": "ALUm4k5mLw1G"
      },
      "source": [
        "Our task is a typical **supervised classification task**. Given a labelled data set (tweets are labelled with their sentiments), we will need to build a model which is able to predict whether a tweet (from a test set on which the model hasnt't been trained) is positive, negative, or neutral. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00034-d28bf8d4-f286-4255-a831-72a6e4f03956",
        "output_cleared": false,
        "id": "rIB1PLNKLw1G"
      },
      "source": [
        "### Know your data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00009-91cb1589-0457-42ab-a0ed-aab0a511f26f",
        "output_cleared": false,
        "id": "6G2sdeqELw1H"
      },
      "source": [
        "The first step of any data mining project is to explore the data. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00036-b7b24ebb-ce41-4e8e-8285-efe004ef241e",
        "output_cleared": false,
        "id": "ehGmbUDGLw1H"
      },
      "source": [
        "Here we set up some visualisation parameters (make a chart bigger). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00010-5004e551-7ef6-4ed8-95a3-e263676a4507",
        "output_cleared": false,
        "source_hash": "297da4b4",
        "execution_millis": 5,
        "execution_start": 1605623349156,
        "id": "3VU_nqt6Lw1I"
      },
      "source": [
        "plot_size = plt.rcParams[\"figure.figsize\"] \n",
        "print(plot_size[0]) \n",
        "print(plot_size[1])\n",
        "\n",
        "plot_size[0] = 8\n",
        "plot_size[1] = 6\n",
        "plt.rcParams[\"figure.figsize\"] = plot_size \n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00038-cebda723-e8f0-473f-b9b2-7346d99d6587",
        "output_cleared": false,
        "id": "8ufvfD7-Lw1I"
      },
      "source": [
        "Now, let's plot the data from the spreadsheet and explore it in details."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00011-06076ea7-0b2f-463c-ab85-64ccb31d7393",
        "output_cleared": false,
        "source_hash": "fe777134",
        "execution_millis": 193,
        "execution_start": 1605623349172,
        "id": "qW_xmGwXLw1J"
      },
      "source": [
        "airline_tweets.airline.value_counts().plot(kind='pie', autopct='%1.0f%%')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00012-4c0c53f2-78aa-4e8c-8e61-572e53b64cdc",
        "output_cleared": false,
        "source_hash": "9a26a303",
        "execution_millis": 116,
        "execution_start": 1605623349421,
        "id": "lDHnhRScLw1L"
      },
      "source": [
        "airline_tweets.airline_sentiment.value_counts().plot(\n",
        "    kind='pie', \n",
        "    autopct='%1.0f%%', \n",
        "    colors=[\"red\", \"yellow\", \"green\"]\n",
        "    )\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00041-034b1ade-d758-46cd-9b7b-719eee328c9e",
        "output_cleared": false,
        "id": "a1ix6cuyLw1M"
      },
      "source": [
        "Now we know what airlines are discussed and also that there are way more negative tweets than positive or neutral.\n",
        "\n",
        "We can also group tweets by airline and see the sentiment breakdown by airline name."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00013-f1034795-b81a-4959-bd5f-64147410da86",
        "output_cleared": false,
        "source_hash": "eff196f5",
        "execution_millis": 370,
        "execution_start": 1605623349546,
        "id": "z1GtFoH6Lw1M"
      },
      "source": [
        "airline_sentiment = airline_tweets.groupby(['airline', 'airline_sentiment']).airline_sentiment.count().unstack()\n",
        "airline_sentiment.plot(kind='bar')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00043-028009ea-2e7b-4b4a-8f96-2fcdaf524232",
        "output_cleared": false,
        "id": "LcMBWjQULw1N"
      },
      "source": [
        "Install seaborn visualisation library and explore the confidence level of the sentiment labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00014-89acf9c3-638a-48e9-b700-817cd86fc976",
        "output_cleared": false,
        "source_hash": "d27c978c",
        "execution_millis": 3561,
        "execution_start": 1605623349923,
        "id": "0OSesAV3Lw1N"
      },
      "source": [
        "!pip install seaborn\n",
        "import seaborn as sns\n",
        "\n",
        "sns.barplot(x='airline_sentiment', y='airline_sentiment_confidence' , data=airline_tweets)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00015-8ebf29bb-b595-48f3-a547-ecfb44adde68",
        "output_cleared": false,
        "id": "oVU-UY7qLw1O"
      },
      "source": [
        "## Dividing dataset into feature and label sets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00016-7a5f3b14-2896-4c02-b690-1017fc7238f7",
        "output_cleared": false,
        "id": "MZ_pk1WhLw1O"
      },
      "source": [
        "A refresher on ML terminology:\n",
        "\n",
        "- **Features** - input (e.g. a column in your dataset, some property of your training data)\n",
        "*(Examples: number of rooms, height of a person, income))*\n",
        "- **Labels** - output (what we are trying to predict)\n",
        "*(Examples: house price, weight, education level)*\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00047-a4cac65c-4696-460d-a306-f4c38bbdb7d3",
        "output_cleared": false,
        "id": "ayRZCBB9Lw1O"
      },
      "source": [
        "After exploring the data through visualising it, we need to think which columns we will use as input features and which column values we will try to predict.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00017-d01e8c2c-e092-474a-8744-c2fb286cffce",
        "output_cleared": false,
        "id": "P6mrXH3jLw1O"
      },
      "source": [
        "As our task is to classify tweets into three classes (neg, pos, or neutral), we can decide that the labels (output) will be exactly that - the column with the sentiment values. \n",
        "\n",
        "As for the input features, let's think: what will we base our decision on?\n",
        "Maybe on the tweet date and time? Does the sentiment depend on it? Or on the airline name? \n",
        "In fact, no. The input will be the tweet **text** (column 'text') and the label we want to predict is 'airline_sentiment'). Have a look at the code below and make sure you understand how we assign values to the features and labels sets of data using Pandas iloc (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iloc.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00016-dfcdbb1b-8fb6-4f70-b616-a0a54c6c3adf",
        "output_cleared": false,
        "source_hash": "5d6a8eeb",
        "execution_millis": 4,
        "execution_start": 1605623353490,
        "id": "Uem9FR86Lw1P"
      },
      "source": [
        "features = airline_tweets.iloc[:, 10].values\n",
        "labels = airline_tweets.iloc[:, 1].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00019-7db801a5-f044-4c7f-91d5-a098d0810e04",
        "output_cleared": false,
        "id": "jDgSKgQ4Lw1P"
      },
      "source": [
        "## Preprocessing (cleaning) data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00020-7740de73-517d-422d-9e9b-2df11d39b75a",
        "output_cleared": false,
        "id": "VBKjUH_hLw1P"
      },
      "source": [
        "...with our beloved regular expressions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00019-9d2d7a74-f48f-4965-bb7c-c1c184119afc",
        "output_cleared": false,
        "source_hash": "f92bd2a5",
        "execution_millis": 429,
        "execution_start": 1605623353515,
        "id": "4OoWh2xxLw1P"
      },
      "source": [
        "processed_features = []\n",
        "\n",
        "for sentence in range(0, len(features)):\n",
        "    # Remove all the special characters\n",
        "    processed_feature = re.sub(r'\\W', ' ', str(features[sentence]))\n",
        "\n",
        "    # remove all single characters\n",
        "    processed_feature= re.sub(r'\\s+[a-zA-Z]\\s+', ' ', processed_feature)\n",
        "\n",
        "    # Remove single characters from the start\n",
        "    processed_feature = re.sub(r'\\^[a-zA-Z]\\s+', ' ', processed_feature) \n",
        "\n",
        "    # Substituting multiple spaces with single space\n",
        "    processed_feature = re.sub(r'\\s+', ' ', processed_feature, flags=re.I)\n",
        "\n",
        "    # Removing prefixed 'b'\n",
        "    processed_feature = re.sub(r'^b\\s+', '', processed_feature)\n",
        "\n",
        "    # Converting to Lowercase\n",
        "    processed_feature = processed_feature.lower()\n",
        "\n",
        "    processed_features.append(processed_feature)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00053-23ac81d9-5231-47d4-9231-20ea0b7f8528",
        "output_cleared": false,
        "id": "jrjlwaajLw1P"
      },
      "source": [
        "Let's see what has been done here.\n",
        "\n",
        "1. Removing all the special characters\n",
        "2. Remove all the single characters left as a result of Step 1.\n",
        "(e.g. if we remove special character ' from Jack's and replace it with space, we are left with Jack s. Here s has no meaning, so we remove it by replacing all single characters with a space)\n",
        "3. If we replace all single characters with space, multiple spaces are created. => let's replace all multiple spaces with single spaces \n",
        "4. Lowercase everything"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00022-e0714eb7-92e5-444a-88d4-68556ec8f22c",
        "output_cleared": false,
        "id": "rABZC6ILLw1Q"
      },
      "source": [
        "## Vectorising text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00023-74470173-a3c8-41e3-863a-a6f6a2de7550",
        "output_cleared": false,
        "id": "Tf4Acqk0Lw1Q"
      },
      "source": [
        "As we know, machine learning systems only work with numbers, not raw texts. We now have to convert our text data into numerical representation. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00024-991489eb-4380-4ab5-8efb-973b247c65a7",
        "output_cleared": false,
        "id": "Em-cAM0xLw1Q"
      },
      "source": [
        "You are already familiar with the **TF-IDF vectoriser**. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00059-98e41dbd-0991-4ba9-8320-2236b8799aff",
        "output_cleared": false,
        "id": "TOAPt2n6Lw1Q"
      },
      "source": [
        "**REFLECT AND WRITE (2)**: what does TF-IDF vectoriser do? How is it different from other vectorising methods?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00025-17cefa62-87df-431c-ada3-1f79b4c91452",
        "output_cleared": false,
        "source_hash": "a7e3ed37",
        "execution_millis": 1188,
        "execution_start": 1605623353951,
        "id": "4XRfFKpaLw1R"
      },
      "source": [
        "nltk.download ('stopwords')\n",
        "from nltk.corpus import stopwords\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "vectorizer = TfidfVectorizer (max_features=2500, min_df=7, max_df=0.8, stop_words=stopwords.words('english'))\n",
        "processed_features = vectorizer.fit_transform(processed_features).toarray()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00027-bc574efc-6afe-4069-92ed-7823a4eb98e4",
        "output_cleared": false,
        "id": "F7dmC30hLw1R"
      },
      "source": [
        "Let's look at some parameters:\n",
        "- **max_features** (set to 2500): the vectoriser will only use the 2500 most frequently occurring words to create a bag of words feature vector. \n",
        "- **max_df** (set to 0.8): only use the words occuring occur in a maximum of 80% of the documents. \n",
        "- **min-df** (set to 7): only use words that occur in at least 7 documents\n",
        "This is one of the ways to exclude words that may be too common to be useful for our task.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00028-ba824a91-c36f-4fc1-8678-778602b334d1",
        "output_cleared": false,
        "id": "92sTPyqzLw1S"
      },
      "source": [
        "## Dividing data into training and testing sets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00029-9e5fe4b4-3205-4107-9b2c-6c0a262e5936",
        "output_cleared": false,
        "id": "QOHcInkyLw1S"
      },
      "source": [
        "- **Training set** - used to train the model\n",
        "- **Test set**  - used to test the model's performance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00030-ba59a83a-2039-4898-9f52-55e16aebfacc",
        "output_cleared": false,
        "source_hash": "5a232803",
        "execution_millis": 574,
        "execution_start": 1605623355153,
        "id": "pIeyvUngLw1S"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(processed_features, labels, test_size=0.2, random_state=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00031-d4f89992-93a9-4f73-82a9-1a369d11778e",
        "output_cleared": false,
        "id": "x50VgBWFLw1S"
      },
      "source": [
        "- **test_size** (set to 0.2): our data set will be split into two sets of 80% (for training) and 20% data (for testing).\n",
        "- **random_state** parameter is used to ensure that every time you run your code the split will end up in subsets of data containing identical values. It can be set to anything but commonly used value is 0 (which means \"I want this code to be reproducible). Some ML folks also like to use 42 (https://www.independent.co.uk/life-style/history/42-the-answer-to-life-the-universe-and-everything-2205734.html). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00032-008d4799-157d-4d24-9431-3f6045c5f94b",
        "output_cleared": false,
        "id": "k0JIyzURLw1T"
      },
      "source": [
        "## Machine learning in action"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00033-a977eeaf-c354-4227-b7e9-9832950760d7",
        "output_cleared": false,
        "id": "ln39zpBSLw1T"
      },
      "source": [
        "Now that we have split our data into training and test sets, we will use a machine learning algorithm to learn from it.\n",
        "Although we can use any ML algorithm, here we will use **Random Forest**.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00065-af0c1008-4a01-4d1f-a688-a251e15ed062",
        "output_cleared": false,
        "id": "L1-EVG-cLw1T"
      },
      "source": [
        "![](https://i.pinimg.com/564x/55/cc/a7/55cca7c380856eb2300caf24fd95097c.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00066-1a5e7449-e392-4449-a114-496aab287fc6",
        "output_cleared": false,
        "id": "QobTB7WMLw1T"
      },
      "source": [
        "(Image source: https://www.pinterest.com.au/pin/458733912018966676/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00067-5fcd7acf-9002-4bd4-ab7a-4b9a2f781ef9",
        "output_cleared": false,
        "id": "tMESHfKhLw1T"
      },
      "source": [
        "Do you know about **decision trees**?\n",
        "This algorithm is a supervised learning one. \n",
        "We create a training model to be used to predict class or value of target variables by learning **decision rules** inferred from the training data.\n",
        "\n",
        "The algorithm solves the problem using **tree representation**. Each internal node of the tree corresponds to an attribute, and each leaf node corresponds to a class label."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00068-f3cd8753-b9ae-4fd7-9771-709833191b3e",
        "output_cleared": false,
        "id": "8BzZh6sQLw1T"
      },
      "source": [
        "**Random forest** algorithm combines the results of multiple decision trees. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00069-a236ec12-3697-45d5-b789-764545a296c1",
        "output_cleared": false,
        "id": "pQwKWBuSLw1U"
      },
      "source": [
        "A short video on decision trees and random forests: https://www.youtube.com/watch?v=J4Wdy0Wc_xQ&t=49s"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00034-f486ae54-9a38-447a-9e8b-8d631bf92107",
        "output_cleared": false,
        "id": "zy3qxsAHLw1U"
      },
      "source": [
        "**RandomForestClassifier** class of the **sklearn.ensemble** module can be used to train a model with the random forest algorithm. \n",
        "We need to call the **fit** method on the RandomForestClassifier and pass it our **training features and labels** as parameters. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00034-8c875931-1876-4ff2-a0f6-b39af319d30b",
        "output_cleared": false,
        "source_hash": "51b151a",
        "execution_millis": 66436,
        "execution_start": 1605623355732,
        "id": "esAvR3mDLw1U"
      },
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "text_classifier = RandomForestClassifier(n_estimators=100, random_state=0)\n",
        "text_classifier.fit(X_train, y_train)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00067-8124bd24-0a04-4801-a022-7fe973624c2f",
        "output_cleared": false,
        "id": "RG62mXdpLw1U"
      },
      "source": [
        "- **n_estimators** = number of trees in your forest (Random Forest is an ensemble method comprising of creating multiple decision trees).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00068-fa760912-f6a0-4813-b65e-a209abc17d7e",
        "output_cleared": false,
        "id": "VpTWXu3NLw1U"
      },
      "source": [
        "## Predictions "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00036-368fff5e-6528-47b2-ace8-168647fb2dae",
        "output_cleared": false,
        "id": "hVD-pbGaLw1V"
      },
      "source": [
        "Once the model has been trained, we can now use the model to make predictions (is this tweet negative? positive? neutral?). \n",
        "To do so, we need to call the **predict** method on the object of the RandomForestClassifier class that we used for training. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00037-1f4f42c6-3deb-49cd-bda1-9068575407d2",
        "output_cleared": false,
        "source_hash": "1214093a",
        "execution_millis": 453,
        "execution_start": 1605623422211,
        "id": "gon3HMziLw1V"
      },
      "source": [
        "predictions = text_classifier.predict(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00071-2ecdbee2-dafc-40e3-aac0-948687412f10",
        "output_cleared": false,
        "id": "CqbMjwx0Lw1V"
      },
      "source": [
        "Done!\n",
        "Now we need to evaluate the performance of our model.\n",
        "You may remember the classification metrics we can use, such as:\n",
        "- confusion matrix\n",
        "- F1 measure\n",
        "- accuracy\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00077-7787d9d0-9e32-456f-b825-3b2d26d43582",
        "output_cleared": false,
        "id": "Gpeb5zZzLw1V"
      },
      "source": [
        "**CONFUSION MATRIX, PRECISION AND RECALL REFRESHER**\n",
        "![](https://miro.medium.com/max/4420/1*btcfBuM5Eqqc6rJ3iw3sNQ.png)\n",
        "(Image source (and a good article on the topic): https://medium.com/@alon.lek/should-i-look-at-precision-recall-or-specificity-sensitivity-3946158aace1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00078-8609ccab-ae02-41ac-bd31-ff2bd28b7eca",
        "output_cleared": false,
        "id": "7o938pvzLw1V"
      },
      "source": [
        "F-score is a measure of the model's accuracy which combines precision and recall. \n",
        "Read more: https://deepai.org/machine-learning-glossary-and-terms/f-score\n",
        "![]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00080-5aac0219-bd30-475f-8e73-a7a29b94763c",
        "output_cleared": false,
        "id": "WqnzigXELw1W"
      },
      "source": [
        "Let's see how our model performed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00038-076cb45d-02d6-4cfb-8179-21d5f9cbd0a1",
        "output_cleared": false,
        "source_hash": "b009a922",
        "execution_millis": 69,
        "execution_start": 1605623422715,
        "id": "GmHQ5LbWLw1W"
      },
      "source": [
        "from sklearn.metrics import classification_report, confusion_matrix, accuracy_score\n",
        "\n",
        "print(confusion_matrix(y_test,predictions))\n",
        "print(classification_report(y_test,predictions))\n",
        "print(accuracy_score(y_test, predictions))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00073-c76a7aeb-6503-4410-adf6-619f44afe580",
        "output_cleared": false,
        "id": "1nxQJHadLw1W"
      },
      "source": [
        "Voila! Our algorithm has achieved an accuracy of 75.30.\n",
        "There may be ways to improve it (add more data, better feature engineering, model parameter tuning etc.). \n",
        "But there are also other ways to do sentiment analysis, so let's move on for now!\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00039-923a0373-d1f1-430f-872f-9ce4f3771530",
        "output_cleared": false,
        "id": "1oEw7vEXLw1W"
      },
      "source": [
        "## Method 3 - Neural Networks "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00075-a9d64784-51ec-4568-b1f1-a858875dfd5e",
        "output_cleared": false,
        "id": "32mXxrgNLw1X"
      },
      "source": [
        "Another method we will look into today is using neural networks.\n",
        "We will train a sentiment classifier for movie reviews in IMDB data set, using **Recurrent Neural Networks**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00085-10d3e34c-e1c8-40ab-8a89-6654c82ea94c",
        "output_cleared": false,
        "id": "MrJsBz-YLw1X"
      },
      "source": [
        "Video on RNNs: https://www.youtube.com/watch?v=LHXXI4-IEns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00045-8a7a1644-3221-4158-8803-25d631d13666",
        "output_cleared": false,
        "id": "h8u0ArapLw1X"
      },
      "source": [
        "The code below is based on this tutorial: https://towardsdatascience.com/a-beginners-guide-on-sentiment-analysis-with-rnn-9e100627c02e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00087-6b02c95d-a2c8-4d96-b791-2a8246fbf779",
        "output_cleared": false,
        "id": "ZNYNAmtnLw1X"
      },
      "source": [
        "Let's start with installing Keras and Tensorflow. \n",
        "Tensorflow is an open source library for numerical computation that makes machine learning and neural network training and running faster and easier.\n",
        "Keras, in turn, is a deep learning framework built on top of Tensorflow.\n",
        "\n",
        "Keras provides some toy datasets (https://keras.io/api/datasets/), including a sentiment analysis one - https://keras.io/api/datasets/imdb/.\n",
        "\"This is a dataset of 25,000 movies reviews from IMDB, labeled by sentiment (positive/negative). Reviews have been preprocessed, and each review is encoded as a list of word indexes (integers).\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00045-cbe4bbf3-09c2-4842-a7a4-e20ac144e536",
        "output_cleared": false,
        "source_hash": "13247c78",
        "execution_millis": 10141,
        "execution_start": 1605623422829,
        "id": "T0PeIOXXLw1X"
      },
      "source": [
        "!pip install keras\n",
        "!pip install tensorflow\n",
        "import tensorflow as tf\n",
        "from keras.datasets import imdb"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00089-d3257186-dbe6-4481-a0c9-5c440ba82490",
        "output_cleared": false,
        "id": "eOa7L2TBLw1X"
      },
      "source": [
        "In this dataset, words are indexed by overall frequency in the dataset.\n",
        "E.g. integer \"3\" encodes the 3rd most frequent word in the data. \n",
        "This helps us do some quick filtering like \"only consider the top 10,000 most common words, except the top 20 most common words\".\n",
        "\n",
        "Below we set up the vocabulary size - the 'num_words\" parameters will do exactly that, command the functiion to only use the top 5000 most frequently used words in the dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00090-15497602-388d-412c-8ea7-c92e31978a5f",
        "output_cleared": false,
        "id": "LXYEYeSLLw1Y"
      },
      "source": [
        "The load function returns a tuple of Numpy arrays: (x_train, y_train), (x_test, y_test).\n",
        "(Read more: https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb/load_data)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00046-f5b5819c-3b82-4bc3-b07b-40a75d55d168",
        "output_cleared": false,
        "source_hash": "59308d79",
        "execution_millis": 8240,
        "id": "B9MGpcDoLw1Y"
      },
      "source": [
        "vocabulary_size = 5000\n",
        "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = vocabulary_size)\n",
        "print('Loaded dataset with {} training samples, {} test samples'.format(len(X_train), len(X_test)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00092-c965b5f1-75b3-4211-b740-87258769b230",
        "output_cleared": false,
        "id": "U3pA6_nILw1Y"
      },
      "source": [
        "Let's have a look at a sample review and its label.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00047-c38dced8-0119-495a-bfd1-c2cdcb6bf8b3",
        "output_cleared": false,
        "source_hash": "78e02a3c",
        "execution_millis": 4,
        "id": "5Dxnez4pLw1Y"
      },
      "source": [
        "print('---review---')\n",
        "print(X_train[6])\n",
        "print('---label---')\n",
        "print(y_train[6])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00094-f0b2b817-73f9-4c85-b3b1-20ad1669eca5",
        "output_cleared": false,
        "id": "qtBEgYWdLw1Z"
      },
      "source": [
        "The review is represented by a sequence of integers. Each of the integers is a word ID. These IDs were assigned to all individual words.\n",
        "The label, in turn, is an integer (0 for negative, 1 for positive).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00095-ca6cb262-e78b-499d-95f7-838cbbb66395",
        "output_cleared": false,
        "id": "EG8zXgRjLw1Z"
      },
      "source": [
        "**imdb.get_word_index()** returns the dictionary where words are mapped to ID integers, and we cal always use it to get a word from its index.\n",
        "to map the review back to the original words.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00048-6acead2e-48f6-4a55-89bf-81ebb433f2f0",
        "output_cleared": false,
        "source_hash": "241304bf",
        "execution_millis": 181,
        "id": "y1NQFV-BLw1Z"
      },
      "source": [
        "word2id = imdb.get_word_index()\n",
        "id2word = {i: word for word, i in word2id.items()}\n",
        "print('---review with words---')\n",
        "print([id2word.get(i, ' ') for i in X_train[6]])\n",
        "print('---label---')\n",
        "print(y_train[6])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00049-102f0aa1-6a90-4f68-b477-9ae5bfe418d2",
        "output_cleared": false,
        "source_hash": "a4e8272b",
        "execution_millis": 3,
        "id": "Tmevo4acLw1Z"
      },
      "source": [
        "print('Maximum review length: {}'.format(\n",
        "len(max((X_train + X_test), key=len))))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00050-4278af8b-0e4c-4254-9749-cdc987f90416",
        "output_cleared": false,
        "source_hash": "bea4d806",
        "execution_millis": 4,
        "id": "JaSzcYUzLw1a"
      },
      "source": [
        "print('Minimum review length: {}'.format(\n",
        "len(min((X_test + X_test), key=len))))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00099-d889d73d-b4de-4c73-898a-3d30f07878fd",
        "output_cleared": false,
        "id": "csahXz7HLw1a"
      },
      "source": [
        "To feed our data into RNN, all input documents must have the same length. \n",
        "Longer reviews will be trunkated, shorter reviws - padded with zeros. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00051-a4addb7e-c66d-44fe-afd5-75b55f8353e7",
        "output_cleared": false,
        "source_hash": "6d644bb6",
        "execution_millis": 1545,
        "id": "B6gkjIxlLw1b"
      },
      "source": [
        "from keras.preprocessing import sequence\n",
        "max_words = 500\n",
        "X_train = sequence.pad_sequences(X_train, maxlen=max_words)\n",
        "X_test = sequence.pad_sequences(X_test, maxlen=max_words)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00101-25bf6f0e-7992-44cb-953e-5a9a11d3c31b",
        "output_cleared": false,
        "id": "L7XkyDwTLw1c"
      },
      "source": [
        "Let's design our RNN model for sentiment analysis!\n",
        "First, we import some layers from Keras.\n",
        "Reminder: our **input** is a sequence of words (technically, integer word IDs) of maximum length = max_words, and our **output** is a binary sentiment label (0 or 1).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00102-d0e55a26-60f8-4bc5-ab78-a71be5c9e8f6",
        "output_cleared": false,
        "id": "t4ovs2vvLw1c"
      },
      "source": [
        "Keras offers an **Embedding layer** that can be used for neural networks on text data.\n",
        "The input data for this layer must be represented as integers. We have done it already. \n",
        "The Embedding layer is initialized with random weights and will learn an embedding for each of the words in the training dataset.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00103-d552ecd4-be29-49e7-9e83-6d02955225fc",
        "output_cleared": false,
        "id": "dih_FBMHLw1c"
      },
      "source": [
        "First layer - **Embedding layer** - must have 3 arguments set:\n",
        "\n",
        "- **input_dim**: size of the vocabulary in the text data. For example, if your data is integer encoded to values between 0-10, then the size of the vocabulary would be 11 words.\n",
        "- **output_dim**: size of the vector space in which words will be embedded (choose any, try different values)\n",
        "- **input_length**: length of input sequences, as you would define for any input layer of a Keras model. For example, if all of your input documents are comprised of 1000 words, this would be 1000."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00104-e8c1cc47-070d-4862-ac02-7b34b8df2e0a",
        "output_cleared": false,
        "id": "cOttbYFPLw1d"
      },
      "source": [
        "Second layer we add is **LSTM - Long Short-Term Memory**.\n",
        "\n",
        "\"Recurrent neural networks are used for \"things\" that happen recurrently so one thing after the other (e.g. time series, but also words). Long Short-Term Memory networks (LSTM) are a specific type of Recurrent Neural Network (RNN) that are capable of learning the relationships between elements in an input sequence. In our case the elements are words. So our next layer is an LSTM layer with 100 memory units\".\n",
        "(https://www.liip.ch/en/blog/sentiment-detection-with-keras-word-embeddings-and-lstm-deep-learning-networks)\n",
        "\n",
        "\n",
        "More about LSTM: http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
        "And an illustrated guide here: https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21\n",
        "\n",
        "\n",
        "The output of the embedding layer is fed to the LSTM layer. \n",
        "Finally, there is a Dense layer with Sigmoid activation function which essentially does the class prediction. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00052-e56254db-55f8-4575-99d2-c2ddcf54c7ad",
        "output_cleared": false,
        "source_hash": "dec37edd",
        "execution_millis": 327,
        "id": "43ZLRTcvLw1d"
      },
      "source": [
        "from keras import Sequential #Sequential model is a sequence of layers\n",
        "from keras.layers import Embedding, LSTM, Dense, Dropout\n",
        "embedding_size=32\n",
        "model=Sequential()\n",
        "# we have initialised a Sequential model and now start adding layers:\n",
        "model.add(Embedding(vocabulary_size, embedding_size, input_length=max_words))\n",
        "model.add(LSTM(100))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "print(model.summary())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00106-69f515c2-d027-4fd2-8f15-11d5accf4a29",
        "output_cleared": false,
        "id": "EKATdYTJLw1d"
      },
      "source": [
        "Let's now compile our model. We specify the **loss function** and **optimizer** we want to use for training, and any **evaluation metrics** we want to measure. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00053-45f1abc4-1467-4c2c-835d-0d93dbec3620",
        "output_cleared": false,
        "source_hash": "c29f3392",
        "execution_millis": 4,
        "id": "w1EhMyYiLw1d"
      },
      "source": [
        "model.compile(loss='binary_crossentropy', \n",
        "             optimizer='adam', \n",
        "             metrics=['accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00108-4803653c-33da-42f7-ad22-74f128abc04c",
        "output_cleared": false,
        "id": "foPQlc9MLw1d"
      },
      "source": [
        "When the model is compiled, we can start training. \n",
        "**Batch size** and **number of training epochs** have to be specified: together with our model architecture these parameteres will determine the training time.\n",
        "\n",
        "Training may take a while - be patient and relax :)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00109-a2315cab-2975-466c-9fa8-ef3f94d55215",
        "output_cleared": false,
        "id": "GfYD4_s8Lw1d"
      },
      "source": [
        "During model training we need to observe the loss function: it should constantly be going down. It means that our model is improving. \n",
        "We show the model our dataset 3 times (**epochs** parameter). \n",
        "The **batch size** (we set it to 64) defines how many reviews (data samples) the model will see at once. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00054-50add25d-5c2b-4a6f-aaa0-06b019cfd90a",
        "output_cleared": false,
        "source_hash": "95769e7d",
        "execution_millis": 1459530,
        "id": "RY3ah6sULw1e"
      },
      "source": [
        "batch_size = 64\n",
        "num_epochs = 3\n",
        "X_valid, y_valid = X_train[:batch_size], y_train[:batch_size]\n",
        "X_train2, y_train2 = X_train[batch_size:], y_train[batch_size:]\n",
        "model.fit(X_train2, y_train2, validation_data=(X_valid, y_valid), batch_size=batch_size, epochs=num_epochs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00110-57d2ae93-52e6-4613-b22d-daaafde709f2",
        "output_cleared": false,
        "id": "JMjnW96bLw1e"
      },
      "source": [
        "The model is trained!\n",
        "Let's check how well it performs on unseen test data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00055-d8b068e5-2082-46a2-ab4a-6d784c6171bc",
        "output_cleared": false,
        "source_hash": "c0ecc7ce",
        "execution_millis": 160643,
        "id": "6UuW2m0-Lw1e"
      },
      "source": [
        "scores = model.evaluate(X_test, y_test, verbose=0)\n",
        "print('Test accuracy:', scores[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00112-0cebba1c-6371-4c2e-8e08-48d32ae2d9a7",
        "output_cleared": false,
        "id": "lFjYuNWwLw1e"
      },
      "source": [
        "There are several ways in which we can build the model. We can experiments with different layers and parameters while considering such factors as training time and the possibility of overfitting. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deepnote_cell_type": "markdown",
        "tags": [],
        "cell_id": "00115-6fe3a848-cbf2-42fa-a5d3-f3a54de1e5ea",
        "output_cleared": false,
        "id": "W0Ca-heFLw1f"
      },
      "source": [
        "**CODE IT (2)**: experiment with a different network architecture and parameter tuning and see if  you can get a more accurate sentiment classification model.\n",
        "\n",
        "Examples of what you can try (choose any):\n",
        "\n",
        "- add Flatten layer before the final Dense one with model.add(Flatten())\n",
        "- replace LSTM with SimpleRNN with *model.add(layers.SimpleRNN(15))*\n",
        "- change number of epochs\n",
        "- change number of vector dimensions (embedding size)\n",
        "- change batch size\n",
        "- change number of LSTM or SimpleRNN units\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deepnote_cell_type": "code",
        "tags": [],
        "cell_id": "00115-94dfd133-3160-4393-a6c3-35a3c9fbf228",
        "output_cleared": false,
        "source_hash": "75d640a1",
        "id": "AhmqmYQCLw1f"
      },
      "source": [
        "#YOUR CODE (there is no right answer)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}