lecture_22_natural_language_processing_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Deep Learning for NLP & Sequence Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<br /><br />\n",
    "\n",
    "### Motivation\n",
    "\n",
    "In this class we are going to cover popular Deep Learning models for **sequential data**, i.e. where each data point $X_i$ represents a sequence of some tokens $X_i={s_0, s_1, ... s_n}$. Note that sequence lengths can **vary** in a a given dataset, i.e $|{X_i}| \\neq |{X_j}|$.\n",
    "\n",
    "NLP is a classical example of sequence modeling, where a dataset represents a set of sentences/paragraphs/documents (sequence of words). In NLP usually we assume that set of tokens (words) come from a predefined fixed **vocabulary**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Learning Objectives\n",
    "\n",
    "The goal of this class is a high-level overview/summary of recent trends in Deep Learning for NLP and sequence modeling. The material is huge, and, unfortunately, most details are beyond this class, however, lot of extra reading links are provided to gain more insights. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T10:29:57.643260Z",
     "start_time": "2020-06-30T10:29:57.641328Z"
    }
   },
   "source": [
    "## Recurrent Neural Networks (RNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T12:20:37.085084Z",
     "start_time": "2020-06-30T12:20:37.083013Z"
    }
   },
   "source": [
    "### Motivation\n",
    "\n",
    "When we work with sequences, we can simply calculate embeddings for individual tokens (word2vec) of a sequence and then sum them up (average/concatenate). This simple aggregation might work on simple tasks but in real world more complex and smart aggregation mechanism is required. RNNs are natural fit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T10:30:12.299263Z",
     "start_time": "2020-06-30T10:30:12.297291Z"
    }
   },
   "source": [
    "### Vanilla RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T12:05:46.021718Z",
     "start_time": "2020-06-30T12:05:46.018855Z"
    }
   },
   "source": [
    "<img src='images/seq_models/rnn.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$X={x_0, x_1, ... x_t}$ is our input data point (individual $x_i$ can be bag-of-words or word2vec representations or something else you come up with), represented as blue circles; A green box is a **RNN cell** which contains a computation and weight; $h_0, h_1, ..., h_t$ is a sequence of **hidden states**, which, intuitively, are information carriers from cell to cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T12:13:27.112209Z",
     "start_time": "2020-06-30T12:13:27.107695Z"
    }
   },
   "source": [
    "Computation in a RNN cell is the follows:  \n",
    "<br />\n",
    "$h_0 = 0$  \n",
    "$h_t = \\sigma(W x_t + U h_{t-1} + b)$, where $\\sigma$ is element-wise sigmoid    \n",
    "<br />\n",
    "$W, U, b$ - are weights  \n",
    "\n",
    "<br />\n",
    "\n",
    "**Q:** How to do sequence classification/regression with RNN?  \n",
    "**A:** Option 1. Just attach a classifier/regressor on the last hidden state. Option 2. Take average/max pooling of hidden layers and pass to classifier/regressor.\n",
    "\n",
    "**Q:** How to do sequence batching?  \n",
    "**A:** You will have a 3D tensor [B, T, E] or [T, B, E], where B - batch size, T - max sequence length in a batch, E - feature vector dim. Due to the fact that sequences are variable length, you take max length in a batch and append rest with zeros or some special token. Refer also https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pack_padded_sequence.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Long Short-Term Memory Networks (LSTM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vanilla RNNs suffer much from so called **vanishing gradient** problem. If you try to write down gradient equation for it, you will bump into a part where lot of small numbers (linear to input sequence size) are multiplied together, which causes the result to go to zero. (Details here https://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture07-fancy-rnn.pdf) For long enough sequences this becomes a serious problem and different architecture is required."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T13:15:17.920819Z",
     "start_time": "2020-06-30T13:15:17.917489Z"
    }
   },
   "source": [
    "<img src='images/seq_models/lstm.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sequence-to-Sequence Framework\n",
    "\n",
    "Sequence-to-Sequence (seq2seq) is a well know method in Deep Learning for transforming one sequence into another (original paper https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf). Classical example is machine translation, where sentence in one language gets transformed in the corresponding sentence in another language. All state-of-the-art translation systems as well as many other systems use seq-to-seq.  \n",
    "\n",
    "The idea is very similar to Autoencoders. We have two networks (LSTM, for instance), encoder and decoder. Encoder accept input sequence and decoder predicts transformed sequence. Decoder's initial hidden state is the last hidden state of the encoder - that's how the information is shared between encoder and decoder. Intuitively, last hidden state consists of compressed representation of input sentence which a decoder should use for transformation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='images/seq_models/seq2seq_trans.jpg' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-04T07:31:58.711694Z",
     "start_time": "2020-07-04T07:31:58.709683Z"
    }
   },
   "source": [
    "#### Attention Mechanism "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-04T07:33:45.178185Z",
     "start_time": "2020-07-04T07:33:45.172872Z"
    }
   },
   "source": [
    "**Q:** When doing classification/regression using RNN, can we use smarter way than just simply using last hidden state or average/max pooling of hidden states for input to final classifier/regressor layer?  \n",
    "  \n",
    "**A:** Attention Mechanism: We can do weighted average of hidden states where weight indicates the importance of the particular hidden state. Weights are learnt jointly along with the mainstream task.\n",
    "<br />\n",
    "<img src='images/seq_models/attn.png' />  \n",
    "<br /><br />\n",
    "Y's are hidden layers, Z is final aggregated output, TANH - is computation for similarity measurement (can be one layer fully-connected network with tanh activation, or, more popularly, just a scalar product). C - depending on a task, can be a learnable parameter or some other thing. **NOTE:** The implementations of attention mechanism can vary from paper to paper.  \n",
    "\n",
    "Examples: https://distill.pub/2016/augmented-rnns/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T16:08:05.297561Z",
     "start_time": "2020-06-30T16:08:05.287760Z"
    }
   },
   "source": [
    "### Suggested Readings about RNNs\n",
    "\n",
    "https://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "\n",
    "\n",
    "### PyTorch Examples\n",
    "\n",
    "https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html  \n",
    "https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-04T08:23:52.905168Z",
     "start_time": "2020-07-04T08:23:52.903461Z"
    }
   },
   "source": [
    "## Transformer Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intro\n",
    "\n",
    "Until 2017, arguably, LSTMs were de-factor standard for NLP tasks, but then https://arxiv.org/abs/1706.03762 has been released, introducing different type of neural network architecture that set new SOTA on neural machine translation. It is called \"Transformer\", which mainly uses attention mechanism, residual connections and feed forward neural networks. Since then, this new approach gained popularity and today, almost all SOTA deep NLP models rely on that.  \n",
    "\n",
    "**Computational Cost** is the main advantage of transformer over LSTM and other RNNs. Modern sequence modeling tasks can have sequence length of order ~1000 tokens and more (https://arxiv.org/pdf/2004.05150.pdf) which is huge. RNN is inherently sequential, meaning that computation of next cell needs result from previous cell, thus making it non-parallelizable. Training on large datasets with big sequence lengths is very very slow. On the other hand, transformer is highly parallelizable and almost all components can be implemented using matrix operations.\n",
    "\n",
    "**Self-Attention Mechanism (Special kind of attention for Transformer)** is a second thing that makes Transformers superior to RNNs. It can relate signals from any arbitrarily distant sequence locations in O(1) whereas, LSTM needs O(|distance between tokens|) operations to do the same thing. It is believed that latter makes it harder to train the model and learn complex sequential relationships.  \n",
    "\n",
    "Code and explanation of original paper can be seen here: http://nlp.seas.harvard.edu/2018/04/03/attention.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BERT (Pre-Trained Transformer for Language Modeling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Language Modeling\n",
    "\n",
    "**Language Modeling** is one of the classical NLP tasks that aims to model a natural language, let's say, an English. What does it mean to model a language? It means to learn a probability distribution over sentences of a given language. Given an arbitrary sequence of words, the model must output a probability that measures the likelihood that this sequence will appear as a sentence in real world.  \n",
    "\n",
    "Mathematically, neural network must learn $P(X)=P(word_0, word_1, ..., word_n)=\\prod_{i=0}^nP(word_i | word_{i-1}, ..., word_{0})$  \n",
    "\n",
    "There are many methods of achieving this task. Two popular choices are so called **\"Next Word Prediction task\"** and **\"Masked Language Modeling task\"**.  \n",
    "\n",
    "RNNs usually learn the former, and Transformers are based on the latter. Attendees of this class are strongly encouraged to get familiar with these tasks in detail.\n",
    "\n",
    "**NOTE:** Language Modeling can be thought as an alternative pre-training method for NLP, as ImageNet is for Computer Vision. It shows great results in transfer learning.  \n",
    "\n",
    "https://arxiv.org/abs/1801.06146 - nice paper from Jeremy Howard (fast.ai) that summarizes lots of tricks and tips to get a good language model using RNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### BERT\n",
    "\n",
    "**BERT** is a Transformer neural network trained on Masked Language Modeling task for Language Modelling by Google. https://arxiv.org/abs/1810.04805\n",
    "\n",
    "BERT became so popular, there's been huge variety of BERT based models released. you can check out them and also the code here: https://huggingface.co/transformers/pretrained_models.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### OpenAI GPT\n",
    "\n",
    "OpenAI released 3 versions of their own variant of Transformer (GPT-1, GPT-2, GPT-3) that achieved really good results on text generation. \n",
    "\n",
    "https://openai.com/blog/better-language-models/\n",
    "https://github.com/openai/gpt-3"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "164px",
    "width": "514px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}