graph_summary.py.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Summarization using Text rank \n",
    "\n",
    "### STEP 1 : Data cleaning ( removing stop words, non letter characters, turning to lower case letters )\n",
    "### STEP 2 : Sentence vector representation\n",
    "### STEP 3 : Graph formation where edges formed using cosine similarity between sentences\n",
    "### STEP 4 : Applying text rank algorithm and output the summary "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial Phase\n",
    "### Importing Libraries and Reading Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "### importing the necessary libraries\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.cluster.util import cosine_distance\n",
    "import numpy as np\n",
    "import pandas\n",
    "import nltk\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import sent_tokenize\n",
    "import networkx as nx\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Reading the data file\n",
    "\n",
    "df = pandas.read_csv('Downloads/tennis_articles_v4.csv')\n",
    "# df['article_text']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 1 : Data Cleaning\n",
    "### Cleaning sentences, by removing Non Alphabet Characters and converting to Lower Case Letters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "### cleaning sentences, by removing non alphabet characters and converting to lower case letters\n",
    "\n",
    "s = \"\"\n",
    "d = {}\n",
    "for a in df['article_text']:\n",
    "      s += a\n",
    "# print s\n",
    "sentences = sent_tokenize(s)\n",
    "clean_sentences = []\n",
    "for s in sentences:\n",
    "    temp = re.sub(\"[^a-zA-Z]\",\" \",s)\n",
    "    temp = temp.lower()\n",
    "    clean_sentences.append(temp)\n",
    "    d[temp] = s \n",
    "# print clean_sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Stop Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "### defined a functiom for removing stop words which are downloaded from NLTk's list of english stop words\n",
    "\n",
    "stop_words = stopwords.words('english')\n",
    "def rem_stop(s):\n",
    "    var = \"\"\n",
    "    words = nltk.word_tokenize(s)\n",
    "    for w in words:\n",
    "        if( w not in stop_words):\n",
    "           var+=w+\" \"\n",
    "    return var\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "### removed the stop words using the function defined above\n",
    "\n",
    "dict = {}\n",
    "clean = []\n",
    "# print clean_sentences\n",
    "for s in clean_sentences:\n",
    "    temp = rem_stop(s)\n",
    "    clean.append(temp)\n",
    "    dict[temp] = d[s]\n",
    "# print clean  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "### loaded pre trained word2vec model from Gensim\n",
    "\n",
    "from gensim.models import KeyedVectors\n",
    "filename = 'Downloads/GoogleNews-vectors-negative300.bin'\n",
    "model = KeyedVectors.load_word2vec_format(filename, binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 2 : Sentence Vector Generation\n",
    "### Vector Representations are created using pre trained word2vec model from Gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/shristi/.local/lib/python2.7/site-packages/ipykernel_launcher.py:5: DeprecationWarning: Call to deprecated `wv` (Attribute will be removed in 4.0.0, use self instead).\n",
      "  \"\"\"\n"
     ]
    }
   ],
   "source": [
    "### creating vector representation of sentences after extracting word vectors\n",
    "\n",
    "# print(model)\n",
    "word_embeddings = {}\n",
    "words = list(model.wv.vocab)\n",
    "# print len(words)\n",
    "for a in words:\n",
    "    word_embeddings[a]=model[a]\n",
    "\n",
    "# print len(word_embeddings)\n",
    "\n",
    "\n",
    "sentence_vectors = []\n",
    "for i in clean:\n",
    "  if len(i) != 0:\n",
    "    v = sum([word_embeddings.get(w, np.zeros((300,))) for w in i.split()])/(len(i.split())+0.001)\n",
    "  else:\n",
    "    v = np.zeros((300,))\n",
    "  sentence_vectors.append(v)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 3 : Graph Formation\n",
    "### Graph is formed where sentences are the nodes and edges are formed using Cosine Similarity between the sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "### generating the final summary after producing the graph using networkx and applying pagerank algo\n",
    "\n",
    "sentence_similarity_martix = np.zeros([len(sentences), len(sentences)])\n",
    "for i in range(len(sentences)):\n",
    "  for j in range(len(sentences)):\n",
    "    if i != j:\n",
    "      sentence_similarity_martix[i][j] = cosine_similarity(sentence_vectors[i].reshape(1,300), sentence_vectors[j].reshape(1,300))[0,0]\n",
    "\n",
    "sentence_similarity_graph = nx.from_numpy_array(sentence_similarity_martix)\n",
    "scores = nx.pagerank(sentence_similarity_graph)\n",
    "ranked_sentence = sorted(((scores[i],s) for i,s in enumerate(clean)), reverse=True)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 4 : Summary Generation\n",
    "### Summary is produced using PageRank algorithm and top 5 ranked sentences are printed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"I don't like being under that kind of pressure,\" Federer said of the deadline Kosmos handed him.Kei Nishikori will try to end his long losing streak in ATP finals and Kevin Anderson will go for his second title of the year at the Erste Bank Open on Sunday.\n",
      "When I'm on the courts or when I'm on the court playing, I'm a competitor and I want to beat every single person whether they're in the locker room or across the net.So I'm not the one to strike up a conversation about the weather and know that in the next few minutes I have to go and try to win a tennis match.\n",
      "The Spaniard broke Anderson twice in the second but didn't get another chance on the South African's serve in the final set.Federer, 37, first broke through on tour over two decades ago and he has since gone on to enjoy a glittering career.\n",
      "Nina and Irina give their opinions on what coaching should look like in the future, on both tours (18:55).Federer won the Swiss Indoors last week by beating Romanian qualifier Marius Copil in the final.\n",
      "Federer had an easier time than in his only previous match against Medvedev, a three-setter at Shanghai two weeks ago.Roger Federer has revealed that organisers of the re-launched and condensed Davis Cup gave him three days to decide if he would commit to the controversial competition.\n"
     ]
    }
   ],
   "source": [
    "### printing the top 5 sentences\n",
    "# print ranked\n",
    "for i in range(5):\n",
    "     print dict[ranked_sentence[i][1]]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}