precooked_replication.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MULTIVAC - Meta-model Unification Learned Through Inquiry Vectorization and Automated Comprehension\n",
    "## Introduction\n",
    "Gallup’s MULTIVAC effort supports the goals of the DARPA ASKE program by developing a system that absorbs scientific knowledge — in the form of facts, relationships, models and equations — from a particular domain corpus into a Markov Logic Network (MLN) ontology and learns to query that ontology in order to accelerate scientific exploration within the target domain. MULTIVAC will consist of an expert query generator trained on a corpus of historical expert queries and tuned dialectically with the use of a Generative Adversarial Network (GAN) architecture. As a prototype system, MULTIVAC will focus on the domain of epidemiological research, and specifically the realm of SIR/SEIR (Susceptible-Infected-Recovered, often with an additional “Exposed” element) compartmental model approaches. It is Gallup’s intent that this system includes a “human-in-the-loop” element, especially during training, to ensure that the system is properly tuned and responsive to the needs and interests of the human researchers it is intended to augment.\n",
    "\n",
    "![MULTIVAC Schematic](images/MULTIVAC_schematic.png)\n",
    "\n",
    "DARPA’s Information Innovation Office’s Automating Scientific Knowledge Extraction (ASKE) program seeks to develop approaches to make it easier for scientists to build, maintain and reason over rich models of complex systems — which could include physical, biological, social, engineered or hybrid systems. By interpreting and exposing scientific knowledge and assumptions in existing model code and documentation, researchers can identify new data and information resources automatically, extracting useful information from these sources, and integrating this useful information into machine-curated expert models for robust modeling.\n",
    "\n",
    "## Replication\n",
    "The MULTIVAC pipeline, to run from beginning to end, is encapsulated in command-line executable code found on Gallup's [MULTIVAC repository](https://github.com/GallupGovt/multivac). To run that file -- `conductor.py`, found in the top-level MULTIVAC directory -- one needs to do the following:\n",
    "1. Within the [MULTIVAC repository](https://github.com/GallupGovt/multivac):\n",
    "  * Clone the Gallup MULTIVAC repository\n",
    "  * Instantiate a virtual environment\n",
    "  * `pip install -r requirements.txt` to install all necessary Python dependencies\n",
    "2. Within the [QG-Net repository](https://github.com/GallupGovt/qgnet) (a secondary, cloned repository that the Gallup team uses for a portion of its modeling effort):\n",
    "  * Clone the Gallup QG-Net repository (this should be at the same level directory as MULTIVAC)\n",
    "  * Clone the Facebook Research [DrQA](https://github.com/facebookresearch/DrQA) repository and follow the [install instructions](https://github.com/facebookresearch/DrQA#installing-drqa) (this should be at the same level directory as MULTIVAC). Note, this requires a Linux/OSX machine.\n",
    "3. Other downloads and dependencies\n",
    "  * The [Stanford Core NLP](https://stanfordnlp.github.io/CoreNLP/download.html) toolset needs to be downloaded and installed. **Be sure to follow the setup instructions** to ensure proper install for this to work.\n",
    "  * The [GloVe pre-trained data](http://nlp.stanford.edu/data/glove.42B.300d.zip) need to be downloaded and placed in the `data_dir` as defined in `settings.py` in the top-level directory of the MULTIVAC repository. \n",
    "  * Have [R](https://cran.r-project.org/) installed on your machine; part of MULTIVAC's code reaches back into R.\n",
    "\n",
    "Finally, QG-Net specifically works on a GPU. **The QG-Net portion of code will not work unless on a GPU** (tested with an Nvidia Quadro Pro-4000 and a Nvidia Tesla K80 cards). Also note, end-to-end, this will take approximately 60 hours to complete. \n",
    "\n",
    "## Piece-by-piece execution\n",
    "If one chooses not to replicate the entire pipeline, then below are pieces to step in and out of particular pieces, given data pre-compiled by Gallup. These files can be accessed through a public, non-secure FTP site hosted at Gallup. Navigate to ftp://ftp.gallup.com in a web browser. You will be prompted for credentials supplied by Gallup; if you do not have these, contact PI [Ben Ryan](ben_ryan@gallup.com). From there, navigate to the `aske` folder where you will see all of the various inputs and outputs of the MULTIVAC system. The instructions are as follows with what to do with these data:\n",
    "* Scraping\n",
    "  * Inputs: There are no inputs per se, since this first step instantiates MULTIVAC by scraping articles from the internet from arXiv, Springer, and PubMed around the epidemilogy topic of choice for MULTIVAC. You will need to set two environment variables: ‘SPRINGER_API_KEY’ and ‘USER_EMAIL’ for the Springer and PubMed APIs, respectively.\n",
    "  * Outputs: The `20181212.json` file encompasses all of the scraped data and are used by the parsing step.\n",
    "* Parsing\n",
    "  * Inputs: The `20181212.json` file.\n",
    "  * Outputs: The `articles-with-equations.json` and the files in the `dim_files` directory. The JSON file is used with GloVe modeling and QG-Net, which the DIM files are used with the MLN.\n",
    "* GloVe modeling\n",
    "  * Inputs: The `articles-with-equations.json` file.\n",
    "  * Outputs: The `da_embeddings.txt` file. This is a domain-adapted word embeddings used in QG-Net.\n",
    "* QG-Net\n",
    "  * Inputs: The `da_embeddings.txt` and the `articles-with-equations.json` files.\n",
    "  * Outputs: The `input.txt` is a processed version of data that is fed through QG-Net itself and the `output_questions_QG-Net.pt.txt` is a set of questions from QG-Net.\n",
    "* MLN\n",
    "  * Inputs: The files in the `dim_files` directory.\n",
    "  * Outputs: The `mln.pkl` file that wraps up the knowledge graph and attributes, including semantic clustering.\n",
    "\n",
    "The rest of this notebook will step through language around the particular component of MULTIVAC and then code -- using the data referenced above -- to exemplify execution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Settings\n",
    "In the cell below, initialize all the necessary paths for operating in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from multivac import settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Scraping\n",
    "To build a robust and diverse set of source models, MULTIVAC set a target of 2,000 scientific journal articles fitting set specifications and filters. These specifications are coded in a user-editable configuration file (by default, `multivac.cfg`), and cover both search parameters (search terms, and sources to search) as well as filter parameters, to weed out duplicates or unrelated content that might return in a more naive match on the specified search terms.\n",
    " \n",
    "To achieve the desired sample document size, MULTIVAC targets three different online sources of epidemiological research: arXiv.org, PubMed, and Springer. MULTIVAC accesses these resources through source-specific APIs and authenticates with user API access keys. Each source is searched for articles containing the specified search terms The results are saved as a combined JSON file. Each article ID is a top-level key, with a Metadata key containing various metadata keys and values and a Text key containing the plain body text. This file then serves as the intermediate \"source\" datastore for subsequent analysis and processing.\n",
    " \n",
    "These three sources also serve to demonstrate MULTIVAC's ability to work with a variety of data storage types: PubMed articles are ingested from XML, Springer from HTML, and arXiv from PDF. In the final version of this prototype system, MULTIVAC scraped a combined 2,740 articles (after pruning malformed or un-parseable results) from these three sources: 897 from arXiv, 1,280 from PubMed, and 573 from Springer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.A. Collect data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "import feedparser\n",
    "import os\n",
    "import pickle\n",
    "import requests\n",
    "import time\n",
    "\n",
    "from bs4 import BeautifulSoup as bs\n",
    "\n",
    "from multivac import settings\n",
    "from multivac.src.data.get import get_total_number_of_results, prep_terms, query_api\n",
    "\n",
    "\n",
    "# load environment variables from .env\n",
    "springer_api_key = os.environ.get('SPRINGER_API_KEY')\n",
    "user_email = os.environ.get('USER_EMAIL')  # courtesy to NIH to include your email\n",
    "\n",
    "wait_time = 3\n",
    "\n",
    "\n",
    "def collect_get_main():\n",
    "    # ------------------------------------------------------------------------\n",
    "    # arxiv\n",
    "\n",
    "    # build query and get metadata of articles matching our search criteria\n",
    "    params = {'start': 0, 'max_results': 100, 'sortBy': 'relevance'\n",
    "             ,'sortOrder': 'descending'}\n",
    "    li = [x.replace('-', ' ').split(' ') for x in settings.terms]\n",
    "    q = 'OR'.join(['%28' + prep_terms(x) + '%29' for x in li])\n",
    "    url = 'http://export.arxiv.org/api/query?search_query=' + q\n",
    "    arxiv_metadata = query_api(url, q, params, wait_time=1, verbose=True)\n",
    "\n",
    "    # save pdfs of articles that matched our search criteria\n",
    "    # we use doi as the filename when that id is present; otherwise we use the\n",
    "    # arxiv id\n",
    "    for ix, md in enumerate(arxiv_metadata):\n",
    "        url = md['id']\n",
    "        pdf_url = url.replace('/abs/', '/pdf/')\n",
    "        article_fn = url.split('/abs/')[-1]\n",
    "        article_fn = '_'.join(article_fn.split('/')) + '.pdf'\n",
    "        # specify filename so we can associate each pdf with its metadata down\n",
    "        # the road\n",
    "        arxiv_metadata[ix]['fn'] = article_fn\n",
    "        dst = settings.raw_dir / 'arxiv' / article_fn\n",
    "        if not os.path.exists(dst):\n",
    "            r = requests.get(pdf_url)\n",
    "            with open(dst, 'wb') as f:\n",
    "                f.write(r.content)\n",
    "            time.sleep(0.3)\n",
    "\n",
    "    # save arxiv metadata\n",
    "    fn = 'arxiv' + '.pkl'\n",
    "    dst = settings.metadata_dir / fn\n",
    "    with open(dst, 'wb') as f:\n",
    "        pickle.dump(arxiv_metadata, f)\n",
    "\n",
    "    # ------------------------------------------------------------------------\n",
    "    # springer\n",
    "\n",
    "    # build query to retrieve metadata\n",
    "    make_q = lambda li: '(' + ' OR '.join(['\"' + s + '\"' for s in li]) + ')'\n",
    "    q = make_q(settings.terms)\n",
    "    base = 'http://api.springernature.com/openaccess/json?q='\n",
    "    url = base + q\n",
    "    params = {\n",
    "        'source': 'springer',\n",
    "        'openaccess': 'true',\n",
    "        'api_key': springer_api_key, 'p': 20, 's': 1\n",
    "    }\n",
    "    params_ = copy.deepcopy(params)\n",
    "\n",
    "    # retrieve metadata\n",
    "    springer_metadata = []\n",
    "    while True:\n",
    "        r = requests.get(url, params_)\n",
    "        if len(r.json()['records']) == 0:\n",
    "            break\n",
    "        params_['s'] = params_['s'] + params_['p']\n",
    "        springer_metadata += r.json()['records']\n",
    "        time.sleep(wait_time)\n",
    "    print('%s total Springer articles' % len(springer_metadata))\n",
    "\n",
    "    # iterate over springer metadata and download html for each article\n",
    "    # we use a generator to increase wait times with each connection error\n",
    "    waits = (2**x for x in range(0,6))\n",
    "    for ix, md in enumerate(springer_metadata):\n",
    "        fn = md['doi'].replace('/', '-')\n",
    "        if len(fn) == 0:\n",
    "            fn = md['identifier']\n",
    "        fn = fn + '.html'\n",
    "        springer_metadata[ix]['fn'] = fn\n",
    "        dst = settings.raw_dir / 'springer' / fn\n",
    "        if not os.path.exists(dst):\n",
    "            try:\n",
    "                r = requests.get(md['url'][0]['value'])\n",
    "            except ConnectionError:\n",
    "                time.sleep(waits.__next__)\n",
    "                r = requests.get(md['url'][0]['value'])\n",
    "            html = bs(r.text).encode('utf-8').decode('utf-8')\n",
    "            with open(dst, 'w', encoding='utf-8') as f:\n",
    "                f.write(html)\n",
    "            time.sleep(3)\n",
    "\n",
    "    # save springer metadata\n",
    "    dst = settings.metadata_dir / 'springer.pkl'\n",
    "    with open(dst, 'wb') as f:\n",
    "        pickle.dump(springer_metadata, f)\n",
    "\n",
    "    # ------------------------------------------------------------------------\n",
    "    # pubmed\n",
    "\n",
    "    # search pubmed central for free full text articles containing selected\n",
    "    # query\n",
    "    # get the ids which we then use to get the xml text data\n",
    "    replace = lambda s: s.replace(' ', '+')\n",
    "    quote = lambda s: '%22' + s + '%22'\n",
    "    terms = [quote(replace(s)) for s in settings.terms]\n",
    "    term = 'term='+ '%28'+ '+OR+'.join(terms) + '%29'\n",
    "    fulltext = 'free+fulltext%5bfilter%5d'\n",
    "    retmax = 'retmax=2000'\n",
    "    base = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pmc'\n",
    "    params = {'retmax': 5000, 'email': user_email}\n",
    "    url = base + '&' + term + '+' + fulltext + '&' + retmax\n",
    "    r = requests.get(url)\n",
    "    ids = [x.contents[0] for x in bs(r.text).find_all('id')]\n",
    "\n",
    "    print('%s Pubmed Central (PMC) articles' % ids)\n",
    "\n",
    "    # get xml text data and save to disk\n",
    "    for i in ids:\n",
    "        pmc_id = 'pmc' + str(i)\n",
    "        fn = (pmc_id + '.xml')\n",
    "        dst = settings.raw_dir / 'pubmed' / fn\n",
    "        if not os.path.exists(dst):\n",
    "            url = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=' + str(i)\n",
    "            r = requests.get(url, params={'id': i})\n",
    "            xml = r.text\n",
    "            with open(dst, 'w') as f:\n",
    "                f.write(xml)\n",
    "            time.sleep(0.5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "989 total arXiv results, 1 second wait time between each call\n",
      "**********\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    376\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 2.7, use buffering of HTTP responses\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 377\u001b[0;31m                 \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbuffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    378\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: getresponse() got an unexpected keyword argument 'buffering'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-b1b112b3acbb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcollect_get_main\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-2-be836e31f595>\u001b[0m in \u001b[0;36mcollect_get_main\u001b[0;34m()\u001b[0m\n\u001b[1;32m     73\u001b[0m     \u001b[0mspringer_metadata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m     \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 75\u001b[0;31m         \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams_\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     76\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'records'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     77\u001b[0m             \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/api.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(url, params, **kwargs)\u001b[0m\n\u001b[1;32m     73\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m     \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetdefault\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'allow_redirects'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 75\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'get'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     76\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     77\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/api.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(method, url, **kwargs)\u001b[0m\n\u001b[1;32m     58\u001b[0m     \u001b[0;31m# cases, and look like a memory leak in others.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     59\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0msessions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 60\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     61\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     62\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m    531\u001b[0m         }\n\u001b[1;32m    532\u001b[0m         \u001b[0msend_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 533\u001b[0;31m         \u001b[0mresp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    534\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    535\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m    644\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    645\u001b[0m         \u001b[0;31m# Send the request\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 646\u001b[0;31m         \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    647\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    648\u001b[0m         \u001b[0;31m# Total elapsed time of the request (approximately)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m    447\u001b[0m                     \u001b[0mdecode_content\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    448\u001b[0m                     \u001b[0mretries\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_retries\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 449\u001b[0;31m                     \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    450\u001b[0m                 )\n\u001b[1;32m    451\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m    598\u001b[0m                                                   \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout_obj\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    599\u001b[0m                                                   \u001b[0mbody\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 600\u001b[0;31m                                                   chunked=chunked)\n\u001b[0m\u001b[1;32m    601\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    602\u001b[0m             \u001b[0;31m# If we're going to release the connection in ``finally:``, then\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    378\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    379\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 380\u001b[0;31m                     \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    381\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    382\u001b[0m                     \u001b[0;31m# Remove the TypeError from the exception chain in Python 3;\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36mgetresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1319\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1320\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1321\u001b[0;31m                 \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbegin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1322\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1323\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36mbegin\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    294\u001b[0m         \u001b[0;31m# read until we get a non-100 response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    295\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 296\u001b[0;31m             \u001b[0mversion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreason\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    297\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mstatus\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mCONTINUE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    298\u001b[0m                 \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36m_read_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    255\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    256\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 257\u001b[0;31m         \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_MAXLINE\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"iso-8859-1\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    258\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0m_MAXLINE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    259\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mLineTooLong\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"status line\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    587\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 589\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    590\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    591\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "collect_get_main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.B. Parse results (note, Part 1.A. must be run first)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "import json\n",
    "import os\n",
    "import pickle\n",
    "import pubmed_parser\n",
    "import slate\n",
    "\n",
    "from bs4 import BeautifulSoup as bs\n",
    "from collections import OrderedDict\n",
    "\n",
    "from multivac import settings\n",
    "from multivac.src import utilities\n",
    "from multivac.src.data.process import (\n",
    "    aggregate_pubmed, filter_arxiv, parse_articles_data, parse_html, parse_pdf, parse_pubmed, save_outputs\n",
    ")\n",
    "\n",
    "\n",
    "def collect_process_main():\n",
    "    output = {}\n",
    "    for source in settings.sources:\n",
    "        data_raw_dir = settings.raw_dir / source\n",
    "        if source in ['arxiv', 'springer']:\n",
    "            data = parse_articles_data(source, data_raw_dir)\n",
    "        elif source == 'pubmed':\n",
    "            srcs = [data_raw_dir / x for x in os.listdir(data_raw_dir)]\n",
    "            data = aggregate_pubmed(srcs)\n",
    "        if len(output) == 0:\n",
    "            output = copy.deepcopy(data)\n",
    "        else:\n",
    "            output.update(data)\n",
    "    arxiv_drops = [x.split()[0] for x in settings.arxiv_drops]\n",
    "    filtered_output = filter_arxiv(output, arxiv_drops)\n",
    "    save_outputs(filtered_output)\n",
    "    return True\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "collect_process_main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Parsing\n",
    "The parsing component of MULTIVAC takes a JSON file of scraped journal articles, and parses the text as well as LaTeX notation contained within the text. It is largely driven by parsing.py which accepts two arguments:\n",
    "\n",
    "```\n",
    "-bp  OPTIONAL  Specify the index of the first document to start processing (useful for stopping and continuing\n",
    "               the parsing process)\n",
    "-c   BOOLEAN   Indicator for whether or not to create a JSON file with tokenized LaTeX equations (defaults to \n",
    "               False)\n",
    "```\n",
    "\n",
    "Initial MULTIVAC text parsing relies on two natural language processing engines -- `stanfordnlp` and `spaCy` -- to construct dependency trees, tag parts of speech and lemmatize tokens. MULTIVAC leverages both NLP engines because of their complementary strengths and value to the overall system. The `spaCy` library provides unparalleled speed in transforming and manipulating texts, while Stanford’s dependency parsing is intentionally designed to emphasize and prioritize semantically meaningful syntactic structure. While `spaCy` also performs dependency tree parsing, the model and strategy are much more purely syntactic, making it a poor fit for semantically parsing our texts into a MLN knowledge graph.\n",
    " \n",
    "Each sentence is processed individually to identify the dependency structure of its tokens. When LaTeX notation occurs in text the notation block is extracted and a \"dummy\" token is substituted, allowing the NLP dependency parsing to interpret the sentence as a proper English language construct. This is especially important for in-line LaTeX notations, which otherwise render many of the most important sentences in an article un-parseable.\n",
    "\n",
    "![LaTeX equation parsing](images/latex_parse_1.png)\n",
    "\n",
    "The LaTeX equation itself is separately parsed and then re-inserted into the sentence, with the root of the LaTeX tree taking the place of the dummy token in the dependency structure. The LaTeX representations are parsed by converting them first into a sympy representation that enables deconstructing expressions into a nested tree structure that contains a series of functions and arguments. For example, the expression *2x + (x*y)* would be expressed as *Add(Pow(Number(2), Entity('x')), Mul(Entity('x'), Entity('y')))* where *Add()*, *Pow()* and *Mul()* are functions; and *Number(2)* and *Symbol('x')* are arguments. MULTIVAC transforms these nested parenthetical representations into a collapsed dependencies format and inserts the entire chain back into the source sentence, updating token indices as appropriate. The individual relationship and entity tokens from these equations are also expanded out in string representation and replace the LaTeX notation in the original text.\n",
    "\n",
    "![LaTeX entity relationshop](images/latex_parse_2.png)\n",
    "\n",
    "The outputs of this translation process are three sets of files: Dependency files, Morphology files, and Input files. Each file represents a parse of one article and is formatted in blocks, with one block for each sentence in the article. \"Input\" files record original word or punctuation as well as the part of speech (POS) tag, while \"Morph\" files record the token lemma, and each line contains a separate token. \"Dep\" files record the Stanford Universal Dependency relationships between pairs of words as well as the indices of the component words in the sentence. The article texts with processed equation tokens re-inserted are also written out to a file (`articles-with-equations.json`) for further use in preparing GloVe embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "import json\n",
    "import pickle\n",
    "import re as reg\n",
    "import spacy\n",
    "import stanfordnlp\n",
    "\n",
    "import multivac.src.data.equationparsing as eq\n",
    "\n",
    "from interruptingcow import timeout\n",
    "\n",
    "from multivac import settings\n",
    "from multivac.src.data.textparsing import clean_doc\n",
    "from multivac.src.data.parsing import (\n",
    "    create_parse_files, get_adjustment_position, get_token_governor, \n",
    "    load_data\n",
    ")\n",
    "\n",
    "def nlp_parse_main(args_dict):\n",
    "    ''' Main run file that orchestrates everything\n",
    "    '''\n",
    "\n",
    "    ## Load NLP engines\n",
    "    spacynlp = spacy.load('en_core_web_sm')\n",
    "    nlp = stanfordnlp.Pipeline(models_dir=settings.stanf_nlp_dir,\n",
    "                               treebank='en_ewt', use_gpu=False,\n",
    "                               pos_batch_size=3000)\n",
    "\n",
    "    ## Load documents\n",
    "    jsonObj, allDocs = load_data(settings.processed_dir / 'data.json')\n",
    "\n",
    "    ## Process and Clean documents\n",
    "    try:\n",
    "        allDocsClean = pickle.load(open('allDocsClean.pkl', \"rb\" ))\n",
    "        print('Loaded pickle!')\n",
    "    except FileNotFoundError:\n",
    "        print('Starting from scratch')\n",
    "        allDocsClean= []\n",
    "        for i, doc in enumerate(allDocs):\n",
    "            if i%10==0:\n",
    "                print(i)\n",
    "            allDocsClean.append(clean_doc(doc, spacynlp))\n",
    "\n",
    "        with open('allDocsClean.pkl', 'wb') as f:\n",
    "            pickle.dump(allDocsClean, f)\n",
    "\n",
    "\n",
    "    allDocs2 = [eq.extract_and_replace_latex(doc) for docNum, doc in\n",
    "                enumerate(allDocsClean)]\n",
    "    print('Number of LateX Equations parsed: {}'.format(len(eq.LATEXMAP)))\n",
    "\n",
    "\n",
    "    ## Put equations back into text - this will be fed to glove embedding\n",
    "    if args_dict['nlp_newjson']:\n",
    "        print('***************\\nBuilding JSON file for glove embedding...')\n",
    "        allDocs3 = []\n",
    "        percentCompletedMultiple = int(len(allDocs2)/10)\n",
    "        for i, doc in enumerate(allDocs2[0:]):\n",
    "            if i%percentCompletedMultiple == 0:\n",
    "                print('{}% completed'.format(round(i/(len(allDocs2))*100, 0)))\n",
    "            newDoc = reg.sub(r'Ltxqtn[a-z]{8}', eq.put_equation_tokens_in_text,\n",
    "                             doc)\n",
    "            allDocs3.append(newDoc)\n",
    "\n",
    "        jsonObj2 = copy.deepcopy(jsonObj)\n",
    "        allDocs3Counter = 0\n",
    "\n",
    "        for key, value in list(jsonObj2.items()):\n",
    "            if value['text']:\n",
    "                jsonObj2[key]['text']=allDocs3[allDocs3Counter]\n",
    "                allDocs3Counter = allDocs3Counter+1\n",
    "\n",
    "        with open('{}/articles-with-equations.json'.format(settings.data_dir),\n",
    "                  'w', encoding='utf8') as fp:\n",
    "            json.dump(jsonObj2, fp)\n",
    "\n",
    "\n",
    "    ## Parse files into DIM\n",
    "    startPoint=-1\n",
    "    if args_dict['nlp_bp'] is not None:\n",
    "        startPoint = args_dict['nlp_bp']\n",
    "\n",
    "    for i, doc in enumerate(allDocs2[0:]):\n",
    "        print('Processing document #{}'.format(i))\n",
    "        if i > startPoint:\n",
    "\n",
    "            # Use exception handling so that the process doesn't get stuck and\n",
    "            # time out because of memory errors\n",
    "            try:\n",
    "                with timeout(300, exception=RuntimeError):\n",
    "                    nlpifiedDoc = nlp(doc)\n",
    "                    thisDocumentData = create_parse_files(\n",
    "                        nlpifiedDoc, i, True, settings.data_dir\n",
    "                    )\n",
    "            except RuntimeError:\n",
    "                print(\"Didn't finish document #{} within five minutes. Moving to next one.\".format(i))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define argument dictionary needed for some processes\n",
    "args_dict = {\n",
    "    'nlp_bp': None, \n",
    "    'nlp_newjson': False, \n",
    "    'subset': None, \n",
    "    'prior_num_conj': 10, \n",
    "    'prior_num_param': 5, \n",
    "    'qgnet_path': settings.qgnet_dir, \n",
    "    'verbose': False\n",
    "}\n",
    "\n",
    "nlp_parse_main(args_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. GloVe models\n",
    "MULTIVAC also trains a 300-dimensional domain-adapted Global Vectors (GloVe) word-embeddings model on the corpus and saves this file in the same folder. GloVe embeddings derive multi-dimensional vector spaces describing word associations based on calculations of word co-occurrences over a large corpus.<sup>[1]</sup> \n",
    " \n",
    "MULTIVAC begins with a pre-trained 300-dimensional GloVe model incorporating 2 million terms found in the Common Crawl corpus, a collection of over 2 billion webpages scraped monthly.<sup>[2]</sup> This model represents a best-in-class embedding model for generic English language text. However, given the specific and highly technical domain we are attempting to understand and model, much domain-specific semantic knowledge – not to mention domain-specific vocabulary – are not accounted for in this generic model. MULTIVAC augments this model by training a domain-specific model on our corpus, and combining embeddings using Canonical Correctional Analysis (CCA) on the intersection of tokens between the two models.<sup>[3]</sup> The vectors for each token of the domain adapted GloVe embedding model are derived from a weighted average of the canonical vectors (N = 100) from the CCA analysis.\n",
    " \n",
    "This alignment occurs on words that exist in both the domain-specific and generic model vocabularies, but for terms that are entirely domain-specific the vector representations are projected into the 100-dimensional canonical vector space from the CCA analysis via matrix multiplication and appended to the domain-adapted embedding vectors. The resulting domain-adapted model encompasses all terms in our corpus and combines semantic meaning from both the domain and wider global context.\n",
    "\n",
    "<hr>\n",
    "\n",
    "<sup>[1]</sup> Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. “GloVe: Global Vectors for Word Representation.” Full text available at https://nlp.stanford.edu/pubs/glove.pdf\n",
    "\n",
    "<sup>[2]</sup> See http://commoncrawl.org/connect/blog/ for up to date statistics on the corpus. As of this report the total is now 3.1 billion pages, though this has varied over time since project inception, and not simply increased monotonically. When the pre-trained GloVe model was created the corpus was closer to 2 billion pages in size.\n",
    "\n",
    "<sup>[3]</sup> Prathusha K Sarma, YIngyu Liang, William A Sethares, “Domain Adapted Word Embeddings for Improved Sentiment Classification,” Submitted on 11 May 2018. arXiv:1805.04576 [cs.CL] Full text available at: https://arxiv.org/pdf/1805.04576"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from rpy2.robjects import r, pandas2ri, numpy2ri\n",
    "from scipy.stats import zscore\n",
    "from sklearn.cross_decomposition import CCA\n",
    "from unidecode import unidecode\n",
    "\n",
    "from multivac import settings\n",
    "from multivac.src.data.glove import domain_adapted_CCA, loadGloveModel\n",
    "\n",
    "def glove_main(verbose=False):\n",
    "    # Load data from nlp parsing\n",
    "    \n",
    "    if verbose:\n",
    "        print(\"Loading JSON with article source texts... \")\n",
    "    with open('{}/articles-with-equations.json'.format(settings.data_dir), 'r',\n",
    "              encoding='utf-8') as jf:\n",
    "        src_data = json.load(jf)\n",
    "        \n",
    "    if verbose:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Processing article source data for GloVe modeling... \")\n",
    "\n",
    "    texts = [src_data[art]['text'] for art in src_data if\n",
    "             src_data[art]['text'] is not None]\n",
    "\n",
    "    # The \"unidecode\" step simplifies non-ASCII chars which\n",
    "    # mess up the R GloVe engine.\n",
    "    texts_df = pd.Series(texts).apply(lambda x: unidecode(x))\n",
    "    texts_df = pd.DataFrame({'text':texts_df})\n",
    "    \n",
    "    if verbose:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Training domain-specific GloVe model... \")\n",
    "\n",
    "    # Source all the functions contained in the 'trainEmbeddings' R file\n",
    "    r(\"source('{}/trainEmbeddings.R')\".format('src/data'))\n",
    "\n",
    "    # Call the main GloVe-embedding function from the R script\n",
    "    trainEmbeddings_R = r(\"trainEmbeddings\")\n",
    "\n",
    "    \n",
    "    if verbose:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Processing GloVe model embeddings... \")\n",
    "        \n",
    "    # Train domain-specific GloVe embedding model and ouput as a Numpy Matrix\n",
    "    pandas2ri.activate()\n",
    "    DS_embeddings_R = trainEmbeddings_R(texts_df)\n",
    "    pandas2ri.deactivate()\n",
    "\n",
    "    DS_embeddings = numpy2ri.rpy2py(DS_embeddings_R[0])\n",
    "\n",
    "    \n",
    "    if verbose:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Loading pre-trained generic GloVe embeddings... \")\n",
    "        \n",
    "    # Get domain-specific GloVe vocabulary\n",
    "    domain_spec_vocab = list(DS_embeddings_R[1])\n",
    "\n",
    "    # Load in Stanford's 'Common Crawl' domain-general Glove Embedding Model\n",
    "    # Only pull out the words that are contained in our corpus\n",
    "    # * This can take a while (~30min) - could use some optimization *\n",
    "    DG_embeddings = loadGloveModel(\n",
    "        '{}/glove.42B.300d.txt'.format(settings.data_dir),\n",
    "        domain_spec_vocab\n",
    "    )\n",
    "\n",
    "    \n",
    "    if verbose:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Merging GloVe models via CCA... \")\n",
    "        \n",
    "    # Processing to ensure rows match between the domain-general and\n",
    "    # domain-specific embeddings\n",
    "    # Convert domain-general embedding from dictionary to array\n",
    "    domain_gen_vocab = np.array([DG_embeddings[i] for i in\n",
    "                                DG_embeddings.keys()])\n",
    "\n",
    "    # Find the indices of matching words\n",
    "    both = set(domain_gen_vocab).intersection(domain_spec_vocab)\n",
    "    indices_gen = [domain_gen_vocab.index(x) for x in both]\n",
    "    indices_spec = [domain_spec_vocab.index(x) for x in both]\n",
    "    indices_spec_notDG = [domain_spec_vocab.index(x) for x in\n",
    "                          domain_spec_vocab if x not in both]\n",
    "\n",
    "    # Sort and subset domain-specific array to match indices of domain-general\n",
    "    # array\n",
    "    DS_embeddings_subset = DS_embeddings[indices_spec,:].copy()\n",
    "    DG_embeddings_subset = DG_embeddings[indices_gen,:].copy()\n",
    "\n",
    "    # fit cca model\n",
    "    cca_res, DA_embeddings = domain_adapted_CCA(DG_embeddings_subset,\n",
    "                                                DS_embeddings_subset, NC=100)\n",
    "\n",
    "    DS_embeddings_notinDG = DS_embeddings[indices_spec_notDG,:]\n",
    "    DS_embeddings_notinDG_norm = zscore(DS_embeddings_notinDG)\n",
    "\n",
    "    DA_notinDG_embeddings = cca_res.y_weights_.T @ DS_embeddings_notinDG_norm.T\n",
    "    DA_embeddings_final = np.append(DA_embeddings,DA_notinDG_embeddings.T,\n",
    "                                    axis=0)\n",
    "\n",
    "    \n",
    "    if verbose:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Writing to disk. \")\n",
    "        \n",
    "    # write data to disk\n",
    "    np.savetxt('{}/da_embeddings.txt'.format(settings.models_dir),\n",
    "               DA_embeddings_final, fmt='%d')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_main(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Build the query generation network (QG-Net)\n",
    "In the next iteration, MULTIVAC will accept properly formatted queries, parse them into compatible formats and match their components to relevant nodes and clusters in its MLN knowledge base to produce answers for those queries. This functionality will be tested and refined in several ways, first using queries auto-extracted from the source texts and then with human expert review.\n",
    " \n",
    "MULTIVAC derives queries from source texts using a modified version of deep learning question-generation system called QG-Net.<su>1</su> QG-Net is a recurrent neural network-based model that takes as inputs a \"context\" and an \"answer\" and outputs a question tailored to produce that answer. For MULTIVAC, we feed in the abstracts of our source articles to the QG-Net context reader, on the theory that the abstract most succinctly encapsulates the purpose and findings of any given article. MULTIVAC uses the most important sentences in the abstract as the \"contexts\" and uses the most important words and phrases in the sentences as the \"answers.\"\n",
    " \n",
    "Term frequency-inverse document frequency scores (TF-IDF) are calculated to determine which terms/sentences in the documents are truly important and differentiating. The sentences with the largest total sum of their terms' TF-IDF scores are used to generate queries. Within these sentences, the terms or phrases (by default, the system calculates TF-IDF scores for *n-grams* up to three words) with the highest TF-IDF scores are tagged as \"answers.\"\n",
    " \n",
    "This approach differs from the method in the original QG-Net system, which depended on words listed in an index and named entity tags supplied by the Stanford NLP parser to select potential answers from context sentences. In research articles, as with the vast majority of texts which are not specifically published books, there is almost never an index or other lengthy annotated list of key words and phrases to guide this type of approach. Named entity recognition, on the other hand, can be limiting in terms of the types of information and relationships supplied for training question generators. TF-IDF scores, however, are format independent and can apply to any type of underlying semantic content, making them more versatile and powerful guides for programmatically generating queries. On top priority is to augment this selection approach with scoring based on co-occurrence with LaTeX equation tokens in the bodies of our source articles.\n",
    " \n",
    "QG-Net uses GloVe embeddings to represent words as vectors coupled with tags for part of speech, named entities, and word case (capitalized or lowercase) calculated using the Stanford natural language processing toolkit as inputs in the context reader. In MULTIVAC's version of this system, the input embeddings are supplied by the domain-adapted GloVe model previously calculated. Finally, the QG-Net input includes a binary-valued indicator to indicate whether a word is the \"answer.\"\n",
    " \n",
    "From these inputs, the question generator generates diverse question text word-by-word given all context word representations. For the prototype system, MULTIVAC generated 2,804 queries based on the source document research abstracts. Generated queries are saved in a text file in the `models_dir` folder as `output_questions_QG-Net.pt.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import corenlp\n",
    "import os\n",
    "import subprocess\n",
    "\n",
    "from multivac import settings\n",
    "from multivac.src.data.parsing import load_data\n",
    "from qgnet.test.testinput.preprocessing_pdf import create_tf_idf, preprocess_pdf\n",
    "\n",
    "\n",
    "def qgnet_main(args_dict):\n",
    "    # first, run shell script, if necessary, in qgnet to create model\n",
    "    \n",
    "    if args_dict['verbose']:\n",
    "        print(\"Establishing qgnet model locally... \")\n",
    "\n",
    "    subprocess.call([\n",
    "        '{}/download_QG-Net.sh'.format(settings.qgnet_dir),\n",
    "        args_dict['qgnet_path']\n",
    "    ])\n",
    "\n",
    "    if args_dict['verbose']:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Loading article abstracts... \")\n",
    "\n",
    "    # second, pre-process the pdfs\n",
    "    jsonObj, allDocs = load_data(settings.processed_dir / 'data.json')\n",
    "    abstracts = []\n",
    "    \n",
    "    for value in jsonObj.values():\n",
    "        if \"summary\" in value['metadata']:\n",
    "            abstracts.append(value['metadata'][\"summary\"])\n",
    "        elif \"abstract\" in value['metadata']:\n",
    "            abstracts.append(value['metadata'][\"abstract\"])\n",
    "\n",
    "    if args_dict['verbose']:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Processing abstracts... \")\n",
    "\n",
    "    nlp = corenlp.CoreNLPClient(output_format='json', properties={\n",
    "        'timeout': '50000'})\n",
    "\n",
    "    features, tfidf = create_tf_idf(abstracts, False)\n",
    "\n",
    "    for i, abstract in enumerate(abstracts):\n",
    "        preprocess_pdf(abstract, features[i,:].toarray(), tfidf, nlp)\n",
    "\n",
    "    if args_dict['verbose']:\n",
    "        print(\"Done.\\n\")\n",
    "        print(\"Generating questions... \")\n",
    "\n",
    "    # third, generate qg-net questions\n",
    "    subprocess.call([\n",
    "        '{}/test/qg_reproduce_LS.sh'.format(settings.qgnet_dir),\n",
    "        args_dict['qgnet_path'],\n",
    "        settings.models_dir\n",
    "    ])\n",
    "\n",
    "    if args_dict['verbose']:\n",
    "        print(\"Done.\")\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qgnet_main({'qgnet_path': '../qgnet', 'verbose': True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Build Markov logic network (MLN)\n",
    "The pymln subsystem reads in the `*.dep`, `*.morph`, and `*.input` files generated by the data parsing step, and compiles these into a network of nodes representing semantic tokens and edges representing their parent-child relationships. These nodes are assigned to initial semantic clusters which link them via their relationships and arguments to other nodes in the knowledge base. Each cluster maps these links via python dictionaries and sets encoding the various types of relationships, types of arguments, and specific Argument nodes in globally tracked indices.\n",
    " \n",
    "To build this probabilistic graphical model knowledge base, MULTIVAC first builds a listing of all sentences and tokens from the dep/input/morph files. Tokens are first read in line by line from the input and morph files, recording form, lemma, part-of-speech and index in each sentence. Then they are joined into dependency trees based on the links defined in the collapsed dependency representations in the dep files.\n",
    " \n",
    "When building these dependency relations, MULTIVAC ignores more purely syntactic dependencies such as punctuation or auxiliary verbs. These dependencies were essential in parsing the sentences originally, but now that the key semantic relationships are identified they would contribute little extra information in our knowledge base.\n",
    " \n",
    "Once the sentences are initially read in, MULTIVAC iterates through them and assigns each token to a semantic cluster (initially one cluster for each unique term). Then MULTIVAC recursively builds argument clusters for each child token, after which it iterates through the argument clusters and scores which argument clusters to merge based on their maximum a posteriori (MAP) log likelihood in the corpus.\n",
    " \n",
    "These initial ingest and clustering steps are relatively quick, and scale linearly with the input data. The next step in the ontology induction process is to create an agenda for merging or composing semantic clusters across the entire Markov Logic Network, using MAP log likelihood. For each cluster, the parents and children of each grounded semantic \"part\" are compared to the parents and children of all other member parts to assess whether their clusters can be merged (where two terms are considered to have the same semantic role or meaning in the network) or composed (where a parent-child combination would become the core form of the parent cluster, and the child cluster would be removed). This operation is O(n2), unfortunately. Calculating these operations over a sample of 200 of our input articles currently takes about five hours. One top priority is to leverage Python’s numpy library and vectorization to mitigate the computational cost in this step as much as possible.\n",
    " \n",
    "Once MULTIVAC has compiled an agenda of merge and compose operations to consider, the system scores each operation according to how it affects the MAP log likelihood of the overall Markov Logic Network, and operations above a certain threshold (-200 by default) are added to the active agenda for execution, highest scoring first. After each operation is executed MULTIVAC checks for any additional operations that could now be possible and adds these to the active agenda based on the same criteria, while removing any operations that are now obsolete. This cycle repeats until there are no more eligible operations in the queue, at which point MULTIVAC writes the resulting MLN to disk as a dictionary of nodes and edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from datetime import datetime\n",
    "\n",
    "from multivac import settings\n",
    "from multivac.pymln.pymln import read_input_files\n",
    "from multivac.pymln.semantic import Parse, MLN, Clust\n",
    "from multivac.pymln.syntax.StanfordParseReader import StanfordParseReader\n",
    "\n",
    "def mln_main(args_dict):\n",
    "    # set variables\n",
    "    verbose = args_dict['verbose']\n",
    "    data_dir = settings.data_dir\n",
    "    results_dir = settings.mln_dir\n",
    "    parser = Parse(args_dict['prior_num_param'], args_dict['prior_num_conj'])\n",
    "\n",
    "    # read in inputs\n",
    "    input_files = read_input_files(data_dir)\n",
    "    input_files.sort()\n",
    "\n",
    "    # set final parameter\n",
    "    if 'subset' in args_dict:\n",
    "        subset = args_dict['subset']\n",
    "    else:\n",
    "        subset = len(input_files)\n",
    "\n",
    "    articles = []\n",
    "    for i, fileName in enumerate(input_files):\n",
    "        try:\n",
    "            a = StanfordParseReader.readParse(fileName, data_dir)\n",
    "        except:\n",
    "            print(\"Error on {}, {}\".format(i, fileName))\n",
    "            raise Exception\n",
    "\n",
    "        if i%100 == 0:\n",
    "            print(\"{} articles parsed.\".format(i))\n",
    "\n",
    "        if i >= subset:\n",
    "            break\n",
    "\n",
    "        articles.append(a)\n",
    "\n",
    "\n",
    "    if verbose:\n",
    "        print(\"{} Initializing...\".format(datetime.now()))\n",
    "    parser.initialize(articles, verbose)\n",
    "\n",
    "    if verbose:\n",
    "        print(\"{}: {} articles parsed, of {} sentences and {} total tokens.\"\n",
    "              .format(datetime.now(),\n",
    "                      len(articles),\n",
    "                      parser.numSents,\n",
    "                      parser.numTkns))\n",
    "    num_arg_clusts = sum([len(x._argClusts) for x in Clust.clusts.values()])\n",
    "\n",
    "    if verbose:\n",
    "        print(\"{}: {} initial clusters, with {} argument clusters.\"\n",
    "              .format(datetime.now(), len(Clust.clusts), num_arg_clusts))\n",
    "        print(\"{} Merging arguments...\".format(datetime.now()))\n",
    "    parser.mergeArgs()\n",
    "    num_arg_clusts = sum([len(x._argClusts) for x in Clust.clusts.values()])\n",
    "\n",
    "    if verbose:\n",
    "        print(\"Now with {} initial clusters, {} argument clusters.\"\n",
    "              .format(len(Clust.clusts), num_arg_clusts))\n",
    "        print(\"{} Creating agenda...\".format(datetime.now()))\n",
    "    parser.agenda.createAgenda(verbose)\n",
    "\n",
    "    if verbose:\n",
    "        print(\"{}: {} possible operations in queue, {} merges and {} composes.\"\n",
    "              .format(datetime.now(),\n",
    "                      len(parser.agenda._agendaToScore),\n",
    "                      len(parser.agenda._mc_neighs),\n",
    "                      len(parser.agenda._compose_cnt)))\n",
    "        print(\"{} Processing agenda...\".format(datetime.now()))\n",
    "    parser.agenda.procAgenda(verbose)\n",
    "\n",
    "    num_arg_clusts = sum([len(x._argClusts) for x in Clust.clusts.values()])\n",
    "\n",
    "    if verbose:\n",
    "        print(\"{}: {} final clusters, with {} argument clusters.\"\n",
    "              .format(datetime.now(), len(Clust.clusts), num_arg_clusts))\n",
    "\n",
    "    MLN.save_mln(data_dir / \"mln.pkl\")\n",
    "    MLN.printModel(results_dir)\n",
    "\n",
    "    if verbose:\n",
    "        print(\"{} Induced MLN saved.\".format(datetime.now()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "args_dict = {'verbose': True, 'prior_num_param': 5, 'prior_num_conj': 10, 'subset': 20}\n",
    "mln_main(args_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "After running all five parts of the MULTIVAC system, a user will have all necessary components to maps questions to the developed ontology. This step -- pre-baked, if you will -- is demonstrated in the `prepared_output.ipynb` in the same top-level MULTIVAC directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}