Day4_scispacy_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Goal: Explore other great models, components and pipelines from other sources like 'scispacy' which comes from the Allen Institute for Artificial Intelligence (AllenAI) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "from spacy import displacy\n",
    "\n",
    "from scispacy.abbreviation import AbbreviationDetector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The scispacy models and included components include many features which are useful for biomedical text and potentially even clinical text since it provides an improved tokenizer for medical tokens and improved sentence boundary detection as trained from large amounts of biomedical literature.\n",
    "## Today we will look at the AbbreviationDetector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Spinal and bulbar muscular atrophy (SBMA) is an \\\n",
    "           inherited motor neuron disease caused by the expansion \\\n",
    "           of a polyglutamine tract within the androgen receptor (AR). \\\n",
    "           SBMA can be caused by this easily.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_nlp = spacy.load(\"en_core_sci_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for pipe in default_nlp.pipeline:\n",
    "    print(pipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scispacy and other components and models than extend beyond the base spacy functionality use a special variable inside Documents, Tokens, and Spans called the extension variable.  This is a class member components can use to register names and variables of new knowledge or linguistic information that it adds.  This name of this member variable is the literal underscore character (_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_document_extensions(spacy_doc):\n",
    "    document_extensions = spacy_doc._._extensions.keys()\n",
    "    print(document_extensions)\n",
    "    return len(document_extensions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = default_nlp(text)\n",
    "\n",
    "default_pipeline_extension_count = count_document_extensions(doc)\n",
    "    \n",
    "print('Total tokens with extension values : {}'.format(default_pipeline_extension_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Add the Abbreviation Detector to our pipeline and we can get extended information from its processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abbreviation_nlp = spacy.load(\"en_core_sci_sm\")\n",
    "\n",
    "# Add the abbreviation pipe to the spacy pipeline.\n",
    "abbreviation_pipe = AbbreviationDetector(abbreviation_nlp)\n",
    "abbreviation_nlp.add_pipe(abbreviation_pipe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's look at the pipeline components\n",
    "for pipe in abbreviation_nlp.pipeline:\n",
    "    print(pipe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc2 = abbreviation_nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abbreviation_pipeline_extension_count = count_document_extensions(doc2)\n",
    "    \n",
    "print('Total tokens with extension values : {}'.format(abbreviation_pipeline_extension_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's look at the abbreviations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Abbreviation\", \"\\t\", \"Definition\")\n",
    "for abrv in doc2._.abbreviations:\n",
    "    print(f\"{abrv} \\t ({abrv.start}, {abrv.end}) {abrv._.long_form}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(doc2._._extensions.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:scispacy]",
   "language": "python",
   "name": "conda-env-scispacy-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}