06_Keyword_Searching_and_ErrorAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keyword Searching and Error Analysis\n",
    "\n",
    "A good starting point to determine the complexity of your problem is a keyword search: if a keyword or set of keywords appear in the document, classify the document as positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#First import packages\n",
    "import urllib.request\n",
    "import os\n",
    "import codecs\n",
    "import zipfile\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "import ipywidgets\n",
    "import sklearn.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nlp_pneumonia_utils import read_doc_annotations\n",
    "from nlp_pneumonia_utils import calculate_prediction_metrics\n",
    "from nlp_pneumonia_utils import mark_text\n",
    "from nlp_pneumonia_utils import pneumonia_annotation_html_markup\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Create a keyword classifier\n",
    "### Team exercise:\n",
    "### iterate through a set of keywords (self.keywords)\n",
    "### If a keyword in the set appears in 'text', return prediction = 1\n",
    "### else, return prediction = 0. \n",
    "Complete the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class KeywordClassifier(object):\n",
    "    def __init__(self):\n",
    "        self.keywords = set()\n",
    "    def predict(self, text):\n",
    "        prediction = 0\n",
    "        \n",
    "#         your code here\n",
    "        return prediction\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the function you just wrote by adding one keyword to the set: 'pneumonia'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading annotations from file : data/training_v2.zip\n",
      "Opening local file : data/training_v2.zip\n",
      "Total Annotated Documents : 70\n",
      "Number of documents labeled positive by keyword classifier: 30\n",
      "Number of documents labeled positive by gold standard: 34\n"
     ]
    }
   ],
   "source": [
    "keyword_classifier = KeywordClassifier()\n",
    "keyword_classifier.keywords.add('pneumonia')\n",
    "annotated_doc_map = read_doc_annotations('data/training_v2.zip')\n",
    "print('Total Annotated Documents : {0}'.format(len(annotated_doc_map)))\n",
    "\n",
    "keyword_positives = 0\n",
    "actual_positives = 0\n",
    "for doc in annotated_doc_map.values():\n",
    "    if keyword_classifier.predict(doc.text) == 1:\n",
    "        keyword_positives+=1\n",
    "        \n",
    "    if doc.positive_label == 1:\n",
    "        actual_positives+=1\n",
    "        \n",
    "print('Number of documents labeled positive by keyword classifier: ' + str(keyword_positives))\n",
    "print('Number of documents labeled positive by gold standard: ' + str(actual_positives))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Error analysis of keyword classifier\n",
    "\n",
    "### Let's first look at false negatives \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def list_false_negatives(gold_docs, prediction_function):\n",
    "    fn_docs={}\n",
    "    for doc_name, gold_doc in gold_docs.items():\n",
    "        gold_label=gold_doc.positive_label;\n",
    "        pred_label = prediction_function(gold_doc.text)\n",
    "        if gold_label==1 and pred_label==0:\n",
    "            fn_docs[doc_name]=gold_doc            \n",
    "    return fn_docs     \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13\n"
     ]
    }
   ],
   "source": [
    "fn=list_false_negatives(annotated_doc_map, keyword_classifier.predict)\n",
    "docs=list(fn.keys())\n",
    "print(len(docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How do we fix false negatives?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look through each document that is a false negative, showing  one document a time:<br/><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae0e9f2e3d744cdf9b63048411812470",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='i', max=12), Output()), _dom_classes=('widget-interact',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(i=ipywidgets.IntSlider(min=0, max=len(docs)-1))\n",
    "def display_doc(i):\n",
    "    doc_name=docs[i]    \n",
    "    print(doc_name)\n",
    "    anno_doc=fn[doc_name]\n",
    "    display(HTML(pneumonia_annotation_html_markup(anno_doc).replace('\\n', '<br>')))    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. More efficient review:\n",
    "Not convenient to read? Let's try snippet view instead. Now we need to make another function to replace \"*pneumonia_annotation_html_markup*\". \n",
    "\n",
    "Although we measuring the document level annotation, we will focus on mention level (\"**EVIDENCE_OF_PNEUMONIA**\") error analyses. Because the later is where the errors originate from.<br/><br/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def snippets_markup(annotated_doc_map):\n",
    "    html = [\"<html>\",\"<table width=100% >\",\n",
    "            \"<col style=\\\"width:25%\\\"><col style=\\\"width:75%\\\">\"\n",
    "            \"<tr><th style=\\\"text-align:center\\\">document name</th><th style=\\\"text-align:center\\\">Snippets</th>\"]\n",
    "    for doc_name, anno_doc in annotated_doc_map.items():\n",
    "        html.extend(snippet_markup(doc_name,anno_doc))\n",
    "    html.append(\"</table>\")\n",
    "    html.append(\"</html>\")\n",
    "    return ''.join(html) \n",
    "\n",
    "\n",
    "def snippet_markup(doc_name,anno_doc):\n",
    "    from pyConTextNLP.display.html import __sort_by_span\n",
    "    from pyConTextNLP.display.html import __insert_color\n",
    "    html=[]\n",
    "    color= 'blue'    \n",
    "    window_size=50    \n",
    "    html.append(\"<tr>\")\n",
    "    html.append(\"<td style=\\\"text-align:left\\\">{0}</td>\".format(doc_name))\n",
    "    html.append(\"<td></td>\")\n",
    "    html.append(\"</tr>\")\n",
    "    for anno in anno_doc.annotations:\n",
    "        if anno.type == 'EVIDENCE_OF_PNEUMONIA':\n",
    "#           make sure the our snippet will be cut inside the text boundary\n",
    "            begin=anno.start_index-window_size\n",
    "            end=anno.end_index+window_size\n",
    "            begin=begin if begin>0 else 0\n",
    "            end=end if end<len(anno_doc.text) else len(anno_doc.text)    \n",
    "#           render a highlighted snippet\n",
    "            cell=__insert_color(anno_doc.text[begin:end],[anno.start_index-begin,anno.end_index-end],color)\n",
    "#           add the snippet into table\n",
    "            html.append(\"<tr>\")\n",
    "            html.append(\"<td></td>\")\n",
    "            html.append(\"<td style=\\\"text-align:left\\\">{0}</td>\".format(cell))\n",
    "            html.append(\"</tr>\") \n",
    "    return html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it out:<br/><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><table width=100% ><col style=\"width:25%\"><col style=\"width:75%\"><tr><th style=\"text-align:center\">document name</th><th style=\"text-align:center\">Snippets</th><tr><td style=\"text-align:left\">subject_id_146_hadm_id_18965</td><td></td></tr><tr><td></td><td style=\"text-align:left\">al effusion.\n",
       "     Right CPA not included on film. <span style=\"color: blue;\">There is obscuration of left hemidiaphragm\n",
       "     likely secondary to atelectasis/consolidation</span> in left lower lobe.\n",
       "\n",
       "</td></tr><tr><td style=\"text-align:left\">subject_id_150_hadm_id_12121</td><td></td></tr><tr><td></td><td style=\"text-align:left\">es are\n",
       "     unremarkable.\n",
       "     \n",
       "     IMPRESSION:  <span style=\"color: blue;\">Small focal opacity in right upper lobe</span> and right paratracheal\n",
       "     opacity.  In the sett</td></tr><tr><td></td><td style=\"text-align:left\">CHEST PA AND LATERAL:  The heart size is normal.  <span style=\"color: blue;\">There is an area of\n",
       "     increased opacity</span> lateral to the right paratracheal stripe.  In the</td></tr><tr><td></td><td style=\"text-align:left\">pacity lateral to the right paratracheal stripe.  <span style=\"color: blue;\">In the right\n",
       "     upper lobe, there is a small focal opacity.</span>  The lungs are otherwise clear.\n",
       "     There are no</td></tr><tr><td style=\"text-align:left\">subject_id_157_hadm_id_26180</td><td></td></tr><tr><td></td><td style=\"text-align:left\">ung is incompletely imaged\n",
       "     on this study and <span style=\"color: blue;\">there is a questionable area of abnormality partially\n",
       "     obscuring the mid portion of the right hemidiaphragm</span>, incompletely evaluated.\n",
       "     \n",
       "     IMPRESSION:  </td></tr><tr><td style=\"text-align:left\">subject_id_261_hadm_id_19250</td><td></td></tr><tr><td></td><td style=\"text-align:left\">ignificant change compared with [**3197-12-9**].  <span style=\"color: blue;\">Bilateral pulmonary\n",
       "     opacities</span> involving the lower and mid lung zones.\n",
       "\n",
       "</td></tr><tr><td></td><td style=\"text-align:left\">c\n",
       "     angle is not included in this radiograph.  <span style=\"color: blue;\">There is left lower lobe\n",
       "     consolidation/collapse</span>.  There is bilateral lower zone and mid zone\n",
       "    </td></tr><tr><td></td><td style=\"text-align:left\"> is left lower lobe\n",
       "     consolidation/collapse.  <span style=\"color: blue;\">There is bilateral lower zone and mid zone\n",
       "     infiltrate.</span>  The heart size is slightly enlarged.\n",
       "     \n",
       "     </td></tr><tr><td style=\"text-align:left\">subject_id_5472_hadm_id_11987</td><td></td></tr><tr><td></td><td style=\"text-align:left\">id SVC. \n",
       "     There is no apparent pneumothorax.  <span style=\"color: blue;\">A right IJ line, NGT, and ETT are\n",
       "     unchanged as are the parenchymal changes in the lungs</span> compared to the earlier\n",
       "     chest x-ray this mor</td></tr><tr><td style=\"text-align:left\">subject_id_6349_hadm_id_20192</td><td></td></tr><tr><td></td><td style=\"text-align:left\">There is no pulmonary vascular\n",
       "     engorgement.  <span style=\"color: blue;\">There is an increasing left-sided pleural effusion with\n",
       "     associated atelectasis.  Pneumonia at this location cannot be excluded.</span>  Noted\n",
       "     is a density in the left upper lung z</td></tr><tr><td style=\"text-align:left\">subject_id_7027_hadm_id_33117</td><td></td></tr><tr><td></td><td style=\"text-align:left\">ossibility of free\n",
       "     intraperitoneal air.\n",
       "     <span style=\"color: blue;\">2) Left lower lobe atelectasis/consolidation.</span>\n",
       "     3) Moderate gastric distention with multiple</td></tr><tr><td></td><td style=\"text-align:left\">stinal and hilar contours are\n",
       "     unremarkable.  <span style=\"color: blue;\">There is patchy opacity at the left lower lobe representing\n",
       "     either atelectasis or consolidation</span>.  No definite free air is identified,\n",
       "     howeve</td></tr><tr><td style=\"text-align:left\">subject_id_7272_hadm_id_19098</td><td></td></tr><tr><td></td><td style=\"text-align:left\">rt failure with bilateral pleural effusions.\n",
       "     <span style=\"color: blue;\">Collapse and/or consolidation at the bases bilaterally.\n",
       "</span>\n",
       "</td></tr><tr><td></td><td style=\"text-align:left\">rall heart size is difficult to assess.  There is <span style=\"color: blue;\">dense retrocardiac\n",
       "     opacity, possibly secondary to collapse and/or consolidation</span> in the left lower\n",
       "     lobe.  There is also a rig</td></tr><tr><td></td><td style=\"text-align:left\">ion in the left lower\n",
       "     lobe.  There is also a <span style=\"color: blue;\">right lower lobe and middle lobe opacity consistent\n",
       "     with collapse and/or consolidation</span>.\n",
       "     \n",
       "     IMPRESSION: Persistent left heart fai</td></tr><tr><td style=\"text-align:left\">subject_id_7525_hadm_id_19141</td><td></td></tr><tr><td></td><td style=\"text-align:left\">n distal superior vena cava, unchanged.  There is <span style=\"color: blue;\">marked\n",
       "     improvement of the bilateral consolidations</span>, especially on the right. The NG\n",
       "     tube tip is</td></tr><tr><td style=\"text-align:left\">subject_id_7675_hadm_id_3</td><td></td></tr><tr><td></td><td style=\"text-align:left\">r contours are stable\n",
       "     since the prior study. <span style=\"color: blue;\">There is persistent opacification in the right lower\n",
       "     lung zone</span>. The pulmonary vascularity is unremarkable. The e</td></tr><tr><td></td><td style=\"text-align:left\">terminating in the left IJ. No\n",
       "     pneumothorax. <span style=\"color: blue;\">Persistent right lower lung zone opacification</span> and probable\n",
       "     small effusion.\n",
       "\n",
       "</td></tr><tr><td style=\"text-align:left\">subject_id_8494_hadm_id_20131</td><td></td></tr><tr><td></td><td style=\"text-align:left\">lung and right mediastinal\n",
       "     lymphadenopathy.  <span style=\"color: blue;\">Patchy opacities are seen throughout the right lung which\n",
       "     could be due to atelectasis from compression of the right lung or represent\n",
       "     air space consolidation.</span>  An ET tube is identified with tip 6.3 cm from th</td></tr><tr><td style=\"text-align:left\">subject_id_8734_hadm_id_8478</td><td></td></tr><tr><td></td><td style=\"text-align:left\">  Fluid overload vs. congestive failure.\n",
       "     2)  <span style=\"color: blue;\">Possible developing left infiltrate</span>.\n",
       "\n",
       "</td></tr><tr><td></td><td style=\"text-align:left\">minent, consistent with fluid overload.  There is <span style=\"color: blue;\">increased opacity within\n",
       "     the left lower lung. It is not clear if this is related to technique,\n",
       "     overlying soft tissue, or an underlying infiltrate</span>. There is no definite\n",
       "     pleural effusion.  The</td></tr><tr><td style=\"text-align:left\">subject_id_9082_hadm_id_29395</td><td></td></tr><tr><td></td><td style=\"text-align:left\">tient with seizure.\n",
       "     \n",
       "     Low lung volumes.  <span style=\"color: blue;\">Bilateral basilar opacities,</span> considerably larger at the\n",
       "     left base than at</td></tr><tr><td></td><td style=\"text-align:left\">hyroid.\n",
       "     \n",
       "     IMPRESSION:  Lung volumes with <span style=\"color: blue;\">bilateral basilar opacities.</span>\n",
       "     \n",
       "     Question substernal thyroid enlargemen</td></tr></table></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fn=list_false_negatives(annotated_doc_map, keyword_classifier.predict)\n",
    "docs=list(fn.keys())\n",
    "display(HTML(snippets_markup(fn)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Add more keywords to your keyword classifier to decrease the number of false negatives\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'pneumonia', 'consolidation'}\n"
     ]
    }
   ],
   "source": [
    "keyword_classifier.keywords = {'pneumonia', 'consolidation'}\n",
    "print(keyword_classifier.keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Compute the outcome metrics using your KeywordClassifier\n",
    "\n",
    "Let's copy and paste the *keyword_classifier* definition below for editing convenience. Reuse the *calculate_prediction_metrics* function. We need to change a little bit of its input parameter, because it needs a list for the 1st parameter, and our *annotated_doc_map* is a dictionary. So we use *list(annotated_doc_map.values())* instead of *annotated_doc_map* directly.<br/><br/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision : 0.6829268292682927\n",
      "Recall :    0.8235294117647058\n",
      "F1:         0.7466666666666667\n",
      "\n",
      "Confusion Matrix : \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>23</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted   0   1\n",
       "Actual           \n",
       "0          23  13\n",
       "1           6  28"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res=calculate_prediction_metrics(list(annotated_doc_map.values()), keyword_classifier.predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision : 0.6829268292682927\n",
      "Recall :    0.8235294117647058\n",
      "F1:         0.7466666666666667\n",
      "\n",
      "Confusion Matrix : \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>23</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted   0   1\n",
       "Actual           \n",
       "0          23  13\n",
       "1           6  28"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "gold_docs=list(annotated_doc_map.values())\n",
    "prediction_function=keyword_classifier.predict\n",
    "gold_labels = [x.positive_label for x in gold_docs]\n",
    "pred_labels = []\n",
    "for gold_doc in gold_docs:\n",
    "    pred_label = prediction_function(gold_doc.text)\n",
    "    pred_labels.append(pred_label)\n",
    "\n",
    "# now let's use scikit-learn to compute some metrics\n",
    "precision = sklearn.metrics.precision_score(gold_labels, pred_labels)\n",
    "recall = sklearn.metrics.recall_score(gold_labels, pred_labels)\n",
    "f1 = sklearn.metrics.f1_score(gold_labels, pred_labels)\n",
    "# let's use Pandas to make a confusion matrix for us\n",
    "confusion_matrix_df = pd.crosstab(pd.Series(gold_labels, name='Actual'),\n",
    "                                  pd.Series(pred_labels, name='Predicted'))\n",
    "\n",
    "print('Precision : {0}'.format(precision))\n",
    "print('Recall :    {0}'.format(recall))\n",
    "print('F1:         {0}'.format(f1))\n",
    "\n",
    "print('\\nConfusion Matrix : ')\n",
    "display(confusion_matrix_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Quiz\n",
    "Try the following questions, see if you've understood this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9ab2ea98aac94982bc3ca94b1fe48444",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "RadioButtons(description='False negative means:', layout=Layout(width='600px'), options=('Negative in both gol…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "771f513e69da48e4ac2ffceea0adc4f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Submit', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_1\n",
    "error_analyses_1()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><body>What type of data will be returned by the function <b>\"list_false_negatives\"</b>?</body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b969def6ce564b3186858a959550bcf4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "RadioButtons(description='Which is corret:', layout=Layout(width='600px'), options=('list', 'array', 'dictiona…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29db1077572644cc887c9f969f6fb5fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Submit', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_2\n",
    "error_analyses_2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><body>If you see the following expert annotation in gold reference, but your KeywordClassifier predict the corresponding document as negative answer. Which keyword above is the best choice to add in your KeywordClassifier?<br/><br/>The mediastinal and hilar contours are\n",
       "unremarkable. There is <span style=\"color: blue;\">patchy opacity at the left lower lobe representing\n",
       "either atelectasis or consolidation</span>. No definite free air is identified.</body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html><body><span style=\"color: green;\">This is a better answer<span></body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_3\n",
    "error_analyses_3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2019.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Kelly Peterson, Alec Chapman, Jianlin Shi <br> Acknowledgement: Many thanks to Olga Patterson because part of the materials are adopted from his previous work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}