load_model_and_predict.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Load Model and Predict\n",
    "\n",
    "This tutorial shows steps of loading the pre-trained models (Logistic Regression, and Hidden Markov Model) and predicting the utterance-level topic labels using the packages in the repository. Test data file is needed to run prediction. \n",
    "\n",
    "\n",
    "## Requirements\n",
    "Currently our package supports `Python 2` with the following packages. `Python 3` will be supported in the near future.\n",
    "- `numpy`\n",
    "- `nltk`\n",
    "- `pandas`\n",
    "- `sklearn`\n",
    "- `csv`\n",
    "- `cPickle`. \n",
    "\n",
    "-------------------------------------------------------_\n",
    "\n",
    "# Data\n",
    "## File Paths\n",
    "Set the path to the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "te_data_file = './data/sample_test_data.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Classes\n",
    "Data preprocessing is done at the initialization step when creating data classes.<br>\n",
    "Training and test data classes are slightly different since labels and vocabulary are determined only at the training step.\n",
    "\n",
    "An object of class `MHDTrainData` should be put in as an argument for `.fit_model` function, <br>\n",
    "and an object of class `MHDTestData` should be plugged into the `.predict_*` function for each model.\n",
    "\n",
    "Since we are going to load the pre-trained model, we only load the test data using `MHDTestData`. <br>\n",
    "When loading is finished, pre-processed test data will be saved to `corpus_pkl` file (in the argument). <br>\n",
    "Saving the preprocessed file into `corpus_pkl` file can save time when loading the same file again. <br> \n",
    "Loading the test data corpus from the pickle file can be disabled by setting the argument `reload_corpus` to `True`.<br>\n",
    "Also, the label and vocabulary from the training data are loaded.  <br>\n",
    "Those files are already available in the current repository as files `label.pkl` and `vocab.pkl`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mhddata import MHDTestData"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mhdtest = MHDTestData(te_data_file, nouns_only=False, ignore_case=True,\n",
    "                 remove_numbers=False, sub_numbers=True, proper_nouns_dir=\"./stopwordlists\",\n",
    "                 min_wlen=1, token_pattern=r\"(?u)[A-Za-z\\?\\!\\-\\.']+\", verbose=3, \n",
    "                 reload_corpus=True, corpus_pkl='./data/corpus_te.pkl', \n",
    "                 tr_label_pkl='./data/label.pkl', tr_vocab_pkl='./data/vocab.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Models\n",
    "\n",
    "Since we are loading the pre-trained models, we only talk about **loading** the model, **not training**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Logistic Regression Models\n",
    "Load the pre-trained model from `./lrdialog_ovr.pkl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from models import LogRegDialogModel\n",
    "\n",
    "lr = LogRegDialogModel(lr_type='ovr')\n",
    "lr.load_model(model_file=\"./model/lrdialog_ovr.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now run prediction using the loaded model with the loaded test data. <br>\n",
    "Utterance-level results will be saved to an output file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "lr.predict(mhdtest, verbose=1, output_filename=\"./utter_level_results_lrovr.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output the scores to see the scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lr.result.scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also can print out the scores as csv and save it to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lr.result.print_scores(filename='./result_in_diff_metrics.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the output probability and predictions to pkl files. (Used in 3.)\n",
    "HMM on top of any base class can be run by loading predictions and out probs. <br>\n",
    "To test the case, we will save the output probabilities and predictions from above (logistic regression results). <br>\n",
    "We can assume that this results are from a recurrent neural network (RNN), for example.<bR>\n",
    "These files will be loaded later in the part 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "predfile = './fake_rnn_pred.pkl'\n",
    "outprobfile = './fake_rnn_prob.pkl'\n",
    "\n",
    "import cPickle as cp\n",
    "with open(outprobfile, 'wb') as f:\n",
    "    cp.dump(lr.result.output_prob, f)\n",
    "with open(predfile, 'wb') as f:\n",
    "    cp.dump(lr.result.predictions, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. HMM on top of LR\n",
    "Running HMM requires you to have an object of **`base_model`**, which should be trained and predicted in advance and given as an argument. <br>\n",
    "The object has to have `.result` field since HMM is using the output probabilities from the model. \n",
    "<br>Here we use the logistic regression model that was trained and predicted above.<br>\n",
    "**NOTE: The base model and the HMM should share the same train and test data!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from models import HMMDialogModel\n",
    "hmmlr = HMMDialogModel(base_model=lr)  # lr: logistic regression model from the previous part."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loads the model. HMM pickle file has transition probabilities as well as start and ending probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hmmlr.load_model(model_file='./model/hmmdialog.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predicts the output labels using HMM and Viterbi decoding. <br>\n",
    "Also outputs the utterance-level results to a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hmmlr.predict_viterbi(mhdtest, output_filename=\"./utter_level_results_hmm_lrovr.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hmmlr.result.scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. HMM on top of other output probabilities\n",
    "\n",
    "If we have a set of results from another base model (independent model) that is trained somewhere else (e.g. output from RNN), <br>\n",
    "we can load the predictions and output probabilities and plug them into HMM. <br>\n",
    "They should be the result of the same data as `mhdtest`.\n",
    "- `predictions`:  Should have a list of sessions, where each session is a 2-d array with size `(N,T)`, where `N` is the number of utterances in the session and `T` is the number of topics (labels). Each entry is the $p(topic|utterance)$ in each session.  <br> Type: `list[ 2-d np.array[float] ]`.\n",
    "- `output_probs`: Should have a list of sessions, where each session is a list of utterance predictions within that session. <br> Type: `list[list[int]]` or `list[np.array[int]]`\n",
    "\n",
    "\n",
    "After loading predictions and probabilities, a base model object should have the following data\n",
    "and it can be plugged in as an argument to HMMDialogModel\n",
    "- base_model.result\n",
    "- base_model.result.output_prob\n",
    "- base_model.model_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models import DialogModel, HMMDialogModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the files that we saved at the end of part 1. <br>\n",
    "Remember these are actually from the Logistic Regression classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predfile = './fake_rnn_pred.pkl'\n",
    "outprobfile = './fake_rnn_prob.pkl'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are not from RNN, but let's say we've loaded the results from RNN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "rnn = DialogModel()\n",
    "rnn.load_results(mhdtest, model_info=\"RNN\", marginals=None, \n",
    "                 predictions=predfile, output_probs=outprobfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load HMM pickle again and predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hmmrnn = HMMDialogModel(base_model=rnn)\n",
    "hmmrnn.load_model(model_file='./model/hmmdialog.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "hmmrnn.predict_viterbi(mhdtest, output_filename=\"./utter_level_results_fake_hmm_rnn.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case we should have the same result as the result at section 2. since we've loaded the same result from LR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hmmlr.result.scores"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}