Deep_Wide_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we learn how to use the tf.esimator API to train a wide linear model and a deep feed-forward neural network. This approach combines the strengths of memorization and generalization.\n",
    "\n",
    "![wide and deep model](wide_n_deep.svg)\n",
    "\n",
    "The explaination of the model from tensorflow website as following:\n",
    "\n",
    "The figure above shows a comparison of a wide model (logistic regression with sparse features and transformations), a deep model (feed-forward neural network with an embedding layer and several hidden layers), and a Wide & Deep model (joint training of both). At a high level, here are the steps using the tf.estimator API:\n",
    "\n",
    "1. Preprocess our movielens dataset in pandas.\n",
    "2. Define features\n",
    "3. Build inputs from the original dataset \n",
    "4. Hash string type categorical features and use int type features value as category id directly.\n",
    "5. Create embeddings of sparse features for the deep model.\n",
    "6. Define features for both the deep and the wide part of the model.\n",
    "7. Train and validate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load dataset and split train and valid set. This is same step we have shown in the previous notebook for collaborative filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def load_movie_lens():\n",
    "    age_desc = {\n",
    "        1: \"Under 18\", 18: \"18-24\", 25: \"25-34\", 35: \"35-44\", 45: \"45-49\", 50: \"50-55\", 56: \"56+\"\n",
    "    }\n",
    "    occupation_desc = { \n",
    "        0: \"other or not specified\", 1: \"academic/educator\", 2: \"artist\", 3: \"clerical/admin\",\n",
    "        4: \"college/grad student\", 5: \"customer service\", 6: \"doctor/health care\",\n",
    "        7: \"executive/managerial\", 8: \"farmer\", 9: \"homemaker\", 10: \"K-12 student\", 11: \"lawyer\",\n",
    "        12: \"programmer\", 13: \"retired\", 14: \"sales/marketing\", 15: \"scientist\", 16: \"self-employed\",\n",
    "        17: \"technician/engineer\", 18: \"tradesman/craftsman\", 19: \"unemployed\", 20: \"writer\"\n",
    "    }\n",
    "    rating_data = pd.read_csv(\n",
    "        \"ml-1m/ratings.dat\",\n",
    "        sep=\"::\",\n",
    "        engine=\"python\",\n",
    "        encoding=\"latin-1\",\n",
    "        names=['userid', 'movieid', 'rating', 'timestamp'])\n",
    "    user_data = pd.read_csv(\n",
    "        \"ml-1m/users.dat\", \n",
    "        sep='::', \n",
    "        engine='python', \n",
    "        encoding='latin-1',\n",
    "        names=['userid', 'gender', 'age', 'occupation', 'zipcode']\n",
    "    )\n",
    "    user_data['age_desc'] = user_data['age'].apply(lambda x: age_desc[x])\n",
    "    user_data['occ_desc'] = user_data['occupation'].apply(lambda x: occupation_desc[x])\n",
    "    movie_data = pd.read_csv(\n",
    "        \"ml-1m/movies.dat\",\n",
    "        sep='::', \n",
    "        engine='python', \n",
    "        encoding='latin-1',\n",
    "        names=['movieid', 'title', 'genre']\n",
    "    )\n",
    "    dataset = pd.merge(pd.merge(rating_data, movie_data, how=\"left\", on=\"movieid\"), user_data, how=\"left\", on=\"userid\")\n",
    "    adj_col = dataset['movieid']\n",
    "    adj_col_uni = adj_col.sort_values().unique()\n",
    "    adj_df = pd.DataFrame(adj_col_uni).reset_index().rename(columns = {0:'movieid','index':'adj_movieid'})\n",
    "    dataset = pd.merge(adj_df,dataset,how=\"right\", on=\"movieid\")\n",
    "    dataset['adj_userid'] = dataset['userid'] - 1\n",
    "    return dataset\n",
    "\n",
    "def split_dataset(dataset, split_frac=.7):\n",
    "    dataset = dataset.sample(frac=1, replace=False)\n",
    "    n_split = int(len(dataset)*split_frac)\n",
    "    trainset = dataset[:n_split]\n",
    "    validset = dataset[n_split:]\n",
    "    return trainset, validset\n",
    "\n",
    "fullset = load_movie_lens()\n",
    "trainset, validset = split_dataset(fullset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. define features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the dataset, we know that we have following features: \"genre\", \"zipcode\", \"gender\", \"age\", \"occupation\".\n",
    " - The data type of \"genre\", \"zipcode\", \"gender\" are string, the data type of \"age\", \"occupation\" are int. So we group the features in STR and INT groups accordingly for further encoding.\n",
    " - We select all features for the deep model\n",
    " - We select some feature transformation for the wide model.<br>\n",
    "\n",
    "<font color=blue>The feature selection for deep and wide parts of the model is flexible, you can try out different combinations.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "CAT_STR_COLS = [\"genre\", \"zipcode\" ,\"gender\"]\n",
    "CAT_INT_COLS = [\"age\", \"occupation\"]\n",
    "LABEL_COL = \"rating\"\n",
    "DEEP_COLS = CAT_STR_COLS + CAT_INT_COLS\n",
    "WIDE_COL_CROSSES = [[\"gender\", \"age\"], [\"gender\", \"occupation\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. build inputs from original dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this Deep and Widel Model API expects sparse tensors as inputs, we convert here all the feature columns and the label column from our original dataset to sparse tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_inputs(dataframe):\n",
    "    \"\"\"\n",
    "    Creates sparse tensors to hold our feature values and constants to hold our label values.\n",
    "    For each feature we have selected for the deep and wide model, we create a sparse tensor. We use tf.SparseTensor \n",
    "    to create sparse tensors for features, and use tf.constant to create a constant with label values.\n",
    "    \n",
    "    Arguments:\n",
    "    dataframe -- pandas dataframe containing the values of features and labels.\n",
    "    \n",
    "    Returns:\n",
    "    feature_inputs -- a dictionary of sparse tensors of features.\n",
    "    label_input -- a constant with shape of [number of training example, 1]\n",
    "    \"\"\" \n",
    "    feature_inputs = {\n",
    "        col_name: tf.SparseTensor(\n",
    "            indices = [[i, 0] for i in range(len(dataframe[col_name]))],\n",
    "            values = dataframe[col_name].values,\n",
    "            dense_shape = [len(dataframe[col_name]), 1]\n",
    "        )\n",
    "        for col_name in CAT_STR_COLS + CAT_INT_COLS\n",
    "    }\n",
    "    label_input = tf.constant(dataframe[LABEL_COL].values-1)\n",
    "    return (feature_inputs, label_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. create hash buckets for categorical features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define two functions to encode string type categorical features and int type categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_hash_columns(CAT_STR_COLS):\n",
    "    \"\"\"\n",
    "    Use tf.feature_column.categorical_column_with_hash_bucket to encode the string type categorical features.\n",
    "    Documentation of this function from tensorflow:\n",
    "        Use this when your sparse features are in string or integer format, and you want to distribute your inputs \n",
    "        into a finite number of buckets by hashing. output_id = Hash(input_feature_string) % bucket_size.\n",
    "    \n",
    "    Arguments:\n",
    "    CAT_STR_COLS -- string type categorical columns.\n",
    "    \n",
    "    Returns:\n",
    "    hashed_layers -- 3 hashed categorical columns in a list.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    hashed_columns = [\n",
    "        tf.feature_column.categorical_column_with_hash_bucket(col_name, hash_bucket_size=1000) \n",
    "        for col_name in CAT_STR_COLS\n",
    "    ]\n",
    "    return hashed_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_int_columns(CAT_INT_COLS):   \n",
    "    \"\"\"\n",
    "    Use tf.feature_column.categorical_column_with_identity to encode the int type categorical features.\n",
    "    Documentation of this function from tensorflow:\n",
    "        Use this when your inputs are integers in the range [0, num_buckets), and you want to use the \n",
    "        input value itself as the categorical ID.\n",
    "    \n",
    "    Arguments:\n",
    "    CAT_INT_COLS -- int type categorical columns.\n",
    "    \n",
    "    Returns:\n",
    "    hashed_layers -- 2 categorical columns in a list.\n",
    "    \n",
    "    \"\"\"\n",
    "    int_columns = [\n",
    "        tf.feature_column.categorical_column_with_identity(col_name, num_buckets=1000, default_value=0)\n",
    "        for col_name in CAT_INT_COLS\n",
    "    ]\n",
    "    return int_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the tensorflow tutorial, they have used tf.feature_column.categorical_column_with_vocabulary_list to create the int type categorical columns.\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    age = tf.feature_column.categorical_column_with_vocabulary_list(\"age\", [1,18, 25, 35, 45, 50, 56])<br>\n",
    "    occupation = tf.feature_column.categorical_column_with_vocabulary_list(\"occupation\", [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20])<br>\n",
    "    age = tf.feature_column.indicator_column(age)<br>\n",
    "    occupation = tf.feature_column.indicator_column(occupation)<br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. create embedding for sparse features "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create dense embeddings for the sparse features to feed into DNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_embeddings(hashed_columns, int_columns, dim=6):\n",
    "    \"\"\"\n",
    "    Create embeddings for sparse features in the deep model. We use function tf.feature_column.embedding_colum to \n",
    "    convert the categorical columns we have created from the above steps to a dense representation.\n",
    "    \n",
    "    Arguments:\n",
    "    hashed_columns -- all categorical columnns that came out of the make_hash_columns function \n",
    "                    and are going to be fed into the DNN.\n",
    "    int_columns -- all categorical columnns that came out of the make_int_columns function \n",
    "                    and are going to be fed into the DNN.\n",
    "    dim -- 6, hyper-parameter dimension for the feature embeddings.\n",
    "    \n",
    "    Returns:\n",
    "    emdedding_layers -- list of columns with dense (embedded) representations.\n",
    "    \n",
    "    \"\"\" \n",
    "    embedding_layers = [\n",
    "        tf.feature_column.embedding_column(\n",
    "            column,\n",
    "            dimension=dim\n",
    "        )\n",
    "        for column in hashed_columns+int_columns\n",
    "    ]\n",
    "    return embedding_layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. define features for the wide part"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our case, all the columns from embedding layers should go into the deep model, so we our deep model input equals to embedding_layers and we are not going to write a function for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_wide_input_layers(WIDE_COL_CROSSES):\n",
    "    \"\"\"\n",
    "    Make input cross features for the wide model. We use the tf.feature_column.crossed_column function to hash the \n",
    "    cross transformation.\n",
    "    \n",
    "    Arguments:\n",
    "    WIDE_COL_CROSSES -- cross feature combinations.\n",
    "    \n",
    "    Returns:\n",
    "    crossed_wide_input_layers -- input cross features for the wide model.\n",
    "    \n",
    "    \"\"\" \n",
    "    crossed_wide_input_layers = [\n",
    "        tf.feature_column.crossed_column([c for c in cs], hash_bucket_size=int(10**(3+len(cs))))\n",
    "        for cs in WIDE_COL_CROSSES\n",
    "    ]\n",
    "    return crossed_wide_input_layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. train and validate the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we provide input features for the deep model and wide model, define the number of layers and layer sizes of DNN \n",
    "and create the model with tf.contrib.learn.DNNLinearCombinedClassifier. We save the model in directory ./model/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "create input layers...done!\n",
      "create model...INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcf41ae5668>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 10, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': './model/'}\n",
      "done!\n",
      "training model...WARNING:tensorflow:From /home/cynthia/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:641: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:From /home/cynthia/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/optimizers.py:160: assert_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.assert_global_step\n",
      "WARNING:tensorflow:From /home/cynthia/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/optimizers.py:160: assert_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.assert_global_step\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from ./model/model.ckpt-1000\n",
      "INFO:tensorflow:Saving checkpoints for 1001 into ./model/model.ckpt.\n",
      "INFO:tensorflow:loss = 1.44681, step = 1001\n",
      "INFO:tensorflow:global_step/sec: 0.881767\n",
      "INFO:tensorflow:loss = 1.44639, step = 1101 (113.410 sec)\n",
      "INFO:tensorflow:global_step/sec: 0.919119\n",
      "INFO:tensorflow:loss = 1.44599, step = 1201 (108.799 sec)\n",
      "INFO:tensorflow:global_step/sec: 0.917763\n",
      "INFO:tensorflow:loss = 1.4456, step = 1301 (108.961 sec)\n",
      "INFO:tensorflow:global_step/sec: 0.913875\n",
      "INFO:tensorflow:loss = 1.44523, step = 1401 (109.424 sec)\n",
      "INFO:tensorflow:global_step/sec: 0.917907\n",
      "INFO:tensorflow:loss = 1.44485, step = 1501 (108.944 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1547 into ./model/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 0.907156\n",
      "INFO:tensorflow:loss = 1.44448, step = 1601 (110.235 sec)\n",
      "INFO:tensorflow:global_step/sec: 0.781231\n",
      "INFO:tensorflow:loss = 1.44411, step = 1701 (128.003 sec)\n",
      "INFO:tensorflow:global_step/sec: 0.629334\n",
      "INFO:tensorflow:loss = 1.44374, step = 1801 (158.898 sec)\n",
      "INFO:tensorflow:global_step/sec: 0.627976\n",
      "INFO:tensorflow:loss = 1.44336, step = 1901 (159.244 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1960 into ./model/model.ckpt.\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into ./model/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 1.44299.\n",
      "done!\n",
      "evaluating model...WARNING:tensorflow:From /home/cynthia/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:641: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "INFO:tensorflow:Starting evaluation at 2017-09-22-18:15:36\n",
      "INFO:tensorflow:Restoring parameters from ./model/model.ckpt-2000\n",
      "INFO:tensorflow:Evaluation [1/1]\n",
      "INFO:tensorflow:Finished evaluation at 2017-09-22-18:15:40\n",
      "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.34907, global_step = 2000, loss = 1.44281\n",
      "done!\n",
      "calculating predictions...INFO:tensorflow:Restoring parameters from ./model/model.ckpt-2000\n",
      "done!\n",
      "calculating probabilites...INFO:tensorflow:Restoring parameters from ./model/model.ckpt-2000\n",
      "done!\n"
     ]
    }
   ],
   "source": [
    "print(\"create input layers...\", end=\"\")\n",
    "hash_columns = make_hash_columns(CAT_STR_COLS)\n",
    "int_columns = make_int_columns(CAT_INT_COLS)\n",
    "embedding_layers = make_embeddings(hash_columns, int_columns,dim =6)\n",
    "deep_input_layers = embedding_layers\n",
    "wide_input_layers = make_wide_input_layers(WIDE_COL_CROSSES)\n",
    "print(\"done!\")\n",
    "print(\"create model...\", end=\"\")\n",
    "model = tf.contrib.learn.DNNLinearCombinedClassifier(\n",
    "    n_classes=5,\n",
    "    linear_feature_columns = wide_input_layers,\n",
    "    dnn_feature_columns = deep_input_layers,\n",
    "    dnn_hidden_units = [32, 16],\n",
    "    fix_global_step_increment_bug=True,\n",
    "    config = tf.contrib.learn.RunConfig(\n",
    "        keep_checkpoint_max = 1,\n",
    "        save_summary_steps = 10,\n",
    "        model_dir = \"./model/\"\n",
    "    )\n",
    ")\n",
    "print(\"done!\")\n",
    "print(\"training model...\", end=\"\")\n",
    "model.fit(input_fn = lambda: make_inputs(trainset), steps=1000)\n",
    "print(\"done!\")\n",
    "print(\"evaluating model...\", end=\"\")\n",
    "results = model.evaluate(input_fn = lambda: make_inputs(validset), steps=1)\n",
    "print(\"done!\")\n",
    "print(\"calculating predictions...\", end=\"\")\n",
    "predictions = model.predict_classes(input_fn = lambda: make_inputs(validset))\n",
    "print(\"done!\")\n",
    "print(\"calculating probabilites...\", end=\"\")\n",
    "probabilities = model.predict_proba(input_fn = lambda: make_inputs(validset))\n",
    "print(\"done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 1.4464459\n",
      "accuracy: 0.34885341\n",
      "global_step: 1000\n"
     ]
    }
   ],
   "source": [
    "for n, r in results.items():\n",
    "    print(\"%s: %a\"%(n, r))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predict = list(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prob = list(probabilities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DNN Accuracy: 34.907003%\n"
     ]
    }
   ],
   "source": [
    "dnw_accuracy = np.sum(np.asarray(predict)+1 == validset.rating.values) / len(validset)\n",
    "print(\"DNW Accuracy: %f%%\"%(dnw_accuracy*100,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>age_desc</th>\n",
       "      <th>occ_desc</th>\n",
       "      <th>title</th>\n",
       "      <th>genre</th>\n",
       "      <th>rating</th>\n",
       "      <th>prediction</th>\n",
       "      <th>rating1</th>\n",
       "      <th>rating2</th>\n",
       "      <th>rating3</th>\n",
       "      <th>rating4</th>\n",
       "      <th>rating5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>223742</th>\n",
       "      <td>M</td>\n",
       "      <td>35-44</td>\n",
       "      <td>clerical/admin</td>\n",
       "      <td>Little Princess, The (1939)</td>\n",
       "      <td>Children's|Drama</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.025210</td>\n",
       "      <td>0.068422</td>\n",
       "      <td>0.234015</td>\n",
       "      <td>0.375578</td>\n",
       "      <td>0.296774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>915512</th>\n",
       "      <td>M</td>\n",
       "      <td>25-34</td>\n",
       "      <td>unemployed</td>\n",
       "      <td>Misery (1990)</td>\n",
       "      <td>Horror</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0.105347</td>\n",
       "      <td>0.148656</td>\n",
       "      <td>0.290903</td>\n",
       "      <td>0.287796</td>\n",
       "      <td>0.167298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209015</th>\n",
       "      <td>M</td>\n",
       "      <td>25-34</td>\n",
       "      <td>technician/engineer</td>\n",
       "      <td>Supercop (1992)</td>\n",
       "      <td>Action|Thriller</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0.050473</td>\n",
       "      <td>0.120541</td>\n",
       "      <td>0.292055</td>\n",
       "      <td>0.350531</td>\n",
       "      <td>0.186400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>719570</th>\n",
       "      <td>M</td>\n",
       "      <td>25-34</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Red Violin, The (Le Violon rouge) (1998)</td>\n",
       "      <td>Drama|Mystery</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0.057457</td>\n",
       "      <td>0.115074</td>\n",
       "      <td>0.279706</td>\n",
       "      <td>0.332208</td>\n",
       "      <td>0.215554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283590</th>\n",
       "      <td>M</td>\n",
       "      <td>35-44</td>\n",
       "      <td>programmer</td>\n",
       "      <td>Manon of the Spring (Manon des sources) (1986)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.025996</td>\n",
       "      <td>0.079624</td>\n",
       "      <td>0.237717</td>\n",
       "      <td>0.393170</td>\n",
       "      <td>0.263494</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       gender age_desc                occ_desc  \\\n",
       "223742      M    35-44          clerical/admin   \n",
       "915512      M    25-34              unemployed   \n",
       "209015      M    25-34     technician/engineer   \n",
       "719570      M    25-34  other or not specified   \n",
       "283590      M    35-44              programmer   \n",
       "\n",
       "                                                 title             genre  \\\n",
       "223742                     Little Princess, The (1939)  Children's|Drama   \n",
       "915512                                   Misery (1990)            Horror   \n",
       "209015                                 Supercop (1992)   Action|Thriller   \n",
       "719570        Red Violin, The (Le Violon rouge) (1998)     Drama|Mystery   \n",
       "283590  Manon of the Spring (Manon des sources) (1986)             Drama   \n",
       "\n",
       "        rating  prediction   rating1   rating2   rating3   rating4   rating5  \n",
       "223742       4           4  0.025210  0.068422  0.234015  0.375578  0.296774  \n",
       "915512       4           3  0.105347  0.148656  0.290903  0.287796  0.167298  \n",
       "209015       3           4  0.050473  0.120541  0.292055  0.350531  0.186400  \n",
       "719570       3           4  0.057457  0.115074  0.279706  0.332208  0.215554  \n",
       "283590       4           4  0.025996  0.079624  0.237717  0.393170  0.263494  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = validset[[\"gender\",\"age_desc\",\"occ_desc\", \"title\", \"genre\", \"rating\"]].copy()\n",
    "results[\"prediction\"] = np.asarray(predict)+1\n",
    "results[\"rating1\"] = np.vstack(prob)[:,0]\n",
    "results[\"rating2\"] = np.vstack(prob)[:,1]\n",
    "results[\"rating3\"] = np.vstack(prob)[:,2]\n",
    "results[\"rating4\"] = np.vstack(prob)[:,3]\n",
    "results[\"rating5\"] = np.vstack(prob)[:,4]\n",
    "results.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}