ALS_Matrix_Factorization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we demonstrate how to use tensorflow API to design and implement a matrix factorization model for predicting the movie ratings on the movie lens 1m dataset.\n",
    "\n",
    "In our dataset, we have user ids and movie ids and ratings from each user on different movies. This gives us information of user preferences. \n",
    "\n",
    "Using this information, we want to learn movie and user features and use these features to predict the rating, so that we can recommend the potentially highest rated movies to our users. \n",
    "\n",
    "![ALS matrix factorization model](Picture1.png)\n",
    "\n",
    "- users $\\{1,...,N\\}$ with user features $\\{\\theta_1,...,\\theta_N\\} =: \\theta$\n",
    "- movies $\\{1,...,M\\}$ with movie features $\\{\\phi_1,...,\\phi_M\\} =: \\phi$\n",
    "- dataset $D = \\{r_{i,j}: user\\ i\\ has\\ rated\\ movie\\ j\\}$ of all ratings\n",
    "- predicted rating of user $i$ of movie $j$: $\\hat{r}_{i,j} = \\theta_i^T \\cdot \\phi_j$\n",
    "- cost function $J(\\theta, \\phi)  = \\sum_{r_{i,j} \\in D} (r_{i,j} - \\hat{r}_{i,j})^2$\n",
    "- We will use ALS method for matrix factorization. This means we calculate two gradients of our cost function $\\nabla_\\theta J$ and $\\nabla_\\phi J$ independently in order to update our parameters $\\theta$ and $\\phi$ one after another.\n",
    " \n",
    "We will achieve this in following steps:\n",
    "1. Load the movielens data into a pandas dataframe for preprocessing\n",
    "2. Set up our parameters: $\\theta$ and $\\phi$\n",
    "3. Load the data into a tensorflow variable\n",
    "4. Define the prediction and cost\n",
    "5. Start the session and run training, evaluation and finally predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### download data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First download the data from movie lens website. Once the data file is unzipped, there are three files that we are using-- user.csv, movie.csv and rating.csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#! wget -q http://files.grouplens.org/datasets/movielens/ml-1m.zip\n",
    "#! unzip ml-1m.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "step 1: load data to pandas dataframes, merge the dataframes to one table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "age_desc = { 1: \"Under 18\", 18: \"18-24\", 25: \"25-34\", 35: \"35-44\", 45: \"45-49\", 50: \"50-55\", 56: \"56+\" }\n",
    "occupation_desc = { 0: \"other or not specified\", 1: \"academic/educator\", 2: \"artist\", 3: \"clerical/admin\",\n",
    "                4: \"college/grad student\", 5: \"customer service\", 6: \"doctor/health care\",\n",
    "                7: \"executive/managerial\", 8: \"farmer\", 9: \"homemaker\", 10: \"K-12 student\", 11: \"lawyer\",\n",
    "                12: \"programmer\", 13: \"retired\", 14: \"sales/marketing\", 15: \"scientist\", 16: \"self-employed\",\n",
    "                17: \"technician/engineer\", 18: \"tradesman/craftsman\", 19: \"unemployed\", 20: \"writer\" }\n",
    "\n",
    "rating_data = pd.read_csv(\n",
    "    \"ml-1m/ratings.dat\",\n",
    "    sep=\"::\",\n",
    "    engine=\"python\",\n",
    "    encoding=\"latin-1\",\n",
    "    names=['userid', 'movieid', 'rating', 'timestamp'])\n",
    "\n",
    "user_data = pd.read_csv(\n",
    "    \"ml-1m/users.dat\", \n",
    "    sep='::', \n",
    "    engine='python', \n",
    "    encoding='latin-1',\n",
    "    names=['userid', 'gender', 'age', 'occupation', 'zipcode']\n",
    ")\n",
    "user_data['age_desc'] = user_data['age'].apply(lambda x: age_desc[x])\n",
    "user_data['occ_desc'] = user_data['occupation'].apply(lambda x: occupation_desc[x])\n",
    "\n",
    "movie_data = pd.read_csv(\n",
    "    \"ml-1m/movies.dat\",\n",
    "    sep='::', \n",
    "    engine='python', \n",
    "    encoding='latin-1',\n",
    "    names=['movieid', 'title', 'genre']\n",
    ")\n",
    "\n",
    "dataset = pd.merge(pd.merge(rating_data, movie_data, how=\"left\", on=\"movieid\"), user_data, how=\"left\", on=\"userid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 2: preprocess the movie id and user id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def check_cols(df, cols):\n",
    "    \"\"\"\n",
    "    check if there are gaps of index, and if the index starts from 0\n",
    "    \n",
    "    Arguments:\n",
    "    df -- dataframe of the dataset\n",
    "    cols -- dataframe columns that needs to be checked, in our case user id and movie id\n",
    "    \n",
    "    Returns:\n",
    "    a list of tuple [('COLUMN_NAME', boolean)], if True, the column needs to be fixed, if False, the column is ok.\n",
    "    \"\"\"\n",
    "    return [(col, False) if len(dataset[col].unique())-1 == dataset[col].max() else (col, True) for col in cols]\n",
    "\n",
    "def remove_gaps(df, col):\n",
    "    \"\"\"\n",
    "    preprocess the index of user id and movie id to start from 0 and eliminate gaps in the index\n",
    "    \n",
    "    Arguments:\n",
    "    df -- dataframe of the dataset\n",
    "    col -- dataframe columns that needs to be adjusted, in our case both user id and movie id\n",
    "    \n",
    "    Returns:\n",
    "    a dataframe with adjusted columns.\n",
    "    \"\"\"\n",
    "    adj_col_uni = df[col].sort_values().unique()\n",
    "    adj_df = pd.DataFrame(adj_col_uni).reset_index().rename(columns = {0: col, 'index': \"adj_%s\"%(col,)})\n",
    "    return pd.merge(adj_df, df, how=\"right\", on=col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before fix:\n",
      "userid needs fix!\n",
      "movieid needs fix!\n",
      "\n",
      "after fix\n",
      "adj_userid ok.\n",
      "adj_movieid ok.\n"
     ]
    }
   ],
   "source": [
    "index_cols = [\"userid\", \"movieid\"]\n",
    "cols_check = check_cols(dataset, index_cols)\n",
    "print_check = lambda check: print(*[\"%s needs fix!\"%(c,) if f else \"%s ok.\"%(c,) for c, f in check], sep=\"\\n\")\n",
    "print(\"before fix:\")\n",
    "print_check(cols_check)\n",
    "for col, needs_fix in cols_check:\n",
    "    if needs_fix:\n",
    "        dataset = remove_gaps(dataset, col)\n",
    "\n",
    "print(\"\\nafter fix\")\n",
    "print_check(check_cols(dataset, [\"adj_userid\", \"adj_movieid\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "step 3: shuffle data and split the data to train and validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dataset = dataset.sample(frac=1, replace=False)\n",
    "n_split = int(len(dataset)*.7)\n",
    "trainset = dataset[:n_split]\n",
    "validset = dataset[n_split:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### build the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Recall that following is our model:\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "- users $\\{1,...,N\\}$ with user features $\\{\\theta_1,...,\\theta_N\\} =: \\theta$<br>\n",
    "- movies $\\{1,...,M\\}$ with movie features $\\{\\phi_1,...,\\phi_M\\} =: \\phi$<br>\n",
    "- dataset $D = \\{r_{i,j}: user\\ i\\ has\\ rated\\ movie\\ j\\}$ of all ratings<br>\n",
    "- predicted rating of user $i$ of movie $j$: $\\hat{r}_{i,j} = \\theta_i^T \\cdot \\phi_j$<br>\n",
    "- cost function $J(\\theta, \\phi)  = \\sum_{r_{i,j} \\in D} (r_{i,j} - \\hat{r}_{i,j})^2$<br>\n",
    "- We will use ALS method for matrix factorization. This means we calculate two gradients of our cost function $\\nabla_\\theta J$ and $\\nabla_\\phi J$ independently in order to update our parameters $\\theta$ and $\\phi$ one after another.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def initialize_features(num_users, num_movies, dim):\n",
    "    \"\"\"\n",
    "    Initialize features. User_features and movie_features need to be trained by the matrix factorization model.\n",
    "    \n",
    "    Arguments:\n",
    "    num_users -- number of users\n",
    "    num_movies -- number of movies\n",
    "    dim -- dimension of learned user and movie features, it's a hyper-parameter\n",
    "    \n",
    "    Returns:\n",
    "    user_features -- a matrix (variable) of shape [number of users, dim]\n",
    "    movie_features -- a matrix (variable) of shape [number of movies, dim]\n",
    "    \"\"\" \n",
    "    user_features = tf.get_variable(\n",
    "        \"theta\",\n",
    "        shape = [num_users, dim],\n",
    "        dtype = tf.float32,\n",
    "        initializer = tf.truncated_normal_initializer(mean=0, stddev=.05)\n",
    "    )\n",
    "    movie_features = tf.get_variable(\n",
    "        \"phi\",\n",
    "        shape = [num_movies, dim],\n",
    "        dtype = tf.float32,\n",
    "        initializer = tf.truncated_normal_initializer(mean=0, stddev=.05)\n",
    "    )\n",
    "    return user_features, movie_features\n",
    "\n",
    "def create_dataset(user_ids, movie_ids, ratings):\n",
    "    \"\"\"\n",
    "    Load user id, movie id and rating values. Turn numpy array to tensors.\n",
    "    \n",
    "    Arguments:\n",
    "    user_ids -- user index\n",
    "    movie_ids -- movies index\n",
    "    ratings -- true rating value\n",
    "    \n",
    "    Returns:\n",
    "    user_id_var -- a constant of shape [number of training examples, 1]\n",
    "    movie_id_var -- a constant of shape [number of training examples, 1]\n",
    "    ratings_var -- a constant of shape [number of training examples, 1]\n",
    "    \"\"\" \n",
    "    user_id_var = tf.constant(name=\"userid\", value=user_ids)\n",
    "    movie_id_var = tf.constant(name=\"movieid\", value=movie_ids)\n",
    "    ratings_var = tf.constant(name=\"ratings\", value=np.asarray(ratings, dtype=np.float32))\n",
    "    return user_id_var, movie_id_var, ratings_var\n",
    "   \n",
    "def lookup_features(user_features, movie_features, user_ids, movie_ids): \n",
    "    \"\"\"\n",
    "    Retrieve embeddings based on user ids and movie ids respectively.\n",
    "    We use tf.gather function for this. tf.gather gathers slices from params according to indices.\n",
    "    \n",
    "    Arguments:\n",
    "    user_features -- shape [number of user ids, dim]\n",
    "    movie_features -- shape [number of movie ids, dim]\n",
    "    user_ids -- user id tensor (in our case loaded user ids from dataset)\n",
    "    movie_ids -- movie id tensor (in our case loaded movie ids from dataset)\n",
    "    \n",
    "    Returns:\n",
    "    selected_user_features -- a tensor of shape [number of examples, dim]\n",
    "    selected_movie_features -- a tensor of shape [number of examples, dim]\n",
    "    \"\"\" \n",
    "    selected_user_features = tf.gather(user_features, user_ids)\n",
    "    selected_movie_features = tf.gather(movie_features, movie_ids)\n",
    "    return selected_user_features, selected_movie_features\n",
    "\n",
    "def predict(selected_user_features, selected_movie_features):\n",
    "    \"\"\"\n",
    "    Calculate predictions. This is the dot product of user features and movie features. \n",
    "    For each training example, this corresponds to a single number.\n",
    "    \n",
    "    Arguments:\n",
    "    selected_user_features -- matrix of user features for each example -- shape [number of examples, dim]\n",
    "    selected_movie_features -- matrix of movies features value for each example -- shape [number of examples, dim]\n",
    "    \n",
    "    Returns:\n",
    "    selected_predictions -- a tensor of shape [number of examples, 1]\n",
    "    \"\"\" \n",
    "    selected_predictions = tf.reduce_sum(\n",
    "        selected_user_features * selected_movie_features,\n",
    "        axis = 1\n",
    "    )\n",
    "    ##alternatively, using tf.reduce_sum(tf.multiply(selected_user_embeddings,selected_movie_embeddings), axis=1)\n",
    "    return selected_predictions\n",
    "\n",
    "def mean_squared_difference(predictions, ratings):\n",
    "    \"\"\"\n",
    "    Calculate cost.\n",
    "    \n",
    "    Arguments:\n",
    "    predictions -- predicted ratings.\n",
    "    ratings -- true ratings.\n",
    "    \n",
    "    Returns:\n",
    "    difference -- mean squared error. It's a real number. \n",
    "    \"\"\" \n",
    "    difference = tf.reduce_mean(tf.squared_difference(predictions, ratings))\n",
    "    return difference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### set hyper parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "emb_dim = 8\n",
    "learning_rate = 50\n",
    "epochs = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### train model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define the tensorflow graph and create the session to compute the values.\n",
    "\n",
    "From the tensorflow documentation:\n",
    " - A graph defines the computation. It doesn’t compute anything, it doesn’t hold any values, it just defines the operations that you specified in your code.\n",
    " - A session allows to execute graphs or part of graphs. It allocates resources (on one or more machines) for that and holds the actual values of intermediate results and variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "valid loss at step 1: 14.061668\n",
      "valid loss at step 101: 1.053597\n",
      "valid loss at step 201: 0.873390\n",
      "valid loss at step 301: 0.833664\n",
      "valid loss at step 401: 0.804873\n",
      "valid loss at step 501: 0.786872\n",
      "valid loss at step 601: 0.775171\n",
      "valid loss at step 701: 0.767974\n",
      "valid loss at step 801: 0.763356\n",
      "valid loss at step 901: 0.760175\n"
     ]
    }
   ],
   "source": [
    "with tf.Graph().as_default():\n",
    "    with tf.variable_scope(\"features\"):\n",
    "        usr_embs, mov_embs = initialize_features(len(dataset.adj_userid.unique()), len(dataset.adj_movieid.unique()), emb_dim)\n",
    "    with tf.variable_scope(\"train_set\"):\n",
    "        train_data = trainset[[\"adj_userid\", \"adj_movieid\", \"rating\"]].values.T #shape(3, 700146)\n",
    "        train_usr_ids, train_mov_ids, train_ratings = create_dataset(*train_data)# expend to 3 lists\n",
    "    with tf.variable_scope(\"valid_set\"):\n",
    "        valid_data = validset[[\"adj_userid\", \"adj_movieid\", \"rating\"]].values.T\n",
    "        valid_usr_ids, valid_mov_ids, valid_ratings = create_dataset(*valid_data)\n",
    "    with tf.variable_scope(\"training\"):\n",
    "        train_sel_usr_emb, train_sel_mov_emb = lookup_features(usr_embs, mov_embs, train_usr_ids, train_mov_ids)\n",
    "        train_preds = predict(train_sel_usr_emb, train_sel_mov_emb)\n",
    "        train_loss = mean_squared_difference(train_preds, train_ratings)\n",
    "        optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "        train_usr_embs = optimizer.minimize(train_loss, var_list=[usr_embs])\n",
    "        train_mov_embs = optimizer.minimize(train_loss, var_list=[mov_embs])   \n",
    "    with tf.variable_scope(\"validation\"):\n",
    "        valid_sel_usr_emb, valid_sel_mov_emb = lookup_features(usr_embs, mov_embs, valid_usr_ids, valid_mov_ids)\n",
    "        valid_preds = predict(valid_sel_usr_emb, valid_sel_mov_emb)\n",
    "        valid_loss = mean_squared_difference(valid_preds, valid_ratings)\n",
    "    with tf.Session() as sess:\n",
    "        writer = tf.summary.FileWriter('Graph/MF',sess.graph)\n",
    "        sess.run(tf.global_variables_initializer())\n",
    "        train_loss_history = []\n",
    "        valid_loss_history = []\n",
    "        for i in range(epochs):\n",
    "            current_train_loss, _ = sess.run([train_loss, train_usr_embs])\n",
    "            current_train_loss, _ = sess.run([train_loss, train_mov_embs])\n",
    "            current_valid_loss = sess.run(valid_loss)\n",
    "            if i%100 ==0:\n",
    "                print(\"valid loss at step %i: %f\"%(i+1, current_valid_loss))\n",
    "            train_loss_history.append(current_train_loss)\n",
    "            valid_loss_history.append(current_valid_loss)\n",
    "        final_user_features, final_movie_features = sess.run([usr_embs, mov_embs])\n",
    "        final_valid_predictions = sess.run(valid_preds) \n",
    "        writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### plot losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "plot the traing loss and valid loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAAJQCAYAAABIJTh6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt43HWd9//XOzPfmSRN2vQQoKWFFtalLbQUCAftTwFB\nbhAVTwjc1NN6y+qigHop6P1bV69dL/Fa9hZxVbarqCsIN1vFw8rKwQXq/gS0xQLFolgotLRAekia\nNJM5fn5/zCRNS9pmkvl+PzPfPB/XNdccMpl5M18Pz3z4zIw55wQAAABg7Jp8DwAAAAA0GiIaAAAA\nqBIRDQAAAFSJiAYAAACqREQDAAAAVSKiAQAAgCoR0QAAAECViGgAAACgSkQ0AAAAUKWk7wHGYtas\nWW7+/Pm+xwAAAEDMrV27drtzrvNQ92uIiJ4/f77WrFnjewwAAADEnJk9P5b7sZ0DAAAAqBIRDQAA\nAFSJiAYAAACq1BB7ogEAAFCWz+e1ZcsWDQ4O+h6loTU3N2vu3LkKgmBcv09EAwAANJAtW7aovb1d\n8+fPl5n5HqchOee0Y8cObdmyRQsWLBjXY7CdAwAAoIEMDg5q5syZBPQEmJlmzpw5odV8IhoAAKDB\nENATN9HXkIgGAAAAqkREAwAAYMx6enr0zW9+c1y/++Y3v1k9PT1jvv8XvvAF3XDDDeN6rrAR0QAA\nABizg0V0oVA46O/efffd6ujoCGOsyBHRAAAAGLPrrrtOGzdu1LJly/TpT39aDz74oF7/+tfrbW97\nmxYvXixJevvb365TTjlFxx9/vFauXDn8u/Pnz9f27du1adMmLVq0SB/+8Id1/PHH67zzzlMmkzno\n865bt05nnHGGli5dqne84x3atWuXJOmmm27S4sWLtXTpUl166aWSpIceekjLli3TsmXLdNJJJ6mv\nr6/mrwMfcQcAANCorrlGWreuto+5bJl0440H/PH111+v9evXa13leR988EE99thjWr9+/fDHxd1y\nyy2aMWOGMpmMTj31VL3rXe/SzJkz93mcZ555Rrfffrv+9V//Ve95z3v0ox/9SCtWrDjg877vfe/T\n17/+dZ155pn6/Oc/ry9+8Yu68cYbdf311+u5555TOp0e3ipyww036Bvf+IaWL1+u/v5+NTc3T/RV\neRVWogEAADAhp5122j6ft3zTTTfpxBNP1BlnnKHNmzfrmWeeedXvLFiwQMuWLZMknXLKKdq0adMB\nH7+3t1c9PT0688wzJUnvf//7tXr1aknS0qVLdfnll+vWW29VMlleH16+fLk++clP6qabblJPT8/w\n7bXESjQAAECjOsiKcZSmTJkyfPnBBx/U/fffr4cfflitra0666yzRv085nQ6PXw5kUgccjvHgfzi\nF7/Q6tWr9fOf/1xf+tKX9OSTT+q6667ThRdeqLvvvlvLly/XPffco4ULF47r8Q+ElWgAAACMWXt7\n+0H3GPf29mr69OlqbW3V008/rUceeWTCzzlt2jRNnz5dv/71ryVJP/jBD3TmmWeqVCpp8+bNOvvs\ns/WVr3xFvb296u/v18aNG7VkyRJde+21OvXUU/X0009PeIb9sRINAACAMZs5c6aWL1+uE044QRdc\ncIEuvPDCfX5+/vnn6+abb9aiRYt03HHH6YwzzqjJ837/+9/XRz7yEQ0MDOiYY47Rd7/7XRWLRa1Y\nsUK9vb1yzumqq65SR0eH/vZv/1YPPPCAmpqadPzxx+uCCy6oyQwjmXOu5g9aa11dXW7NmjW+xwAA\nAPBuw4YNWrRoke8xYmG019LM1jrnug71u2znAAAAAKpERAMAAABVIqIBAACAKhHRAAAAQJWIaAAA\nAKBKoX3EnZndIuktkl5xzp2w388+JekGSZ3Oue1hzTART37tv/TTf3hSyYRTMql9TumU09xZg1p4\nYlrz3rpMOussqYm/RwAAACaLMD8n+nuS/lnSv4280czmSTpP0gshPveE/f7lOfrb7W88+J0ekJbe\n+Lg+f+Qn9K5f/JV04onRDAcAANBA2tra1N/fr61bt+qqq67SqlWrXnWfs846SzfccIO6urrGdLtv\noS2fOudWS9o5yo++Kukzkur6A6rf+6WFKhSkwUGpv1/atUvq7pa2bZOefVZ68EHpn76clZt3lN79\n4tf0ldfeJR3kO98BAAAmuzlz5owa0I0o0j0IZnaRpBedc49H+bzjYSYlElI6LU2ZInV0SLNmSUcc\nIS1YIJ15pvTJ69Jau3G6Lntrn67LfEG/+Z//7HtsAACAUF133XX6xje+MXz9C1/4gm644Qb19/fr\nnHPO0cknn6wlS5bopz/96at+d9OmTTrhhPIu30wmo0svvVSLFi3SO97xDmUymUM+9+23364lS5bo\nhBNO0LXXXitJKhaL+sAHPqATTjhBS5Ys0Ve/+lVJ0k033aTFixdr6dKluvTSS2vxj76PyL7228xa\nJX1O5a0cY7n/FZKukKSjjjoqxMkmJgiklT9s1+ojd+uzD79VD61bJy1b5nssAAAwCVxzjbRuXW0f\nc9ky6cYbD/zzSy65RNdcc42uvPJKSdKdd96pe+65R83Nzbrrrrs0depUbd++XWeccYbe9ra3ycxG\nfZxvfetbam1t1YYNG/TEE0/o5JNPPuhcW7du1bXXXqu1a9dq+vTpOu+88/STn/xE8+bN04svvqj1\n69dLknp6eiRJ119/vZ577jml0+nh22opypXoYyUtkPS4mW2SNFfSY2Z2xGh3ds6tdM51Oee6Ojs7\nIxyzem1t0tWfTGq1ztRTX7vf9zgAAAChOemkk/TKK69o69atevzxxzV9+nTNmzdPzjl97nOf09Kl\nS3XuuefqxRdf1Msvv3zAx1m9erVWrFghSVq6dKmWLl160Of93e9+p7POOkudnZ1KJpO6/PLLtXr1\nah1zzDF69tln9fGPf1y//OUvNXXq1OHHvPzyy3Xrrbcqmaz9unFkK9HOuSclHTZ0vRLSXfX66RzV\net9HWvWZL0h3/cR0/Hd9TwMAACaDg60Yh+niiy/WqlWr9NJLL+mSSy6RJN12223q7u7W2rVrFQSB\n5s+fr8HBwdBnmT59uh5//HHdc889uvnmm3XnnXfqlltu0S9+8QutXr1aP//5z/WlL31JTz75ZE1j\nOrSVaDO7XdLDko4zsy1m9qGwnqseHH64dMq8l/XLntOlzZt9jwMAABCaSy65RHfccYdWrVqliy++\nWJLU29urww47TEEQ6IEHHtDzzz9/0Md4wxveoB/+8IeSpPXr1+uJJ5446P1PO+00PfTQQ9q+fbuK\nxaJuv/12nXnmmdq+fbtKpZLe9a536R/+4R/02GOPqVQqafPmzTr77LP1la98Rb29verv76/NP3xF\naCvRzrnLDvHz+WE9ty/nvtHpn75/urIP3qX0e+f5HgcAACAUxx9/vPr6+nTkkUdq9uzZkqTLL79c\nb33rW7VkyRJ1dXVp4cKFB32Mj370o/rgBz+oRYsWadGiRTrllFMOev/Zs2fr+uuv19lnny3nnC68\n8EJddNFFevzxx/XBD35QpVJJkvTlL39ZxWJRK1asUG9vr5xzuuqqq9TR0VGbf/gKc66uP2lOktTV\n1eXWrFnje4xDWvV/i7r40oTWXPZPOuWHn/I9DgAAiKENGzZo0aJFvseIhdFeSzNb65w75IdS8zV7\nNXTyqQlJ0mOPJzxPAgAAgDAR0TW0YIHUnszoiRdq+68LAAAAUF+I6Boyk46d1auN/YeVv+YQAAAg\nBI2wHbfeTfQ1JKJr7C/m57VRx0obN/oeBQAAxFBzc7N27NhBSE+Ac047duxQc3PzuB8jss+JniyO\nfU1CP31kgYqb7lXixBN9jwMAAGJm7ty52rJli7q7u32P0tCam5s1d+7ccf8+EV1jxy5pVV4pbXly\nl46+yPc0AAAgboIg0IIFC3yPMemxnaPG5i5qlyRt/RN7ogEAAOKKiK6x2XPLH2+37fmc50kAAAAQ\nFiK6xo44ony+bZvfOQAAABAeIrrGOjulJhW1bdf43+0JAACA+kZE11giIR3e2qeX+tt8jwIAAICQ\nENEhOKJ9QNsGp0ulku9RAAAAEAIiOgSHT8/pFXVKPT2+RwEAAEAIiOgQzJjhtEvTJT4EHQAAIJaI\n6BBMn5nQTs2Qtm/3PQoAAABCQESHYMbhgXrUodLLrEQDAADEEREdghmz03JqUu+WPt+jAAAAIARE\ndAimz05Lkna+xLcWAgAAxBERHYIZs8tftLJre9HzJAAAAAgDER2CGZ0JSdLO7XxONAAAQBwR0SGY\nMaN8vnOn3zkAAAAQDiI6BFOnls937/Y7BwAAAMJBRIegvb183seHcwAAAMQSER2Ctrbyed8eXl4A\nAIA4ovJC0NQkTUkOqm8w6XsUAAAAhICIDkl7kFXfYOB7DAAAAISAiA5Je3NOfdm07zEAAAAQAiI6\nJO3NefWVWqUiX7gCAAAQN0R0SNpbiupXm7Rnj+9RAAAAUGNEdEjaWkvqUzsRDQAAEENEdEja24ho\nAACAuCKiQ9LeJiIaAAAgpojokLRPNSIaAAAgpojokLRNbdIetcn1E9EAAABxQ0SHpKW9/G2F2Z6M\n50kAAABQa0R0SFqnliN6YFfW8yQAAACoNSI6JK3Tyl/5PdCT8zwJAAAAao2IDknLtJQkIhoAACCO\niOiQtE5PS5Iyu/OeJwEAAECtEdEhGd4T3Vf0PAkAAABqjYgOSUtL+ZyIBgAAiB8iOiStreXzzIDz\nOwgAAABqjogOyVBEDwz4nQMAAAC1R0SHZHg7R8b8DgIAAICaI6JDMrydI0tEAwAAxA0RHZLhlejB\nhN9BAAAAUHNEdEiGIzpLRAMAAMQNER2SIJACyyuTI6IBAADihogOUUsir4Fc0vcYAAAAqDEiOkQt\nyZwGCoHvMQAAAFBjRHSImpMFZQts5wAAAIgbIjpE6WRR2QLbOQAAAOKGiA5Rc1DUYJHtHAAAAHFD\nRIcoHZSULSUl53yPAgAAgBoiokPUnCopq7SUy/keBQAAADVERIconXIaVLOUyfgeBQAAADVERIco\nnXLllejBQd+jAAAAoIaI6BA1N6u8Ek1EAwAAxAoRHaJ0WuWVaLZzAAAAxAoRHaLmZmNPNAAAQAwR\n0SFKN4tP5wAAAIghIjpEwyvR2azvUQAAAFBDRHSI0s3GSjQAAEAMEdEham5tUl4plTKsRAMAAMRJ\naBFtZreY2Stmtn7Ebf9oZk+b2RNmdpeZdYT1/PUg3VJ+ebP9ec+TAAAAoJbCXIn+nqTz97vtPkkn\nOOeWSvqTpM+G+PzeNbdWInqg6HkSAAAA1FJoEe2cWy1p53633eucK1SuPiJpbljPXw/SrQlJ0uAe\nIhoAACBOfO6J/itJ/3mgH5rZFWa2xszWdHd3RzhW7QxFNCvRAAAA8eIlos3sf0sqSLrtQPdxzq10\nznU557o6OzujG66GmtuSkliJBgAAiJtk1E9oZh+Q9BZJ5zjnXNTPH6X0lPLLm82UPE8CAACAWoo0\nos3sfEmfkXSmc24gyuf2gYgGAACIpzA/4u52SQ9LOs7MtpjZhyT9s6R2SfeZ2Tozuzms568HqdZy\nROcH2c4BAAAQJ6GtRDvnLhvl5u+E9Xz1KEiZJCk3yEo0AABAnPCNhSFKpcrnuSwRDQAAECdEdIiG\nIjqfjfX7JwEAACYdIjpEQVA+zxHRAAAAsUJEh2jvSjTbOQAAAOKEiA7R8Ep0zu8cAAAAqC0iOkTD\nK9E5tnMAAADECREdouGV6LzfOQAAAFBbRHSIhj/iLmd+BwEAAEBNEdEhGt7OwUo0AABArBDRIdq7\nnYOVaAAAgDghokM0vBJd8DsHAAAAaouIDlEiIZlKyhUSvkcBAABADRHRIUs1FZQv8jIDAADECXUX\nsqCpqBwRDQAAECvUXchSiaJyxaTvMQAAAFBDRHTIUoki2zkAAABihroLWdBUVK7EGwsBAADihIgO\nWSpRUr5IRAMAAMQJER2yIFFSzrEnGgAAIE6I6JClkiXl2c4BAAAQK0R0yIJkSTmlpGLR9ygAAACo\nESI6ZKmhiM7nfY8CAACAGiGiQ5ZKOuUVSLmc71EAAABQI0R0yIKkYyUaAAAgZojokKWCyko0EQ0A\nABAbRHTIgoCVaAAAgLghokOWCsRKNAAAQMwQ0SELArESDQAAEDNEdMhSqUpE8+kcAAAAsUFEhyyV\nYjsHAABA3BDRIQtSbOcAAACIGyI6ZKmUsRINAAAQM0R0yIKUsRINAAAQM0R0yFJpVqIBAADihogO\nWZBuUkkJFTN8OgcAAEBcENEhS6VNkpTLFD1PAgAAgFohokOWai6/xPlBIhoAACAuiOiQBenyS5wb\nLHmeBAAAALVCRIeMlWgAAID4IaJDxko0AABA/BDRIUu1JCSxEg0AABAnRHTIguZyRLMSDQAAEB9E\ndMiGVqJzWed5EgAAANQKER2yVGtSkpTPshINAAAQF0R0yIa3c7ASDQAAEBtEdMiGP+IuR0QDAADE\nBREdsiAon7MSDQAAEB9EdMhSqfI5K9EAAADxQUSHbHglmogGAACIDSI6ZEMr0bmc+R0EAAAANUNE\nh2x4O0eelWgAAIC4IKJDtnc7ByvRAAAAcUFEh2zvSrTfOQAAAFA7RHTIhlei86xEAwAAxAURHbKh\niM4X/M4BAACA2iGiQzYc0axEAwAAxAYRHbK9K9FENAAAQFwQ0SFLJsvn+SIRDQAAEBdEdMjMpIQV\nlS/wUgMAAMQFZReBwAoq8MZCAACA2CCiIxA0FZUv8lIDAADEBWUXASIaAAAgXii7CARNReVLvNQA\nAABxQdlFoLwSnfA9BgAAAGoktIg2s1vM7BUzWz/ithlmdp+ZPVM5nx7W89eTIFFiJRoAACBGwiy7\n70k6f7/brpP0K+fcayT9qnI99oJEiZVoAACAGAktop1zqyXt3O/miyR9v3L5+5LeHtbz15PySjQR\nDQAAEBdR7zE43Dm3rXL5JUmHR/z8XhDRAAAA8eJto65zzklyB/q5mV1hZmvMbE13d3eEk9VekCgp\n74hoAACAuIg6ol82s9mSVDl/5UB3dM6tdM51Oee6Ojs7IxswDEHCKe+Skjvg3wwAAABoIFFH9M8k\nvb9y+f2Sfhrx83sRJJ3yCsR3fwMAAMRDmB9xd7ukhyUdZ2ZbzOxDkq6X9CYze0bSuZXrsZcciuh8\n3vcoAAAAqIFkWA/snLvsAD86J6znrFdB0qmgJBENAAAQE3wDSAQCVqIBAABihYiOQJAUEQ0AABAj\nRHQEgoCIBgAAiBMiOgJ8OgcAAEC8ENERCFKsRAMAAMQJER0BtnMAAADECxEdgSAwIhoAACBGiOgI\nsJ0DAAAgXojoCAyvRPPGQgAAgFggoiMQpNjOAQAAECdEdASSKVNRSbkcEQ0AABAHRHQEglT5ZS4M\nsp0DAAAgDojoCAQpkyTlB4ueJwEAAEAtENERCNLllzmfYSUaAAAgDojoCATpykp0tuR5EgAAANQC\nER2BIJ2QREQDAADEBREdgeHtHOyJBgAAiAUiOgJBcyWis0Q0AABAHBDRERjezjHIdg4AAIA4IKIj\nEDRXIjrnPE8CAACAWiCiIzAc0byxEAAAIBaI6AiwEg0AABAvRHQEWIkGAACIFyI6Ask0K9EAAABx\nQkRHIAjK54UcK9EAAABxQERHYCii83m/cwAAAKA2iOgIDEc02zkAAABigYiOACvRAAAA8UJER4CI\nBgAAiBciOgJ7I5rtHAAAAHFAREdgb0Sb30EAAABQE0R0BNjOAQAAEC9EdASGI7rASjQAAEAcENER\n2BvRfucAAABAbRDREWBPNAAAQLwQ0RFgOwcAAEC8ENERSCbL5/kiLzcAAEAcUHURMJMSVlShyEo0\nAABAHBDREQmswHYOAACAmCCiIxI0FdnOAQAAEBNUXUSIaAAAgPig6iISNBWVL/FyAwAAxAFVF5Hy\nSnTC9xgAAACoASI6IkFTiZVoAACAmKDqIhIkWIkGAACICyI6IkGipHyJiAYAAIgDIjoiRDQAAEB8\nENERCRIl5V3S9xgAAACoASI6IkHCKe9YiQYAAIgDIjoiyYQrr0Q753sUAAAATBARHZEg6VRQUioU\nfI8CAACACSKiIxIknfIKpHze9ygAAACYICI6IkQ0AABAfBDRERmOaLZzAAAANDwiOiJBwEo0AABA\nXBDREQkCEdEAAAAxQURHhIgGAACIDyI6IkQ0AABAfBDRERmOaN5YCAAA0PCI6IgEgbESDQAAEBNE\ndESCFBENAAAQF0R0RIIUe6IBAADigoiOSDJoUlFJuRwRDQAA0OiI6IgEKZMkFQZ5YyEAAECjI6Ij\nMhTR+cGi50kAAAAwUV4i2sw+YWZPmdl6M7vdzJp9zBGlIF1+qYloAACAxhd5RJvZkZKuktTlnDtB\nUkLSpVHPETUiGgAAID58bedISmoxs6SkVklbPc0RGSIaAAAgPiKPaOfci5JukPSCpG2Sep1z9+5/\nPzO7wszWmNma7u7uqMesOSIaAAAgPnxs55gu6SJJCyTNkTTFzFbsfz/n3ErnXJdzrquzszPqMWtu\nOKJzzvMkAAAAmCgf2znOlfScc67bOZeX9GNJr/MwR6RYiQYAAIgPHxH9gqQzzKzVzEzSOZI2eJgj\nUkFzQpKUz5Y8TwIAAICJ8rEn+lFJqyQ9JunJygwro54jakQ0AABAfCR9PKlz7u8k/Z2P5/ZlOKJz\nRDQAAECj4xsLI5JsLv+9ks/yxkIAAIBGR0RHZGglupAnogEAABodER0RPuIOAAAgPojoiAQpk0RE\nAwAAxAERHZEgKJ8T0QAAAI2PiI4IEQ0AABAfRHREhiO64HcOAAAATBwRHZG9K9F+5wAAAMDEEdER\nGY7ovN85AAAAMHFEdESIaAAAgPggoiOyd0+0+R0EAAAAEzamiDazq81sqpV9x8weM7Pzwh4uTliJ\nBgAAiI+xrkT/lXNut6TzJE2X9F5J14c2VQwNR3SRlWgAAIBGN9aIHiq/N0v6gXPuqRG3YQySyfI5\n2zkAAAAa31gjeq2Z3atyRN9jZu2SSuGNFT9DK9EFIhoAAKDhJcd4vw9JWibpWefcgJnNkPTB8MaK\nHzMpoSIr0QAAADEw1pXo10r6o3Oux8xWSPp/JfWGN1Y8BU0F9kQDAADEwFgj+luSBszsREmfkrRR\n0r+FNlVMBVZUvpjwPQYAAAAmaKwRXXDOOUkXSfpn59w3JLWHN1Y8sRINAAAQD2PdE91nZp9V+aPt\nXm9mTZKC8MaKp6CpqHyR77cBAABodGMtukskZVX+vOiXJM2V9I+hTRVTQVOJ7RwAAAAxMKaIroTz\nbZKmmdlbJA0659gTXaUgwUo0AABAHIz1a7/fI+m3ki6W9B5Jj5rZu8McLI6CREn5EhENAADQ6Ma6\nJ/p/SzrVOfeKJJlZp6T7Ja0Ka7A4YjsHAABAPIx1WbRpKKArdlTxu6gIkqxEAwAAxMFYV6J/aWb3\nSLq9cv0SSXeHM1J8BQmnfImVaAAAgEY3poh2zn3azN4laXnlppXOubvCGyuekkkiGgAAIA7GuhIt\n59yPJP0oxFliL0g4FZSUCgUpOeaXHgAAAHXmoCVnZn2S3Gg/kuScc1NDmSqmgsApo0DK5YhoAACA\nBnbQknPO8dXeNRQkpd1DEd3a6nscAAAAjBMfFRGhIHDKD0U0AAAAGhYRHaEgUDmis1nfowAAAGAC\niOgIBYGxEg0AABADRHSEUikppxQRDQAA0OCI6Ail01JWaSIaAACgwRHREUqnKyvR7IkGAABoaER0\nhFLNxko0AABADBDREUoT0QAAALFAREco3dykvFIqDRLRAAAAjYyIjlC62SRJ+YG850kAAAAwEUR0\nhFItCUlSdk/B8yQAAACYCCI6QumW8sudHSh6ngQAAAATQURHKN1aWYkmogEAABoaER2hoYjODbCd\nAwAAoJER0RFKtSYlSdlMyfMkAAAAmAgiOkLD2zmIaAAAgIZGREcoPaWyEj3oPE8CAACAiSCiIzQU\n0bkMbywEAABoZER0hFLp8petZLOsRAMAADQyIjpC6XT5PDvodw4AAABMDBEdoeGIzvqdAwAAABND\nREdoKKJzbOcAAABoaER0hFKp8jkr0QAAAI2NiI7Q8HaOnPkdBAAAABNCREdoeDtHzu8cAAAAmBgi\nOkLDK9F5XnYAAIBGRs1FaHhPNNs5AAAAGhoRHaHhiGYlGgAAoKFRcxEyk1KWU67Ayw4AANDIqLmI\npRMFZYloAACAhkbNRSzVVFA2n/A9BgAAACaAiI5YOlFQtkhEAwAANDIiOmLpREE5IhoAAKChEdER\nSyeKyhaTvscAAADABHiJaDPrMLNVZva0mW0ws9f6mMOHVJKIBgAAaHS+au5rkn7pnHu3maUktXqa\nI3LpZFHZYuB7DAAAAExA5BFtZtMkvUHSByTJOZeTlIt6Dl/SQUk5x0o0AABAI/OxnWOBpG5J3zWz\n35vZt81sioc5vEgHJWVLrEQDAAA0Mh8RnZR0sqRvOedOkrRH0nX738nMrjCzNWa2pru7O+oZQ5MO\nSsoqLRUKvkcBAADAOPmI6C2StjjnHq1cX6VyVO/DObfSOdflnOvq7OyMdMAwpQJXjujcpNnBAgAA\nEDuRR7Rz7iVJm83suMpN50j6Q9Rz+JJOOeWUkrJZ36MAAABgnHy9w+3jkm6rfDLHs5I+6GmOyKVT\nKq9EE9EAAAANy0tEO+fWSery8dy+pZutHNGZjO9RAAAAME58Y2HEUmkrb+cYHPQ9CgAAAMaJiI5Y\nuqWJlWgAAIAGR0RHbHg7ByvRAAAADYuIjli6NaG8UirtYSUaAACgURHREUu1JCRJ+T18TjQAAECj\nIqIjlm4tR3R2Nx9xBwAA0KiI6Ig1t5U/VXCwL+95EgAAAIwXER2xlvZyRGf6Cp4nAQAAwHgR0RFr\nmRpIIqIBAAAaGREdsaGIHugveZ4EAAAA40VER6x1WmUlur/oeRIAAACMFxEdsZYp5Zc8M8BKNAAA\nQKMioiPW0lI+z+xxfgcBAADAuBHRERuOaL6wEAAAoGER0REbiuiBAb9zAAAAYPyI6IgNr0QPmt9B\nAAAAMG6w1Y5QAAAb9UlEQVREdMRaW8vnRDQAAEDjIqIjNrwSneWlBwAAaFSUXMSCQEpYUZkcLz0A\nAECjouQ8aGnKaiCX9D0GAAAAxomI9qAlmVeGiAYAAGhYRLQHrcmcMgUiGgAAoFER0R60JAvK5APf\nYwAAAGCciGgPWlIFZYpENAAAQKMioj1oCYoaKKZ9jwEAAIBxIqI9aEkXlSmlJed8jwIAAIBxIKI9\naE2XlFGLlMv5HgUAAADjQER70NLsyhGdyfgeBQAAAONARHswHNGDg75HAQAAwDgQ0R60tEgDamUl\nGgAAoEER0R60tIiVaAAAgAZGRHvQOsXYEw0AANDAiGgPWlpNRSWV78/6HgUAAADjQER70NKWkCRl\neohoAACARkREezAU0QO9ec+TAAAAYDyIaA9apgaSWIkGAABoVES0B60dKUlENAAAQKMioj1o6UhL\nkjK7+dpvAACARkREezAc0eyJBgAAaEhEtAdDe6IH+oqeJwEAAMB4ENEeTGkzSdJAf8nzJAAAABgP\nItqDtrbyeX+f8zsIAAAAxoWI9mAoovv6ze8gAAAAGBci2oP29vJ5/wARDQAA0IiIaA9aWiRTSf2Z\nhO9RAAAAMA5EtAdNTdKUxKD6M0nfowAAAGAciGhP2oKs+rIp32MAAABgHIhoT9pTWfXniGgAAIBG\nRER70pbKqz+f9j0GAAAAxoGI9qStOa/+YrPvMQAAADAORLQn7S1F9ZWmSEW++hsAAKDRENGetLWW\n1K82ac8e36MAAACgSkS0J21TXDmi+/t9jwIAAIAqEdGetLWJlWgAAIAGRUR70j7V1Kd2uT5WogEA\nABoNEe1J27QmFZVUdteA71EAAABQJSLak7Zp5a/87t8+6HkSAAAAVIuI9qRteiWid+Y8TwIAAIBq\nEdGetM8of+V3386850kAAABQLSLak7ZKRPfvIqIBAAAaDRHtSdus8ld+9/fyjYUAAACNhoj2pG1m\nWpLU10NEAwAANBoi2pP2aeWXvn93yfMkAAAAqBYR7UlbW/m8fzcr0QAAAI2GiPZkOKL7/M4BAACA\n6nmLaDNLmNnvzew/fM3gU0uL1KSidvfzdwwAAECj8VlwV0va4PH5vTKTpgUD6h0IfI8CAACAKnmJ\naDObK+lCSd/28fz1oiOVUc9g2vcYAAAAqJKvlegbJX1G0gE/msLMrjCzNWa2pru7O7rJItTRMqie\nbIvvMQAAAFClyCPazN4i6RXn3NqD3c85t9I51+Wc6+rs7Ixoumh1tObVk5/iewwAAABUycdK9HJJ\nbzOzTZLukPRGM7vVwxzedbQV1OOmSYODvkcBAABAFSKPaOfcZ51zc51z8yVdKum/nHMrop6jHnRM\nLalHHVJvr+9RAAAAUAU+X82jjg4R0QAAAA3Ia0Q75x50zr3F5ww+dcxoUr/aVdhBRAMAADQSVqI9\n6piZkCT1bhvwPAkAAACqQUR71HFYSpLUsy3jeRIAAABUg4j2qOPw8het9Lyc9TwJAAAAqkFEe9Qx\np1WS1NOd9zwJAAAAqkFEezQc0TuKnicBAABANYhoj4beWNizy3meBAAAANUgoj3q6Cif9/Sa30EA\nAABQFSLao7Y2qUlF9ezmMAAAADQS6s2jpiZpWnKPevYkfY8CAACAKhDRnnWkBtSzJ/A9BgAAAKpA\nRHvW0ZzVrkyz7zEAAABQBSLasxltOe3MtfkeAwAAAFUgoj3r7Mhre2mGNDDgexQAAACMERHt2ayZ\nTt3qlHbs8D0KAAAAxoiI9qzzcFOPpiu/bbvvUQAAADBGRLRns44ofzLHjk19nicBAADAWBHRnnXO\nK38yx/YX2BMNAADQKIhoz2YdPUWS1L150PMkAAAAGCsi2rPOY9olSd3bCp4nAQAAwFgR0Z7Nml3e\nE72923meBAAAAGNFRHs2c2b5vHsHhwIAAKBRUG6eBYHUkdit7b1J36MAAABgjIjoOtCZ3q3uvhbf\nYwAAAGCMiOg6MKs1o+2ZVt9jAAAAYIyI6DrQOXVQ3blpvscAAADAGBHRdWDWjJK2l2ZIe/b4HgUA\nAABjQETXgcMOb9IrOkylrS/5HgUAAABjQETXgTlHJVVQoO1Pb/c9CgAAAMaAiK4Dc/6i/MkcW5/e\n7XkSAAAAjAURXQfmHDdVkrTt2YznSQAAADAWRHQdmLO4Q5K09YWC50kAAAAwFkR0HThiTvkwbN1m\nnicBAADAWBDRdSCdlmYld2nr9pTvUQAAADAGRHSdmNPSo629U3yPAQAAgDEgouvEnGl7tHWgw/cY\nAAAAGAMiuk7MmZXV1kKnVCz6HgUAAACHQETXiTmznV7SESq+1O17FAAAABwCEV0n5hwVqKSEXln/\nsu9RAAAAcAhEdJ2Yu7BNkrR53U7PkwAAAOBQiOg6Mb9rliRp0x8GPE8CAACAQyGi68TRS8pf/b3p\n2ZLnSQAAAHAoRHSdmDrNNCPRo+deDHyPAgAAgEMgouvIgind2rSj3fcYAAAAOAQiuo7Mn9mnTXs6\nfY8BAACAQyCi68j8I/PaVJwn17/H9ygAAAA4CCK6jsw/NqFBtejlx170PQoAAAAOgoiuIwsWt0iS\nNq3d4XkSAAAAHAwRXUfmnzJTkvTsk2znAAAAqGdEdB055ozDZCrpmQ0F36MAAADgIIjoOtIypUlH\nB9v0xxeafY8CAACAgyCi68zCGa/o6R18zB0AAEA9I6LrzMKjBvTH7HyVsnnfowAAAOAAiOg6c9yi\nJg1oirY8vNn3KAAAADgAIrrOLDxtqiTpj79+xfMkAAAAOBAius4sPHu2JOnp32c8TwIAAIADIaLr\nzOGLZmia9WrDH833KAAAADgAIrrOmElL2p/XE5un+x4FAAAAB0BE16FlR+/S433HqFQo+R4FAAAA\noyCi69BJJ5v61a6ND23xPQoAAABGQUTXoWVvnCFJWvfLlzxPAgAAgNEQ0XVo8YULlFRe6x7N+h4F\nAAAAoyCi61DzzClalNqodX9q9T0KAAAARkFE16mTZ7+k33UfLed8TwIAAID9EdF16nVdOXWXZmnj\nb3f4HgUAAAD7iTyizWyemT1gZn8ws6fM7OqoZ2gEr3vbLEnSb+54wfMkAAAA2J+PleiCpE855xZL\nOkPSlWa22MMcdW3xOxdqqnr1m4fyvkcBAADAfiKPaOfcNufcY5XLfZI2SDoy6jnqXVNbq1479Sn9\n5plZvkcBAADAfrzuiTaz+ZJOkvSozznq1euO26n1/fO1a3vR9ygAAAAYwVtEm1mbpB9JusY5t3uU\nn19hZmvMbE13d3f0A9aBN16QllOT/uuWTb5HAQAAwAheItrMApUD+jbn3I9Hu49zbqVzrss519XZ\n2RntgHXi9P+1RO3arfvv6vM9CgAAAEbw8ekcJuk7kjY45/5P1M/fSIJ5R+istrW674nDfI8CAACA\nEXysRC+X9F5JbzSzdZXTmz3M0RDOPWmHNg7M0XN/zPkeBQAAABU+Pp3jv51z5pxb6pxbVjndHfUc\njeL8S6ZJkv7jm3xeNAAAQL3gGwvr3F++93Qt0gbddZfvSQAAADCEiK53U6fqHQvWafXm+drBN4AD\nAADUBSK6Abzj4qSKSupn/7LN9ygAAAAQEd0QTrnyDB2jjfrhdwd9jwIAAAAR0Q3Bjpqn9859UL/6\n89HassX3NAAAACCiG8R7P5SSU5Nu/Ue2dAAAAPhGRDeIY688X6/Xr/XtfwtUKvmeBgAAYHIjohtF\nZ6euPOURbeyZpf/8Wd73NAAAAJMaEd1A3vmFpZqjF/X1v9vuexQAAIBJjYhuIMGb36QrZ9yhe56Y\nrTVrfE8DAAAweRHRjaSpSR/7TKtmaIf+7uN88woAAIAvRHSDmfrx9+vTbd/S3Y/M1KOP+p4GAABg\nciKiG01rqz722amapW599m965JzvgQAAACYfIroBtV39IX2x7QY98FiH7ridigYAAIgaEd2IpkzR\nX1+/QF36nT7xN4Pq6fE9EAAAwORCRDeoxEc+rJsX3qju3pQ+dVXO9zgAAACTChHdqBIJnfK9q3St\nvqJbfpDSHXf4HggAAGDyIKIb2emn64tXduu1+o2u+Ku8nnnG90AAAACTAxHd4IIbvqzbF/29UoO7\ndeH/KGg7X2YIAAAQOiK60TU36+gff1U/TV+iFzYVddFbixoY8D0UAABAvBHRcbBwoZb/+zW6TSv0\nyCPSmy9w6u/3PRQAAEB8EdFx8Za36F03nalbtUL//euSznuTY2sHAABASIjoOPnYx3TZ9ct0p7tY\nj/02r9NOdXrqKd9DAQAAxA8RHTfXXqt3/tP/o4dKr1fmxR067TSnlSvF14MDAADUEBEdR5/8pE7/\nv5/SmqbT9brif+uv/1q66CLpxRd9DwYAABAPRHRcvec9OvL/u1P3zPtf+qo+oXvvzuu445y+/GUp\nm/U9HAAAQGMjouPslFPU9Pu1uubDe/RUcaHOdffrc5+T/vIvnW6+mZgGAAAYLyI67trapJUrdey9\nN+snR1+t+3Su5vRs0Ec/Kh1zjPT3fy9t2+Z7SAAAgMZCRE8Wb3qT9PjjOvfGt+o36bN1n87V8Znf\n6fOfl446yund75buukvKZHwPCgAAUP+I6MkkCKSrr5Ztek7nfvUtunfKO/UnvUZXp/9FD969R+98\np3TYYdJll0n//u/Sjh2+BwYAAKhP5hrgs8+6urrcmjVrfI8RP4WCdPfd0s03K/+f9+tBnal/n/HX\numvwAm0fmCIzp5NPNr3pTdLrXy+ddpo0a5bvoQEAAMJjZmudc12HvB8RDUnlz79btUq6804VfvOo\nfqdTdd+Ut+v+1ov08M6/VKFY/pcWxx4rnX661NUlHX+8dMIJ0uzZkpnn+QEAAGqAiMb4bdsm3Xef\ndO+90r33ak/3Hq1Rlx5tOUuPTn2THhlYqq19U4fvPn16OaiPO678ZsWRp5kzCWwAANA4iGjURqkk\nbdggPfqo9Mgj0sMPS089pW43U0/peK0PTtb6acu13pboz5k5erm/bZ9fb2uTjj66vFo9dJozZ+/l\nI44oh3ZHh9TEDn0AAOAZEY3w7Nkj/eEP0vr15dOTT5ZDe8sW7VGrntMCPacFejZYqGfbT9Tm5Hxt\nc0doW3amtu6Zqnwx8aqHNCuvaM+cKc2YUT4fujxjhjR1qtTefvBTOs2qNwAAmBgiGtEbHJSef17a\nuFF69tny6bnnyttDtm6VXnpJLp/XTs3QVs3RNs3WSzpCOxOHaUfzkdqZOkI7kodpp2Zqp+vQjvxU\n7Rycot3Z9JiePpksx3Rbm9TSUv2ptVVqbi7HeCr16vPRbhv5MwIeAIDGN9aITkYxDCaJ5ubyxujj\njhv956WSbMcOzdy2TTO3btWSbduk7dvLn6W34w/Szv+uXN4h7dwp9e2QslkVlFCf2g942m3T1Jea\npb7UDPVpuvYMtCuTmaKMtSijVmVcs3a7Zg2U0soU08oUA2UK5dNoq+LjFQQHD+10eu99gmDf01hu\nC+v3kkn+AAAAoFpENKLT1CR1dpZPS5ce+v7OSQMDSvb2anpfn6b39Um7d0t9fXtPw9d3SLufK1/O\nZMqngYFXX85n9vlGmaKalFHLPqecUsOnrNKjn1uLcslW5ZKtyiZalUu0KNvUolyiufwz16xcNq1s\nNq1cf1rZocd0KQ0oqZwLlHfJ8qmUUK5UPs8XE8qXmpQvNilXSChfjGajeDI5vmgPI+yTyfGfhn4/\nkeAPAwBAuIho1C8zacqU8qmWnCtvPclklBgYUFsmo7aRoZ3LSdnsoU+5nJTtkbIvj/G+o9yWzx98\nVElFJZRXoLwC5ZQavjza9eHbkq3KBy3KJ8qxn08073PKJZqVb0orb+nyeVNKOWtW3gLllVLe9nvs\nXKB8NqmcSypfSirjEtpdCf/ccPgnKvHfpHzRlC+UT7m8n3eMJhITC/GxnhKJ0U9NTdXd7vN3mprK\nJ7N9L/OHCAAcGBGNycds70boGTP8zuJcOaSHIjuX2+dkuZyS2aySuZxa9vvZaPc/8O27xnb/7Ci3\nZ7PlOcf7j6h9/xAYjv2mFuWDVuVTU5QPyqv6hWTz3lPQokIiXT4l0yo0pfdeT6RVaEqp0JRS3lLD\nlwsW7D0pWbmcLF9WUgWXVEGJveelhAouoYIr/xFQcE0qlJpUyDdpIGsqFE2FgqlQ0EFPxeK+p1Kp\nfN4Abzk5qKGQHgrrkafRbo/yvtX+flinsB8/itPQP8PQMR95/DmP//mQaq6Hdd+R15uapGnTVNeI\naMAns73vTGxrO/T9fSkWxxnvOVnlj4BX/SFw0MfZOfrtmYM87yFW9cdt/3eQplJSOiW1j3L7fu9A\ndUFKxaBZxWS6fB40qxSky9eHTonyfUrJVPnyq06BSomUik3B3lMipaIlVUoEKlpSRdd0wJAfy+3O\nlW8vlfa9PPI02u1jva0W9y2Vyn+wTOS5wjqN9fEBjN0xx5Q/p6CeEdEADi2R2Lt6X68Osaof+u0D\nA9KuXa/+NwmVf5tQi1X9A0okRo/9/T9WZui0/76VQ11OjfF+Y7081vslavfG33oRZsxP5I+AodlG\nzsl5/M+HVHM9rPvuf33qVNU9IhpAPEyCVf2a3L5nz757UfL5Q18uFv28VmZji/Ch4B65SX3/DesH\n+tlELo/jdyyZlE34sZr23dw+dL7/vxcHECoiGgCi1Air+vsb2gNysNAea5BP9HcO9PtDe1P2v5zN\njn77yP0sB/v9kfdrBCPfKTpaaI92Xqv7RH3foQ3dB9oQX+31WjxGFI/JH0t1g4gGABzc0P95B0Fj\nxX8tDe17qEWQjzXaR/vZyI3hIze5jzw/2M9qdd98PvwZcGC1DPOR7y6t9vJ4f28sj3HEEdLXv+77\nlT4oIhoAgEMx27ulIpXyPc3k4NzBg3u0d5FO9Ppke8yhdxYP/Ww8lw/080M99qEet7fX938CD4mI\nBgAA9WdoTzxQp5p8DwAAAAA0GiIaAAAAqBIRDQAAAFSJiAYAAACqREQDAAAAVSKiAQAAgCoR0QAA\nAECViGgAAACgSkQ0AAAAUCUiGgAAAKgSEQ0AAABUiYgGAAAAqkREAwAAAFUiogEAAIAqEdEAAABA\nlYhoAAAAoEpENAAAAFAlIhoAAACokpeINrPzzeyPZvZnM7vOxwwAAADAeEUe0WaWkPQNSRdIWizp\nMjNbHPUcAAAAwHj5WIk+TdKfnXPPOudyku6QdJGHOQAAAIBx8RHRR0raPOL6lspt+zCzK8xsjZmt\n6e7ujmw4AAAA4FDq9o2FzrmVzrku51xXZ2en73EAAACAYUkPz/mipHkjrs+t3HZAa9eu3W5mz4c6\n1ehmSdru4XkRLY7z5MBxnhw4zpMDx3ly8HWcjx7Lncw5F/Yg+z6hWVLSnySdo3I8/07S/3TOPRXp\nIGNgZmucc12+50C4OM6TA8d5cuA4Tw4c58mh3o9z5CvRzrmCmX1M0j2SEpJuqceABgAAAA7Ex3YO\nOefulnS3j+cGAAAAJqpu31hYJ1b6HgCR4DhPDhznyYHjPDlwnCeHuj7Oke+JBgAAABodK9EAAABA\nlYjoAzCz883sj2b2ZzO7zvc8GB8zm2dmD5jZH8zsKTO7unL7DDO7z8yeqZxPH/E7n60c9z+a2f/w\nNz2qZWYJM/u9mf1H5TrHOWbMrMPMVpnZ02a2wcxey3GOHzP7ROV/s9eb2e1m1sxxbnxmdouZvWJm\n60fcVvVxNbNTzOzJys9uMjOL+p9FIqJHZWYJSd+QdIGkxZIuM7PFfqfCOBUkfco5t1jSGZKurBzL\n6yT9yjn3Gkm/qlxX5WeXSjpe0vmSvln5zwMaw9WSNoy4znGOn69J+qVzbqGkE1U+3hznGDGzIyVd\nJanLOXeCyp/kdak4znHwPZWP0UjjOa7fkvRhSa+pnPZ/zEgQ0aM7TdKfnXPPOudyku6QdJHnmTAO\nzrltzrnHKpf7VP4/3CNVPp7fr9zt+5LeXrl8kaQ7nHNZ59xzkv6s8n8eUOfMbK6kCyV9e8TNHOcY\nMbNpkt4g6TuS5JzLOed6xHGOo6Sklsp3S7RK2iqOc8Nzzq2WtHO/m6s6rmY2W9JU59wjrvzGvn8b\n8TuRIqJHd6SkzSOub6nchgZmZvMlnSTpUUmHO+e2VX70kqTDK5c59o3rRkmfkVQacRvHOV4WSOqW\n9N3Ktp1vm9kUcZxjxTn3oqQbJL0gaZukXufcveI4x1W1x/XIyuX9b48cEY1JwczaJP1I0jXOud0j\nf1b5S5aPqWlgZvYWSa8459Ye6D4c51hISjpZ0reccydJ2qPKv/odwnFufJU9sRep/EfTHElTzGzF\nyPtwnOOp0Y4rET26FyXNG3F9buU2NCAzC1QO6Nuccz+u3Pxy5V8JqXL+SuV2jn1jWi7pbWa2SeXt\nV280s1vFcY6bLZK2OOcerVxfpXJUc5zj5VxJzznnup1zeUk/lvQ6cZzjqtrj+mLl8v63R46IHt3v\nJL3GzBaYWUrlje0/8zwTxqHyjt3vSNrgnPs/I370M0nvr1x+v6Sfjrj9UjNLm9kCld+w8Nuo5sX4\nOOc+65yb65ybr/J/X//LObdCHOdYcc69JGmzmR1XuekcSX8QxzluXpB0hpm1Vv43/ByV38/CcY6n\nqo5rZevHbjM7o/Kfj/eN+J1Iefna73rnnCuY2cck3aPyu4Jvcc495XksjM9ySe+V9KSZravc9jlJ\n10u608w+JOl5Se+RJOfcU2Z2p8r/x1yQdKVzrhj92KgRjnP8fFzSbZUFjmclfVDlBSGOc0w45x41\ns1WSHlP5uP1e5W+uaxPHuaGZ2e2SzpI0y8y2SPo7je9/p/9G5U/6aJH0n5VT5PjGQgAAAKBKbOcA\nAAAAqkREAwAAAFUiogEAAIAqEdEAAABAlYhoAAAAoEpENABMUmZ2lpn9h+85AKAREdEAAABAlYho\nAKhzZrbCzH5rZuvM7F/MLGFm/Wb2VTN7ysx+ZWadlfsuM7NHzOwJM7vLzKZXbv8LM7vfzB43s8fM\n7NjKw7eZ2Soze9rMbqt8AxgA4BCIaACoY2a2SNIlkpY79/+3d/+uPsVxHMefL4v8igwWAxkpP1IG\nZfIPGK6FbjJbbFIs/gfFeMUgxa4M37oTBqWMplvqLhKKdL0M3zNcFs4t935dz8d0zvt8evf5LOe8\ne59PfXocWAEuAjuAl22PABOmJ38B3AOutT0KvF4VfwDcbnsMOA28G+IngKvAYeAQ01M+JUm/4bHf\nkjTbzgIngRdDk3gbsAx8Bx4OY+4Dj5PsBva0nQzxBeBRkl3A/rZPANp+ARjyPW+7NNy/Ag4Ci39/\nWZL0b7OIlqTZFmCh7fWfgsnNX8Z1jfm/rrpewe+CJP0Rt3NI0mx7Bswl2QeQZG+SA0zf33PDmAvA\nYtsPwPskZ4b4PDBp+xFYSnJuyLE1yfZ1XYUkbTJ2HCRphrV9k+QG8DTJFuAbcAX4DJwani0z3TcN\ncAm4MxTJb4HLQ3weuJvk1pDj/DouQ5I2nbRr/QMoSdooST613bnR85Ck/5XbOSRJkqSR7ERLkiRJ\nI9mJliRJkkayiJYkSZJGsoiWJEmSRrKIliRJkkayiJYkSZJGsoiWJEmSRvoBrU2O1eukuE4AAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11ee1d518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,10))\n",
    "plt.plot(train_loss_history, color=\"red\", label=\"train loss\")\n",
    "plt.plot(valid_loss_history, color=\"blue\", label=\"valid loss\")\n",
    "plt.xlabel(\"epoch\")\n",
    "plt.ylabel(\"loss\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF Accuracy: 45.008881%\n"
     ]
    }
   ],
   "source": [
    "mf_accuracy = np.sum(np.round(final_valid_predictions) == validset.rating.values) / len(final_valid_predictions)\n",
    "print(\"MF Accuracy: %f%%\"%(mf_accuracy*100,))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### results on validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>userid</th>\n",
       "      <th>movieid</th>\n",
       "      <th>age_desc</th>\n",
       "      <th>occ_desc</th>\n",
       "      <th>title</th>\n",
       "      <th>genre</th>\n",
       "      <th>rating</th>\n",
       "      <th>prediction (rnd.)</th>\n",
       "      <th>prediction (prc.)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>323753</th>\n",
       "      <td>M</td>\n",
       "      <td>3432</td>\n",
       "      <td>1220</td>\n",
       "      <td>25-34</td>\n",
       "      <td>programmer</td>\n",
       "      <td>Blues Brothers, The (1980)</td>\n",
       "      <td>Action|Comedy|Musical</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3.595490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>572412</th>\n",
       "      <td>M</td>\n",
       "      <td>3579</td>\n",
       "      <td>2089</td>\n",
       "      <td>18-24</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Rescuers Down Under, The (1990)</td>\n",
       "      <td>Animation|Children's</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2.909302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>473636</th>\n",
       "      <td>M</td>\n",
       "      <td>3931</td>\n",
       "      <td>1680</td>\n",
       "      <td>25-34</td>\n",
       "      <td>customer service</td>\n",
       "      <td>Sliding Doors (1998)</td>\n",
       "      <td>Drama|Romance</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3.858740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>308120</th>\n",
       "      <td>M</td>\n",
       "      <td>4378</td>\n",
       "      <td>1203</td>\n",
       "      <td>18-24</td>\n",
       "      <td>technician/engineer</td>\n",
       "      <td>12 Angry Men (1957)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3.197440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>621235</th>\n",
       "      <td>M</td>\n",
       "      <td>4251</td>\n",
       "      <td>2301</td>\n",
       "      <td>45-49</td>\n",
       "      <td>executive/managerial</td>\n",
       "      <td>History of the World: Part I (1981)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3.245275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>280736</th>\n",
       "      <td>M</td>\n",
       "      <td>1482</td>\n",
       "      <td>1127</td>\n",
       "      <td>25-34</td>\n",
       "      <td>executive/managerial</td>\n",
       "      <td>Abyss, The (1989)</td>\n",
       "      <td>Action|Adventure|Sci-Fi|Thriller</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3.565892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251324</th>\n",
       "      <td>M</td>\n",
       "      <td>2414</td>\n",
       "      <td>1033</td>\n",
       "      <td>25-34</td>\n",
       "      <td>academic/educator</td>\n",
       "      <td>Fox and the Hound, The (1981)</td>\n",
       "      <td>Animation|Children's</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3.941283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178892</th>\n",
       "      <td>M</td>\n",
       "      <td>4732</td>\n",
       "      <td>648</td>\n",
       "      <td>25-34</td>\n",
       "      <td>sales/marketing</td>\n",
       "      <td>Mission: Impossible (1996)</td>\n",
       "      <td>Action|Adventure|Mystery</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3.208387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>723124</th>\n",
       "      <td>M</td>\n",
       "      <td>4439</td>\n",
       "      <td>2694</td>\n",
       "      <td>35-44</td>\n",
       "      <td>doctor/health care</td>\n",
       "      <td>Big Daddy (1999)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3.135849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83939</th>\n",
       "      <td>M</td>\n",
       "      <td>727</td>\n",
       "      <td>317</td>\n",
       "      <td>35-44</td>\n",
       "      <td>lawyer</td>\n",
       "      <td>Santa Clause, The (1994)</td>\n",
       "      <td>Children's|Comedy|Fantasy</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3.934950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>541832</th>\n",
       "      <td>M</td>\n",
       "      <td>4098</td>\n",
       "      <td>2001</td>\n",
       "      <td>35-44</td>\n",
       "      <td>executive/managerial</td>\n",
       "      <td>Lethal Weapon 2 (1989)</td>\n",
       "      <td>Action|Comedy|Crime|Drama</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4.141999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>915511</th>\n",
       "      <td>M</td>\n",
       "      <td>1579</td>\n",
       "      <td>3499</td>\n",
       "      <td>25-34</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Misery (1990)</td>\n",
       "      <td>Horror</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4.010313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>405738</th>\n",
       "      <td>M</td>\n",
       "      <td>3934</td>\n",
       "      <td>1379</td>\n",
       "      <td>25-34</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Young Guns II (1990)</td>\n",
       "      <td>Action|Comedy|Western</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.088194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285899</th>\n",
       "      <td>F</td>\n",
       "      <td>1899</td>\n",
       "      <td>1147</td>\n",
       "      <td>45-49</td>\n",
       "      <td>doctor/health care</td>\n",
       "      <td>When We Were Kings (1996)</td>\n",
       "      <td>Documentary</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4.463424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181963</th>\n",
       "      <td>F</td>\n",
       "      <td>4048</td>\n",
       "      <td>673</td>\n",
       "      <td>35-44</td>\n",
       "      <td>academic/educator</td>\n",
       "      <td>Space Jam (1996)</td>\n",
       "      <td>Adventure|Animation|Children's|Comedy|Fantasy</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4.013864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>931626</th>\n",
       "      <td>M</td>\n",
       "      <td>3125</td>\n",
       "      <td>3555</td>\n",
       "      <td>25-34</td>\n",
       "      <td>executive/managerial</td>\n",
       "      <td>U-571 (2000)</td>\n",
       "      <td>Action|Thriller</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3.929566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>602570</th>\n",
       "      <td>M</td>\n",
       "      <td>5788</td>\n",
       "      <td>2193</td>\n",
       "      <td>25-34</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Willow (1988)</td>\n",
       "      <td>Action|Adventure|Fantasy</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3.773484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>516839</th>\n",
       "      <td>F</td>\n",
       "      <td>5530</td>\n",
       "      <td>1924</td>\n",
       "      <td>18-24</td>\n",
       "      <td>college/grad student</td>\n",
       "      <td>Plan 9 from Outer Space (1958)</td>\n",
       "      <td>Horror|Sci-Fi</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2.927153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>596378</th>\n",
       "      <td>M</td>\n",
       "      <td>2242</td>\n",
       "      <td>2162</td>\n",
       "      <td>18-24</td>\n",
       "      <td>college/grad student</td>\n",
       "      <td>NeverEnding Story II: The Next Chapter, The (1...</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3.229770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>336866</th>\n",
       "      <td>F</td>\n",
       "      <td>1516</td>\n",
       "      <td>1240</td>\n",
       "      <td>25-34</td>\n",
       "      <td>programmer</td>\n",
       "      <td>Terminator, The (1984)</td>\n",
       "      <td>Action|Sci-Fi|Thriller</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3.214885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>403557</th>\n",
       "      <td>F</td>\n",
       "      <td>4208</td>\n",
       "      <td>1376</td>\n",
       "      <td>35-44</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Star Trek IV: The Voyage Home (1986)</td>\n",
       "      <td>Action|Adventure|Sci-Fi</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>3.898657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>532357</th>\n",
       "      <td>M</td>\n",
       "      <td>3101</td>\n",
       "      <td>1968</td>\n",
       "      <td>18-24</td>\n",
       "      <td>scientist</td>\n",
       "      <td>Breakfast Club, The (1985)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>4.322102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>898240</th>\n",
       "      <td>M</td>\n",
       "      <td>5648</td>\n",
       "      <td>3421</td>\n",
       "      <td>25-34</td>\n",
       "      <td>technician/engineer</td>\n",
       "      <td>Animal House (1978)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4.268606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>877298</th>\n",
       "      <td>M</td>\n",
       "      <td>2095</td>\n",
       "      <td>3301</td>\n",
       "      <td>56+</td>\n",
       "      <td>retired</td>\n",
       "      <td>Whole Nine Yards, The (2000)</td>\n",
       "      <td>Comedy|Crime</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3.500426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>796428</th>\n",
       "      <td>M</td>\n",
       "      <td>1395</td>\n",
       "      <td>2968</td>\n",
       "      <td>25-34</td>\n",
       "      <td>artist</td>\n",
       "      <td>Time Bandits (1981)</td>\n",
       "      <td>Adventure|Fantasy|Sci-Fi</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>4.660578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>732618</th>\n",
       "      <td>M</td>\n",
       "      <td>523</td>\n",
       "      <td>2716</td>\n",
       "      <td>50-55</td>\n",
       "      <td>executive/managerial</td>\n",
       "      <td>Ghostbusters (1984)</td>\n",
       "      <td>Comedy|Horror</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>4.430697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>627537</th>\n",
       "      <td>F</td>\n",
       "      <td>3713</td>\n",
       "      <td>2324</td>\n",
       "      <td>25-34</td>\n",
       "      <td>executive/managerial</td>\n",
       "      <td>Life Is Beautiful (La Vita è bella) (1997)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>4.849105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40433</th>\n",
       "      <td>M</td>\n",
       "      <td>5677</td>\n",
       "      <td>141</td>\n",
       "      <td>25-34</td>\n",
       "      <td>academic/educator</td>\n",
       "      <td>Birdcage, The (1996)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>3.652851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>621897</th>\n",
       "      <td>F</td>\n",
       "      <td>3464</td>\n",
       "      <td>2302</td>\n",
       "      <td>25-34</td>\n",
       "      <td>sales/marketing</td>\n",
       "      <td>My Cousin Vinny (1992)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3.703601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>904828</th>\n",
       "      <td>F</td>\n",
       "      <td>3205</td>\n",
       "      <td>3450</td>\n",
       "      <td>35-44</td>\n",
       "      <td>self-employed</td>\n",
       "      <td>Grumpy Old Men (1993)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3.459073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387733</th>\n",
       "      <td>M</td>\n",
       "      <td>5111</td>\n",
       "      <td>1334</td>\n",
       "      <td>35-44</td>\n",
       "      <td>artist</td>\n",
       "      <td>Blob, The (1958)</td>\n",
       "      <td>Horror|Sci-Fi</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3.665181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>482628</th>\n",
       "      <td>F</td>\n",
       "      <td>1894</td>\n",
       "      <td>1721</td>\n",
       "      <td>35-44</td>\n",
       "      <td>executive/managerial</td>\n",
       "      <td>Titanic (1997)</td>\n",
       "      <td>Drama|Romance</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3.531790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851826</th>\n",
       "      <td>M</td>\n",
       "      <td>4774</td>\n",
       "      <td>3176</td>\n",
       "      <td>45-49</td>\n",
       "      <td>sales/marketing</td>\n",
       "      <td>Talented Mr. Ripley, The (1999)</td>\n",
       "      <td>Drama|Mystery|Thriller</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4.008468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>566304</th>\n",
       "      <td>M</td>\n",
       "      <td>4957</td>\n",
       "      <td>2072</td>\n",
       "      <td>25-34</td>\n",
       "      <td>artist</td>\n",
       "      <td>'burbs, The (1989)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2.680943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>523888</th>\n",
       "      <td>M</td>\n",
       "      <td>4533</td>\n",
       "      <td>1954</td>\n",
       "      <td>25-34</td>\n",
       "      <td>programmer</td>\n",
       "      <td>Rocky (1976)</td>\n",
       "      <td>Action|Drama</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3.779227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999132</th>\n",
       "      <td>M</td>\n",
       "      <td>2362</td>\n",
       "      <td>3948</td>\n",
       "      <td>25-34</td>\n",
       "      <td>sales/marketing</td>\n",
       "      <td>Meet the Parents (2000)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>4.275411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142695</th>\n",
       "      <td>F</td>\n",
       "      <td>2657</td>\n",
       "      <td>527</td>\n",
       "      <td>Under 18</td>\n",
       "      <td>K-12 student</td>\n",
       "      <td>Schindler's List (1993)</td>\n",
       "      <td>Drama|War</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>4.537353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169208</th>\n",
       "      <td>F</td>\n",
       "      <td>2926</td>\n",
       "      <td>595</td>\n",
       "      <td>18-24</td>\n",
       "      <td>artist</td>\n",
       "      <td>Beauty and the Beast (1991)</td>\n",
       "      <td>Animation|Children's|Musical</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4.074231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>950386</th>\n",
       "      <td>M</td>\n",
       "      <td>3554</td>\n",
       "      <td>3676</td>\n",
       "      <td>25-34</td>\n",
       "      <td>scientist</td>\n",
       "      <td>Eraserhead (1977)</td>\n",
       "      <td>Drama|Horror</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>2.755942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>830422</th>\n",
       "      <td>F</td>\n",
       "      <td>5643</td>\n",
       "      <td>3098</td>\n",
       "      <td>35-44</td>\n",
       "      <td>academic/educator</td>\n",
       "      <td>Natural, The (1984)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3.336859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974878</th>\n",
       "      <td>M</td>\n",
       "      <td>1820</td>\n",
       "      <td>3770</td>\n",
       "      <td>25-34</td>\n",
       "      <td>writer</td>\n",
       "      <td>Dreamscape (1984)</td>\n",
       "      <td>Adventure|Crime|Sci-Fi|Thriller</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3.603387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129671</th>\n",
       "      <td>M</td>\n",
       "      <td>1352</td>\n",
       "      <td>480</td>\n",
       "      <td>35-44</td>\n",
       "      <td>writer</td>\n",
       "      <td>Jurassic Park (1993)</td>\n",
       "      <td>Action|Adventure|Sci-Fi</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2.773809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>629803</th>\n",
       "      <td>M</td>\n",
       "      <td>5488</td>\n",
       "      <td>2333</td>\n",
       "      <td>25-34</td>\n",
       "      <td>scientist</td>\n",
       "      <td>Gods and Monsters (1998)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>3.892385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80676</th>\n",
       "      <td>F</td>\n",
       "      <td>4208</td>\n",
       "      <td>302</td>\n",
       "      <td>35-44</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Queen Margot (La Reine Margot) (1994)</td>\n",
       "      <td>Drama|Romance</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>4.066545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851072</th>\n",
       "      <td>M</td>\n",
       "      <td>1548</td>\n",
       "      <td>3176</td>\n",
       "      <td>35-44</td>\n",
       "      <td>academic/educator</td>\n",
       "      <td>Talented Mr. Ripley, The (1999)</td>\n",
       "      <td>Drama|Mystery|Thriller</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>4.229926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>902891</th>\n",
       "      <td>M</td>\n",
       "      <td>4030</td>\n",
       "      <td>3444</td>\n",
       "      <td>35-44</td>\n",
       "      <td>executive/managerial</td>\n",
       "      <td>Bloodsport (1988)</td>\n",
       "      <td>Action</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3.480523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>516319</th>\n",
       "      <td>F</td>\n",
       "      <td>4472</td>\n",
       "      <td>1923</td>\n",
       "      <td>45-49</td>\n",
       "      <td>self-employed</td>\n",
       "      <td>There's Something About Mary (1998)</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3.330519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>674849</th>\n",
       "      <td>M</td>\n",
       "      <td>5102</td>\n",
       "      <td>2474</td>\n",
       "      <td>25-34</td>\n",
       "      <td>programmer</td>\n",
       "      <td>Color of Money, The (1986)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4.380075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184396</th>\n",
       "      <td>F</td>\n",
       "      <td>1051</td>\n",
       "      <td>708</td>\n",
       "      <td>25-34</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Truth About Cats &amp; Dogs, The (1996)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3.434905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>737397</th>\n",
       "      <td>M</td>\n",
       "      <td>1854</td>\n",
       "      <td>2723</td>\n",
       "      <td>25-34</td>\n",
       "      <td>other or not specified</td>\n",
       "      <td>Mystery Men (1999)</td>\n",
       "      <td>Action|Adventure|Comedy</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.071134</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       gender  userid  movieid  age_desc                occ_desc  \\\n",
       "323753      M    3432     1220     25-34              programmer   \n",
       "572412      M    3579     2089     18-24  other or not specified   \n",
       "473636      M    3931     1680     25-34        customer service   \n",
       "308120      M    4378     1203     18-24     technician/engineer   \n",
       "621235      M    4251     2301     45-49    executive/managerial   \n",
       "280736      M    1482     1127     25-34    executive/managerial   \n",
       "251324      M    2414     1033     25-34       academic/educator   \n",
       "178892      M    4732      648     25-34         sales/marketing   \n",
       "723124      M    4439     2694     35-44      doctor/health care   \n",
       "83939       M     727      317     35-44                  lawyer   \n",
       "541832      M    4098     2001     35-44    executive/managerial   \n",
       "915511      M    1579     3499     25-34  other or not specified   \n",
       "405738      M    3934     1379     25-34  other or not specified   \n",
       "285899      F    1899     1147     45-49      doctor/health care   \n",
       "181963      F    4048      673     35-44       academic/educator   \n",
       "931626      M    3125     3555     25-34    executive/managerial   \n",
       "602570      M    5788     2193     25-34  other or not specified   \n",
       "516839      F    5530     1924     18-24    college/grad student   \n",
       "596378      M    2242     2162     18-24    college/grad student   \n",
       "336866      F    1516     1240     25-34              programmer   \n",
       "403557      F    4208     1376     35-44  other or not specified   \n",
       "532357      M    3101     1968     18-24               scientist   \n",
       "898240      M    5648     3421     25-34     technician/engineer   \n",
       "877298      M    2095     3301       56+                 retired   \n",
       "796428      M    1395     2968     25-34                  artist   \n",
       "732618      M     523     2716     50-55    executive/managerial   \n",
       "627537      F    3713     2324     25-34    executive/managerial   \n",
       "40433       M    5677      141     25-34       academic/educator   \n",
       "621897      F    3464     2302     25-34         sales/marketing   \n",
       "904828      F    3205     3450     35-44           self-employed   \n",
       "387733      M    5111     1334     35-44                  artist   \n",
       "482628      F    1894     1721     35-44    executive/managerial   \n",
       "851826      M    4774     3176     45-49         sales/marketing   \n",
       "566304      M    4957     2072     25-34                  artist   \n",
       "523888      M    4533     1954     25-34              programmer   \n",
       "999132      M    2362     3948     25-34         sales/marketing   \n",
       "142695      F    2657      527  Under 18            K-12 student   \n",
       "169208      F    2926      595     18-24                  artist   \n",
       "950386      M    3554     3676     25-34               scientist   \n",
       "830422      F    5643     3098     35-44       academic/educator   \n",
       "974878      M    1820     3770     25-34                  writer   \n",
       "129671      M    1352      480     35-44                  writer   \n",
       "629803      M    5488     2333     25-34               scientist   \n",
       "80676       F    4208      302     35-44  other or not specified   \n",
       "851072      M    1548     3176     35-44       academic/educator   \n",
       "902891      M    4030     3444     35-44    executive/managerial   \n",
       "516319      F    4472     1923     45-49           self-employed   \n",
       "674849      M    5102     2474     25-34              programmer   \n",
       "184396      F    1051      708     25-34  other or not specified   \n",
       "737397      M    1854     2723     25-34  other or not specified   \n",
       "\n",
       "                                                    title  \\\n",
       "323753                         Blues Brothers, The (1980)   \n",
       "572412                    Rescuers Down Under, The (1990)   \n",
       "473636                               Sliding Doors (1998)   \n",
       "308120                                12 Angry Men (1957)   \n",
       "621235                History of the World: Part I (1981)   \n",
       "280736                                  Abyss, The (1989)   \n",
       "251324                      Fox and the Hound, The (1981)   \n",
       "178892                         Mission: Impossible (1996)   \n",
       "723124                                   Big Daddy (1999)   \n",
       "83939                            Santa Clause, The (1994)   \n",
       "541832                             Lethal Weapon 2 (1989)   \n",
       "915511                                      Misery (1990)   \n",
       "405738                               Young Guns II (1990)   \n",
       "285899                          When We Were Kings (1996)   \n",
       "181963                                   Space Jam (1996)   \n",
       "931626                                       U-571 (2000)   \n",
       "602570                                      Willow (1988)   \n",
       "516839                     Plan 9 from Outer Space (1958)   \n",
       "596378  NeverEnding Story II: The Next Chapter, The (1...   \n",
       "336866                             Terminator, The (1984)   \n",
       "403557               Star Trek IV: The Voyage Home (1986)   \n",
       "532357                         Breakfast Club, The (1985)   \n",
       "898240                                Animal House (1978)   \n",
       "877298                       Whole Nine Yards, The (2000)   \n",
       "796428                                Time Bandits (1981)   \n",
       "732618                                Ghostbusters (1984)   \n",
       "627537         Life Is Beautiful (La Vita è bella) (1997)   \n",
       "40433                                Birdcage, The (1996)   \n",
       "621897                             My Cousin Vinny (1992)   \n",
       "904828                              Grumpy Old Men (1993)   \n",
       "387733                                   Blob, The (1958)   \n",
       "482628                                     Titanic (1997)   \n",
       "851826                    Talented Mr. Ripley, The (1999)   \n",
       "566304                                 'burbs, The (1989)   \n",
       "523888                                       Rocky (1976)   \n",
       "999132                            Meet the Parents (2000)   \n",
       "142695                            Schindler's List (1993)   \n",
       "169208                        Beauty and the Beast (1991)   \n",
       "950386                                  Eraserhead (1977)   \n",
       "830422                                Natural, The (1984)   \n",
       "974878                                  Dreamscape (1984)   \n",
       "129671                               Jurassic Park (1993)   \n",
       "629803                           Gods and Monsters (1998)   \n",
       "80676               Queen Margot (La Reine Margot) (1994)   \n",
       "851072                    Talented Mr. Ripley, The (1999)   \n",
       "902891                                  Bloodsport (1988)   \n",
       "516319                There's Something About Mary (1998)   \n",
       "674849                         Color of Money, The (1986)   \n",
       "184396                Truth About Cats & Dogs, The (1996)   \n",
       "737397                                 Mystery Men (1999)   \n",
       "\n",
       "                                                genre  rating  \\\n",
       "323753                          Action|Comedy|Musical       4   \n",
       "572412                           Animation|Children's       3   \n",
       "473636                                  Drama|Romance       4   \n",
       "308120                                          Drama       3   \n",
       "621235                                         Comedy       2   \n",
       "280736               Action|Adventure|Sci-Fi|Thriller       3   \n",
       "251324                           Animation|Children's       4   \n",
       "178892                       Action|Adventure|Mystery       3   \n",
       "723124                                         Comedy       3   \n",
       "83939                       Children's|Comedy|Fantasy       4   \n",
       "541832                      Action|Comedy|Crime|Drama       4   \n",
       "915511                                         Horror       4   \n",
       "405738                          Action|Comedy|Western       2   \n",
       "285899                                    Documentary       4   \n",
       "181963  Adventure|Animation|Children's|Comedy|Fantasy       4   \n",
       "931626                                Action|Thriller       4   \n",
       "602570                       Action|Adventure|Fantasy       3   \n",
       "516839                                  Horror|Sci-Fi       3   \n",
       "596378                   Adventure|Children's|Fantasy       1   \n",
       "336866                         Action|Sci-Fi|Thriller       4   \n",
       "403557                        Action|Adventure|Sci-Fi       5   \n",
       "532357                                   Comedy|Drama       5   \n",
       "898240                                         Comedy       4   \n",
       "877298                                   Comedy|Crime       3   \n",
       "796428                       Adventure|Fantasy|Sci-Fi       5   \n",
       "732618                                  Comedy|Horror       3   \n",
       "627537                                   Comedy|Drama       5   \n",
       "40433                                          Comedy       5   \n",
       "621897                                         Comedy       3   \n",
       "904828                                         Comedy       4   \n",
       "387733                                  Horror|Sci-Fi       4   \n",
       "482628                                  Drama|Romance       4   \n",
       "851826                         Drama|Mystery|Thriller       4   \n",
       "566304                                         Comedy       3   \n",
       "523888                                   Action|Drama       3   \n",
       "999132                                         Comedy       5   \n",
       "142695                                      Drama|War       5   \n",
       "169208                   Animation|Children's|Musical       4   \n",
       "950386                                   Drama|Horror       4   \n",
       "830422                                          Drama       3   \n",
       "974878                Adventure|Crime|Sci-Fi|Thriller       4   \n",
       "129671                        Action|Adventure|Sci-Fi       3   \n",
       "629803                                          Drama       5   \n",
       "80676                                   Drama|Romance       2   \n",
       "851072                         Drama|Mystery|Thriller       5   \n",
       "902891                                         Action       4   \n",
       "516319                                         Comedy       2   \n",
       "674849                                          Drama       4   \n",
       "184396                                 Comedy|Romance       4   \n",
       "737397                        Action|Adventure|Comedy       2   \n",
       "\n",
       "        prediction (rnd.)  prediction (prc.)  \n",
       "323753                  4           3.595490  \n",
       "572412                  3           2.909302  \n",
       "473636                  4           3.858740  \n",
       "308120                  3           3.197440  \n",
       "621235                  3           3.245275  \n",
       "280736                  4           3.565892  \n",
       "251324                  4           3.941283  \n",
       "178892                  3           3.208387  \n",
       "723124                  3           3.135849  \n",
       "83939                   4           3.934950  \n",
       "541832                  4           4.141999  \n",
       "915511                  4           4.010313  \n",
       "405738                  2           2.088194  \n",
       "285899                  4           4.463424  \n",
       "181963                  4           4.013864  \n",
       "931626                  4           3.929566  \n",
       "602570                  4           3.773484  \n",
       "516839                  3           2.927153  \n",
       "596378                  3           3.229770  \n",
       "336866                  3           3.214885  \n",
       "403557                  4           3.898657  \n",
       "532357                  4           4.322102  \n",
       "898240                  4           4.268606  \n",
       "877298                  4           3.500426  \n",
       "796428                  5           4.660578  \n",
       "732618                  4           4.430697  \n",
       "627537                  5           4.849105  \n",
       "40433                   4           3.652851  \n",
       "621897                  4           3.703601  \n",
       "904828                  3           3.459073  \n",
       "387733                  4           3.665181  \n",
       "482628                  4           3.531790  \n",
       "851826                  4           4.008468  \n",
       "566304                  3           2.680943  \n",
       "523888                  4           3.779227  \n",
       "999132                  4           4.275411  \n",
       "142695                  5           4.537353  \n",
       "169208                  4           4.074231  \n",
       "950386                  3           2.755942  \n",
       "830422                  3           3.336859  \n",
       "974878                  4           3.603387  \n",
       "129671                  3           2.773809  \n",
       "629803                  4           3.892385  \n",
       "80676                   4           4.066545  \n",
       "851072                  4           4.229926  \n",
       "902891                  3           3.480523  \n",
       "516319                  3           3.330519  \n",
       "674849                  4           4.380075  \n",
       "184396                  3           3.434905  \n",
       "737397                  2           2.071134  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = validset[[\"gender\", \"userid\",\"movieid\",\"age_desc\",\"occ_desc\", \"title\", \"genre\", \"rating\"]].copy()\n",
    "results[\"prediction (rnd.)\"] = np.asarray(np.round(final_valid_predictions), dtype=np.int16)\n",
    "results[\"prediction (prc.)\"] = final_valid_predictions\n",
    "results.head(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Measures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### Precision, Recall, MAE, RMSE measures:\n",
    "\n",
    "def compute_recall(prediction_col, target_col):\n",
    "    recall=[]\n",
    "    for i in range(5):\n",
    "        rating_df = results[results[target_col]==i+1]\n",
    "        num_true_rating = len(rating_df)+0.0\n",
    "        current_recall = (len(rating_df[rating_df[prediction_col]==i+1]))/num_true_rating\n",
    "        recall.append(current_recall)\n",
    "    return recall\n",
    "\n",
    "def compute_precision(prediction_col, target_col): \n",
    "    precision=[]\n",
    "    for i in range(5):\n",
    "        pred_df = results[results[prediction_col]==i+1]\n",
    "        pred_rating = len(pred_df)+0.0\n",
    "        current_precision = (len(pred_df[pred_df[target_col]==i+1]))/pred_rating\n",
    "        precision.append(current_precision)\n",
    "    return precision    \n",
    "\n",
    "def compute_mae(prediction_col, target_col):\n",
    "    return np.mean(np.abs(results[prediction_col]-results[target_col]))\n",
    "\n",
    "def compute_rmse(prediction_col, target_col):\n",
    "    return np.sqrt(1/len(results)*np.sum((results[prediction_col]- results[target_col])**2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.12242835057676299,\n",
       " 0.2602769297190613,\n",
       " 0.502085911485909,\n",
       " 0.6385712646755134,\n",
       " 0.2782322852636389]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_recall('prediction (rnd.)', 'rating')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.6914036265950302,\n",
       " 0.3469681397738952,\n",
       " 0.4058722824965967,\n",
       " 0.45429626657053657,\n",
       " 0.6664430478073582]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_precision('prediction (rnd.)', 'rating')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6391991015220138"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_mae('prediction (rnd.)', 'rating')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91609156911837586"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_rmse('prediction (rnd.)', 'rating')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}