notebooks/measuring-quality-item-based-CF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measuring the quality of recommender systems\n",
    "\n",
    "In this notebook, I explore how the quality of recommender systems can be measured.\n",
    "\n",
    "Recommender systems rely on predicting how users rate items. \n",
    "\n",
    "Based on the rating predictions, the recommender system recommends the items to a user with the highest predicted ratings.\n",
    "\n",
    "So, measuring the quality of a recommender system can be translated into measuring the predicted rating of an item against the real rating that a user gave. If the recommender predicts a 5-star rating when the true rating is 1-star, then that signals the quality of the predictions made by the recommender system is not that great.\n",
    "\n",
    "Some other principles of the general methodology to measure the quality:\n",
    "- ratings could be predicted for a set of (user, item) tuples. Aggregate metrics like __root mean square error__ (RMSE) are used to report the overall quality of the predictions.\n",
    "- rating predictions should be applied to an _unseen_ set of (user, item) observations. Otherwise, the prediction is trivial; the prediction could simply repeat the observation that the recommender had already seen. Hence, the dataset could be split into train / test sets, with the __train__ set shown to the recommender system, and the __test__ set used to report the quality of the recommender system.\n",
    "- Splitting the dataset into train / test could introduce unwanted __bias__. As a result, the metrics could not reflect the real quality of the recommender system. For example, if the train set only contains ratings of 1 and 2 stars, the recommender system may never be able to predict 5-star ratings. The dataset contains many more 4-star ratings than 1-star ratings. If we randomly sampled without taking this into account, we may end up under or overrepresenting certain x-star ratings. Hence, the train-test datasets are created by preserving the distribution of the ratings in both datasets. In other words, a __stratified__ split is applied.\n",
    "\n",
    "Some principles specific to measuring the quality of recommender systems based on __user-based collaborative filtering__:\n",
    "- __the test set cannot contain users that were not part of the train set__. This is because the algorithm searches the ratings dataset for the most similar users to the current user. If the current user is not part of the ratings dataset, finding the most similar users becomes impossible.\n",
    "- __the test set cannot contain items that were not part of the train set__. If the train set does not contain ratings of the item of other users, it becomes impossible to rate that specific item. This is because the predicted rating relies on the ratings of that specific item of other users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# data manipulation\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from math import sqrt\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "RANDOM_SEED = 42\n",
    "np.random.seed(RANDOM_SEED)\n",
    "\n",
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, os.path.abspath('../src'))\n",
    "\n",
    "from item_based_collaborative_filtering import compute_similarities\n",
    "from item_based_collaborative_filtering import predict\n",
    "from utils import read_ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# paths to the files. download them from http://files.grouplens.org/datasets/movielens/ml-100k.zip if you don't have them yet\n",
    "MOVIE_RATINGS_PATH = '../ml-100k/u.data'\n",
    "MOVIE_NAMES_PATH = '../ml-100k/u.item'\n",
    "\n",
    "# test sample size\n",
    "TEST_SIZE = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>movie</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  movie  rating\n",
       "0   196    242       3\n",
       "1   186    302       3\n",
       "2    22    377       1\n",
       "3   244     51       2\n",
       "4   166    346       1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings = read_ratings(MOVIE_RATINGS_PATH)\n",
    "ratings = pd.DataFrame(data=ratings, columns=['user', 'movie', 'rating'])\n",
    "ratings = ratings.astype(int)\n",
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train-test-split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# stratified to keep the distribution of movie ratings the same in train and test datasets.\n",
    "train, test = train_test_split(\n",
    "    ratings, \n",
    "    test_size=TEST_SIZE, \n",
    "    stratify=ratings.rating, \n",
    "    random_state=RANDOM_SEED\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>movie</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>74736</th>\n",
       "      <td>294</td>\n",
       "      <td>881</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63657</th>\n",
       "      <td>263</td>\n",
       "      <td>260</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85869</th>\n",
       "      <td>872</td>\n",
       "      <td>845</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98751</th>\n",
       "      <td>660</td>\n",
       "      <td>946</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20694</th>\n",
       "      <td>346</td>\n",
       "      <td>394</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  movie  rating\n",
       "74736   294    881       3\n",
       "63657   263    260       2\n",
       "85869   872    845       3\n",
       "98751   660    946       2\n",
       "20694   346    394       4"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    0.336693\n",
       "5    0.282031\n",
       "2    0.151261\n",
       "3    0.148735\n",
       "1    0.081281\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.rating.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    0.498\n",
       "3    0.497\n",
       "5    0.003\n",
       "2    0.001\n",
       "1    0.001\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.rating.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 3)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(999, 3)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove items from test set that are not part of train set\n",
    "test = test.loc[test.movie.isin(train.movie)]\n",
    "test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>movie</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>39841</th>\n",
       "      <td>608</td>\n",
       "      <td>673</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7731</th>\n",
       "      <td>269</td>\n",
       "      <td>293</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9953</th>\n",
       "      <td>231</td>\n",
       "      <td>252</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27256</th>\n",
       "      <td>417</td>\n",
       "      <td>255</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80742</th>\n",
       "      <td>805</td>\n",
       "      <td>708</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  movie  rating\n",
       "39841   608    673       4\n",
       "7731    269    293       3\n",
       "9953    231    252       4\n",
       "27256   417    255       3\n",
       "80742   805    708       3"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test = test.sample(100)\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computing item-item similarities on the train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "similarities = compute_similarities(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting the rating for a sample (user, item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = ratings.sample(random_state=RANDOM_SEED)\n",
    "user_id = sample.user.values[0]\n",
    "item_id = sample.movie.values[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_id, item_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict(381, 12, train, similarities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rating predictions on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "test['pred'] = test.apply(lambda row: predict(row.user, row.movie, train, similarities), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove items that could not be rated\n",
    "test = test.loc[test.pred != -1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quality Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rms = sqrt(mean_squared_error(test.rating, test.pred))\n",
    "rms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stars = train.rating.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make random guesses of ratings, and compute RMSE as a basis for comparison\n",
    "test['random'] = np.random.choice(stars.index, p=stars.values, size=test.shape[0])\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_rms = sqrt(mean_squared_error(test.rating, test.random))\n",
    "random_rms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}