examples/00_quick_start/dkn_MIND.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DKN : Deep Knowledge-Aware Network for News Recommendation\n",
    "\n",
    "DKN \\[1\\] is a deep learning model which incorporates information from knowledge graph for better news recommendation. Specifically, DKN uses TransX \\[2\\] method for knowledge graph representation learning, then applies a CNN framework, named KCNN, to combine entity embedding with word embedding and generate a final embedding vector for a news article. CTR prediction is made via an attention-based neural scorer. \n",
    "\n",
    "## Properties of DKN:\n",
    "\n",
    "- DKN is a content-based deep model for CTR prediction rather than traditional ID-based collaborative filtering. \n",
    "- It makes use of knowledge entities and common sense in news content via joint learning from semantic-level and knowledge-level representations of news articles.\n",
    "- DKN uses an attention module to dynamically calculate a user's aggregated historical representaition.\n",
    "\n",
    "\n",
    "## Data format:\n",
    "\n",
    "### DKN takes several files as input as follows:\n",
    "\n",
    "- **training / validation / test files**: each line in these files represents one instance. Impressionid is used to evaluate performance within an impression session, so it is only used when evaluating, you can set it to 0 for training data. The format is : <br> \n",
    "`[label] [userid] [CandidateNews]%[impressionid] `<br> \n",
    "e.g., `1 train_U1 N1%0` <br> \n",
    "\n",
    "- **user history file**: each line in this file represents a users' click history. You need to set `history_size` parameter in the config file, which is the max number of user's click history we use. We will automatically keep the last `history_size` number of user click history, if user's click history is more than `history_size`, and we will automatically pad with 0 if user's click history is less than `history_size`. the format is : <br> \n",
    "`[Userid] [newsid1,newsid2...]`<br>\n",
    "e.g., `train_U1 N1,N2` <br> \n",
    "\n",
    "- **document feature file**: It contains the word and entity features for news articles. News articles are represented by aligned title words and title entities. To take a quick example, a news title may be: <i>\"Trump to deliver State of the Union address next week\"</i>, then the title words value may be `CandidateNews:34,45,334,23,12,987,3456,111,456,432` and the title entitie value may be: `entity:45,0,0,0,0,0,0,0,0,0`. Only the first value of entity vector is non-zero due to the word \"Trump\". The title value and entity value is hashed from 1 to `n` (where `n` is the number of distinct words or entities). Each feature length should be fixed at k (`doc_size` parameter), if the number of words in document is more than k, you should truncate the document to k words, and if the number of words in document is less than k, you should pad 0 to the end. \n",
    "the format is like: <br> \n",
    "`[Newsid] [w1,w2,w3...wk] [e1,e2,e3...ek]`\n",
    "\n",
    "- **word embedding/entity embedding/ context embedding files**: These are `*.npy` files of pretrained embeddings. After loading, each file is a `[n+1,k]` two-dimensional matrix, n is the number of words(or entities) of their hash dictionary, k is dimension of the embedding, note that we keep embedding 0 for zero padding. \n",
    "\n",
    "In this experiment, we used GloVe\\[4\\] vectors to initialize the word embedding. We trained entity embedding using TransE\\[2\\] on knowledge graph and context embedding is the average of the entity's neighbors in the knowledge graph.<br>\n",
    "\n",
    "## MIND dataset\n",
    "\n",
    "MIND dataset\\[3\\] is a large-scale English news dataset. It was collected from anonymized behavior logs of Microsoft News website. MIND contains 1,000,000 users, 161,013 news articles and 15,777,377 impression logs. Every news article contains rich textual content including title, abstract, body, category and entities. Each impression log contains the click events, non-clicked events and historical news click behaviors of this user before this impression.\n",
    "\n",
    "In this notebook we are going to use a subset of MIND dataset, **MIND demo**. MIND demo contains 500 users, 9,432 news articles  and 6,134 impression logs. \n",
    "\n",
    "For this quick start notebook, we are providing directly all the necessary word embeddings, entity embeddings and context embedding files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global settings and imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42) \n",
      "[GCC 7.5.0]\n",
      "Tensorflow version: 1.15.2\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "from tempfile import TemporaryDirectory\n",
    "import scrapbook as sb\n",
    "import tensorflow as tf\n",
    "tf.get_logger().setLevel('ERROR') # only show error messages\n",
    "\n",
    "from reco_utils.recommender.deeprec.deeprec_utils import download_deeprec_resources, prepare_hparams\n",
    "from reco_utils.recommender.deeprec.models.dkn import DKN\n",
    "from reco_utils.recommender.deeprec.io.dkn_iterator import DKNTextIterator\n",
    "\n",
    "print(f\"System version: {sys.version}\")\n",
    "print(f\"Tensorflow version: {tf.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 11.3k/11.3k [00:08<00:00, 1.40kKB/s]\n"
     ]
    }
   ],
   "source": [
    "tmpdir = TemporaryDirectory()\n",
    "data_path = os.path.join(tmpdir.name, \"mind-demo-dkn\")\n",
    "\n",
    "yaml_file = os.path.join(data_path, r'dkn.yaml')\n",
    "train_file = os.path.join(data_path, r'train_mind_demo.txt')\n",
    "valid_file = os.path.join(data_path, r'valid_mind_demo.txt')\n",
    "test_file = os.path.join(data_path, r'test_mind_demo.txt')\n",
    "news_feature_file = os.path.join(data_path, r'doc_feature.txt')\n",
    "user_history_file = os.path.join(data_path, r'user_history.txt')\n",
    "wordEmb_file = os.path.join(data_path, r'word_embeddings_100.npy')\n",
    "entityEmb_file = os.path.join(data_path, r'TransE_entity2vec_100.npy')\n",
    "contextEmb_file = os.path.join(data_path, r'TransE_context2vec_100.npy')\n",
    "if not os.path.exists(yaml_file):\n",
    "    download_deeprec_resources(r'https://recodatasets.z20.web.core.windows.net/deeprec/', tmpdir.name, 'mind-demo-dkn.zip')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "is_executing": false
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "epochs = 10\n",
    "history_size = 50\n",
    "batch_size = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kg_file=None,user_clicks=None,FEATURE_COUNT=None,FIELD_COUNT=None,data_format=dkn,PAIR_NUM=None,DNN_FIELD_NUM=None,n_user=None,n_item=None,n_user_attr=None,n_item_attr=None,iterator_type=None,SUMMARIES_DIR=None,MODEL_DIR=None,wordEmb_file=/tmp/tmpm_qyjfgp/mind-demo-dkn/word_embeddings_100.npy,entityEmb_file=/tmp/tmpm_qyjfgp/mind-demo-dkn/TransE_entity2vec_100.npy,contextEmb_file=/tmp/tmpm_qyjfgp/mind-demo-dkn/TransE_context2vec_100.npy,news_feature_file=/tmp/tmpm_qyjfgp/mind-demo-dkn/doc_feature.txt,user_history_file=/tmp/tmpm_qyjfgp/mind-demo-dkn/user_history.txt,use_entity=True,use_context=True,doc_size=10,history_size=50,word_size=12600,entity_size=3987,entity_dim=100,entity_embedding_method=None,transform=True,train_ratio=None,dim=100,layer_sizes=[300],cross_layer_sizes=None,cross_layers=None,activation=['sigmoid'],cross_activation=identity,user_dropout=False,dropout=[0.0],attention_layer_sizes=100,attention_activation=relu,attention_dropout=0.0,model_type=dkn,method=classification,load_saved_model=False,load_model_name=None,filter_sizes=[1, 2, 3],num_filters=100,mu=None,fast_CIN_d=0,use_Linear_part=False,use_FM_part=False,use_CIN_part=False,use_DNN_part=False,init_method=uniform,init_value=0.1,embed_l2=1e-06,embed_l1=0.0,layer_l2=1e-06,layer_l1=0.0,cross_l2=0.0,cross_l1=0.0,reg_kg=0.0,learning_rate=0.0005,lr_rs=1,lr_kg=0.5,kg_training_interval=5,max_grad_norm=2,is_clip_norm=0,dtype=32,loss=log_loss,optimizer=adam,epochs=10,batch_size=100,enable_BN=True,show_step=10000,save_model=False,save_epoch=2,metrics=['auc'],write_tfevents=False,item_embedding_dim=None,cate_embedding_dim=None,user_embedding_dim=None,train_num_ngs=4,need_sample=True,embedding_dropout=0.0,user_vocab=None,item_vocab=None,cate_vocab=None,pairwise_metrics=['group_auc', 'mean_mrr', 'ndcg@5;10'],EARLY_STOP=100,max_seq_length=None,hidden_size=None,L=None,T=None,n_v=None,n_h=None,min_seq_length=1,attention_size=None,att_fcn_layer_sizes=None,dilations=None,kernel_size=None,embed_size=None,n_layers=None,decay=None,eval_epoch=None,top_k=None\n"
     ]
    }
   ],
   "source": [
    "hparams = prepare_hparams(yaml_file,\n",
    "                          news_feature_file = news_feature_file,\n",
    "                          user_history_file = user_history_file,\n",
    "                          wordEmb_file=wordEmb_file,\n",
    "                          entityEmb_file=entityEmb_file,\n",
    "                          contextEmb_file=contextEmb_file,\n",
    "                          epochs=epochs,\n",
    "                          history_size=history_size,\n",
    "                          batch_size=batch_size)\n",
    "print(hparams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the DKN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "model = DKN(hparams, DKNTextIterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.5437, 'group_auc': 0.5163, 'mean_mrr': 0.157, 'ndcg@5': 0.1527, 'ndcg@10': 0.2191}\n"
     ]
    }
   ],
   "source": [
    "print(model.run_eval(valid_file))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "is_executing": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "at epoch 1\n",
      "train info: logloss loss:0.6685948745679047\n",
      "eval info: auc:0.5426, group_auc:0.5296, mean_mrr:0.181, ndcg@10:0.2502, ndcg@5:0.1856\n",
      "at epoch 1 , train time: 11.8 eval time: 4.8\n",
      "at epoch 2\n",
      "train info: logloss loss:0.6204990633463455\n",
      "eval info: auc:0.5473, group_auc:0.5139, mean_mrr:0.1706, ndcg@10:0.2333, ndcg@5:0.1829\n",
      "at epoch 2 , train time: 11.1 eval time: 4.7\n",
      "at epoch 3\n",
      "train info: logloss loss:0.5874787417508788\n",
      "eval info: auc:0.5389, group_auc:0.493, mean_mrr:0.1639, ndcg@10:0.2214, ndcg@5:0.1691\n",
      "at epoch 3 , train time: 11.0 eval time: 4.7\n",
      "at epoch 4\n",
      "train info: logloss loss:0.5630691410121271\n",
      "eval info: auc:0.5673, group_auc:0.5183, mean_mrr:0.176, ndcg@10:0.2357, ndcg@5:0.1854\n",
      "at epoch 4 , train time: 11.0 eval time: 4.7\n",
      "at epoch 5\n",
      "train info: logloss loss:0.5432774212400792\n",
      "eval info: auc:0.5959, group_auc:0.564, mean_mrr:0.1926, ndcg@10:0.2562, ndcg@5:0.1977\n",
      "at epoch 5 , train time: 11.0 eval time: 4.8\n",
      "at epoch 6\n",
      "train info: logloss loss:0.5274657431547924\n",
      "eval info: auc:0.6, group_auc:0.5659, mean_mrr:0.1895, ndcg@10:0.2528, ndcg@5:0.1917\n",
      "at epoch 6 , train time: 11.0 eval time: 4.7\n",
      "at epoch 7\n",
      "train info: logloss loss:0.5117715953265206\n",
      "eval info: auc:0.6041, group_auc:0.5679, mean_mrr:0.1757, ndcg@10:0.2447, ndcg@5:0.1774\n",
      "at epoch 7 , train time: 11.0 eval time: 4.7\n",
      "at epoch 8\n",
      "train info: logloss loss:0.49649867617477805\n",
      "eval info: auc:0.6061, group_auc:0.5749, mean_mrr:0.1834, ndcg@10:0.25, ndcg@5:0.1831\n",
      "at epoch 8 , train time: 10.9 eval time: 4.7\n",
      "at epoch 9\n",
      "train info: logloss loss:0.4817596108226453\n",
      "eval info: auc:0.5967, group_auc:0.5703, mean_mrr:0.1759, ndcg@10:0.2417, ndcg@5:0.1769\n",
      "at epoch 9 , train time: 11.0 eval time: 4.7\n",
      "at epoch 10\n",
      "train info: logloss loss:0.46655569288690213\n",
      "eval info: auc:0.6043, group_auc:0.5742, mean_mrr:0.1796, ndcg@10:0.2474, ndcg@5:0.1822\n",
      "at epoch 10 , train time: 11.1 eval time: 4.8\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<reco_utils.recommender.deeprec.models.dkn.DKN at 0x7f6d41a62ba8>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_file, valid_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the DKN model\n",
    "\n",
    "Now we can check the performance on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.597, 'group_auc': 0.5887, 'mean_mrr': 0.1883, 'ndcg@5': 0.1904, 'ndcg@10': 0.2608}\n"
     ]
    }
   ],
   "source": [
    "res = model.run_eval(test_file)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sb.glue(\"res\", res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "\\[1\\] Wang, Hongwei, et al. \"DKN: Deep Knowledge-Aware Network for News Recommendation.\" Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2018.<br>\n",
    "\\[2\\] Knowledge Graph Embeddings including TransE, TransH, TransR and PTransE. https://github.com/thunlp/KB2E <br>\n",
    "\\[3\\] Wu, Fangzhao, et al. \"MIND: A Large-scale Dataset for News Recommendation\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://msnews.github.io/competition.html <br>\n",
    "\\[4\\] GloVe: Global Vectors for Word Representation. https://nlp.stanford.edu/projects/glove/"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python (reco_gpu)",
   "language": "python",
   "name": "reco_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}