3-2_dense(dpr)_bert(fine-tuned).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dense retrieval + bert(rerank)\n",
    "\n",
    "작성일자: 2101\\\n",
    "작성자: 조진욱\\\n",
    "목표: dpr 같은 모델을 만들자\n",
    "순서: \n",
    "3-1 과 동일\n",
    "\n",
    "비고:\n",
    "1. load 함수에서 json 형태로 불러오도록 되어있어 필요없지만 retrieval 의 결과를 json 으로 저장하는 과정을 거침. \n",
    "2. reader(mrc) 모델은 encoder 모델에서 topk 를 뽑은 뒤 합쳐서 rerank 만 도입한 것. 구현하기 쉬우나 학습하는데 시간이 걸려서 일단 사전 훈련된 모델을 가져옴 \n",
    "3. 그러나 1-2 best metric 보다 성능이 안나와야 정상. 왜냐하면 retrieval 과정에서 잘못된 context 들이 선택되었을 것이기 때문."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step1 retrieval 모델 불러와서 각 query 에 대한 document 찾기 \n",
    "현재 dense retrieval은 기본값으로 하나의 query 당 5개의 doc을 불러오도록 해뒀으나, 우리는 mrc 형식으로 best 1개 document 만을 선택하도록 할 것이므로 n_doc 값을 1로 조정해야함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from retrieval import DenseRetrieval\n",
    "from utils import save_json\n",
    "import config as cfg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "mode = 'dev'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "searching indexed dataset\n",
      "passages and index is exist\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3569f4a15a744692b66539be6e97218a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=4639.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "model_name = 'facebook/dpr-question_encoder-single-nq-base'\n",
    "qas_file =  f'{mode}_qa.json'\n",
    "passages_path = os.path.join(cfg.dense_dir, \"dpr_dataset\")\n",
    "index_path = os.path.join(cfg.dense_dir, \"dpr_dataset_hnsw_index.faiss\")\n",
    "ret_ds = DenseRetrieval.from_pretrained(model_name,\n",
    "                                        passages_path=passages_path,\n",
    "                                        index_path=index_path,\n",
    "                                       mode=mode)\n",
    "cqa_df = ret_ds.retrieve(qas_file)\n",
    "res_path = os.path.join(cfg.dense_dir, f\"retrieved_{mode}_dense.csv\")\n",
    "\n",
    "cqa_df.to_csv(res_path, sep='\\t', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>que_id</th>\n",
       "      <th>answers</th>\n",
       "      <th>context</th>\n",
       "      <th>context_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Who is the creator of American Idol?</td>\n",
       "      <td>56d21eb1e7d4791d00902667</td>\n",
       "      <td>{'answer_start': 67, 'text': 'Simon Fuller'}</td>\n",
       "      <td>American Idol was based on the British show Po...</td>\n",
       "      <td>[6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What company produces American idol?</td>\n",
       "      <td>56d21eb1e7d4791d00902668</td>\n",
       "      <td>{'answer_start': 96, 'text': '19 Entertainment'}</td>\n",
       "      <td>19 Recordings, a recording label owned by 19 E...</td>\n",
       "      <td>[172]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What year did American Idol begin airing?</td>\n",
       "      <td>56d21eb1e7d4791d00902669</td>\n",
       "      <td>{'answer_start': 201, 'text': '2002'}</td>\n",
       "      <td>American Idol is an American singing competiti...</td>\n",
       "      <td>[1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What British show is American Idols format bas...</td>\n",
       "      <td>56d21eb1e7d4791d0090266a</td>\n",
       "      <td>{'answer_start': 270, 'text': 'Pop Idol'}</td>\n",
       "      <td>American Idol was based on the British show Po...</td>\n",
       "      <td>[6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What television network does American Idol air...</td>\n",
       "      <td>56d21eb1e7d4791d0090266b</td>\n",
       "      <td>{'answer_start': 185, 'text': 'Fox'}</td>\n",
       "      <td>American Idol is broadcast to over 100 nations...</td>\n",
       "      <td>[176]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0               Who is the creator of American Idol?   \n",
       "1               What company produces American idol?   \n",
       "2          What year did American Idol begin airing?   \n",
       "3  What British show is American Idols format bas...   \n",
       "4  What television network does American Idol air...   \n",
       "\n",
       "                     que_id                                           answers  \\\n",
       "0  56d21eb1e7d4791d00902667      {'answer_start': 67, 'text': 'Simon Fuller'}   \n",
       "1  56d21eb1e7d4791d00902668  {'answer_start': 96, 'text': '19 Entertainment'}   \n",
       "2  56d21eb1e7d4791d00902669             {'answer_start': 201, 'text': '2002'}   \n",
       "3  56d21eb1e7d4791d0090266a         {'answer_start': 270, 'text': 'Pop Idol'}   \n",
       "4  56d21eb1e7d4791d0090266b              {'answer_start': 185, 'text': 'Fox'}   \n",
       "\n",
       "                                             context context_id  \n",
       "0  American Idol was based on the British show Po...        [6]  \n",
       "1  19 Recordings, a recording label owned by 19 E...      [172]  \n",
       "2  American Idol is an American singing competiti...        [1]  \n",
       "3  American Idol was based on the British show Po...        [6]  \n",
       "4  American Idol is broadcast to over 100 nations...      [176]  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "cqa_df = pd.read_csv(res_path, sep='\\t', index_col=False)\n",
    "cqa_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "filename: ./data/dense/retrieved_dev_dense.json\n"
     ]
    }
   ],
   "source": [
    "result_filename = f\"retrieved_{mode}_dense.json\"\n",
    "save_json(cqa_df, result_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step2 bert 모델이 만들어진 json 파일을 다시 불러와 MRC 진행\n",
    "\n",
    "torch Dataset의 형식으로 변환, mrc 형식으로 각 인스턴스에 대한 start end position 을 구한뒤 그에 대한 평가 진행 EM, F1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DPRReader, DPRReaderTokenizer\n",
    "tokenizer = DPRReaderTokenizer.from_pretrained('facebook/dpr-reader-single-nq-base')\n",
    "model = DPRReader.from_pretrained('facebook/dpr-reader-single-nq-base')\n",
    "model = model.to(cfg.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n",
    "from tqdm import tqdm, trange\n",
    "from utils import load_and_cache_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from transformers.data.metrics.squad_metrics import (\n",
    "    \n",
    "from squad_metrics import(\n",
    "    compute_predictions_logits,\n",
    "    squad_evaluate,\n",
    ")\n",
    "\n",
    "\n",
    "\n",
    "from transformers.data.processors.squad import SquadResult, SquadProcessor, squad_convert_examples_to_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/1 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating features from dataset file at %s ./data/squad\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00,  1.14it/s]\n",
      "convert squad examples to features: 100%|██████████| 4639/4639 [00:07<00:00, 661.31it/s]\n",
      "add example index and unique id: 100%|██████████| 4639/4639 [00:00<00:00, 1314598.76it/s]\n"
     ]
    }
   ],
   "source": [
    "dataset, examples, features = load_and_cache_examples(cfg, tokenizer, mode_or_filename=result_filename, output_examples=True)\n",
    "eval_sampler = SequentialSampler(dataset)\n",
    "eval_dataloader = DataLoader(dataset, sampler=eval_sampler, batch_size=cfg.eval_batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, tokenizer):\n",
    "    print(\"***** Running evaluation *****\")\n",
    "    print(\"  Num examples = \", len(dataset))\n",
    "    print(\"  Batch size = \", cfg.eval_batch_size)\n",
    "    all_results = []\n",
    "    for batch in tqdm(eval_dataloader, desc=\"Evaluating\"):\n",
    "        model.eval()\n",
    "        batch = tuple(t.to(cfg.device) for t in batch)\n",
    "\n",
    "        with torch.no_grad():\n",
    "            inputs = {\n",
    "                \"input_ids\": batch[0],\n",
    "                \"attention_mask\": batch[1],\n",
    "#                 \"token_type_ids\": batch[2],\n",
    "            }\n",
    "\n",
    "            feature_indices = batch[3]\n",
    "            outputs = model(**inputs)\n",
    "\n",
    "        for i, feature_index in enumerate(feature_indices):\n",
    "            eval_feature = features[feature_index.item()]\n",
    "            unique_id = int(eval_feature.unique_id)\n",
    "\n",
    "            start_logits = outputs.start_logits[i]\n",
    "            end_logits = outputs.end_logits[i]\n",
    "            result = SquadResult(unique_id, start_logits, end_logits)\n",
    "\n",
    "            all_results.append(result)\n",
    "            \n",
    "    predictions = compute_predictions_logits(examples,\n",
    "                                            features,\n",
    "                                            all_results,\n",
    "                                            cfg.n_best_size,\n",
    "                                            cfg.max_answer_length,\n",
    "                                            True,\n",
    "                                            None,\n",
    "                                            None,\n",
    "                                            None,\n",
    "                                            cfg.verbose_logging,\n",
    "                                            False,\n",
    "                                            cfg.null_score_diff_threshold,\n",
    "                                            tokenizer,)\n",
    "    results = squad_evaluate(examples, predictions)\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluating:   0%|          | 0/580 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** Running evaluation *****\n",
      "  Num examples =  4639\n",
      "  Batch size =  8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluating: 100%|██████████| 580/580 [00:20<00:00, 28.73it/s]\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "compute_predictions_logits() takes 11 positional arguments but 13 were given",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-48bbb59c5c2b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtokenizer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-23-6e054686a2ca>\u001b[0m in \u001b[0;36mevaluate\u001b[0;34m(model, tokenizer)\u001b[0m\n\u001b[1;32m     28\u001b[0m             \u001b[0mall_results\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 30\u001b[0;31m     predictions = compute_predictions_logits(examples,\n\u001b[0m\u001b[1;32m     31\u001b[0m                                             \u001b[0mfeatures\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m                                             \u001b[0mall_results\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: compute_predictions_logits() takes 11 positional arguments but 13 were given"
     ]
    }
   ],
   "source": [
    "results = evaluate(model, tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key, value in results.items():\n",
    "    print(\"eval_{}: {}\".format(key, value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}