03-question-answering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14410483",
   "metadata": {},
   "source": [
    "# Hugging Face - Question Answering in Japanese\n",
    "\n",
    "This source code builds the fine-tuned model to perform extractive question answering (extractive QA) for Japanese language.<br>\n",
    "For more background and details, see [here](https://tsmatz.wordpress.com/2022/12/12/huggingface-japanese-question-answering/).\n",
    "\n",
    "For the reason of learning performance, I have used [rinna/japanese-roberta-base](https://huggingface.co/rinna/japanese-roberta-base) for pre-trained transformer, which is well-trained and optimized for Japanese corpus.<br>\n",
    "You can also use [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) for other languages.\n",
    "\n",
    "*back to [index](https://github.com/tsmatz/huggingface-finetune-japanese/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bdb4369",
   "metadata": {},
   "source": [
    "## Install required packages\n",
    "\n",
    "In order to install core components, see [Readme](https://github.com/tsmatz/huggingface-finetune-japanese/).<br>\n",
    "Install additional packages for running this notebook as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5295c488",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c04ef40e",
   "metadata": {},
   "source": [
    "## Check device\n",
    "\n",
    "Check whether GPU is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f5bc098b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU is enabled.\n",
      "device count: 1, current device: 0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    print(\"GPU is enabled.\")\n",
    "    print(\"device count: {}, current device: {}\".format(torch.cuda.device_count(), torch.cuda.current_device()))\n",
    "else:\n",
    "    print(\"GPU is not enabled.\")\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5163ab77",
   "metadata": {},
   "source": [
    "## Prepare data\n",
    "\n",
    "In this example, we use [JaQuAD](https://huggingface.co/datasets/SkelterLabsInc/JaQuAD) (Japanese Question Answering Dataset) in Hugging Face, which is annotated extractive question answering dataset like famous SQuAD.<br>\n",
    "This dataset has over 30000 samples for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "20a1866b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!\n",
      "  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a417191473248f098ac93e48c1de36b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script:   0%|          | 0.00/4.48k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0166f24620f40f187c98ac44542a068",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/6.74k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset ja_qu_ad/default to /home/tsmatsuz/.cache/huggingface/datasets/SkelterLabsInc___ja_qu_ad/default/0.1.0/5847b2e2ab5e02de284395bb15f87f13eae8f6f6ff1f01e4ee9c5c0dcf8ef8eb...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6faca19bc3174308955728cc16b64bea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c694c3f9040e4a758634df9b600b052c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/790k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "652c32f2025d44db8f5843094a262949",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/815k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d634dc46061241f7be2da5ec5d813a13",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/844k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d5fe0a8f9e84e8a8a4c25198d888b79",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/658k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f9d3bb46eaf4f5b950d40a497f5458b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/791k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a8623cfa4b01497a91c7c5802c8c0652",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/776k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "236145e0a8204462a05ebad8672a6cf2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/718k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73b4e65b4a7e440eb58932e304ec70fa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/800k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53efe04a20a8416c822f2ca918c4aebc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/752k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8b69b04381324a519e10e77281d4fe52",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/797k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e32f5564910645ce858a30582bd08d5a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/797k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea459600d1aa47f0a42207bcaa178d9a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/785k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6e2962138494b9e87d39a33af646a4b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/580k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "40abfc8e9a014656bcd6e96c351e6fc2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/743k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2e28524e39244e15912552fe87a000a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/728k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95177790e70b4f1fa56fd66df6c5d0a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/757k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c2e9e59bd6cb4b39a51e76892a580806",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/763k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba098f609a404b79a17b489c71b6a35d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/772k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ceedb00406a9447bb8d1b146fc1f0f1f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/807k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cf7735bfe6b245a88e44c6f814e8b1cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/796k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afce764673054ce6be52ced9018cd67b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/775k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c28c71ad770a4af8abec948d652acda9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/804k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dbf325afa9714c32b1f9eaa7dab5a762",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/855k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc4e1f74066244b7bf304e69361f510d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/745k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "776e6355c7064082964540eeab423be0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/771k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3fbb8173f15541ad87187290fe2e5a0c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/778k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1f8e8c50dc24ccd913c341c50e1a463",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/763k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c85d070a97544a06be247455009c5829",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/762k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10882dc84d9f4c37b80ea7f9b8f8d7ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/945k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8fb03d98cd84223acbf079476e4f2e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/246k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72eb4fc23e3c4f6a9b093c4078663684",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/855k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f45a75abc6d42daa0f5494c2dded6fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/801k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b8c20c678384bedb1ee5c8df11a0d60",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/746k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b974876cf1ed4a5585bbe655c2a3afe7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/621k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "de6ac85d064547d7b063de0e9773a5b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting data files:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating validation split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset ja_qu_ad downloaded and prepared to /home/tsmatsuz/.cache/huggingface/datasets/SkelterLabsInc___ja_qu_ad/default/0.1.0/5847b2e2ab5e02de284395bb15f87f13eae8f6f6ff1f01e4ee9c5c0dcf8ef8eb. Subsequent calls will reuse this data.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a64bbb0d811d4b78a25d29ad173f314b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['id', 'title', 'context', 'question', 'question_type', 'answers'],\n",
       "        num_rows: 31748\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['id', 'title', 'context', 'question', 'question_type', 'answers'],\n",
       "        num_rows: 3939\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "ds = load_dataset(\"SkelterLabsInc/JaQuAD\")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9834be7c",
   "metadata": {},
   "source": [
    "In extractive QA, both ```context``` and ```question``` are provided for inputs, and it then predicts the span (start position and end position) of answer in ```context``` text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c8da181d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'tr-000-00-000',\n",
       " 'title': '手塚治虫',\n",
       " 'context': '手塚治虫(てづかおさむ、本名:手塚治(読み同じ)、1928年(昭和3年)11月3日-1989年(平成元年)2月9日)は、日本の漫画家、アニメーター、アニメ監督である。\\n戦後日本においてストーリー漫画の第一人者として、漫画表現の開拓者的な存在として活躍した。\\n\\n兵庫県宝塚市出身(出生は大阪府豊能郡豊中町、現在の豊中市)同市名誉市民である。\\n大阪帝国大学附属医学専門部を卒業。\\n医師免許取得のち医学博士(奈良県立医科大学・1961年)。',\n",
       " 'question': '戦後日本のストーリー漫画の第一人者で、医学博士の一面もある漫画家は誰?',\n",
       " 'question_type': 'Multiple sentence reasoning',\n",
       " 'answers': {'text': ['手塚治虫'], 'answer_start': [0], 'answer_type': ['Person']}}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds[\"train\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "453c922d",
   "metadata": {},
   "source": [
    "To generate inputs for fine-tuning, now I tokenize each text and convert into token ids.\n",
    "\n",
    "First, load tokenizer in pre-trained ```rinna/japanese-roberta-base``` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c42045d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b34ce938d2a44f482eefa9008899da2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/259 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "04c5eaa2a6534c3f894f846213931efb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/806k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e88fef6b431491eb6f7b6e9f9146365",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/153 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"rinna/japanese-roberta-base\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd8231f7",
   "metadata": {},
   "source": [
    "Before tokenize dataset, let's see how input is formed.<br>\n",
    "As you can see below, the inputs take the format :\n",
    "\n",
    "```question text </s> context text </s> pad```\n",
    "\n",
    "In this task, we need the complete form for the question, but the context can be incomplete. (If the answer happens not to be included in the context, the answer will be empty.)<br>\n",
    "By setting ```max_length``` property and ```truncation=\"only_second\"``` as follows, the first sequence (i.e, question) won't be truncated, but the second sequence (i.e, context) is truncated by the maximum length of tokens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9a7b06fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "戦後日本のストーリー漫画の第一人者で、医学博士の一面もある漫画家は誰?</s>手塚治虫(てづかおさむ、本名:手塚治(読み同じ)、1928年(昭和3年)11月3日-1989年(平成元年)2月9日)は、日本の漫画家、アニメーター、アニメ監督である。戦後日本においてストーリー漫画の第一人者として、漫画表現の開拓者的な存在として活躍した。兵庫県宝塚市出身(出生は大阪府豊能郡豊中町、現在の豊中市)同市名誉市民である。大阪帝国大学附属医学専門部を卒業。医師免許取得のち医学博士(奈良県立医科大学・1961年)。</s>[PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD]\n"
     ]
    }
   ],
   "source": [
    "features = tokenizer(\n",
    "    ds[\"train\"][0][\"question\"],\n",
    "    ds[\"train\"][0][\"context\"],\n",
    "    max_length = 384,\n",
    "    truncation=\"only_second\",\n",
    "    padding = \"max_length\",\n",
    ")\n",
    "print(\"\".join(tokenizer.batch_decode(features[\"input_ids\"])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dcb1ae0",
   "metadata": {},
   "source": [
    "Now let's tokenize and convert dataset for fine-tuning.\n",
    "\n",
    "Question Answering model in Hugging Face expects answer's ```start_positions``` and ```end_positions``` which indicate the positions in the input's sequence. (These positions should be indices in ```input_ids```.)\n",
    "\n",
    "When the context is longer than the maximum sequence length (here, it's 384), it's simply truncated and the overflowing tokens will be returned as the next sequence, by setting ```return_overflowing_tokens=True```.<br>\n",
    "However, it might happen that the answer tokens are separated into multiple sequences. To prevent this occurence, the size of stride in sliding window can be controlled by ```stride``` property. For instance, the last n tokens are not fit and overflow, m + n tokens will be in the next sequence when ```stride=m```. These m tokens are then the overlapped tokens between windows.<br>\n",
    "The position of answer will then be either of first sequence or second sequence, or in both sequences.\n",
    "\n",
    "In this example, I have also removed the sequence, in which the answer doesn't exist.\n",
    "\n",
    "> Note : To get token index for each character, you can also use ```char_to_token()``` method in tokenizer, instead of using ```return_offsets_mapping``` property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "714d1453",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Modified the following source code for supporting the case of overflowing\n",
    "# https://huggingface.co/docs/transformers/tasks/question_answering\n",
    "\n",
    "def tokenize_sample_data(data):\n",
    "    # tokenize\n",
    "    tokenized_feature = tokenizer(\n",
    "        data[\"question\"],\n",
    "        data[\"context\"],\n",
    "        max_length = 384,\n",
    "        return_overflowing_tokens=True,\n",
    "        stride=128,\n",
    "        truncation=\"only_second\",\n",
    "        padding = \"max_length\",\n",
    "        return_offsets_mapping=True,\n",
    "    )\n",
    "\n",
    "    # When it overflows, multiple rows will be returned for a single example.\n",
    "    # The following then gets the array of corresponding the original sample index.\n",
    "    sample_mapping = tokenized_feature.pop(\"overflow_to_sample_mapping\")\n",
    "    # Get the array of [start_char, end_char + 1] in each token.\n",
    "    # The shape is [returned_row_size, max_length]\n",
    "    offset_mapping = tokenized_feature.pop(\"offset_mapping\")\n",
    "\n",
    "    start_positions = []\n",
    "    end_positions = []\n",
    "    for i, offset in enumerate(offset_mapping):\n",
    "        sample_index = sample_mapping[i]\n",
    "        answers = data[\"answers\"][sample_index]\n",
    "        start_char = answers[\"answer_start\"][0]\n",
    "        end_char = start_char + len(answers[\"text\"][0]) - 1\n",
    "        # The format of sequence_ids is [None, 0, ..., 0, None, None, 1, ..., 1, None, None, ...]\n",
    "        # in which question's token is 0 and contex's token is 1\n",
    "        sequence_ids = tokenized_feature.sequence_ids(i)\n",
    "        # find the start and end index of context\n",
    "        idx = 0\n",
    "        while sequence_ids[idx] != 1:\n",
    "            idx += 1\n",
    "        context_start = idx\n",
    "        while sequence_ids[idx] == 1:\n",
    "            idx += 1\n",
    "        context_end = idx - 1\n",
    "        # Set start positions and end positions in inputs_ids\n",
    "        # Note: The second element in offset is end_char + 1\n",
    "        #if offset[context_start][0] > end_char or offset[context_end][1] <= start_char:\n",
    "        if not (offset[context_start][0] <= start_char and end_char < offset[context_end][1]):\n",
    "            # The case that answer is not inside the context\n",
    "            ## Note : Some tokenizer (such as, tokenizer in rinna model) doesn't place CLS\n",
    "            ## for the first token in sequence, and I then set -1 as positions.\n",
    "            ## (Later I'll process rows with start_positions=-1.)\n",
    "            start_positions.append(-1)\n",
    "            end_positions.append(-1)\n",
    "            #start_positions.append(0)\n",
    "            #end_positions.append(0)\n",
    "        else:\n",
    "            # The case that answer is found in the context\n",
    "\n",
    "            # Set start position\n",
    "            idx = context_start\n",
    "            while offset[idx][0] < start_char:\n",
    "                idx += 1\n",
    "            if offset[idx][0] == start_char:\n",
    "                start_positions.append(idx)\n",
    "            else:\n",
    "                start_positions.append(idx - 1)\n",
    "\n",
    "            # Set end position\n",
    "            idx = context_end\n",
    "            while offset[idx][1] > end_char + 1:\n",
    "                idx -= 1\n",
    "            if offset[idx][1] == end_char + 1:\n",
    "                end_positions.append(idx)\n",
    "            else:\n",
    "                end_positions.append(idx + 1)\n",
    "\n",
    "    # build result\n",
    "    tokenized_feature[\"start_positions\"] = start_positions\n",
    "    tokenized_feature[\"end_positions\"] = end_positions   \n",
    "    return tokenized_feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "295ae32e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c027e963e0446f3b2a61be4495db411",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/249 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4c5eedfc0e0d4837a83abeab130efb85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "76dc5ddbcebc41d281b17bca3693b59c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/35 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b9f8fb2a5a24e9da8b7463c7391ce45",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********** input_ids **********\n",
      "[5242, 216, 1879, 1302, 11091, 63, 147, 19, 7, 27730, 10, 13134, 553, 6768, 11, 5943, 3017, 2, 9, 28889, 15, 58, 16154, 13726, 561, 7, 3875, 76, 25649, 687, 15, 3009, 851, 14, 15481, 16, 15, 112, 31, 16, 3824, 22, 31, 33, 61, 4485, 16, 15, 16980, 14, 25, 22, 52, 33, 14, 11, 7, 14122, 7, 21244, 7, 1047, 464, 27, 8, 5242, 15045, 1879, 1302, 11091, 63, 5905, 7, 1302, 1030, 10, 19365, 132, 1763, 12963, 8, 9, 3656, 6227, 3202, 15, 8967, 11, 2882, 1311, 1624, 139, 1311, 82, 87, 7, 373, 1311, 82, 69, 14, 11661, 2957, 1296, 27, 8, 8773, 1744, 22585, 2966, 1673, 126, 8236, 8, 9, 2800, 4459, 3113, 2089, 27730, 15, 7907, 356, 14689, 13, 9117, 16, 14, 8, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]\n",
      "********** start_positions **********\n",
      "18\n",
      "********** end_positions **********\n",
      "19\n"
     ]
    }
   ],
   "source": [
    "# Run conversion\n",
    "tokenized_ds = ds.map(\n",
    "    tokenize_sample_data,\n",
    "    remove_columns=[\"id\", \"title\", \"context\", \"question\", \"question_type\", \"answers\"],\n",
    "    batched=True,\n",
    "    batch_size=128)\n",
    "# Remove rows with start_positions=-1 (see above)\n",
    "tokenized_ds = tokenized_ds.filter(lambda x: x[\"start_positions\"] != -1)\n",
    "\n",
    "print(\"********** input_ids **********\")\n",
    "print(tokenized_ds[\"train\"][\"input_ids\"][0])\n",
    "print(\"********** start_positions **********\")\n",
    "print(tokenized_ds[\"train\"][\"start_positions\"][0])\n",
    "print(\"********** end_positions **********\")\n",
    "print(tokenized_ds[\"train\"][\"end_positions\"][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "692dd056",
   "metadata": {},
   "source": [
    "## Fine-tune"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bf379a1",
   "metadata": {},
   "source": [
    "In this example, we use ```AutoModelForQuestionAnswering``` class with pre-trained RobertaModel.\n",
    "\n",
    "Like [token classification example](./01-named-entity.ipynb), this model consists of pre-trained RobertaModel and classification head.<br>\n",
    "However, the final output (which shape is ```[batch_length, sequence_length, 2]```) is split into 2 parts, and each of them then has the shape ```[batch_length, sequence_length]```. These two tensors are then used as start logits and end logits, and the token classification loss between these logits and true labels (```start_positions``` and ```end_positions```, respectively) are then computed for optimization. (See [here](https://tsmatz.wordpress.com/2022/12/12/huggingface-japanese-question-answering/) for model architecture.)\n",
    "\n",
    "> Note : The following ```num_labels``` and ```hidden_size``` is the default values in ```AutoModelForQuestionAnswering```, and you can then skip these config settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "39cab1c7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "316963c93aa04aee822fec958d247f91",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/663 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9ad72e1c95f143a9a8cdde327a0fdaf1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/443M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at rinna/japanese-roberta-base were not used when initializing RobertaForQuestionAnswering: ['lm_head.layer_norm.weight', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.bias', 'lm_head.bias', 'lm_head.dense.bias']\n",
      "- This IS expected if you are initializing RobertaForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of RobertaForQuestionAnswering were not initialized from the model checkpoint at rinna/japanese-roberta-base and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoConfig, AutoModelForQuestionAnswering\n",
    "\n",
    "# see https://huggingface.co/docs/transformers/main_classes/configuration\n",
    "config = AutoConfig.from_pretrained(\n",
    "    \"rinna/japanese-roberta-base\",\n",
    "    num_labels=2,\n",
    "    hidden_size=768,\n",
    ")\n",
    "model = (AutoModelForQuestionAnswering\n",
    "         .from_pretrained(\"rinna/japanese-roberta-base\", config=config)\n",
    "         .to(device))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30cb5336",
   "metadata": {},
   "source": [
    "We prepare data collator, which works for preprocessing data.<br>\n",
    "Unlike other examples, here we use default data collator which doesn't do any extra works - such as, filling -100 in padded tokens -, because we don't need to skip loss or evaluation computation in padded tokens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0c64860f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DefaultDataCollator\n",
    "\n",
    "data_collator = DefaultDataCollator()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f4bfded",
   "metadata": {},
   "source": [
    "We prepare training arguments for fine-tuning.<br>\n",
    "In this example, we use HuggingFace transformer trainer class, with which you can run training without manually writing training loop.\n",
    "\n",
    "First we prepare trainer's arguments.<br>\n",
    "The checkpoint files (in each 50 steps) are saved in the folder named ```rinna-roberta-qa-ja```.\n",
    "\n",
    "> Note : In general, the saved checkpoints in the training will become so large.<br>\n",
    "> Set ```save_total_limit``` property (which limits the total amount of checkpoints by deleting the older ones) to save disk spaces, or expand disks in Azure VM. (See [here](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/expand-disks) to expand disks in Azure.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "78f28731",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "    output_dir = \"rinna-roberta-qa-ja\",\n",
    "    log_level = \"error\",\n",
    "    num_train_epochs = 3,\n",
    "    learning_rate = 7e-5,\n",
    "    lr_scheduler_type = \"linear\",\n",
    "    warmup_steps = 100,\n",
    "    per_device_train_batch_size = 2,\n",
    "    per_device_eval_batch_size = 1,\n",
    "    gradient_accumulation_steps = 16,\n",
    "    evaluation_strategy = \"steps\",\n",
    "    eval_steps = 150,\n",
    "    save_steps = 500,\n",
    "    logging_steps = 50,\n",
    "    push_to_hub = False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3978f071",
   "metadata": {},
   "source": [
    "Build trainer. (Put it all together.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bca8a572",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import Trainer\n",
    "\n",
    "trainer = Trainer(\n",
    "    model = model,\n",
    "    args = training_args,\n",
    "    data_collator = data_collator,\n",
    "    train_dataset = tokenized_ds[\"train\"],\n",
    "    eval_dataset = tokenized_ds[\"validation\"].select(range(100)),\n",
    "    tokenizer = tokenizer,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce8b530f",
   "metadata": {},
   "source": [
    "Now let's run training.<br>\n",
    "As I have mentioned above, make sure that you have enough disk space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a66059ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tsmatsuz/.local/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='3045' max='3045' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [3045/3045 3:54:47, Epoch 2/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>2.152600</td>\n",
       "      <td>1.098189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>1.157300</td>\n",
       "      <td>0.635520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>1.077800</td>\n",
       "      <td>0.613432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>0.946600</td>\n",
       "      <td>0.531952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>0.817200</td>\n",
       "      <td>0.610254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>0.819700</td>\n",
       "      <td>0.478506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>0.644200</td>\n",
       "      <td>0.522829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>0.514000</td>\n",
       "      <td>0.495377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>0.536700</td>\n",
       "      <td>0.461020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>0.537700</td>\n",
       "      <td>0.475981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>0.481500</td>\n",
       "      <td>0.470509</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1800</td>\n",
       "      <td>0.457100</td>\n",
       "      <td>0.437890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1950</td>\n",
       "      <td>0.465400</td>\n",
       "      <td>0.420714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2100</td>\n",
       "      <td>0.302000</td>\n",
       "      <td>0.461616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2250</td>\n",
       "      <td>0.351700</td>\n",
       "      <td>0.428793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2400</td>\n",
       "      <td>0.300400</td>\n",
       "      <td>0.455939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2550</td>\n",
       "      <td>0.296800</td>\n",
       "      <td>0.468101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2700</td>\n",
       "      <td>0.266200</td>\n",
       "      <td>0.495079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2850</td>\n",
       "      <td>0.252800</td>\n",
       "      <td>0.471554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3000</td>\n",
       "      <td>0.256400</td>\n",
       "      <td>0.453438</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=3045, training_loss=0.7215310818651822, metrics={'train_runtime': 14091.9134, 'train_samples_per_second': 6.918, 'train_steps_per_second': 0.216, 'total_flos': 1.910223004956365e+16, 'train_loss': 0.7215310818651822, 'epoch': 3.0})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44578fb6",
   "metadata": {},
   "source": [
    "In order to use it later, you can save the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0710012a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.makedirs(\"./trained_for_qa_jp\", exist_ok=True)\n",
    "if hasattr(trainer.model, \"module\"):\n",
    "    trainer.model.module.save_pretrained(\"./trained_for_qa_jp\")\n",
    "else:\n",
    "    trainer.model.save_pretrained(\"./trained_for_qa_jp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf2efe66",
   "metadata": {},
   "source": [
    "Load pre-trained model from local."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2be7c430",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForQuestionAnswering\n",
    "\n",
    "model = (AutoModelForQuestionAnswering\n",
    "         .from_pretrained(\"./trained_for_qa_jp\")\n",
    "         .to(device))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca6f566",
   "metadata": {},
   "source": [
    "## Perform Question Answering\n",
    "\n",
    "Now let's predict the answer fot the given context and question (which has not seen in the training set) with fine-tuned model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ed0fbcd",
   "metadata": {},
   "source": [
    "Instead manually running through forward pass, here I use a dedicated pipeline, in which preprocessing and postprocessing (such as, skipping padded tokens) are wrapped.<br>\n",
    "For Asian languages (such as, Chinese, Korean, and Japanese) which doesn't have an explicit white space, specify ```align_to_words=False```.\n",
    "\n",
    "As you can see below, this also returns scores.\n",
    "\n",
    "> Note : Picking up argmax of start and end indicies will fail to take correct answer. For instance, if span (9, 11), (5, 7), and (3, 7) are the top 3 candidates for the answer, 7th token might be picked up as end's index, and it might then return the span (9, 7).<br>\n",
    "> The QA pipeline in Hugging Face automatically picks up the best combination to avoid these mistakes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "26d98624",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** context *****\n",
      "本項東大寺の仏像では、奈良県奈良市にある聖武天皇ゆかりの寺院・東大寺に伝来する仏像について説明する。\n",
      "\n",
      "8世紀に日本の首都であった奈良を代表する寺院である東大寺は、「古都奈良の文化財」の一部として世界遺産に登録されている。東大寺には、「奈良の大仏」として知られる、高さ約15メートルの盧舎那仏像をはじめ、日本仏教美術史を代表する著名作品が多く所蔵されている。\n",
      "\n",
      "本項では東大寺に所在する仏像彫刻について概観する。なお、東大寺の概要については「東大寺」の項を、大仏については「東大寺盧舎那仏像」の項を参照のこと。\n",
      "\n",
      "***** question *****\n",
      "8世紀に日本の首都はどこでしたか。\n",
      "\n",
      "***** true answer *****\n",
      "奈良\n",
      "\n",
      "***** predicted top3 answer *****\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'score': 0.9808037877082825, 'start': 65, 'end': 67, 'answer': '奈良'},\n",
       " {'score': 0.011853429488837719,\n",
       "  'start': 65,\n",
       "  'end': 80,\n",
       "  'answer': '奈良を代表する寺院である東大寺'},\n",
       " {'score': 0.00010151458263862878,\n",
       "  'start': 65,\n",
       "  'end': 74,\n",
       "  'answer': '奈良を代表する寺院'}]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "qa_pipeline = pipeline(\n",
    "    \"question-answering\",\n",
    "    model=model,\n",
    "    tokenizer=tokenizer,\n",
    "    device=0)\n",
    "\n",
    "idx = 0\n",
    "print(\"***** context *****\")\n",
    "print(ds[\"validation\"][\"context\"][idx])\n",
    "print(\"\")\n",
    "print(\"***** question *****\")\n",
    "print(ds[\"validation\"][\"question\"][idx])\n",
    "print(\"\")\n",
    "print(\"***** true answer *****\")\n",
    "print(ds[\"validation\"][\"answers\"][idx][\"text\"][0])\n",
    "print(\"\")\n",
    "print(\"***** predicted top3 answer *****\")\n",
    "qa_pipeline(\n",
    "    question = ds[\"validation\"][\"question\"][idx],\n",
    "    context = ds[\"validation\"][\"context\"][idx],\n",
    "    align_to_words = False,\n",
    "    top_k=3,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04ad95eb",
   "metadata": {},
   "source": [
    "Run prediction without pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e88df8d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** question *****\n",
      "8世紀に日本の首都はどこでしたか。\n",
      "\n",
      "***** context *****\n",
      "本項東大寺の仏像では、奈良県奈良市にある聖武天皇ゆかりの寺院・東大寺に伝来する仏像について説明する。\n",
      "\n",
      "8世紀に日本の首都であった奈良を代表する寺院である東大寺は、「古都奈良の文化財」の一部として世界遺産に登録されている。東大寺には、「奈良の大仏」として知られる、高さ約15メートルの盧舎那仏像をはじめ、日本仏教美術史を代表する著名作品が多く所蔵されている。\n",
      "\n",
      "本項では東大寺に所在する仏像彫刻について概観する。なお、東大寺の概要については「東大寺」の項を、大仏については「東大寺盧舎那仏像」の項を参照のこと。\n",
      "\n",
      "***** true answer *****\n",
      "奈良\n",
      "\n",
      "***** predicted answer *****\n",
      "奈良\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "def inference_answer(question, context):\n",
    "    question = question\n",
    "    context = context\n",
    "    test_feature = tokenizer(\n",
    "        question,\n",
    "        context,\n",
    "        max_length=318,\n",
    "    )\n",
    "    with torch.no_grad():\n",
    "        outputs = model(torch.tensor([test_feature[\"input_ids\"]]).to(device))\n",
    "    start_logits = outputs.start_logits.cpu().numpy()\n",
    "    end_logits = outputs.end_logits.cpu().numpy()\n",
    "    answer_ids = test_feature[\"input_ids\"][np.argmax(start_logits):np.argmax(end_logits)+1]\n",
    "    return \"\".join(tokenizer.batch_decode(answer_ids))\n",
    "\n",
    "idx = 0\n",
    "question = ds[\"validation\"][\"question\"][idx]\n",
    "context = ds[\"validation\"][\"context\"][idx]\n",
    "answer_pred = inference_answer(question, context)\n",
    "\n",
    "print(\"***** question *****\")\n",
    "print(question)\n",
    "print(\"\")\n",
    "print(\"***** context *****\")\n",
    "print(context)\n",
    "print(\"\")\n",
    "print(\"***** true answer *****\")\n",
    "print(ds[\"validation\"][\"answers\"][idx][\"text\"][0])\n",
    "print(\"\")\n",
    "print(\"***** predicted answer *****\")\n",
    "print(answer_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9c5b81aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** question *****\n",
      "「奈良の大仏」の高さは何メートルなの?\n",
      "\n",
      "***** context *****\n",
      "本項東大寺の仏像では、奈良県奈良市にある聖武天皇ゆかりの寺院・東大寺に伝来する仏像について説明する。\n",
      "\n",
      "8世紀に日本の首都であった奈良を代表する寺院である東大寺は、「古都奈良の文化財」の一部として世界遺産に登録されている。東大寺には、「奈良の大仏」として知られる、高さ約15メートルの盧舎那仏像をはじめ、日本仏教美術史を代表する著名作品が多く所蔵されている。\n",
      "\n",
      "本項では東大寺に所在する仏像彫刻について概観する。なお、東大寺の概要については「東大寺」の項を、大仏については「東大寺盧舎那仏像」の項を参照のこと。\n",
      "\n",
      "***** true answer *****\n",
      "約15メートル\n",
      "\n",
      "***** predicted answer *****\n",
      "約15メートルの\n"
     ]
    }
   ],
   "source": [
    "idx = 1\n",
    "question = ds[\"validation\"][\"question\"][idx]\n",
    "context = ds[\"validation\"][\"context\"][idx]\n",
    "answer_pred = inference_answer(question, context)\n",
    "\n",
    "print(\"***** question *****\")\n",
    "print(question)\n",
    "print(\"\")\n",
    "print(\"***** context *****\")\n",
    "print(context)\n",
    "print(\"\")\n",
    "print(\"***** true answer *****\")\n",
    "print(ds[\"validation\"][\"answers\"][idx][\"text\"][0])\n",
    "print(\"\")\n",
    "print(\"***** predicted answer *****\")\n",
    "print(answer_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e5ea750",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}