5.klue-bert-base-mlm-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "97695445",
   "metadata": {},
   "source": [
    "# BERT의 MLM을 사용한 Classification\n",
    "\n",
    "BERT Pretrain 과정에서는 masked token을 맞추는 Masked Language Modeling이라는 방식을 사용합니다.  \n",
    "예를 들어, \"my dog is hairy\" 라는 문장이 있을 때, \"my dog is `[MASK]`\" 로 문장을 변경하고, 모델이 `[MASK]` 토큰을 `hairy`라고 예측할 수 있게 학습하는 것입니다.  \n",
    "\n",
    "이 방식을 Classification에도 사용할 수 있습니다.   \n",
    "\"인천→핀란드 항공기 결항…휴가철 여행객 분통\" 문장의 label은 `세계` 입니다.   \n",
    "\"인천→핀란드 항공기 결항…휴가철 여행객 분통\" 문장의 뒤에 \"[SEP]이 문장은 `[MASK]`\"라는 문장을 붙였습니다.  \n",
    "그 후 모델이 `[MASK]`를 `세계`로 예측하도록 학습시켰습니다.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "47216418",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "사용가능한 GPU수 :  1\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "from tqdm.notebook import tqdm, tnrange\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from transformers import AutoTokenizer, AutoModelForPreTraining\n",
    "\n",
    "from transformers import AdamW\n",
    "from transformers.optimization import get_cosine_schedule_with_warmup\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import Dataset,TensorDataset, DataLoader, RandomSampler\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    print(\"사용가능한 GPU수 : \",torch.cuda.device_count())\n",
    "    device = torch.device(\"cuda\")\n",
    "else:\n",
    "    print(\"CPU 사용\")\n",
    "    device = torch.device(\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78a18e6c",
   "metadata": {},
   "source": [
    "Reproduction을 위한 Seed 고정  \n",
    "출처 : https://dacon.io/codeshare/2363?dtype=vote&s_id=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "088ef286",
   "metadata": {},
   "outputs": [],
   "source": [
    "RANDOM_SEED = 42\n",
    "\n",
    "def seed_everything(seed: int = 42):\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed(seed)  # type: ignore\n",
    "    torch.backends.cudnn.deterministic = True  # type: ignore\n",
    "    torch.backends.cudnn.benchmark = True  # type: ignore\n",
    "    \n",
    "seed_everything(RANDOM_SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12315cdb",
   "metadata": {},
   "source": [
    "모델은 klue/bert-base을 사용하였습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ba1748e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_checkpoint = \"klue/bert-base\"\n",
    "save_checkpoint_path = \"./checkpoints\"\n",
    "batch_size = 32"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf22959",
   "metadata": {},
   "source": [
    "huggingface 에서 tokenizer를 불러옵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7bd602d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4078413a",
   "metadata": {},
   "source": [
    "huggingface 에서 model를 불러옵니다.  \n",
    "`AutoModelForSequenceClassification`이 아닌 `AutoModelForPreTraining`을 사용합니다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e5c86880",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = AutoModelForPreTraining.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "19d28699",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertForPreTraining(\n",
       "  (bert): BertModel(\n",
       "    (embeddings): BertEmbeddings(\n",
       "      (word_embeddings): Embedding(32000, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (token_type_embeddings): Embedding(2, 768)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (encoder): BertEncoder(\n",
       "      (layer): ModuleList(\n",
       "        (0): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (1): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (2): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (3): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (4): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (5): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (6): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (7): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (8): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (9): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (10): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (11): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (pooler): BertPooler(\n",
       "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (activation): Tanh()\n",
       "    )\n",
       "  )\n",
       "  (cls): BertPreTrainingHeads(\n",
       "    (predictions): BertLMPredictionHead(\n",
       "      (transform): BertPredictionHeadTransform(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      )\n",
       "      (decoder): Linear(in_features=768, out_features=32000, bias=True)\n",
       "    )\n",
       "    (seq_relationship): Linear(in_features=768, out_features=2, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86ae45cb",
   "metadata": {},
   "source": [
    "dataset을 가져옵니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "081f2499",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pd.read_csv(\"data/train_data.csv\")\n",
    "test = pd.read_csv(\"data/test_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b472c1fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>title</th>\n",
       "      <th>topic_idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>인천→핀란드 항공기 결항…휴가철 여행객 분통</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>실리콘밸리 넘어서겠다…구글 15조원 들여 美전역 거점화</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>이란 외무 긴장완화 해결책은 미국이 경제전쟁 멈추는 것</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index                           title  topic_idx\n",
       "0      0        인천→핀란드 항공기 결항…휴가철 여행객 분통          4\n",
       "1      1  실리콘밸리 넘어서겠다…구글 15조원 들여 美전역 거점화          4\n",
       "2      2  이란 외무 긴장완화 해결책은 미국이 경제전쟁 멈추는 것          4"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af973e04",
   "metadata": {},
   "source": [
    "데이터를 MLM Classification 방식에 맞게 전처리해야합니다.  \n",
    "`[SEP]이 문장은 [MASK]` 문장을 모든 데이터 뒤에 붙여줍니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8c1c857e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Workspace\\guan\\sentimental_classification\\venv\\lib\\site-packages\\pandas\\core\\indexing.py:1732: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self._setitem_single_block(indexer, value, name)\n"
     ]
    }
   ],
   "source": [
    "for idx in range(len(dataset[\"title\"])):\n",
    "    dataset[\"title\"].iloc[idx] += \".[SEP] 이 문장은 [MASK]\"\n",
    "\n",
    "for idx in range(len(test[\"title\"])):\n",
    "    test[\"title\"].iloc[idx] += \".[SEP] 이 문장은 [MASK]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e098f165",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>title</th>\n",
       "      <th>topic_idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>인천→핀란드 항공기 결항…휴가철 여행객 분통.[SEP] 이 문장은 [MASK]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>실리콘밸리 넘어서겠다…구글 15조원 들여 美전역 거점화.[SEP] 이 문장은 [MASK]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>이란 외무 긴장완화 해결책은 미국이 경제전쟁 멈추는 것.[SEP] 이 문장은 [MASK]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index                                              title  topic_idx\n",
       "0      0        인천→핀란드 항공기 결항…휴가철 여행객 분통.[SEP] 이 문장은 [MASK]          4\n",
       "1      1  실리콘밸리 넘어서겠다…구글 15조원 들여 美전역 거점화.[SEP] 이 문장은 [MASK]          4\n",
       "2      2  이란 외무 긴장완화 해결책은 미국이 경제전쟁 멈추는 것.[SEP] 이 문장은 [MASK]          4"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "376a40a7",
   "metadata": {},
   "source": [
    "klue/bert-base의 Tokenizer에서 각 라벨들의 토큰을 확인합니다.  \n",
    "하나의 토큰으로 예측을 하기 위해 `IT과학` -> `과학`으로, `생활문화` -> `문화`로 바꿨습니다.   \n",
    "\n",
    "`과학`은 4038  \n",
    "`경제`는 3674  \n",
    "`사회`는 3647  \n",
    "`문화`는 3697  \n",
    "`세계`는 3665  \n",
    "`스포츠`는 4559  \n",
    "`정치`는 3713\n",
    "\n",
    "입니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "08738015",
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_token_dict = {0:4038,1:3674,2:3647,3:3697,4:3665,5:4559,6:3713}\n",
    "token_topic_dict = {4038 : 0, 3674 : 1, 3647 : 2, 3697 : 3, 3665 : 4, 4559 : 5, 3713 : 6}\n",
    "topic_dict = {0: \"과학\", 1:\"경제\", 2:\"사회\", 3:\"문화\", 4:\"세계\", 5:\"스포츠\", 6 : \"정치\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8899404a",
   "metadata": {},
   "source": [
    "train과 validation을 나눠줍니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "dac63b6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_train, dataset_val = train_test_split(dataset,test_size = 0.2,random_state = RANDOM_SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ead25147",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>title</th>\n",
       "      <th>topic_idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4824</th>\n",
       "      <td>4824</td>\n",
       "      <td>KT 화상회의 가능한 기업용 전화 출시.[SEP] 이 문장은 [MASK]</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21714</th>\n",
       "      <td>21714</td>\n",
       "      <td>문 대통령 해리 해리스 주한 미국대사와 함께.[SEP] 이 문장은 [MASK]</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29145</th>\n",
       "      <td>29145</td>\n",
       "      <td>김기식 사퇴 찬성 51%…文대통령 지지율 66.2%로 하락리얼미터.[SEP] 이 문...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18342</th>\n",
       "      <td>18342</td>\n",
       "      <td>민주콩고서 수백명 에볼라 사태 끝내자 거리행진.[SEP] 이 문장은 [MASK]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9010</th>\n",
       "      <td>9010</td>\n",
       "      <td>충남 올겨울 첫 한파주의보…눈 최대 15㎝ 쌓일 듯.[SEP] 이 문장은 [MASK]</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       index                                              title  topic_idx\n",
       "4824    4824           KT 화상회의 가능한 기업용 전화 출시.[SEP] 이 문장은 [MASK]          1\n",
       "21714  21714        문 대통령 해리 해리스 주한 미국대사와 함께.[SEP] 이 문장은 [MASK]          6\n",
       "29145  29145  김기식 사퇴 찬성 51%…文대통령 지지율 66.2%로 하락리얼미터.[SEP] 이 문...          6\n",
       "18342  18342       민주콩고서 수백명 에볼라 사태 끝내자 거리행진.[SEP] 이 문장은 [MASK]          4\n",
       "9010    9010    충남 올겨울 첫 한파주의보…눈 최대 15㎝ 쌓일 듯.[SEP] 이 문장은 [MASK]          3"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9b569b",
   "metadata": {},
   "source": [
    "입력 문장을 tokenize 해줍니다. \n",
    "이때, [MASK]토큰의 위치를 기억해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b5e9cfcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def bert_tokenize(dataset,sent_key,label_key,tokenizer):\n",
    "    if label_key is None :\n",
    "        labels = [np.int64(0) for i in dataset[sent_key]]\n",
    "    else :\n",
    "        labels = [np.int64(i) for i in dataset[label_key]]\n",
    "    \n",
    "    sentences = tokenizer(dataset[sent_key].tolist(),truncation=True,padding=True)\n",
    "\n",
    "    input_ids = sentences.input_ids\n",
    "    token_type_ids = sentences.token_type_ids\n",
    "    attention_mask = sentences.attention_mask\n",
    "    \n",
    "    # [MASK] 토큰의 인덱스 저장. \n",
    "    masked_token_idx = []\n",
    "    for input_id in input_ids:\n",
    "        masked_token_idx.append(input_id.index(4))\n",
    "    \n",
    "    return list([input_ids, token_type_ids, attention_mask, labels, masked_token_idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b94b1212",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_inputs = bert_tokenize(dataset_train,\"title\",\"topic_idx\",tokenizer)\n",
    "validation_inputs = bert_tokenize(dataset_val,\"title\",\"topic_idx\",tokenizer)\n",
    "test_inputs = bert_tokenize(test,\"title\",None,tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c170772",
   "metadata": {},
   "source": [
    "토큰화가 잘 되었습니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "57b65a71",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "인천→핀란드 항공기 결항…휴가철 여행객 분통.[SEP] 이 문장은 [MASK]\n",
      "[2, 5108, 10948, 7288, 3662, 2470, 3646, 2048, 4117, 4542, 18, 3, 1504, 6265, 2073, 4, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "print(dataset_train[\"title\"][0])\n",
    "print(train_inputs[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d63b328",
   "metadata": {},
   "source": [
    "padding된 데이터의 길이를 확인해줍니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "25815df1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36 35 35\n",
      "max length : 35\n"
     ]
    }
   ],
   "source": [
    "print(len(test_inputs[0][1]), len(validation_inputs[0][1]), len(train_inputs[0][1]))\n",
    "max_length = max(len(validation_inputs[0][1]), len(train_inputs[0][1]))\n",
    "print(\"max length :\",max_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "46e0e0bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(train_inputs)):\n",
    "    train_inputs[i] = torch.tensor(train_inputs[i])\n",
    "    \n",
    "for i in range(len(validation_inputs)):\n",
    "    validation_inputs[i] = torch.tensor(validation_inputs[i])\n",
    "    \n",
    "for i in range(len(test_inputs)):\n",
    "    test_inputs[i] = torch.tensor(test_inputs[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ad82b562",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = TensorDataset(*train_inputs)\n",
    "train_sampler = RandomSampler(train_data)\n",
    "train_dataloader = DataLoader(train_data,sampler=train_sampler,batch_size=batch_size)\n",
    "\n",
    "validation_data = TensorDataset(*validation_inputs)\n",
    "validation_sampler = RandomSampler(validation_data)\n",
    "validation_dataloader = DataLoader(validation_data,sampler=validation_sampler,batch_size=batch_size)\n",
    "\n",
    "test_data = TensorDataset(*test_inputs)\n",
    "test_dataloader = DataLoader(test_data,batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ce3e9ee",
   "metadata": {},
   "source": [
    "hyper parameter를 설정해줍니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "26ce8a8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-5\n",
    "adam_epsilon = 1e-8\n",
    "epochs = 5\n",
    "\n",
    "num_warmup_steps = 0\n",
    "\n",
    "warmup_ratio = 0.1\n",
    "num_training_steps = len(train_dataloader)*epochs\n",
    "warmup_step = int(num_training_steps * warmup_ratio)\n",
    "\n",
    "optimizer = AdamW(model.parameters(), lr=lr,eps=adam_epsilon) \n",
    "scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=warmup_step, num_training_steps=num_training_steps) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d78c8e9",
   "metadata": {},
   "source": [
    "학습을 진행합니다.  \n",
    "`AutoModelForPreTraining`에 데이터를 입력했을 때 나오는 output은 이 문장이 다음 문장인지 아닌지를 판단하는 확률과 [MASK] 토큰을 예측한 값입니다.  \n",
    "output과 정답에 대한 Loss를 계산하고 Loss를 줄이는 방향으로 학습이 진행됩니다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a89f960a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c0adac4c8414484aaf3511ff6e7305a7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<====================== Epoch 1 ======================>\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d5e4845fe6474fa695d1f2ecd24c2426",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1142 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAverage Training loss: 0.7418069830833038\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df77f969fa594aeca000587e83ce49e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/286 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAccuracy : 0.8761362391851933\n",
      "\n",
      "\tAverage validation loss: 3.3212648837768532\n",
      "<====================== Epoch 2 ======================>\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fde588a232ee4f039bb67a038425538b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1142 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAverage Training loss: 0.28430189007987805\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f7f7853b846486aa6033994d7ac3dc8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/286 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAccuracy : 0.8888402146533786\n",
      "\n",
      "\tAverage validation loss: 1.4593685174725586\n",
      "<====================== Epoch 3 ======================>\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "902d5ee9bffd444dbf9e6833884f49eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1142 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAverage Training loss: 0.18635195633151852\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f50f7260ac71453da9ddfe4e45045ea6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/286 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAccuracy : 0.8950826853575731\n",
      "\n",
      "\tAverage validation loss: 1.1027775633090233\n",
      "<====================== Epoch 4 ======================>\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8ba971b36cc45158977e7e2f3b4d4bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1142 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAverage Training loss: 0.11452150013462531\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b485632668fc4c94a96fee5ffcedff6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/286 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAccuracy : 0.8923447596101194\n",
      "\n",
      "\tAverage validation loss: 0.9058767256206488\n",
      "<====================== Epoch 5 ======================>\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0a904025e9c41609c537b325336590f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1142 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAverage Training loss: 0.07596644382413531\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d92c17062a9445278f37a817ab6ef980",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/286 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\tAccuracy : 0.893549446938999\n",
      "\n",
      "\tAverage validation loss: 0.8153319107517161\n"
     ]
    }
   ],
   "source": [
    "train_loss_set = []\n",
    "# learning_rate = []\n",
    "\n",
    "criterion_lm = torch.nn.CrossEntropyLoss(ignore_index=-1, reduction='mean')\n",
    "criterion_cls = torch.nn.CrossEntropyLoss()\n",
    "\n",
    "model.zero_grad()\n",
    "\n",
    "for _ in tnrange(1,epochs+1,desc='Epoch'):\n",
    "    print(\"<\" + \"=\"*22 + F\" Epoch {_} \"+ \"=\"*22 + \">\")\n",
    "    batch_loss = 0\n",
    "    \n",
    "    # train\n",
    "    model.train()\n",
    "    for step, batch in enumerate(tqdm(train_dataloader)):\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        b_input_ids, b_token_type_ids, b_input_mask, b_labels, b_masked_token_idx = batch\n",
    "        \n",
    "        outputs = model(b_input_ids, token_type_ids=b_token_type_ids, attention_mask=b_input_mask)\n",
    "        \n",
    "        # calculate loss\n",
    "        logits_cls, logits_lm = outputs[1], outputs[0]\n",
    "        \n",
    "        # topic -> token \n",
    "        # Ex.6 -> 3713\n",
    "        mask_label = [topic_token_dict[lb] for lb in b_labels.to('cpu').numpy() ]\n",
    "        \n",
    "        labels_lms = []\n",
    "        \n",
    "        # label 만들기\n",
    "        for idx, label in zip(b_masked_token_idx.to('cpu').numpy(),mask_label):\n",
    "            labels_lm = np.full(max_length, dtype=np.int, fill_value=-1)\n",
    "            labels_lm[idx] = label\n",
    "            labels_lms.append(labels_lm)\n",
    "        label_lms_pt = torch.tensor(labels_lms,dtype=torch.int64).to(device)\n",
    "        \n",
    "        # lm loss 계산\n",
    "        loss_lm = criterion_lm(logits_lm.view(-1, logits_lm.size(2)), label_lms_pt.view(-1))\n",
    "        \n",
    "        # cls loss 계산\n",
    "        labels_cls = [1 for _ in range(len(b_input_ids))]\n",
    "        labels_cls = torch.tensor(labels_cls).to(device)\n",
    "        loss_cls = criterion_cls(logits_cls, labels_cls)\n",
    "        \n",
    "        loss = loss_cls + loss_lm\n",
    "        loss.backward()        \n",
    "        \n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
    "        optimizer.step()\n",
    "        \n",
    "        scheduler.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        batch_loss += loss.item()\n",
    "        \n",
    "    avg_train_loss = batch_loss / len(train_dataloader)\n",
    "    train_loss_set.append(avg_train_loss)\n",
    "    print(F'\\n\\tAverage Training loss: {avg_train_loss}')\n",
    "    \n",
    "    # eval\n",
    "    model.eval()\n",
    "    \n",
    "    predict_list = []\n",
    "    label_list = []\n",
    "    \n",
    "    for step, batch in enumerate(tqdm(validation_dataloader)):\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        b_input_ids, b_token_type_ids, b_input_mask, b_labels, b_masked_token_idx = batch\n",
    "\n",
    "        outputs = model(b_input_ids, token_type_ids=b_token_type_ids, attention_mask=b_input_mask)\n",
    "\n",
    "        logits_cls, logits_lm = outputs[1], outputs[0]\n",
    "        \n",
    "        # 예측한 값.\n",
    "        logits_lm_np = logits_lm.to('cpu').detach().numpy()\n",
    "        pred = np.argmax(logits_lm_np,axis=2)\n",
    "        \n",
    "        # [MASK] 토큰의 인덱스\n",
    "        masked_token_idx_np = b_masked_token_idx.to('cpu').numpy()\n",
    "        \n",
    "        # label\n",
    "        labels_np = b_labels.to('cpu').numpy()\n",
    "\n",
    "        for i in range(len(pred)):\n",
    "            # 예측한 토큰을 label로 바꾸기\n",
    "            l = token_topic_dict[pred[i][masked_token_idx_np[i]]]\n",
    "            predict_list.append(l)\n",
    "\n",
    "        for l in labels_np:\n",
    "            # 정답을 넣기\n",
    "            label_list.append(l)\n",
    "\n",
    "        # topic -> token \n",
    "        # Ex.6 -> 3713\n",
    "        mask_label = [ topic_token_dict[lb] for lb in labels_np ]\n",
    "        \n",
    "        labels_lms = []\n",
    "        for idx, label in zip(masked_token_idx_np,mask_label):\n",
    "            \n",
    "            labels_lm = np.full(max_length, dtype=np.int, fill_value=-1)\n",
    "            labels_lm[idx] = label\n",
    "            labels_lms.append(labels_lm)\n",
    "\n",
    "        #mlm loss 계산\n",
    "        labels_lms_pt = torch.tensor(labels_lms,dtype=torch.int64).to(device)\n",
    "        loss_lm = criterion_lm(logits_lm.view(-1, logits_lm.size(2)), labels_lms_pt.view(-1))\n",
    "\n",
    "        #nsp loss 계산\n",
    "        labels_cls = [1 for _ in range(len(b_input_ids))]\n",
    "        labels_cls = torch.tensor(labels_cls).to(device)\n",
    "        loss_cls = criterion_cls(logits_cls, labels_cls)\n",
    "\n",
    "        loss = loss_cls + loss_lm\n",
    "        \n",
    "        batch_loss += loss.item()\n",
    "\n",
    "    print(\"\\n\\tAccuracy : {}\".format(accuracy_score(predict_list,label_list)))\n",
    "    avg_validation_loss = batch_loss / len(validation_dataloader)\n",
    "    print(F'\\n\\tAverage validation loss: {avg_validation_loss}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff0b2afd",
   "metadata": {},
   "source": [
    "훈련이 끝나고, 예측을 진행합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ca02b5ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ea7f89e9a454da68dfd508f67f42669",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/286 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predict_li = []\n",
    "\n",
    "model.eval()\n",
    "for step, batch in enumerate(tqdm(test_dataloader)):\n",
    "    batch = tuple(t.to(device) for t in batch)\n",
    "    b_input_ids, b_token_type_ids, b_input_mask, b_labels, b_masked_token_idx = batch\n",
    "    \n",
    "    outputs = model(b_input_ids, token_type_ids=b_token_type_ids, attention_mask=b_input_mask)\n",
    "    \n",
    "    logits_cls, logits_lm = outputs[1], outputs[0]\n",
    "    \n",
    "    logits_lm = logits_lm.to('cpu').detach().numpy()\n",
    "    out = np.argmax(logits_lm,axis=2)\n",
    "    \n",
    "    masked_token_idx_np = b_masked_token_idx.to('cpu').numpy()\n",
    "    \n",
    "#     print(out)\n",
    "    \n",
    "    for i in range(len(out)):\n",
    "        l = token_topic_dict[out[i][masked_token_idx_np[i]]]\n",
    "        predict_li.append(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f9d898cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9131"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(predict_li)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "95b33b13",
   "metadata": {},
   "outputs": [],
   "source": [
    "submission = pd.read_csv('data/sample_submission.csv')\n",
    "submission['topic_idx'] = predict_li\n",
    "submission.to_csv(\"results/klue-bert-mlm-classification.csv\",index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14fa7d76",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}