NN-中文文本分類.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3bd90bd1-448f-401f-b5e8-f897f918dbd7",
   "metadata": {},
   "source": [
    "# NN-自然語言處理\n",
    "## 教學目標\n",
    "- 本教學著重於自然語言處理，其中涵蓋`MLP`、`RNN`以及`Transformers`。\n",
    "- 這份教學的目標是介紹如何以 Python 和 PyTorch 實作神經網路。\n",
    "\n",
    "## 適用對象\n",
    "已經有基本的機器學習知識，且擁有 Python、`numpy`、`pandas`、`scikit-learn` 以及 `PyTorch` 基礎的學生。\n",
    "\n",
    "若沒有先學過 Python，請參考 [python-入門語法](./python-入門語法.ipynb) 教學。\n",
    "\n",
    "若沒有先學過 `pandas`，請參考 [pandas-基本功能](./pandas-基本功能.ipynb) 教學。\n",
    "\n",
    "若沒有先學過 `numpy`，請參考 [numpy-基本功能](./numpy-基本功能.ipynb) 教學。\n",
    "\n",
    "若沒有先學過 `scikit-learn`，請參考 [scikit-learn-基本功能](./scikit-learn-基本功能.ipynb) 教學。\n",
    "\n",
    "若沒有先學過  `PyTorch` ，請參考 [PyTorch-基本功能](./PyTorch-基本功能.ipynb) 教學。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "260b72e0-36da-44e4-9d3e-75844fb954b6",
   "metadata": {},
   "source": [
    "## 類神經網路 (NN)\n",
    "- 類神經網路屬於一種機器學習方法\n",
    "- 基本的神經網路模型由許多個非線性轉換的單元所構成 (如下圖所示)\n",
    "\n",
    "![Imgur](https://i.imgur.com/tE4EdPh.png)\n",
    "- 可參閱 [簡介](https://docs.google.com/presentation/d/1QeO_6cmset9AJB5qWWSaNwJbL8n2IePMc5t0vfzBBNA/edit#slide=id.p) 與 [實作](https://github.com/IKMLab/Feedforward-Tutorial/blob/master/text-classfication/text-classfication.ipynb) by IKMLab\n",
    "- 神經網路又可以稱為多層感知機 (Perceptron)\n",
    "- 深度學習則是模型透過深層神經網路來進行學習的方法 \n",
    "- NN 的實作\n",
    "    - 使用 TensorFlow/Keras/PyTorch 等深度學習函式庫\n",
    "    - 也可以使用 scikit-learn 套件\n",
    "        - `from sklearn.neural_network import MLPClassifier`\n",
    "        - 但是要額外實作：\n",
    "            1. 把 word 轉為 id\n",
    "            2. embedding_lookup (或使用 torch.nn.embedding)\n",
    "            3. 以 mini-batch 進行訓練 (fit 模型)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758e8f94-8fcb-4ed0-a4a3-01d442a196a2",
   "metadata": {},
   "source": [
    "## 使用 NN 來進行中文的分類任務\n",
    "\n",
    "- 我們將在這個教學裡讓大家實作中文情緒分析（Sentiment Analysis）\n",
    "- 本資料集爲外賣平臺用戶評價分析，[下載連結](https://raw.githubusercontent.com/SophonPlus/ChineseNlpCorpus/master/datasets/waimai_10k/waimai_10k.csv)。\n",
    "- 資料集欄位爲標籤（label）和評價（review），\n",
    "- 標籤 1 爲正向，0 爲負向。\n",
    "- 正向 4000 條，負向約 8000 條。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fff29fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p data\n",
    "!wget https://raw.githubusercontent.com/SophonPlus/ChineseNlpCorpus/master/datasets/waimai_10k/waimai_10k.csv -O data/waimai_10k.csv\n",
    "!pip install jieba\n",
    "!pip install torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tribal-cradle",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. 導入所需套件\n",
    "\n",
    "# Python 套件\n",
    "import os\n",
    "import math\n",
    "import random\n",
    "\n",
    "# 第3方套件\n",
    "import jieba\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "from torch.nn import TransformerEncoder, TransformerEncoderLayer\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "floral-particular",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. 以 pandas 讀取資料\n",
    "# 請先下載資料集\n",
    "\n",
    "df = pd.read_csv(\"./data/waimai_10k.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5764cbb8-a764-43a6-a84d-f20e5b38c209",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3. 觀察資料\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55c6f14d-298f-435f-b361-d418be65de83",
   "metadata": {},
   "source": [
    "## 建立字典\n",
    "- 電腦無法僅透過字符來區分不同字之間的意涵\n",
    "- 電腦視覺領域依賴的是影像資料本身的像素值\n",
    "- 我們讓電腦理解文字的方法是透過向量\n",
    "- 文字的意義藉由向量來進行表達的形式稱為 word embeddings\n",
    "- 舉例:\n",
    "$\\textrm{apple}=[0.123, 0.456,0.789,\\dots,0.111]$\n",
    "\n",
    "- 如何建立每個文字所屬的向量？\n",
    "    - 傳統方法: 計數法則\n",
    "    - 近代方法 (2013-至今): 使用(淺層)神經網路訓練 word2vec ([參考](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/))，稱為 word embeddings\n",
    "    - 現代方法 (2018-至今): 使用(深層)神經網路訓練 Transformers，也就是BERT ([參考](https://youtu.be/gh0hewYkjgo))，又稱為 contexualized embeddings\n",
    "- 在那之前，要先建立分散式字詞的字典\n",
    "    - 可粗分兩種斷詞方式 (tokenization):\n",
    "        1. 每個字都斷 (character-level)\n",
    "        2. 斷成字詞 (word-level)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db1b10a5-e5cc-4151-baf4-00852493ab6e",
   "metadata": {},
   "source": [
    "## Word embeddings\n",
    "- 著名的方法有:\n",
    "    1. word2vec: Skip-gram, CBOW (continuous bag-of-words)\n",
    "    2. GloVe\n",
    "    3. fastText"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20845258",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_to_idx = {\"好吃\": 0, \"棒\": 1, \"给力\": 2}\n",
    "embeds = torch.nn.Embedding(3, 5)  # 2 words in vocab, 5 dimensional embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23a1e9e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "lookup_tensor = torch.tensor(\n",
    "    [\n",
    "        word_to_idx[\"好吃\"],\n",
    "        word_to_idx[\"棒\"],\n",
    "        word_to_idx[\"给力\"],\n",
    "    ],\n",
    "    dtype=torch.long,\n",
    ")\n",
    "word_embed = embeds(lookup_tensor)\n",
    "print(word_embed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e765695-c297-4ec9-9209-9567a95769a6",
   "metadata": {},
   "source": [
    "### 分水嶺\n",
    "1. 自己先建字典，透過模型中的 nn.embeddings 針對任務進行訓練 (本教學)\n",
    "2. 自己先建字典，接著使用預訓練的 word embeddings 來初始化 nn.embeddings，然後針對任務進行訓練\n",
    "    - 請參閱 [連結](https://colab.research.google.com/drive/13Fa0w7-AKtC0O06vCQHmOKAlPoV7PqOz?usp=sharing)\n",
    "3. 不先建字典，直接針對任務的資料集預先訓練一個 word embeddings，接著使用預訓練的 word embeddings 來初始化 nn.embeddings，然後針對任務進行訓練\n",
    "    - 請參閱 [連結](https://colab.research.google.com/drive/1DhNLBMnf5UwbF6xHYuxaa5VSCa51c4aS?usp=sharing)\n",
    "4. 不先建字典，針對大規模通用資料集預先訓練一個 word embeddings，接著使用預訓練的 word embeddings 來初始化 nn.embeddings，然後針對任務進行訓練\n",
    "    - 請參閱 [連結](http://zake7749.github.io/2016/08/28/word2vec-with-gensim/)，完成訓練後再至分水嶺2進行載入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50474fc7-d3ba-4f4c-ab47-e1823fe04c59",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4. 設定隨機種子 (定義 function)\n",
    "seed = 42\n",
    "\n",
    "def set_seed(seed):\n",
    "    \"\"\" 這個 function 可以使程式碼中有使用到 PyTorch 和 Numpy 的隨機過程受到控制\n",
    "    Args:\n",
    "        seed: 初始化 pseudorandom number generator 的正整數\n",
    "    \"\"\"\n",
    "    os.environ['PYTHONHASHSEED'] = str(seed)\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "\n",
    "    if torch.cuda.is_available():\n",
    "        torch.cuda.manual_seed(seed)\n",
    "        #torch.cuda.manual_seed_all(seed)  # 如果有使用多個 GPU\n",
    "        torch.backends.cudnn.deterministic = True\n",
    "        torch.backends.cudnn.benchmark = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d20a4d6-103e-4e9e-8ca7-d18c7750c1be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 5. 建立字典\n",
    "use_jieba=True\n",
    "\n",
    "vocab = {'<pad>':0, '<unk>':1}\n",
    "\n",
    "if use_jieba:\n",
    "    words = []\n",
    "    for sent in df['review']:\n",
    "        tokens = jieba.lcut(sent, cut_all=False)\n",
    "        words.extend(tokens)\n",
    "\n",
    "else:\n",
    "    # 以 character-level 斷詞\n",
    "    words = df['review'].str.cat()\n",
    "\n",
    "# 使字詞不重複\n",
    "words = sorted(set(words))\n",
    "for idx, word in enumerate(words):\n",
    "    # 一開始已經放兩個進去 dictionary 了\n",
    "    idx = idx + 2\n",
    "    # 將 word to id 放到 dictionary\n",
    "    vocab[word] = idx\n",
    "        \n",
    "# 查看字典大小\n",
    "print(\"The vocab size is {}.\".format(len(vocab)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035a31d6-38d3-44f2-a98d-d487b2a90987",
   "metadata": {},
   "source": [
    "## 使用 PyTorch 建立 Dataset\n",
    "![Imgur](https://i.imgur.com/wGnfCmH.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e40cb6e1-bd26-44c2-886c-b3651f204d50",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 6. 將資料分成 train/ validation/ test\n",
    "\n",
    "train_data, test_data = train_test_split(\n",
    "    df,\n",
    "    test_size=0.2,\n",
    "    random_state=seed\n",
    ")\n",
    "train_data, validation_data = train_test_split(\n",
    "    train_data,\n",
    "    test_size=0.1,\n",
    "    random_state=seed\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c81e62ed-13ad-483c-a47a-0407d7b3ef35",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 7. 定義超參數\n",
    "\n",
    "parameters = {\n",
    "    \"padding_idx\": 0,\n",
    "    \"vocab_size\": len(vocab),\n",
    "    # Hyperparameters\n",
    "    \"embed_dim\": 300,\n",
    "    \"hidden_dim\": 256,\n",
    "    \"module_name\": 'rnn', # 選項: rnn, lstm, gru, transformer\n",
    "    \"num_layers\": 2,\n",
    "    \"learning_rate\": 5e-4, # 使用 Transformer 時建議改成 5e-5\n",
    "    \"epochs\": 10,\n",
    "    \"max_seq_len\": 50,\n",
    "    \"batch_size\": 64,\n",
    "    # Transformers\n",
    "    \"nhead\": 2,\n",
    "    \"dropout\": 0.2,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "standard-joining",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 8. 建立 PyTorch Dataset (定義 class)\n",
    "\n",
    "class WaimaiDataset(torch.utils.data.Dataset):\n",
    "    # 繼承 torch.utils.data.Dataset\n",
    "    def __init__(self, data, max_seq_len, use_jieba):\n",
    "        self.df = data\n",
    "        self.max_seq_len = max_seq_len\n",
    "        # 可以選擇要不要使用結巴進行斷詞\n",
    "        self.use_jieba = use_jieba\n",
    "        \n",
    "    # 改寫繼承的 __getitem__ function\n",
    "    def __getitem__(self, idx):\n",
    "        # dataframe 的第一個 column 是 label\n",
    "        # dataframe 的第一個 column 是 評論的句子\n",
    "        label, sent = self.df.iloc[idx, 0:2]\n",
    "        # 先將 label 轉為 float32 以方便後面進行 loss function 的計算\n",
    "        label_tensor = torch.tensor(label, dtype=torch.float32)\n",
    "        if self.use_jieba:\n",
    "            # 使用 lcut 可以 return list\n",
    "            tokens = jieba.lcut(sent, cut_all=False)\n",
    "        else:\n",
    "            # 每個字都斷詞\n",
    "            tokens = list(sent)\n",
    "        \n",
    "        # 控制最大的序列長度\n",
    "        tokens = tokens[:self.max_seq_len]\n",
    "\n",
    "        # 根據 vocab 轉換 word id\n",
    "        # vocab 是一個 list\n",
    "        tokens_id = [vocab[word] for word in tokens]\n",
    "        tokens_tensor = torch.LongTensor(tokens_id)\n",
    "\n",
    "        # 所以 第 0 個index是句子，第 1 個index是 label\n",
    "        return tokens_tensor, label_tensor\n",
    "    \n",
    "    # 改寫繼承的 __len__ function\n",
    "    def __len__(self):\n",
    "        return len(self.df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee767846-421c-4eff-a4ff-3986308f1497",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 9. 建立 PyTorch Dataset (執行 class)\n",
    "use_jieba=use_jieba\n",
    "\n",
    "trainset = WaimaiDataset(\n",
    "    train_data,\n",
    "    parameters[\"max_seq_len\"],\n",
    "    use_jieba=use_jieba\n",
    ")\n",
    "validset = WaimaiDataset(\n",
    "    validation_data,\n",
    "    parameters[\"max_seq_len\"], \n",
    "    use_jieba=use_jieba\n",
    ")\n",
    "testset = WaimaiDataset(\n",
    "    test_data, \n",
    "    parameters[\"max_seq_len\"], \n",
    "    use_jieba=use_jieba\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d98e478-770e-4d23-9f76-b20b3b88b906",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 10. 整理 batch 的資料 (定義 function)\n",
    "\n",
    "def collate_batch(batch):\n",
    "    # 抽每一個 batch 的第 0 個(注意順序)\n",
    "    text = [i[0] for i in batch]\n",
    "    # 進行 padding\n",
    "    text = pad_sequence(text, batch_first=True)\n",
    "    \n",
    "    # 抽每一個 batch 的第 1 個(注意順序)\n",
    "    label = [i[1] for i in batch]\n",
    "    # 把每一個 batch 的答案疊成一個 tensor\n",
    "    label = torch.stack(label)\n",
    "    \n",
    "    return text, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4540bea8-399e-4759-ab63-351040f20042",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 11. 建立資料分批 (mini-batches)\n",
    "\n",
    "# 因為會針對 trainloader 進行 shuffle\n",
    "# 所以在這個 cell 也執行一次 set_seed\n",
    "# 對 trainloader 進行 shuffle 有助於降低 overfitting\n",
    "set_seed(seed)\n",
    "\n",
    "trainloader = DataLoader(\n",
    "    trainset,\n",
    "    batch_size=parameters[\"batch_size\"],\n",
    "    collate_fn=collate_batch,\n",
    "    shuffle=True\n",
    ")\n",
    "validloader = DataLoader(\n",
    "    validset,\n",
    "    batch_size=parameters[\"batch_size\"],\n",
    "    collate_fn=collate_batch,\n",
    "    shuffle=False\n",
    ")\n",
    "testloader = DataLoader(\n",
    "    testset,\n",
    "    batch_size=parameters[\"batch_size\"],\n",
    "    collate_fn=collate_batch,\n",
    "    shuffle=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4670a6c-7747-423e-9008-293b44b8a79c",
   "metadata": {},
   "source": [
    "## 建立模型\n",
    "![Imgur](https://i.imgur.com/OgLBBm7.png)\n",
    "- 模型建置的流程如上圖所示\n",
    "- 文字的部份會透過 Dataset 及 DataLoader 進行處理\n",
    "- embedding 層經由 nn.embedding 來實現 embedding lookup 的功能\n",
    "- embedding 層再接上模型，最後接上分類層，即可進行分類任務\n",
    "- 本範例提供的 Model class 可以藉由更換 module_name 來呼叫不同的 RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d077a1a8-b1e7-4f94-9c1e-001bb04cf5c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 12. 建立 RNN 模型 (定義 class)\n",
    "\n",
    "class RNNModel(torch.nn.Module):\n",
    "    def __init__(self, args):\n",
    "        \"\"\"定義能夠處理句子分類任務的 RNN 模型架構\n",
    "        Arguments:\n",
    "            - args (dict): 所需要的模型參數 (parameters)\n",
    "        Returns:\n",
    "            - None\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        # 模型參數\n",
    "        self.padding_idx = args[\"padding_idx\"]\n",
    "        self.vocab_size = args[\"vocab_size\"]\n",
    "        self.embed_dim = args[\"embed_dim\"]\n",
    "        self.hidden_dim = args[\"hidden_dim\"]\n",
    "        self.module_name = args[\"module_name\"]\n",
    "        self.num_layers = args[\"num_layers\"]\n",
    "        self.dropout = args[\"dropout\"]\n",
    "\n",
    "        # 定義 Embedding 層\n",
    "        self.embedding_layer = torch.nn.Embedding(\n",
    "            num_embeddings=self.vocab_size,\n",
    "            embedding_dim=self.embed_dim,\n",
    "            padding_idx=self.padding_idx\n",
    "        )\n",
    "        # 定義 dropout 層\n",
    "        self.embedding_dropout = torch.nn.Dropout(self.dropout)\n",
    "\n",
    "        # 使用 RNN 系列的模型 (RNN/GRU/LSTM)\n",
    "        module = self.get_hidden_layer_module()\n",
    "        self.hidden_layer = module(\n",
    "            input_size=self.embed_dim,\n",
    "            hidden_size=self.hidden_dim,\n",
    "            num_layers=self.num_layers,\n",
    "            bidirectional=True,\n",
    "            batch_first=True,\n",
    "            dropout=self.dropout\n",
    "        )\n",
    "        # 將模型輸出到 output space\n",
    "        self.output_layer = torch.nn.Linear(\n",
    "            # 因為是 bi-directional，所以 self.hidden_dim*2\n",
    "            in_features=self.hidden_dim*2,\n",
    "            out_features=1\n",
    "        )\n",
    "\n",
    "    def get_hidden_layer_module(self):\n",
    "        \"\"\"根據指定的 module_name 回傳所使用的 PyTorch RNN 系列模型\n",
    "        Arguments:\n",
    "            - module_name (str): 模型名稱，選項為 rnn, gru, lstm\n",
    "        Returns:\n",
    "            - PyTorch 的模型模組 torch.nn.Module\n",
    "        \"\"\"\n",
    "        if self.module_name == \"rnn\":\n",
    "            return torch.nn.RNN\n",
    "        elif self.module_name == \"lstm\":\n",
    "            return torch.nn.LSTM\n",
    "        elif self.module_name == \"gru\":\n",
    "            return torch.nn.GRU\n",
    "        raise ValueError(\"Invalid module name!\")\n",
    "\n",
    "    def forward(self, X):\n",
    "        \"\"\"定義神經網路的前向傳遞的進行流程\n",
    "        Arguments:\n",
    "            - X: 輸入值，維度為(B, S)，其中 B 為 batch size，S 為 sentence length\n",
    "        Returns:\n",
    "            - logits: 模型的輸出值，維度為(B, 1)，其中 B 為 batch size\n",
    "            - Y: 模型的輸出值但經過非線性轉換 (這邊是用 sigmoid)，維度為(B, 1)，其中 B 為 batch size\n",
    "        \"\"\"\n",
    "        # 維度: (B, S) -> (B, S, E)\n",
    "        # B: batch size; S: sentence length; E: embedding dimension\n",
    "        E = self.embedding_layer(X)\n",
    "        E = self.embedding_dropout(E)\n",
    "        \n",
    "        # 使用 RNN 系列\n",
    "        H_out, H_n = self.hidden_layer(E)\n",
    "\n",
    "        # 取第一個和最後一個 hidden states做相加 (bi-directional)\n",
    "        logits = self.output_layer(H_out[:, -1, :]+H_out[:, 0, :])\n",
    "        Y = torch.sigmoid(logits)\n",
    "\n",
    "        return logits, Y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d25fd96-0f9c-45b3-b1b5-fe895d4d25f2",
   "metadata": {},
   "source": [
    "## 使用 Transformer\n",
    "![Imgur](https://i.imgur.com/58DPGG6.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8685bb8-e7a6-43d3-bc10-4e384dca90e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 13. 建立 Transformer 模型 (定義 class)\n",
    "\n",
    "class Transformer(torch.nn.Module):\n",
    "    def __init__(self, args):\n",
    "        \"\"\"定義能夠處理句子分類任務的 Transformer encoder 模型架構\n",
    "        Arguments:\n",
    "            - args (dict): 所需要的模型參數 (parameters)        \n",
    "        Returns:\n",
    "            - None\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        # 模型參數\n",
    "        self.padding_idx = args[\"padding_idx\"]\n",
    "        self.vocab_size = args[\"vocab_size\"]\n",
    "        self.embed_dim = args[\"embed_dim\"]\n",
    "        self.hidden_dim = args[\"hidden_dim\"]\n",
    "        self.num_layers = args[\"num_layers\"]\n",
    "        self.nhead = args[\"nhead\"]\n",
    "        self.dropout = args[\"dropout\"]\n",
    "        \n",
    "        # 定義 Embedding 層\n",
    "        self.embedding_layer = torch.nn.Embedding(\n",
    "            num_embeddings=self.vocab_size,\n",
    "            embedding_dim=self.embed_dim,\n",
    "            padding_idx=self.padding_idx\n",
    "        )\n",
    "        # 定義 dropout 層\n",
    "        self.embedding_dropout = torch.nn.Dropout(self.dropout)\n",
    "\n",
    "        # 定義 Positional Encoding\n",
    "        self.pos_encoder = PositionalEncoding(\n",
    "            d_model=self.embed_dim,\n",
    "            dropout=self.dropout\n",
    "        )\n",
    "        encoder_layer = TransformerEncoderLayer(\n",
    "            d_model=self.embed_dim,\n",
    "            nhead=self.nhead,\n",
    "            dim_feedforward=self.hidden_dim,\n",
    "            dropout=self.dropout\n",
    "        )\n",
    "        self.transformer_encoder = TransformerEncoder(\n",
    "            encoder_layer=encoder_layer, \n",
    "            num_layers=self.num_layers\n",
    "        )\n",
    "        self.linear_layer = torch.nn.Linear(\n",
    "            in_features=self.embed_dim,\n",
    "            out_features=self.embed_dim\n",
    "        )\n",
    "        self.output_layer = torch.nn.Linear(\n",
    "            in_features=self.embed_dim,\n",
    "            out_features=1\n",
    "        )\n",
    "\n",
    "    def forward(self, X):\n",
    "        \"\"\"定義神經網路的前向傳遞的進行流程\n",
    "        Arguments:\n",
    "            - X: 輸入值，維度為(B, S)，其中 B 為 batch size，S 為 sentence length\n",
    "        Returns:\n",
    "            - logits: 模型的輸出值，維度為(B, 1)，其中 B 為 batch size\n",
    "            - Y: 模型的輸出值但經過非線性轉換 (這邊是用 sigmoid)，維度為(B, 1)，其中 B 為 batch size\n",
    "        \"\"\"\n",
    "        # 維度: (B * S) -> (B * S * E)\n",
    "        # B: batch size; S: sentence length; E: embedding dimension\n",
    "        E = self.embedding_layer(X)\n",
    "        E = self.embedding_dropout(E)\n",
    "\n",
    "        # 使用 Transformer\n",
    "        # 輸出維度為 (B, S, E)\n",
    "        E = self.pos_encoder(E)\n",
    "        # 輸出維度為 (B, S, E)\n",
    "        E = self.transformer_encoder(E)\n",
    "        # 輸出維度為 (B, S, E)\n",
    "        H_out = self.linear_layer(E)\n",
    "        # 輸出維度為 (B, S, E)\n",
    "        \n",
    "        # 取第一個 hidden state\n",
    "        logits = self.output_layer(H_out[:, 0, :])\n",
    "        Y = torch.sigmoid(logits)\n",
    "        \n",
    "        return logits, Y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "909234cc",
   "metadata": {},
   "source": [
    "## Positional Encoding\n",
    "- 功能: Transformer 使用 self-attention 機制中沒有考慮到序列順序，因此以 Positional Encoding 來加入順序資訊\n",
    "- 數學公式如下所示\n",
    "\n",
    "$PE_{pos, 2i}=sin(pos/10000^{2i/d_{model}})$\n",
    "\n",
    "$PE_{pos, 2i+1}=cos(pos/10000^{2i/d_{model}})$\n",
    "\n",
    "- 其中 d_model 是 embedding 的維度，i 是 embedding 的 index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc7b9ba8-0d53-4daa-b234-03d3b61154ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 14. Positional Encoding (定義 class)\n",
    "\n",
    "class PositionalEncoding(torch.nn.Module):\n",
    "    r\"\"\"Inject some information about the relative or absolute position of the tokens\n",
    "        in the sequence. The positional encodings have the same dimension as\n",
    "        the embeddings, so that the two can be summed. Here, we use sine and cosine\n",
    "        functions of different frequencies.\n",
    "\n",
    "    Args:\n",
    "        d_model: the embed dim (required).\n",
    "        dropout: the dropout value (default=0.1).\n",
    "        max_len: the max. length of the incoming sequence (default=5000).\n",
    "    Examples:\n",
    "        >>> pos_encoder = PositionalEncoding(d_model)\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, d_model, dropout=0.1, max_len=5000):\n",
    "        super().__init__()\n",
    "        self.dropout = torch.nn.Dropout(p=dropout)\n",
    "\n",
    "        pe = torch.zeros(max_len, d_model)\n",
    "        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n",
    "        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n",
    "        pe[:, 0::2] = torch.sin(position * div_term)\n",
    "        pe[:, 1::2] = torch.cos(position * div_term)\n",
    "        pe = pe.unsqueeze(0).transpose(0, 1)\n",
    "        self.register_buffer('pe', pe)\n",
    "\n",
    "    def forward(self, x):\n",
    "        r\"\"\"Inputs of forward function\n",
    "        Args:\n",
    "            x: the sequence fed to the positional encoder model (required).\n",
    "        Shape:\n",
    "            x: [sequence length, batch size, embed dim]\n",
    "            output: [sequence length, batch size, embed dim]\n",
    "        Examples:\n",
    "            >>> output = pos_encoder(x)\n",
    "        \"\"\"\n",
    "\n",
    "        x = x + self.pe[:x.size(0), :]\n",
    "        return self.dropout(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0fcde55-fd80-4995-bce2-cd5e2092d5e9",
   "metadata": {},
   "source": [
    "## 設定訓練流程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07b2579d-b998-4d48-9ad3-52f780823872",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 15. 設定訓練流程 (定義 function)\n",
    "\n",
    "def train(trainloader, model, optimizer):\n",
    "    \"\"\"定義訓練時的進行流程\n",
    "    Arguments:\n",
    "        - trainloader: 具備 mini-batches 的 dataset，由 PyTorch DataLoader 所建立\n",
    "        - model: 要進行訓練的模型\n",
    "        - optimizer: 最佳化目標函數的演算法\n",
    "    Returns:\n",
    "        - train_loss: 模型在一個 epoch 的 training loss\n",
    "    \"\"\"\n",
    "    # 設定模型的訓練模式\n",
    "    model.train()\n",
    "    \n",
    "    # 記錄一個 epoch中 training 過程的 loss\n",
    "    train_loss = 0\n",
    "    # 從 trainloader 一次一次抽\n",
    "    for x, y in trainloader:\n",
    "        # 將變數丟到指定的裝置位置\n",
    "        x = x.to(device)\n",
    "        y = y.to(device)\n",
    "\n",
    "        # 重新設定模型的梯度\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # 1. 前向傳遞 (Forward Pass)\n",
    "        logits, pred = model(x)\n",
    "\n",
    "        # 2. 計算 loss (loss function 為二元交叉熵)\n",
    "        loss_fn = torch.nn.BCELoss()\n",
    "        loss = loss_fn(pred.squeeze(-1), y)\n",
    "\n",
    "        # 3. 計算反向傳播的梯度\n",
    "        loss.backward()\n",
    "        # 4. \"更新\"模型的權重\n",
    "        optimizer.step()\n",
    "\n",
    "        # 一個 epoch 會抽很多次 batch，所以每個 batch 計算完都要加起來\n",
    "        # .item() 在 PyTorch 中可以獲得該 tensor 的數值\n",
    "        train_loss += loss.item()\n",
    "\n",
    "    return train_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4ba00e6-0263-4b81-be42-359de3257859",
   "metadata": {},
   "source": [
    "## 設定驗證流程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e751e9b5-7628-4320-8b20-917bb0f3087f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 16. 設定驗證流程 (定義 function)\n",
    "\n",
    "def evaluate(dataloader, model):\n",
    "    \"\"\"定義驗證時的進行流程\n",
    "    Arguments:\n",
    "        - dataloader: 具備 mini-batches 的 dataset，由 PyTorch DataLoader 所建立\n",
    "        - model: 要進行驗證的模型\n",
    "    Returns:\n",
    "        - loss: 模型在驗證/測試集的 loss\n",
    "        - acc: 模型在驗證/測試集的正確率\n",
    "    \"\"\"\n",
    "    # 設定模型的驗證模式\n",
    "    # 此時 dropout 會自動關閉\n",
    "    model.eval()\n",
    "    \n",
    "    # 設定現在不計算梯度\n",
    "    with torch.no_grad():\n",
    "        # 把每個 batch 的 label 儲存成一維 tensor\n",
    "        y_true = torch.tensor([])\n",
    "        y_pred = torch.tensor([])\n",
    "\n",
    "        # 從 dataloader 一次一次抽\n",
    "        for x, y in dataloader:\n",
    "            # 把正確的 label concat 起來\n",
    "            y_true = torch.cat([y_true, y])\n",
    "\n",
    "            x = x.to(device)\n",
    "            y = y.to(device)\n",
    "\n",
    "\n",
    "            _, pred = model(x)\n",
    "            # 預測的數值大於 0.5 則視為類別1，反之為類別0\n",
    "            pred = (pred>0.5)*1\n",
    "            # 把預測的 label concat 起來\n",
    "            # 注意: 如果使用 gpu 計算的話，要先用 .cpu 把 tensor 轉回 cpu\n",
    "            y_pred = torch.cat([y_pred, pred.cpu()])\n",
    "            \n",
    "    # 計算 loss (loss function 為二元交叉熵)\n",
    "    loss_fn = torch.nn.BCELoss()\n",
    "    # 模型輸出的維度是 (B, 1)，使用.squeeze(-1)可以讓維度變 (B,)\n",
    "    loss = loss_fn(y_pred.squeeze(-1), y_true)\n",
    "    # 計算正確率\n",
    "    acc = accuracy_score(y_true, y_pred.squeeze(-1))\n",
    "            \n",
    "    return loss, acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0fec17d-6bb5-4e48-b96d-ce0800d75c52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 17. 執行訓練所需要的準備工作\n",
    "\n",
    "set_seed(seed)\n",
    "\n",
    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
    "\n",
    "if parameters[\"module_name\"] == 'transformer':\n",
    "    model = Transformer(args=parameters)\n",
    "else:\n",
    "    model = RNNModel(args=parameters)\n",
    "\n",
    "model = model.to(device)\n",
    "\n",
    "# 使用 Adam 的演算法進行最佳化\n",
    "opt = torch.optim.Adam(\n",
    "    model.parameters(),\n",
    "    lr=parameters[\"learning_rate\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9557c9-c97d-4111-be93-2f256cd113f0",
   "metadata": {},
   "source": [
    "## 開始訓練"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "694f6cfd-9002-4fb5-8e98-f73e267970ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 18. 整個訓練及驗證過程的 script\n",
    "\n",
    "train_loss_history = []\n",
    "valid_loss_history = []\n",
    "\n",
    "for epoch in range(parameters[\"epochs\"]):\n",
    "    train_loss = train(\n",
    "        trainloader,\n",
    "        model,\n",
    "        optimizer=opt\n",
    "    )\n",
    "    \n",
    "    print(\"Training loss at epoch {} is {}.\".format(epoch+1, train_loss))\n",
    "    train_loss_history.append(train_loss)\n",
    "    \n",
    "    if epoch % 2 == 1:\n",
    "        print(\"=====Start validation=====\")\n",
    "        valid_loss, valid_acc = evaluate(\n",
    "            dataloader=validloader,\n",
    "            model=model\n",
    "        )\n",
    "        valid_loss_history.append(valid_loss)\n",
    "        print(\"Validation accuracy at epoch {} is {}, and validation loss is {}.\"\\\n",
    "              .format(epoch+1, valid_acc, valid_loss))\n",
    "        \n",
    "    torch.save(model.state_dict(), \"model_epoch_{}.pkl\".format(epoch))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08cafd4e-a1e7-4fcf-9fdd-7dec429fb1d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 19. 預測測試集\n",
    "\n",
    "best_epoch = np.argmin(valid_loss_history)\n",
    "model.load_state_dict(\n",
    "    torch.load(\"model_epoch_{}.pkl\".format(best_epoch))\n",
    ")\n",
    "\n",
    "print(\"=====Start testing=====\")\n",
    "test_loss, test_acc = evaluate(testloader, model)\n",
    "print(\"Testing accuracy is {}.\".format(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d892d47a-db39-4c8b-9353-675468264c17",
   "metadata": {},
   "source": [
    "## 任務完成，恭喜你學會了如何使用 `PyTorch` 來搭建神經網路\n",
    "- 若想學習如何使用 RNN 來生成文字，請參考 [char-RNN-文本生成](./char-RNN-文本生成.ipynb) 教學。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24d3ec68-3541-4920-827f-75dc2da976dd",
   "metadata": {},
   "source": [
    "## 什麼？你想使用 BERT？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fa0fd2a-bb11-48ca-b2ef-122342df610f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install simpletransformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47157c4a-db46-45e3-9140-f4ce47d4655f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 載入所需的資料格式\n",
    "\n",
    "train_bert = pd.DataFrame({\n",
    "    \"text\": train_data[\"review\"],\n",
    "    \"label\": train_data[\"label\"]\n",
    "})\n",
    "valid_bert = pd.DataFrame({\n",
    "    \"text\": validation_data[\"review\"],\n",
    "    \"label\": validation_data[\"label\"]\n",
    "})\n",
    "test_bert = list(test_data[\"review\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "688384bf-55de-49d0-99ef-d6413e222321",
   "metadata": {},
   "outputs": [],
   "source": [
    "from simpletransformers.classification import ClassificationModel, ClassificationArgs\n",
    "\n",
    "\n",
    "model_args = ClassificationArgs(sliding_window=True)\n",
    "\n",
    "model = ClassificationModel(\n",
    "    \"bert\",\n",
    "    \"bert-base-chinese\",\n",
    "    args=model_args,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "487c6984-2e05-48bb-b6cd-108ef237e8bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train_model(\n",
    "    train_df=train_bert,\n",
    "    eval_df=valid_bert\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4f9a656-2079-4821-abc0-e2d50303e1ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = model.predict(test_bert)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "899e44bb-60a7-4f25-80ca-fe05b6685105",
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_score(test_data['label'], result[0])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "c722923b96b9c31396b2182f935fa631109324cb0f0f8144167b2ddca282865c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}