hw4_RNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hw4_RNN.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Iallen520/lhy_DL_Hw/blob/master/hw4_RNN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r67y9UpchZ38",
        "colab_type": "text"
      },
      "source": [
        "# Recurrent Neural Networks\n",
        "\n",
        "本次作業是要讓同學接觸NLP當中一個簡單的task——句子分類(文本分類)\n",
        "\n",
        "給定一個句子，判斷他有沒有惡意(負面標1，正面標0)\n",
        "\n",
        "若有任何問題，歡迎來信至助教信箱ntu-ml-2020spring-ta@googlegroups.com"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ajS_WskRo0S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# from google.colab import drive\n",
        "# drive.mount('/content/drive')\n",
        "# path_prefix = 'drive/My Drive/Colab Notebooks/hw4 - Recurrent Neural Network'\n",
        "path_prefix = './'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9YrAlczfM_w6",
        "colab_type": "text"
      },
      "source": [
        "### Download Dataset\n",
        "有三個檔案，分別是training_label.txt、training_nolabel.txt、testing_data.txt\n",
        "\n",
        "training_label.txt：有label的training data(句子配上0 or 1)\n",
        "\n",
        "training_nolabel.txt：沒有label的training data(只有句子)，用來做semi-supervise learning\n",
        "\n",
        "testing_data.txt：你要判斷testing data裡面的句子是0 or 1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x2gwKORmuViJ",
        "colab_type": "code",
        "outputId": "4db7c645-cd05-4f90-8826-e9a45d40440b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "source": [
        "# !wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=1dPHIl8ZnfDz_fxNd2ZeBYedTat2lfxcO' -O 'drive/My Drive/Colab Notebooks/hw8-RNN/data/training_label.txt'\n",
        "# !wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=1x1rJOX_ETqnOZjdMAbEE2pqIjRNa8xcc' -O 'drive/My Drive/Colab Notebooks/hw8-RNN/data/training_nolabel.txt'\n",
        "# !wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=16CtnQwSDCob9xmm6EdHHR7PNFNiOrQ30' -O 'drive/My Drive/Colab Notebooks/hw8-RNN/data/testing_data.txt'\n",
        "\n",
        "!gdown --id '1lz0Wtwxsh5YCPdqQ3E3l_nbfJT1N13V8' --output data.zip\n",
        "!unzip data.zip\n",
        "!ls\n",
        "\n",
        "# !ls 'drive/My Drive/Colab Notebooks/hw4 - Recurrent Neural Network/data'"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1lz0Wtwxsh5YCPdqQ3E3l_nbfJT1N13V8\n",
            "To: /content/data.zip\n",
            "45.1MB [00:00, 203MB/s]\n",
            "Archive:  data.zip\n",
            "  inflating: training_label.txt      \n",
            "  inflating: testing_data.txt        \n",
            "  inflating: training_nolabel.txt    \n",
            "data.zip     testing_data.txt\t training_nolabel.txt\n",
            "sample_data  training_label.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8hDIokoP6464",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# this is for filtering the warnings\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fc143hSvNGr6",
        "colab_type": "text"
      },
      "source": [
        "### Utils"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ICDIhhgCY2-M",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# utils.py\n",
        "# 這個block用來先定義一些等等常用到的函式\n",
        "import torch\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "\n",
        "def load_training_data(path='training_label.txt'):\n",
        "    # 把training時需要的data讀進來\n",
        "    # 如果是'training_label.txt'，需要讀取label，如果是'training_nolabel.txt'，不需要讀取label\n",
        "    if 'training_label' in path:\n",
        "        with open(path, 'r') as f:\n",
        "            lines = f.readlines()\n",
        "            lines = [line.strip('\\n').split(' ') for line in lines]\n",
        "        x = [line[2:] for line in lines]\n",
        "        y = [line[0] for line in lines]\n",
        "        return x, y\n",
        "    else:\n",
        "        with open(path, 'r') as f:\n",
        "            lines = f.readlines()\n",
        "            x = [line.strip('\\n').split(' ') for line in lines]\n",
        "        return x\n",
        "\n",
        "def load_testing_data(path='testing_data'):\n",
        "    # 把testing時需要的data讀進來\n",
        "    with open(path, 'r') as f:\n",
        "        lines = f.readlines()\n",
        "        X = [\"\".join(line.strip('\\n').split(\",\")[1:]).strip() for line in lines[1:]]\n",
        "        X = [sen.split(' ') for sen in X]\n",
        "    return X\n",
        "\n",
        "def evaluation(outputs, labels):\n",
        "    #outputs => probability (float)\n",
        "    #labels => labels\n",
        "    outputs[outputs>=0.5] = 1 # 大於等於0.5為有惡意\n",
        "    outputs[outputs<0.5] = 0 # 小於0.5為無惡意\n",
        "    correct = torch.sum(torch.eq(outputs, labels)).item()\n",
        "    return correct"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oYE8UYQsNIxM",
        "colab_type": "text"
      },
      "source": [
        "### Train Word to Vector"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "cgGWaF8_2S3q",
        "outputId": "d25efb6b-de4f-43f2-8a98-e0aaa47f704e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "# w2v.py\n",
        "# 這個block是用來訓練word to vector 的 word embedding\n",
        "# 注意！這個block在訓練word to vector時是用cpu，可能要花到10分鐘以上\n",
        "import os\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import argparse\n",
        "from gensim.models import word2vec\n",
        "\n",
        "def train_word2vec(x):\n",
        "    # 訓練word to vector 的 word embedding\n",
        "    model = word2vec.Word2Vec(x, size=250, window=5, min_count=5, workers=12, iter=10, sg=1)\n",
        "    return model\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    print(\"loading training data ...\")\n",
        "    train_x, y = load_training_data('training_label.txt')\n",
        "    train_x_no_label = load_training_data('training_nolabel.txt')\n",
        "\n",
        "    print(\"loading testing data ...\")\n",
        "    test_x = load_testing_data('testing_data.txt')\n",
        "\n",
        "    model = train_word2vec(train_x + train_x_no_label + test_x)\n",
        "    \n",
        "    print(\"saving model ...\")\n",
        "    # model.save(os.path.join(path_prefix, 'model/w2v_all.model'))\n",
        "    model.save(os.path.join(path_prefix, 'w2v_all.model'))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loading training data ...\n",
            "loading testing data ...\n",
            "saving model ...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3wHLtS0wNR6w",
        "colab_type": "text"
      },
      "source": [
        "### Data Preprocess"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CfGKiOitk5ob",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# preprocess.py\n",
        "# 這個block用來做data的預處理\n",
        "from torch import nn\n",
        "from gensim.models import Word2Vec\n",
        "\n",
        "class Preprocess():\n",
        "    def __init__(self, sentences, sen_len, w2v_path=\"./w2v.model\"):\n",
        "        self.w2v_path = w2v_path\n",
        "        self.sentences = sentences\n",
        "        self.sen_len = sen_len\n",
        "        self.idx2word = []\n",
        "        self.word2idx = {}\n",
        "        self.embedding_matrix = []\n",
        "    def get_w2v_model(self):\n",
        "        # 把之前訓練好的word to vec 模型讀進來\n",
        "        self.embedding = Word2Vec.load(self.w2v_path)\n",
        "        self.embedding_dim = self.embedding.vector_size\n",
        "    def add_embedding(self, word):\n",
        "        # 把word加進embedding，並賦予他一個隨機生成的representation vector\n",
        "        # word只會是\"<PAD>\"或\"<UNK>\"\n",
        "        vector = torch.empty(1, self.embedding_dim)\n",
        "        torch.nn.init.uniform_(vector)\n",
        "        self.word2idx[word] = len(self.word2idx)\n",
        "        self.idx2word.append(word)\n",
        "        self.embedding_matrix = torch.cat([self.embedding_matrix, vector], 0)\n",
        "    def make_embedding(self, load=True):\n",
        "        print(\"Get embedding ...\")\n",
        "        # 取得訓練好的 Word2vec word embedding\n",
        "        if load:\n",
        "            print(\"loading word to vec model ...\")\n",
        "            self.get_w2v_model()\n",
        "        else:\n",
        "            raise NotImplementedError\n",
        "        # 製作一個 word2idx 的 dictionary\n",
        "        # 製作一個 idx2word 的 list\n",
        "        # 製作一個 word2vector 的 list\n",
        "        for i, word in enumerate(self.embedding.wv.vocab):\n",
        "            print('get words #{}'.format(i+1), end='\\r')\n",
        "            #e.g. self.word2index['魯'] = 1 \n",
        "            #e.g. self.index2word[1] = '魯'\n",
        "            #e.g. self.vectors[1] = '魯' vector\n",
        "            self.word2idx[word] = len(self.word2idx)\n",
        "            self.idx2word.append(word)\n",
        "            self.embedding_matrix.append(self.embedding[word])\n",
        "        print('')\n",
        "        self.embedding_matrix = torch.tensor(self.embedding_matrix)\n",
        "        # 將\"<PAD>\"跟\"<UNK>\"加進embedding裡面\n",
        "        self.add_embedding(\"<PAD>\")\n",
        "        self.add_embedding(\"<UNK>\")\n",
        "        print(\"total words: {}\".format(len(self.embedding_matrix)))\n",
        "        return self.embedding_matrix\n",
        "    def pad_sequence(self, sentence):\n",
        "        # 將每個句子變成一樣的長度\n",
        "        if len(sentence) > self.sen_len:\n",
        "            sentence = sentence[:self.sen_len]\n",
        "        else:\n",
        "            pad_len = self.sen_len - len(sentence)\n",
        "            for _ in range(pad_len):\n",
        "                sentence.append(self.word2idx[\"<PAD>\"])\n",
        "        assert len(sentence) == self.sen_len\n",
        "        return sentence\n",
        "    def sentence_word2idx(self):\n",
        "        # 把句子裡面的字轉成相對應的index\n",
        "        sentence_list = []\n",
        "        for i, sen in enumerate(self.sentences):\n",
        "            print('sentence count #{}'.format(i+1), end='\\r')\n",
        "            sentence_idx = []\n",
        "            for word in sen:\n",
        "                if (word in self.word2idx.keys()):\n",
        "                    sentence_idx.append(self.word2idx[word])\n",
        "                else:\n",
        "                    sentence_idx.append(self.word2idx[\"<UNK>\"])\n",
        "            # 將每個句子變成一樣的長度\n",
        "            sentence_idx = self.pad_sequence(sentence_idx)\n",
        "            sentence_list.append(sentence_idx)\n",
        "        return torch.LongTensor(sentence_list)\n",
        "    def labels_to_tensor(self, y):\n",
        "        # 把labels轉成tensor\n",
        "        y = [int(label) for label in y]\n",
        "        return torch.LongTensor(y)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3WJB7go5NWL0",
        "colab_type": "text"
      },
      "source": [
        "### Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XketwKs4lFfB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# data.py\n",
        "# 實作了dataset所需要的'__init__', '__getitem__', '__len__'\n",
        "# 好讓dataloader能使用\n",
        "import torch\n",
        "from torch.utils import data\n",
        "\n",
        "class TwitterDataset(data.Dataset):\n",
        "    \"\"\"\n",
        "    Expected data shape like:(data_num, data_len)\n",
        "    Data can be a list of numpy array or a list of lists\n",
        "    input data shape : (data_num, seq_len, feature_dim)\n",
        "    \n",
        "    __len__ will return the number of data\n",
        "    \"\"\"\n",
        "    def __init__(self, X, y):\n",
        "        self.data = X\n",
        "        self.label = y\n",
        "    def __getitem__(self, idx):\n",
        "        if self.label is None: return self.data[idx]\n",
        "        return self.data[idx], self.label[idx]\n",
        "    def __len__(self):\n",
        "        return len(self.data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uNJ8xWIMNa2r",
        "colab_type": "text"
      },
      "source": [
        "### Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZS6RJADulIq1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# model.py\n",
        "# 這個block是要拿來訓練的模型\n",
        "import torch\n",
        "from torch import nn\n",
        "class LSTM_Net(nn.Module):\n",
        "    def __init__(self, embedding, embedding_dim, hidden_dim, num_layers, dropout=0.5, fix_embedding=True):\n",
        "        super(LSTM_Net, self).__init__()\n",
        "        # 製作 embedding layer\n",
        "        self.embedding = torch.nn.Embedding(embedding.size(0),embedding.size(1))\n",
        "        self.embedding.weight = torch.nn.Parameter(embedding)\n",
        "        # 是否將 embedding fix住，如果fix_embedding為False，在訓練過程中，embedding也會跟著被訓練\n",
        "        self.embedding.weight.requires_grad = False if fix_embedding else True\n",
        "        self.embedding_dim = embedding.size(1)\n",
        "        self.hidden_dim = hidden_dim\n",
        "        self.num_layers = num_layers\n",
        "        self.dropout = dropout\n",
        "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=num_layers, batch_first=True)\n",
        "        self.classifier = nn.Sequential( nn.Dropout(dropout),\n",
        "                                         nn.Linear(hidden_dim, 1),\n",
        "                                         nn.Sigmoid() )\n",
        "    def forward(self, inputs):\n",
        "        inputs = self.embedding(inputs)\n",
        "        x, _ = self.lstm(inputs, None)\n",
        "        # x 的 dimension (batch, seq_len, hidden_size)\n",
        "        # 取用 LSTM 最後一層的 hidden state\n",
        "        x = x[:, -1, :] \n",
        "        x = self.classifier(x)\n",
        "        return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aWlpEL0sNc10",
        "colab_type": "text"
      },
      "source": [
        "### Train"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4QR4MMz-lR7i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# train.py\n",
        "# 這個block是用來訓練模型的\n",
        "import torch\n",
        "from torch import nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "\n",
        "def training(batch_size, n_epoch, lr, model_dir, train, valid, model, device):\n",
        "    total = sum(p.numel() for p in model.parameters())\n",
        "    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "    print('\\nstart training, parameter total:{}, trainable:{}\\n'.format(total, trainable))\n",
        "    model.train() # 將model的模式設為train，這樣optimizer就可以更新model的參數\n",
        "    criterion = nn.BCELoss() # 定義損失函數，這裡我們使用binary cross entropy loss\n",
        "    t_batch = len(train) \n",
        "    v_batch = len(valid) \n",
        "    optimizer = optim.Adam(model.parameters(), lr=lr) # 將模型的參數給optimizer，並給予適當的learning rate\n",
        "    total_loss, total_acc, best_acc = 0, 0, 0\n",
        "    for epoch in range(n_epoch):\n",
        "        total_loss, total_acc = 0, 0\n",
        "        # 這段做training\n",
        "        for i, (inputs, labels) in enumerate(train):\n",
        "            inputs = inputs.to(device, dtype=torch.long) # device為\"cuda\"，將inputs轉成torch.cuda.LongTensor\n",
        "            labels = labels.to(device, dtype=torch.float) # device為\"cuda\"，將labels轉成torch.cuda.FloatTensor，因為等等要餵進criterion，所以型態要是float\n",
        "            optimizer.zero_grad() # 由於loss.backward()的gradient會累加，所以每次餵完一個batch後需要歸零\n",
        "            outputs = model(inputs) # 將input餵給模型\n",
        "            outputs = outputs.squeeze() # 去掉最外面的dimension，好讓outputs可以餵進criterion()\n",
        "            loss = criterion(outputs, labels) # 計算此時模型的training loss\n",
        "            loss.backward() # 算loss的gradient\n",
        "            optimizer.step() # 更新訓練模型的參數\n",
        "            correct = evaluation(outputs, labels) # 計算此時模型的training accuracy\n",
        "            total_acc += (correct / batch_size)\n",
        "            total_loss += loss.item()\n",
        "            print('[ Epoch{}: {}/{} ] loss:{:.3f} acc:{:.3f} '.format(\n",
        "            \tepoch+1, i+1, t_batch, loss.item(), correct*100/batch_size), end='\\r')\n",
        "        print('\\nTrain | Loss:{:.5f} Acc: {:.3f}'.format(total_loss/t_batch, total_acc/t_batch*100))\n",
        "\n",
        "        # 這段做validation\n",
        "        model.eval() # 將model的模式設為eval，這樣model的參數就會固定住\n",
        "        with torch.no_grad():\n",
        "            total_loss, total_acc = 0, 0\n",
        "            for i, (inputs, labels) in enumerate(valid):\n",
        "                inputs = inputs.to(device, dtype=torch.long) # device為\"cuda\"，將inputs轉成torch.cuda.LongTensor\n",
        "                labels = labels.to(device, dtype=torch.float) # device為\"cuda\"，將labels轉成torch.cuda.FloatTensor，因為等等要餵進criterion，所以型態要是float\n",
        "                outputs = model(inputs) # 將input餵給模型\n",
        "                outputs = outputs.squeeze() # 去掉最外面的dimension，好讓outputs可以餵進criterion()\n",
        "                loss = criterion(outputs, labels) # 計算此時模型的validation loss\n",
        "                correct = evaluation(outputs, labels) # 計算此時模型的validation accuracy\n",
        "                total_acc += (correct / batch_size)\n",
        "                total_loss += loss.item()\n",
        "\n",
        "            print(\"Valid | Loss:{:.5f} Acc: {:.3f} \".format(total_loss/v_batch, total_acc/v_batch*100))\n",
        "            if total_acc > best_acc:\n",
        "                # 如果validation的結果優於之前所有的結果，就把當下的模型存下來以備之後做預測時使用\n",
        "                best_acc = total_acc\n",
        "                #torch.save(model, \"{}/val_acc_{:.3f}.model\".format(model_dir,total_acc/v_batch*100))\n",
        "                torch.save(model, \"{}/ckpt.model\".format(model_dir))\n",
        "                print('saving model with acc {:.3f}'.format(total_acc/v_batch*100))\n",
        "        print('-----------------------------------------------')\n",
        "        model.train() # 將model的模式設為train，這樣optimizer就可以更新model的參數（因為剛剛轉成eval模式）"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qF5YQrupNfCS",
        "colab_type": "text"
      },
      "source": [
        "### Test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2X2wkdAYxHYA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# test.py\n",
        "# 這個block用來對testing_data.txt做預測\n",
        "import torch\n",
        "from torch import nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "\n",
        "def testing(batch_size, test_loader, model, device):\n",
        "    model.eval()\n",
        "    ret_output = []\n",
        "    with torch.no_grad():\n",
        "        for i, inputs in enumerate(test_loader):\n",
        "            inputs = inputs.to(device, dtype=torch.long)\n",
        "            outputs = model(inputs)\n",
        "            outputs = outputs.squeeze()\n",
        "            outputs[outputs>=0.5] = 1 # 大於等於0.5為負面\n",
        "            outputs[outputs<0.5] = 0 # 小於0.5為正面\n",
        "            ret_output += outputs.int().tolist()\n",
        "    \n",
        "    return ret_output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dfnKj0KXNeoz",
        "colab_type": "text"
      },
      "source": [
        "### Main"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EztIWqCmlZof",
        "colab_type": "code",
        "outputId": "961deee1-4726-445b-cd49-ab4fcc0e14cf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 578
        }
      },
      "source": [
        "# main.py\n",
        "import os\n",
        "import torch\n",
        "import argparse\n",
        "import numpy as np\n",
        "from torch import nn\n",
        "from gensim.models import word2vec\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# 通過torch.cuda.is_available()的回傳值進行判斷是否有使用GPU的環境，如果有的話device就設為\"cuda\"，沒有的話就設為\"cpu\"\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "# 處理好各個data的路徑\n",
        "train_with_label = os.path.join(path_prefix, 'training_label.txt')\n",
        "train_no_label = os.path.join(path_prefix, 'training_nolabel.txt')\n",
        "testing_data = os.path.join(path_prefix, 'testing_data.txt')\n",
        "\n",
        "w2v_path = os.path.join(path_prefix, 'w2v_all.model') # 處理word to vec model的路徑\n",
        "\n",
        "# 定義句子長度、要不要固定embedding、batch大小、要訓練幾個epoch、learning rate的值、model的資料夾路徑\n",
        "sen_len = 30\n",
        "fix_embedding = True # fix embedding during training\n",
        "batch_size = 128\n",
        "epoch = 5\n",
        "lr = 0.001\n",
        "# model_dir = os.path.join(path_prefix, 'model/') # model directory for checkpoint model\n",
        "model_dir = path_prefix # model directory for checkpoint model\n",
        "\n",
        "print(\"loading data ...\") # 把'training_label.txt'跟'training_nolabel.txt'讀進來\n",
        "train_x, y = load_training_data(train_with_label)\n",
        "train_x_no_label = load_training_data(train_no_label)\n",
        "\n",
        "# 對input跟labels做預處理\n",
        "preprocess = Preprocess(train_x, sen_len, w2v_path=w2v_path)\n",
        "embedding = preprocess.make_embedding(load=True)\n",
        "train_x = preprocess.sentence_word2idx()\n",
        "y = preprocess.labels_to_tensor(y)\n",
        "\n",
        "# 製作一個model的對象\n",
        "model = LSTM_Net(embedding, embedding_dim=250, hidden_dim=250, num_layers=1, dropout=0.5, fix_embedding=fix_embedding)\n",
        "model = model.to(device) # device為\"cuda\"，model使用GPU來訓練(餵進去的inputs也需要是cuda tensor)\n",
        "\n",
        "# 把data分為training data跟validation data(將一部份training data拿去當作validation data)\n",
        "X_train, X_val, y_train, y_val = train_x[:190000], train_x[190000:], y[:190000], y[190000:]\n",
        "\n",
        "# 把data做成dataset供dataloader取用\n",
        "train_dataset = TwitterDataset(X=X_train, y=y_train)\n",
        "val_dataset = TwitterDataset(X=X_val, y=y_val)\n",
        "\n",
        "# 把data 轉成 batch of tensors\n",
        "train_loader = torch.utils.data.DataLoader(dataset = train_dataset,\n",
        "                                            batch_size = batch_size,\n",
        "                                            shuffle = True,\n",
        "                                            num_workers = 8)\n",
        "\n",
        "val_loader = torch.utils.data.DataLoader(dataset = val_dataset,\n",
        "                                            batch_size = batch_size,\n",
        "                                            shuffle = False,\n",
        "                                            num_workers = 8)\n",
        "\n",
        "# 開始訓練\n",
        "training(batch_size, epoch, lr, model_dir, train_loader, val_loader, model, device)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loading data ...\n",
            "Get embedding ...\n",
            "loading word to vec model ...\n",
            "get words #55777\n",
            "total words: 55779\n",
            "sentence count #200000\n",
            "start training, parameter total:14447001, trainable:502251\n",
            "\n",
            "\n",
            "Train | Loss:0.48037 Acc: 76.344\n",
            "Valid | Loss:0.43569 Acc: 79.391 \n",
            "saving model with acc 79.391\n",
            "-----------------------------------------------\n",
            "[ Epoch2: 1485/1485 ] loss:0.445 acc:28.906 \n",
            "Train | Loss:0.41297 Acc: 81.141\n",
            "Valid | Loss:0.40639 Acc: 80.736 \n",
            "saving model with acc 80.736\n",
            "-----------------------------------------------\n",
            "\n",
            "Train | Loss:0.39345 Acc: 82.240\n",
            "Valid | Loss:0.39597 Acc: 81.517 \n",
            "saving model with acc 81.517\n",
            "-----------------------------------------------\n",
            "[ Epoch4: 1485/1485 ] loss:0.457 acc:30.469 \n",
            "Train | Loss:0.37718 Acc: 83.130\n",
            "Valid | Loss:0.39099 Acc: 81.646 \n",
            "saving model with acc 81.646\n",
            "-----------------------------------------------\n",
            "\n",
            "Train | Loss:0.36063 Acc: 83.993\n",
            "Valid | Loss:0.39466 Acc: 81.695 \n",
            "saving model with acc 81.695\n",
            "-----------------------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8fQeaQNeNm3L",
        "colab_type": "text"
      },
      "source": [
        "### Predict and Write to csv file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vFvjFQopxVrt",
        "colab_type": "code",
        "outputId": "3bf30696-bf76-463b-fc7f-f0b6375a0e00",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "# 開始測試模型並做預測\n",
        "print(\"loading testing data ...\")\n",
        "test_x = load_testing_data(testing_data)\n",
        "preprocess = Preprocess(test_x, sen_len, w2v_path=w2v_path)\n",
        "embedding = preprocess.make_embedding(load=True)\n",
        "test_x = preprocess.sentence_word2idx()\n",
        "test_dataset = TwitterDataset(X=test_x, y=None)\n",
        "test_loader = torch.utils.data.DataLoader(dataset = test_dataset,\n",
        "                                            batch_size = batch_size,\n",
        "                                            shuffle = False,\n",
        "                                            num_workers = 8)\n",
        "print('\\nload model ...')\n",
        "model = torch.load(os.path.join(model_dir, 'ckpt.model'))\n",
        "outputs = testing(batch_size, test_loader, model, device)\n",
        "\n",
        "# 寫到csv檔案供上傳kaggle\n",
        "tmp = pd.DataFrame({\"id\":[str(i) for i in range(len(test_x))],\"label\":outputs})\n",
        "print(\"save csv ...\")\n",
        "tmp.to_csv(os.path.join(path_prefix, 'predict.csv'), index=False)\n",
        "print(\"Finish Predicting\")\n",
        "\n",
        "# 以下是使用command line上傳到kaggle的方式\n",
        "# 需要先pip install kaggle、Create API Token，詳細請看https://github.com/Kaggle/kaggle-api以及https://www.kaggle.com/code1110/how-to-submit-from-google-colab\n",
        "# kaggle competitions submit [competition-name] -f [csv file path]] -m [message]\n",
        "# ex: kaggle competitions submit ml-2020spring-hw4 -f output/predict.csv -m \"......\""
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loading testing data ...\n",
            "Get embedding ...\n",
            "loading word to vec model ...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "get words #1\rget words #2\rget words #3\rget words #4\rget words #5\rget words #6\rget words #7\rget words #8\rget words #9\rget words #10\rget words #11\rget words #12\rget words #13\rget words #14\rget words #15\rget words #16\rget words #17\rget words #18\rget words #19\rget words #20\rget words #21\rget words #22\rget words #23\rget words #24\rget words #25\rget words #26\rget words #27\rget words #28\rget words #29\rget words #30\rget words #31\rget words #32\rget words #33\rget words #34\rget words #35\rget words #36\rget words #37\rget words #38\rget words #39\rget words #40\rget words #41\rget words #42\rget words #43\rget words #44\rget words #45\rget words #46\rget words #47\rget words #48\rget words #49\rget words #50\rget words #51\rget words #52\rget words #53\rget words #54\rget words #55\rget words #56\rget words #57\rget words #58\rget words #59\rget words #60\rget words #61\rget words #62\rget words #63\rget words #64\rget words #65\rget words #66\rget words #67\rget words #68\rget words #69\rget words #70\rget words #71\rget words #72\rget words #73\rget words #74\rget words #75\rget words #76\rget words #77\rget words #78\rget words #79\rget words #80\rget words #81\rget words #82\rget words #83\rget words #84\rget words #85\rget words #86\rget words #87\rget words #88\rget words #89\rget words #90\rget words #91\rget words #92\rget words #93\rget words #94\rget words #95\rget words #96\rget words #97\rget words #98\rget words #99\rget words #100\rget words #101\rget words #102\rget words #103\rget words #104\rget words #105\rget words #106\rget words #107\rget words #108\rget words #109\rget words #110\rget words #111\rget words #112\rget words #113\rget words #114\rget words #115\rget words #116\rget words #117\rget words #118\rget words #119\rget words #120\rget words #121\rget words #122\rget words #123\rget words #124\rget words #125\rget words #126\rget words #127\rget words #128\rget words #129\rget words #130\rget words #131\rget words #132\rget words #133\rget words #134\rget words #135\rget words #136\rget words #137\rget words #138\rget words #139\rget words #140\rget words #141\rget words #142\rget words #143\rget words #144\rget words #145\rget words #146\rget words #147\rget words #148\rget words #149\rget words #150\rget words #151\rget words #152\rget words #153\rget words #154\rget words #155\rget words #156\rget words #157\rget words #158\rget words #159\rget words #160\rget words #161\rget words #162\rget words #163\rget words #164\rget words #165\rget words #166\rget words #167\rget words #168\rget words #169\rget words #170\rget words #171\rget words #172\rget words #173\rget words #174\rget words #175\rget words #176\rget words #177\rget words #178\rget words #179\rget words #180\rget words #181\rget words #182\rget words #183\rget words #184\rget words #185\rget words #186\rget words #187\rget words #188\rget words #189\rget words #190\rget words #191\rget words #192\rget words #193\rget words #194\rget words #195\rget words #196\rget words #197\rget words #198\rget words #199\rget words #200\rget words #201\rget words #202\rget words #203\rget words #204\rget words #205\rget words #206\rget words #207\rget words #208\rget words #209\rget words #210\rget words #211\rget words #212\rget words #213\rget words #214\rget words #215\rget words #216\rget words #217\rget words #218\rget words #219\rget words #220\rget words #221\rget words #222\rget words #223\rget words #224\rget words #225\rget words #226\rget words #227\rget words #228\rget words #229\rget words #230\rget words #231\rget words #232\rget words #233\rget words #234\rget words #235\rget words #236\rget words #237\rget words #238\rget words #239\rget words #240\rget words #241\rget words #242\rget words #243\rget words #244\rget words #245\rget words #246\rget words #247\rget words #248\rget words #249\rget words #250\rget words #251\rget words #252\rget words #253\rget words #254\rget words #255\rget words #256\rget words #257\rget words #258\rget words #259\rget words #260\rget words #261\rget words #262\rget words #263\rget words #264\rget words #265\rget words #266\rget words #267\rget words #268\rget words #269\rget words #270\rget words #271\rget words #272\rget words #273\rget words #274\rget words #275\rget words #276\rget words #277\rget words #278\rget words #279\rget words #280\rget words #281\rget words #282\rget words #283\rget words #284\rget words #285\rget words #286\rget words #287\rget words #288\rget words #289\rget words #290\rget words #291\rget words #292\rget words #293\rget words #294\rget words #295\rget words #296\rget words #297\rget words #298\rget words #299\rget words #300\rget words #301\rget words #302\rget words #303\rget words #304\rget words #305\rget words #306\rget words #307\rget words #308\rget words #309\rget words #310\rget words #311\rget words #312\rget words #313\rget words #314\rget words #315\rget words #316\rget words #317\rget words #318\rget words #319\rget words #320\rget words #321\rget words #322\rget words #323\rget words #324\rget words #325\rget words #326\rget words #327\rget words #328\rget words #329\rget words #330\rget words #331\rget words #332\rget words #333\rget words #334\rget words #335\rget words #336\rget words #337\rget words #338\rget words #339\rget words #340\rget words #341\rget words #342\rget words #343\rget words #344\rget words #345\rget words #346\rget words #347\rget words #348\rget words #349\rget words #350\rget words #351\rget words #352\rget words #353\rget words #354\rget words #355\rget words #356\rget words #357\rget words #358\rget words #359\rget words #360\rget words #361\rget words #362\rget words #363\rget words #364\rget words #365\rget words #366\rget words #367\rget words #368\rget words #369\rget words #370\rget words #371\rget words #372\rget words #373\rget words #374\rget words #375\rget words #376\rget words #377\rget words #378\rget words #379\rget words #380\rget words #381\rget words #382\rget words #383\rget words #384\rget words #385\rget words #386\rget words #387\rget words #388\rget words #389\rget words #390\rget words #391\rget words #392\rget words #393\rget words #394\rget words #395\rget words #396\rget words #397\rget words #398\rget words #399\rget words #400\rget words #401\rget words #402\rget words #403\rget words #404\rget words #405\rget words #406\rget words #407\rget words #408\rget words #409\rget words #410\rget words #411\rget words #412\rget words #413\rget words #414\rget words #415\rget words #416\rget words #417\rget words #418\rget words #419\rget words #420\rget words #421\rget words #422\rget words #423\rget words #424\rget words #425\rget words #426\rget words #427\rget words #428\rget words #429\rget words #430\rget words #431\rget words #432\rget words #433\rget words #434\rget words #435\rget words #436\rget words #437\rget words #438\rget words #439\rget words #440\rget words #441\rget words #442\rget words #443\rget words #444\rget words #445\rget words #446\rget words #447\rget words #448\rget words #449\rget words #450\rget words #451\rget words #452\rget words #453\rget words #454\rget words #455\rget words #456\rget words #457\rget words #458\rget words #459\rget words #460\rget words #461\rget words #462\rget words #463\rget words #464\rget words #465\rget words #466\rget words #467\rget words #468\rget words #469\rget words #470\rget words #471\rget words #472\rget words #473\rget words #474\rget words #475\rget words #476\rget words #477\rget words #478\rget words #479\rget words #480\rget words #481\rget words #482\rget words #483\rget words #484\rget words #485\rget words #486\rget words #487\rget words #488\rget words #489\rget words #490\rget words #491\rget words #492\rget words #493\rget words #494\rget words #495\rget words #496\rget words #497\rget words #498\rget words #499\rget words #500\rget words #501\rget words #502\rget words #503\rget words #504\rget words #505\rget words #506\rget words #507\rget words #508\rget words #509\rget words #510\rget words #511\rget words #512\rget words #513\rget words #514\rget words #515\rget words #516\rget words #517\rget words #518\rget words #519\rget words #520\rget words #521\rget words #522\rget words #523\rget words #524\rget words #525\rget words #526\rget words #527\rget words #528\rget words #529\rget words #530\rget words #531\rget words #532\rget words #533\rget words #534\rget words #535\rget words #536\rget words #537\rget words #538\rget words #539\rget words #540\rget words #541\rget words #542\rget words #543\rget words #544\rget words #545\rget words #546\rget words #547\rget words #548\rget words #549\rget words #550\rget words #551\rget words #552\rget words #553\rget words #554\rget words #555\rget words #556\rget words #557\rget words #558\rget words #559\rget words #560\rget words #561\rget words #562\rget words #563\rget words #564\rget words #565\rget words #566\rget words #567\rget words #568\rget words #569\rget words #570\rget words #571\rget words #572\rget words #573\rget words #574\rget words #575\rget words #576\rget words #577\rget words #578\rget words #579\rget words #580\rget words #581\rget words #582\rget words #583\rget words #584\rget words #585\rget words #586\rget words #587\rget words #588\rget words #589\rget words #590\rget words #591\rget words #592\rget words #593\rget words #594\rget words #595\rget words #596\rget words #597\rget words #598\rget words #599\rget words #600\rget words #601\rget words #602\rget words #603\rget words #604\rget words #605\rget words #606\rget words #607\rget words #608\rget words #609\rget words #610\rget words #611\rget words #612\rget words #613\rget words #614\rget words #615\rget words #616\rget words #617\rget words #618\rget words #619\rget words #620\rget words #621\rget words #622\rget words #623\rget words #624\rget words #625\rget words #626\rget words #627\rget words #628\rget words #629\rget words #630\rget words #631\rget words #632\rget words #633\rget words #634\rget words #635\rget words #636\rget words #637\rget words #638\rget words #639\rget words #640\rget words #641\rget words #642\rget words #643\rget words #644\rget words #645\rget words #646\rget words #647\rget words #648\rget words #649\rget words #650\rget words #651\rget words #652\rget words #653\rget words #654\rget words #655\rget words #656\rget words #657\rget words #658\rget words #659\rget words #660\rget words #661\rget words #662\rget words #663\rget words #664\rget words #665\rget words #666\rget words #667\rget words #668\rget words #669\rget words #670\rget words #671\rget words #672\rget words #673\rget words #674\rget words #675\rget words #676\rget words #677\rget words #678\rget words #679\rget words #680\rget words #681\rget words #682\rget words #683\rget words #684\rget words #685\rget words #686\rget words #687\rget words #688\rget words #689\rget words #690\rget words #691\rget words #692\rget words #693\rget words #694\rget words #695\rget words #696\rget words #697\rget words #698\rget words #699\rget words #700\rget words #701\rget words #702\rget words #703\rget words #704\rget words #705\rget words #706\rget words #707\rget words #708\rget words #709\rget words #710\rget words #711\rget words #712\rget words #713\rget words #714\rget words #715\rget words #716\rget words #717\rget words #718\rget words #719\rget words #720\rget words #721\rget words #722\rget words #723\rget words #724\rget words #725\rget words #726\rget words #727\rget words #728\rget words #729\rget words #730\rget words #731\rget words #732\rget words #733\rget words #734\rget words #735\rget words #736\rget words #737\rget words #738\rget words #739\rget words #740\rget words #741\rget words #742\rget words #743\rget words #744\rget words #745\rget words #746\rget words #747\rget words #748\rget words #749\rget words #750\rget words #751\rget words #752\rget words #753\rget words #754\rget words #755\rget words #756\rget words #757\rget words #758\rget words #759\rget words #760\rget words #761\rget words #762\rget words #763\rget words #764\rget words #765\rget words #766\rget words #767\rget words #768\rget words #769\rget words #770\rget words #771\rget words #772\rget words #773\rget words #774\rget words #775\rget words #776\rget words #777\rget words #778\rget words #779\rget words #780\rget words #781\rget words #782\rget words #783\rget words #784\rget words #785\rget words #786\rget words #787\rget words #788\rget words #789\rget words #790\rget words #791\rget words #792\rget words #793\rget words #794\rget words #795\rget words #796\rget words #797\rget words #798\rget words #799\rget words #800\rget words #801\rget words #802\rget words #803\rget words #804\rget words #805\rget words #806\rget words #807\rget words #808\rget words #809\rget words #810\rget words #811\rget words #812\rget words #813\rget words #814\rget words #815\rget words #816\rget words #817\rget words #818\rget words #819\rget words #820\rget words #821\rget words #822\rget words #823\rget words #824\rget words #825\rget words #826\rget words #827\rget words #828\rget words #829\rget words #830\rget words #831\rget words #832\rget words #833\rget words #834\rget words #835\rget words #836\rget words #837\rget words #838\rget words #839\rget words #840\rget words #841\rget words #842\rget words #843\rget words #844\rget words #845\rget words #846\rget words #847\rget words #848\rget words #849\rget words #850\rget words #851\rget words #852\rget words #853\rget words #854\rget words #855\rget words #856\rget words #857\rget words #858\rget words #859\rget words #860\rget words #861\rget words #862\rget words #863\rget words #864\rget words #865\rget words #866\rget words #867\rget words #868\rget words #869\rget words #870\rget words #871\rget words #872\rget words #873\rget words #874\rget words #875\rget words #876\rget words #877\rget words #878\rget words #879\rget words #880\rget words #881\rget words #882\rget words #883\rget words #884\rget words #885\rget words #886\rget words #887\rget words #888\rget words #889\rget words #890\rget words #891\rget words #892\rget words #893\rget words #894\rget words #895\rget words #896\rget words #897\rget words #898\rget words #899\rget words #900\rget words #901\rget words #902\rget words #903\rget words #904\rget words #905\rget words #906\rget words #907\rget words #908\rget words #909\rget words #910\rget words #911\rget words #912\rget words #913\rget words #914\rget words #915\rget words #916\rget words #917\rget words #918\rget words #919\rget words #920\rget words #921\rget words #922\rget words #923\rget words #924\rget words #925\rget words #926\rget words #927\rget words #928\rget words #929\rget words #930\rget words #931\rget words #932\rget words #933\rget words #934\rget words #935\rget words #936\rget words #937\rget words #938\rget words #939\rget words #940\rget words #941\rget words #942\rget words #943\rget words #944\rget words #945\rget words #946\rget words #947\rget words #948\rget words #949\rget words #950\rget words #951\rget words #952\rget words #953\rget words #954\rget words #955\rget words #956\rget words #957\rget words #958\rget words #959\rget words #960\rget words #961\rget words #962\rget words #963\rget words #964\rget words #965\rget words #966\rget words #967\rget words #968\rget words #969\rget words #970\rget words #971\rget words #972\rget words #973\rget words #974\rget words #975\rget words #976\rget words #977\rget words #978\rget words #979\rget words #980\rget words #981\rget words #982\rget words #983\rget words #984\rget words #985\rget words #986\rget words #987\rget words #988\rget words #989\rget words #990\rget words #991\rget words #992\rget words #993\rget words #994\rget words #995\rget words #996\rget words #997\rget words #998\rget words #999\rget words #1000\rget words #1001\rget words #1002\rget words #1003\rget words #1004\rget words #1005\rget words #1006\rget words #1007\rget words #1008\rget words #1009\rget words #1010\rget words #1011\rget words #1012\rget words #1013\rget words #1014\rget words #1015\rget words #1016\rget words #1017\rget words #1018\rget words #1019\rget words #1020\rget words #1021\rget words #1022\rget words #1023\rget words #1024\rget words #1025\rget words #1026\rget words #1027\rget words #1028\rget words #1029\rget words #1030\rget words #1031\rget words #1032\rget words #1033\rget words #1034\rget words #1035\rget words #1036\rget words #1037\rget words #1038\rget words #1039\rget words #1040\rget words #1041\rget words #1042\rget words #1043\rget words #1044\rget words #1045\rget words #1046\rget words #1047\rget words #1048\rget words #1049\rget words #1050\rget words #1051\rget words #1052\rget words #1053\rget words #1054\rget words #1055\rget words #1056\rget words #1057\rget words #1058\rget words #1059\rget words #1060\rget words #1061\rget words #1062\rget words #1063\rget words #1064\rget words #1065\rget words #1066\rget words #1067\rget words #1068\rget words #1069\rget words #1070\rget words #1071\rget words #1072\rget words #1073\rget words #1074\rget words #1075\rget words #1076\rget words #1077\rget words #1078\rget words #1079\rget words #1080\rget words #1081\rget words #1082\rget words #1083\rget words #1084\rget words #1085\rget words #1086\rget words #1087\rget words #1088\rget words #1089\rget words #1090\rget words #1091\rget words #1092\rget words #1093\rget words #1094\rget words #1095\rget words #1096\rget words #1097\rget words #1098\rget words #1099\rget words #1100\rget words #1101\rget words #1102\rget words #1103\rget words #1104\rget words #1105\rget words #1106\rget words #1107\rget words #1108\rget words #1109\rget words #1110\rget words #1111\rget words #1112\rget words #1113\rget words #1114\rget words #1115\rget words #1116\rget words #1117\rget words #1118\rget words #1119\rget words #1120\rget words #1121\rget words #1122\rget words #1123\rget words #1124\rget words #1125\rget words #1126\rget words #1127\rget words #1128\rget words #1129\rget words #1130\rget words #1131\rget words #1132\rget words #1133\rget words #1134\rget words #1135\rget words #1136\rget words #1137\rget words #1138\rget words #1139\rget words #1140\rget words #1141\rget words #1142\rget words #1143\rget words #1144\rget words #1145\rget words #1146\rget words #1147\rget words #1148\rget words #1149\rget words #1150\rget words #1151\rget words #1152\rget words #1153\rget words #1154\rget words #1155\rget words #1156\rget words #1157\rget words #1158\rget words #1159\rget words #1160\rget words #1161\rget words #1162\rget words #1163\rget words #1164\rget words #1165\rget words #1166\rget words #1167\rget words #1168\rget words #1169\rget words #1170\rget words #1171\rget words #1172\rget words #1173\rget words #1174\rget words #1175\rget words #1176\rget words #1177\rget words #1178\rget words #1179\rget words #1180\rget words #1181\rget words #1182\rget words #1183\rget words #1184\rget words #1185\rget words #1186\rget words #1187\rget words #1188\rget words #1189\rget words #1190\rget words #1191\rget words #1192\rget words #1193\rget words #1194\rget words #1195\rget words #1196\rget words #1197\rget words #1198\rget words #1199\rget words #1200\rget words #1201\rget words #1202\rget words #1203\rget words #1204\rget words #1205\rget words #1206\rget words #1207\rget words #1208\rget words #1209\rget words #1210\rget words #1211\rget words #1212\rget words #1213\rget words #1214\rget words #1215\rget words #1216\rget words #1217\rget words #1218\rget words #1219\rget words #1220\rget words #1221\rget words #1222\rget words #1223\rget words #1224\rget words #1225\rget words #1226\rget words #1227\rget words #1228\rget words #1229\rget words #1230\rget words #1231\rget words #1232\rget words #1233\rget words #1234\rget words #1235\rget words #1236\rget words #1237\rget words #1238\rget words #1239\rget words #1240\rget words #1241\rget words #1242\rget words #1243\rget words #1244\rget words #1245\rget words #1246\rget words #1247\rget words #1248\rget words #1249\rget words #1250\rget words #1251\rget words #1252\rget words #1253\rget words #1254\rget words #1255\rget words #1256\rget words #1257\rget words #1258\rget words #1259\rget words #1260\rget words #1261\rget words #1262\rget words #1263\rget words #1264\rget words #1265\rget words #1266\rget words #1267\rget words #1268\rget words #1269\rget words #1270\rget words #1271\rget words #1272\rget words #1273\rget words #1274\rget words #1275\rget words #1276\rget words #1277\rget words #1278\rget words #1279\rget words #1280\rget words #1281\rget words #1282\rget words #1283\rget words #1284\rget words #1285\rget words #1286\rget words #1287\rget words #1288\rget words #1289\rget words #1290\rget words #1291\rget words #1292\rget words #1293\rget words #1294\rget words #1295\rget words #1296\rget words #1297\rget words #1298\rget words #1299\rget words #1300\rget words #1301\rget words #1302\rget words #1303\rget words #1304\rget words #1305\rget words #1306\rget words #1307\rget words #1308\rget words #1309\rget words #1310\rget words #1311\rget words #1312\rget words #1313\rget words #1314\rget words #1315\rget words #1316\rget words #1317\rget words #1318\rget words #1319\rget words #1320\rget words #1321\rget words #1322\rget words #1323\rget words #1324\rget words #1325\rget words #1326\rget words #1327\rget words #1328\rget words #1329\rget words #1330\rget words #1331\rget words #1332\rget words #1333\rget words #1334\rget words #1335\rget words #1336\rget words #1337\rget words #1338\rget words #1339\rget words #1340\rget words #1341\rget words #1342\rget words #1343\rget words #1344\rget words #1345\rget words #1346\rget words #1347\rget words #1348\rget words #1349\rget words #1350\rget words #1351\rget words #1352\rget words #1353\rget words #1354\rget words #1355\rget words #1356\rget words #1357\rget words #1358\rget words #1359\rget words #1360\rget words #1361\rget words #1362\rget words #1363\rget words #1364\rget words #1365\rget words #1366\rget words #1367\rget words #1368\rget words #1369\rget words #1370\rget words #1371\rget words #1372\rget words #1373\rget words #1374\rget words #1375\rget words #1376\rget words #1377\rget words #1378\rget words #1379\rget words #1380\rget words #1381\rget words #1382\rget words #1383\rget words #1384\rget words #1385\rget words #1386\rget words #1387\rget words #1388\rget words #1389\rget words #1390\rget words #1391\rget words #1392\rget words #1393\rget words #1394\rget words #1395\rget words #1396\rget words #1397\rget words #1398\rget words #1399\rget words #1400\rget words #1401\rget words #1402\rget words #1403\rget words #1404\rget words #1405\rget words #1406\rget words #1407\rget words #1408\rget words #1409\rget words #1410\rget words #1411\rget words #1412\rget words #1413\rget words #1414\rget words #1415\rget words #1416\rget words #1417\rget words #1418\rget words #1419\rget words #1420\rget words #1421\rget words #1422\rget words #1423\rget words #1424\rget words #1425\rget words #1426\rget words #1427\rget words #1428\rget words #1429\rget words #1430\rget words #1431\rget words #1432\rget words #1433\rget words #1434\rget words #1435\rget words #1436\rget words #1437\rget words #1438\rget words #1439\rget words #1440\rget words #1441\rget words #1442\rget words #1443\rget words #1444\rget words #1445\rget words #1446\rget words #1447\rget words #1448\rget words #1449\rget words #1450\rget words #1451\rget words #1452\rget words #1453\rget words #1454\rget words #1455\rget words #1456\rget words #1457\rget words #1458\rget words #1459\rget words #1460\rget words #1461\rget words #1462\rget words #1463\rget words #1464\rget words #1465\rget words #1466\rget words #1467\rget words #1468\rget words #1469\rget words #1470\rget words #1471\rget words #1472\rget words #1473\rget words #1474\rget words #1475\rget words #1476\rget words #1477\rget words #1478\rget words #1479\rget words #1480\rget words #1481\rget words #1482\rget words #1483\rget words #1484\rget words #1485\rget words #1486\rget words #1487\rget words #1488\rget words #1489\rget words #1490\rget words #1491\rget words #1492\rget words #1493\rget words #1494\rget words #1495\rget words #1496\rget words #1497\rget words #1498\rget words #1499\rget words #1500\rget words #1501\rget words #1502\rget words #1503\rget words #1504\rget words #1505\rget words #1506\rget words #1507\rget words #1508\rget words #1509\rget words #1510\rget words #1511\rget words #1512\rget words #1513\rget words #1514\rget words #1515\rget words #1516\rget words #1517\rget words #1518\rget words #1519\rget words #1520\rget words #1521\rget words #1522\rget words #1523\rget words #1524\rget words #1525\rget words #1526\rget words #1527\rget words #1528\rget words #1529\rget words #1530\rget words #1531\rget words #1532\rget words #1533\rget words #1534\rget words #1535\rget words #1536\rget words #1537\rget words #1538\rget words #1539\rget words #1540\rget words #1541\rget words #1542\rget words #1543\rget words #1544\rget words #1545\rget words #1546\rget words #1547\rget words #1548\rget words #1549\rget words #1550\rget words #1551\rget words #1552\rget words #1553\rget words #1554\rget words #1555\rget words #1556\rget words #1557\rget words #1558\rget words #1559\rget words #1560\rget words #1561\rget words #1562\rget words #1563\rget words #1564\rget words #1565\rget words #1566\rget words #1567\rget words #1568\rget words #1569\rget words #1570\rget words #1571\rget words #1572\rget words #1573\rget words #1574\rget words #1575\rget words #1576\rget words #1577\rget words #1578\rget words #1579\rget words #1580\rget words #1581\rget words #1582\rget words #1583\rget words #1584\rget words #1585\rget words #1586\rget words #1587\rget words #1588\rget words #1589\rget words #1590\rget words #1591\rget words #1592\rget words #1593\rget words #1594\rget words #1595\rget words #1596\rget words #1597\rget words #1598\rget words #1599\rget words #1600\rget words #1601\rget words #1602\rget words #1603\rget words #1604\rget words #1605\rget words #1606\rget words #1607\rget words #1608\rget words #1609\rget words #1610\rget words #1611\rget words #1612\rget words #1613\rget words #1614\rget words #1615\rget words #1616\rget words #1617\rget words #1618\rget words #1619\rget words #1620\rget words #1621\rget words #1622\rget words #1623\rget words #1624\rget words #1625\rget words #1626\rget words #1627\rget words #1628\rget words #1629\rget words #1630\rget words #1631\rget words #1632\rget words #1633\rget words #1634\rget words #1635\rget words #1636\rget words #1637\rget words #1638\rget words #1639\rget words #1640\rget words #1641\rget words #1642\rget words #1643\rget words #1644\rget words #1645\rget words #1646\rget words #1647\rget words #1648"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:42: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "get words #55777\n",
            "total words: 55779\n",
            "sentence count #200000\n",
            "load model ...\n",
            "save csv ...\n",
            "Finish Predicting\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qSvgTRuGu2Qb",
        "colab_type": "text"
      },
      "source": [
        "#### Check where the files are"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8SZYJQ62utiK",
        "colab_type": "code",
        "outputId": "f42a12be-91af-49fa-d4ce-d300d16dc812",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "!pwd\n",
        "!ls"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n",
            "ckpt.model   testing_data.txt\t   w2v_all.model.trainables.syn1neg.npy\n",
            "data.zip     training_label.txt    w2v_all.model.wv.vectors.npy\n",
            "predict.csv  training_nolabel.txt\n",
            "sample_data  w2v_all.model\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0iCyLSWFcEmP",
        "colab_type": "text"
      },
      "source": [
        "### Run 20 epochs on n98\n",
        "real\t3m33.317s\n",
        "\n",
        "user\t3m29.813s\n",
        "\n",
        "sys\t1m9.469s"
      ]
    }
  ]
}