model-detect-ai-text.ipynb

{"cells":[{"cell_type":"markdown","metadata":{},"source":["    This is a kaggle notebook ran on accelerator GPU T4 X2 without using Internet"]},{"cell_type":"markdown","metadata":{},"source":["# **LLM - Detect AI Generated Text**\n","\n","- This is a **Kaggle competition** hosted by Vanderbilt University and The Learning Agency Lab (October 31,2023 to January 23,2024). Developed a machine learning model to accurately **differentiate essays written by students from those generated by large language models**, addressing concerns about plagiarism and the impact of LLMs on education.\n","\n","- **DataSet Details:** The competition dataset comprises about **10,000 essays**, some written by students and some generated by a variety of large language models (LLMs). The \n","                       goal of the competition is to determine whether or not essay was generated by an LLM.\n","    - All of the essays were written in response to one of **seven essay prompts**.\n","    - **Training Data:** Essays from **two of the prompts** compose the training set; the remaining essays compose the hidden test set. Nearly all of the training set essays   \n","                         were written by students, with only a few generated essays given as examples. You may wish to generate more essays to use as training data.\n","    - **Testing Data:** The data in test_essays.csv is only dummy data.There are about **9,000 essays** in the test set, both student written and LLM generated.\n","\n","- **Accuracy:**\n","    - **Public Score:**  0.963797 (Calculated on only 46% of test data)\n","    - **Private Score:** 0.908580 (Calculated on 54% of test data)\n","\n","- **Leaderboard Details**\n","    - We secured **Silver medal** in this competition **ranking 125 out of 4359** participants **globally**, representing the **Top 3%** of the teams\n","\n","- **Contributors:**\n","    1. Yash Shrivastava (Team Leader)\n","    2. Devendra Virani\n","    3. Toshan Gupta\n","    4. Smit Shah\n","    5. Garv Gupta\n"]},{"cell_type":"markdown","metadata":{},"source":["    Importing the required Libraries"]},{"cell_type":"code","execution_count":1,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:00:19.412747Z","iopub.status.busy":"2024-02-16T15:00:19.412368Z","iopub.status.idle":"2024-02-16T15:00:25.614818Z","shell.execute_reply":"2024-02-16T15:00:25.613815Z","shell.execute_reply.started":"2024-02-16T15:00:19.412715Z"},"trusted":true},"outputs":[{"name":"stderr","output_type":"stream","text":["/opt/conda/lib/python3.10/site-packages/dask/dataframe/_pyarrow_compat.py:23: UserWarning: You are using pyarrow version 11.0.0 which is known to be insecure. See https://www.cve.org/CVERecord?id=CVE-2023-47248 for further details. Please upgrade to pyarrow>=14.0.1 or install pyarrow-hotfix to patch your current version.\n","  warnings.warn(\n"]}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Importing the libraries\n","import sys\n","import gc\n","import pandas as pd\n","from sklearn.model_selection import StratifiedKFold\n","import numpy as np\n","from sklearn.metrics import roc_auc_score\n","import numpy as np\n","from lightgbm import LGBMClassifier\n","from sklearn.feature_extraction.text import TfidfVectorizer\n","#<------------------------------------------------------------------------------------------------------------------------------------------>\n","from tokenizers import (\n","    decoders,\n","    models,\n","    normalizers,\n","    pre_tokenizers,\n","    processors,\n","    trainers,\n","    Tokenizer,\n",")\n","from datasets import Dataset\n","from tqdm.auto import tqdm\n","from transformers import PreTrainedTokenizerFast\n","\n","from sklearn.linear_model import SGDClassifier\n","from sklearn.naive_bayes import MultinomialNB\n","from sklearn.ensemble import VotingClassifier\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"markdown","metadata":{},"source":["# Data Loading and Preprocessing for Text Classification\n","\n","1. **Data Loading**: We load the data from the following CSV files:\n","    - `train_v2_drcat_02.csv` for Training data\n","    - `test_essays.csv` for Testing data\n","    - `sample_submission.csv` for sample submission format\n","2. **Data Cleaning**: \n","    - Duplicate rows in the training data are removed based on the 'text' column\n","    - Additionally, certain prompts are excluded from the training set.\n","3. **Preprocessing**: Preprocessing steps such as setting lowercase conversion and defining the vocabulary size are applied."]},{"cell_type":"code","execution_count":2,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:00:32.701028Z","iopub.status.busy":"2024-02-16T15:00:32.700217Z","iopub.status.idle":"2024-02-16T15:00:33.830571Z","shell.execute_reply":"2024-02-16T15:00:33.829534Z","shell.execute_reply.started":"2024-02-16T15:00:32.700995Z"},"trusted":true},"outputs":[],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Reading data\n","test = pd.read_csv('/kaggle/input/llm-detect-ai-generated-text/test_essays.csv')   #Testing data\n","sub = pd.read_csv('/kaggle/input/llm-detect-ai-generated-text/sample_submission.csv') #Predicted data\n","train = pd.read_csv(\"/kaggle/input/daigt-v2-train-dataset/train_v2_drcat_02.csv\", sep=',') #Training data\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":3,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:00:37.972172Z","iopub.status.busy":"2024-02-16T15:00:37.971471Z","iopub.status.idle":"2024-02-16T15:00:37.996879Z","shell.execute_reply":"2024-02-16T15:00:37.996035Z","shell.execute_reply.started":"2024-02-16T15:00:37.972139Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Length of training data: 44868\n","Training data:\n"]},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","      <th>label</th>\n","      <th>prompt_name</th>\n","      <th>source</th>\n","      <th>RDizzl3_seven</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Phones\\n\\nModern humans today are always on th...</td>\n","      <td>0</td>\n","      <td>Phones and driving</td>\n","      <td>persuade_corpus</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>This essay will explain if drivers should or s...</td>\n","      <td>0</td>\n","      <td>Phones and driving</td>\n","      <td>persuade_corpus</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Driving while the use of cellular devices\\n\\nT...</td>\n","      <td>0</td>\n","      <td>Phones and driving</td>\n","      <td>persuade_corpus</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Phones &amp; Driving\\n\\nDrivers should not be able...</td>\n","      <td>0</td>\n","      <td>Phones and driving</td>\n","      <td>persuade_corpus</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Cell Phone Operation While Driving\\n\\nThe abil...</td>\n","      <td>0</td>\n","      <td>Phones and driving</td>\n","      <td>persuade_corpus</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>44863</th>\n","      <td>Dear Senator,\\n\\nI am writing to you today to ...</td>\n","      <td>1</td>\n","      <td>Does the electoral college work?</td>\n","      <td>kingki19_palm</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>44864</th>\n","      <td>Dear Senator,\\n\\nI am writing to you today to ...</td>\n","      <td>1</td>\n","      <td>Does the electoral college work?</td>\n","      <td>kingki19_palm</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>44865</th>\n","      <td>Dear Senator,\\n\\nI am writing to you today to ...</td>\n","      <td>1</td>\n","      <td>Does the electoral college work?</td>\n","      <td>kingki19_palm</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>44866</th>\n","      <td>Dear Senator,\\n\\nI am writing to you today to ...</td>\n","      <td>1</td>\n","      <td>Does the electoral college work?</td>\n","      <td>kingki19_palm</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>44867</th>\n","      <td>Dear Senator,\\n\\nI am writing to you today to ...</td>\n","      <td>1</td>\n","      <td>Does the electoral college work?</td>\n","      <td>kingki19_palm</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>44868 rows × 5 columns</p>\n","</div>"],"text/plain":["                                                    text  label  \\\n","0      Phones\\n\\nModern humans today are always on th...      0   \n","1      This essay will explain if drivers should or s...      0   \n","2      Driving while the use of cellular devices\\n\\nT...      0   \n","3      Phones & Driving\\n\\nDrivers should not be able...      0   \n","4      Cell Phone Operation While Driving\\n\\nThe abil...      0   \n","...                                                  ...    ...   \n","44863  Dear Senator,\\n\\nI am writing to you today to ...      1   \n","44864  Dear Senator,\\n\\nI am writing to you today to ...      1   \n","44865  Dear Senator,\\n\\nI am writing to you today to ...      1   \n","44866  Dear Senator,\\n\\nI am writing to you today to ...      1   \n","44867  Dear Senator,\\n\\nI am writing to you today to ...      1   \n","\n","                            prompt_name           source  RDizzl3_seven  \n","0                    Phones and driving  persuade_corpus          False  \n","1                    Phones and driving  persuade_corpus          False  \n","2                    Phones and driving  persuade_corpus          False  \n","3                    Phones and driving  persuade_corpus          False  \n","4                    Phones and driving  persuade_corpus          False  \n","...                                 ...              ...            ...  \n","44863  Does the electoral college work?    kingki19_palm           True  \n","44864  Does the electoral college work?    kingki19_palm           True  \n","44865  Does the electoral college work?    kingki19_palm           True  \n","44866  Does the electoral college work?    kingki19_palm           True  \n","44867  Does the electoral college work?    kingki19_palm           True  \n","\n","[44868 rows x 5 columns]"]},"execution_count":3,"metadata":{},"output_type":"execute_result"}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","print(\"Length of training data:\",len(train))\n","print(\"Training data:\")\n","train\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":4,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:00:43.216665Z","iopub.status.busy":"2024-02-16T15:00:43.215843Z","iopub.status.idle":"2024-02-16T15:00:43.226980Z","shell.execute_reply":"2024-02-16T15:00:43.225990Z","shell.execute_reply.started":"2024-02-16T15:00:43.216623Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Sample Testing Data:\n"]},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>prompt_id</th>\n","      <th>text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0000aaaa</td>\n","      <td>2</td>\n","      <td>Aaa bbb ccc.</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1111bbbb</td>\n","      <td>3</td>\n","      <td>Bbb ccc ddd.</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2222cccc</td>\n","      <td>4</td>\n","      <td>CCC ddd eee.</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["         id  prompt_id          text\n","0  0000aaaa          2  Aaa bbb ccc.\n","1  1111bbbb          3  Bbb ccc ddd.\n","2  2222cccc          4  CCC ddd eee."]},"execution_count":4,"metadata":{},"output_type":"execute_result"}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","print(\"Sample Testing Data:\")\n","test\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":5,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:00:46.915290Z","iopub.status.busy":"2024-02-16T15:00:46.914892Z","iopub.status.idle":"2024-02-16T15:00:46.926080Z","shell.execute_reply":"2024-02-16T15:00:46.925128Z","shell.execute_reply.started":"2024-02-16T15:00:46.915258Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Submission Format:\n"]},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>generated</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0000aaaa</td>\n","      <td>0.1</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1111bbbb</td>\n","      <td>0.9</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2222cccc</td>\n","      <td>0.4</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["         id  generated\n","0  0000aaaa        0.1\n","1  1111bbbb        0.9\n","2  2222cccc        0.4"]},"execution_count":5,"metadata":{},"output_type":"execute_result"}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","print(\"Submission Format:\")\n","sub\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":6,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:01:20.862914Z","iopub.status.busy":"2024-02-16T15:01:20.861724Z","iopub.status.idle":"2024-02-16T15:01:20.942962Z","shell.execute_reply":"2024-02-16T15:01:20.941837Z","shell.execute_reply.started":"2024-02-16T15:01:20.862869Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Initial length of training data: 44868\n","Initial Prompts:\n"]},{"data":{"text/plain":["{'\"A Cowboy Who Rode the Waves\"',\n"," 'Car-free cities',\n"," 'Cell phones at school',\n"," 'Community service',\n"," 'Distance learning',\n"," 'Does the electoral college work?',\n"," 'Driverless cars',\n"," 'Exploring Venus',\n"," 'Facial action coding system',\n"," 'Grades for extracurricular activities',\n"," 'Mandatory extracurricular activities',\n"," 'Phones and driving',\n"," 'Seeking multiple opinions',\n"," 'Summer projects',\n"," 'The Face on Mars'}"]},"execution_count":6,"metadata":{},"output_type":"execute_result"}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Deleting duplicate rows\n","train = train.drop_duplicates(subset=['text'])\n","train.reset_index(drop=True, inplace=True)\n","print(\"Initial length of training data:\",len(train))\n","print(\"Initial Prompts:\")\n","set(train[\"prompt_name\"])\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":7,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:01:48.091906Z","iopub.status.busy":"2024-02-16T15:01:48.091518Z","iopub.status.idle":"2024-02-16T15:01:48.119981Z","shell.execute_reply":"2024-02-16T15:01:48.119018Z","shell.execute_reply.started":"2024-02-16T15:01:48.091875Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Final length of training data: 34497\n","Final prompts:\n"]},{"data":{"text/plain":["{'\"A Cowboy Who Rode the Waves\"',\n"," 'Car-free cities',\n"," 'Cell phones at school',\n"," 'Community service',\n"," 'Does the electoral college work?',\n"," 'Driverless cars',\n"," 'Exploring Venus',\n"," 'Facial action coding system',\n"," 'Mandatory extracurricular activities',\n"," 'Phones and driving',\n"," 'Seeking multiple opinions',\n"," 'The Face on Mars'}"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Removing few prompts (Because these prompts are irrelevant not included in testing data)\n","excluded_prompt_name_list = ['Distance learning','Grades for extracurricular activities','Summer projects']\n","train = train[~(train['prompt_name'].isin(excluded_prompt_name_list))]\n","train = train.drop_duplicates(subset=['text'])\n","train.reset_index(drop=True, inplace=True)\n","print(\"Final length of training data:\",len(train))\n","print(\"Final prompts:\")\n","set(train[\"prompt_name\"])\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":8,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:01:53.210459Z","iopub.status.busy":"2024-02-16T15:01:53.209815Z","iopub.status.idle":"2024-02-16T15:01:53.214823Z","shell.execute_reply":"2024-02-16T15:01:53.213701Z","shell.execute_reply.started":"2024-02-16T15:01:53.210424Z"},"trusted":true},"outputs":[],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Setting up parameters to lowercase=false and giving vocab size\n","LOWERCASE = False\n","VOCAB_SIZE = 14000000\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"markdown","metadata":{},"source":["# Creating Byte-Pair Encoding Tokenizer with Hugging Face\n","\n","1. Initialization of the raw tokenizer with BPE model and special tokens.\n","2. Addition of normalization and pre-tokenization processes, such as NFC normalization and byte-level pre-tokenization.\n","3. Training of the tokenizer using a trainer instance and a provided dataset.\n","4. Creation of a Hugging Face dataset object from a pandas DataFrame.\n","5. Iterative training of the tokenizer from the dataset using a custom corpus iterator.\n","6. Initialization of a PreTrainedTokenizerFast object using the trained raw tokenizer.\n","7. Tokenization of test and train text data using the tokenizer."]},{"cell_type":"code","execution_count":9,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:01:57.937520Z","iopub.status.busy":"2024-02-16T15:01:57.936686Z","iopub.status.idle":"2024-02-16T15:01:58.007640Z","shell.execute_reply":"2024-02-16T15:01:58.006700Z","shell.execute_reply.started":"2024-02-16T15:01:57.937489Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["\n","\n","\n"]}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","# Creating Byte-Pair Encoding tokenizer\n","raw_tokenizer = Tokenizer(models.BPE(unk_token=\"[UNK]\"))\n","#<------------------------------------------------------------------------------------------------------------------------------------------>\n","# Adding normalization and pre_tokenizer\n","raw_tokenizer.normalizer = normalizers.Sequence([normalizers.NFC()] + [normalizers.Lowercase()] if LOWERCASE else [])\n","raw_tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel()\n","#<------------------------------------------------------------------------------------------------------------------------------------------>\n","# Adding special tokens and creating trainer instance\n","special_tokens = [\"[UNK]\", \"[PAD]\", \"[CLS]\", \"[SEP]\", \"[MASK]\"]\n","trainer = trainers.BpeTrainer(vocab_size=VOCAB_SIZE, special_tokens=special_tokens)\n","#<------------------------------------------------------------------------------------------------------------------------------------------>\n","# Creating huggingface dataset object\n","dataset = Dataset.from_pandas(test[['text']])\n","def train_corp_iter(): \n","    for i in range(0, len(dataset), 1000):\n","        yield dataset[i : i + 1000][\"text\"]\n","raw_tokenizer.train_from_iterator(train_corp_iter(), trainer=trainer)\n","tokenizer = PreTrainedTokenizerFast(\n","    tokenizer_object=raw_tokenizer,\n","    unk_token=\"[UNK]\",\n","    pad_token=\"[PAD]\",\n","    cls_token=\"[CLS]\",\n","    sep_token=\"[SEP]\",\n","    mask_token=\"[MASK]\",\n",")\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"markdown","metadata":{},"source":["    Converting training data to Byte Pair Encodings"]},{"cell_type":"code","execution_count":11,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:04:35.047835Z","iopub.status.busy":"2024-02-16T15:04:35.046728Z","iopub.status.idle":"2024-02-16T15:06:06.606924Z","shell.execute_reply":"2024-02-16T15:06:06.605976Z","shell.execute_reply.started":"2024-02-16T15:04:35.047799Z"},"trusted":true},"outputs":[{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"4e5bca80cf744436a7607967d887f7ab","version_major":2,"version_minor":0},"text/plain":["  0%|          | 0/34497 [00:00<?, ?it/s]"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Training tokens for 1st essay:\n","['Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'd', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'd', 'a', '[UNK]', 'Ġ', 'a', '[UNK]', 'e', 'Ġ', 'a', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '.', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'a', '[UNK]', 'e', 'Ġ', 'a', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', 'd', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '.', 'A', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'd', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', 'b', 'a', 'c', '[UNK]', 'Ġ', 'a', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'a', '[UNK]', 'd', 'Ġ', 'a', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'C', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'c', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', 'e', 'd', '[UNK]', 'a', '.', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'e', '[UNK]', 'e', '[UNK]', 'Ġ', 'd', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', 'd', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'e', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'b', 'a', 'd', 'Ġ', 'c', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'e', '[UNK]', 'c', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'c', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '.', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', 'c', 'e', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', 'a', '[UNK]', 'e', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'd', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', 'e', '[UNK]', 'Ġ', 'b', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'c', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'b', 'e', 'c', 'a', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '.', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'a', 'b', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'c', 'e', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', '.', 'A', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'a', 'c', 'e', 'b', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'a', '[UNK]', 'Ġ', 'a', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', 'c', '[UNK]', 'a', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'a', '[UNK]', 'a', '[UNK]', 'Ġ', 'a', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'b', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'c', '[UNK]', '[UNK]', '[UNK]', 'a', 'c', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'c', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'b', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'c', '[UNK]', '[UNK]', '[UNK]', 'a', 'c', '[UNK]', 'Ġ', 'b', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'd', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', 'a', '[UNK]', 'e', '[UNK]', '.', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', 'a', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', 'd', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'c', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'c', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '.', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', 'c', '[UNK]', 'a', '[UNK]', '[UNK]', 'e', 'd', 'Ġ', 'd', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', 'e', '[UNK]', 'a', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '.', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', 'Ġ', 'a', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'd', '.', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', 'a', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', 'b', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', 'd', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'c', '[UNK]', 'Ġ', 'c', 'a', '[UNK]', 'Ġ', 'c', 'a', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'b', '[UNK]', 'e', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'c', 'a', '[UNK]', '[UNK]', 'e', 'd', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', 'd', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', '[UNK]', 'e', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', 'e', '[UNK]', 'b', 'e', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'd', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'b', 'e', 'c', 'a', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'd', '.', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'd', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'b', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'b', 'e', 'c', 'a', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'd', '[UNK]', 'd', 'Ġ', '[UNK]', 'e', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', '.', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'a', '[UNK]', 'Ġ', 'a', 'cc', '[UNK]', 'd', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'c', '[UNK]', 'd', 'e', '.', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'ee', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'd', 'e', 'a', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'b', 'e', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'a', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', 'b', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'd', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'c', '[UNK]', '[UNK]', '[UNK]', 'a', 'c', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'd', 'Ġ', 'a', '[UNK]', 'd', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'c', '[UNK]', 'Ġ', 'c', 'a', '[UNK]', 'Ġ', 'e', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'b', 'a', 'd', '[UNK]', '[UNK]', '.', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', 'a', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', 'b', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'c', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'a', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'b', '[UNK]', 'e', '[UNK]', 'Ġ', 'a', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'c', 'a', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'd', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', 'a', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'e', 'd', 'Ġ', 'a', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'e', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', 'd', 'a', '[UNK]', 'Ġ', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'd', 'Ġ', 'c', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'b', '[UNK]', 'e', '.', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'd', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'Ġ', 'b', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'dd', '[UNK]', 'e', 'Ġ', '[UNK]', '[UNK]', 'Ġ', 'd', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', 'a', '[UNK]', '[UNK]', 'a', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'd', 'a', '[UNK]', 'e', 'd', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', 'e', 'Ġ', 'd', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'd', 'Ġ', 'a', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'd', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', '[UNK]', '[UNK]', 'e', '[UNK]', '.', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'e', 'Ġ', 'b', 'e', '[UNK]', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'Ġ', '[UNK]', '[UNK]', 'a', '[UNK]', 'Ġ', '[UNK]', 'a', '[UNK]', 'e', '.', 'Ġ', 'Ġ', 'Ġ', 'Ġ']\n"]}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Create BPE for training data\n","tokenized_texts_train = []\n","for text in tqdm(train['text'].tolist()):\n","    tokenized_texts_train.append(tokenizer.tokenize(text))\n","print(\"Training tokens for 1st essay:\")\n","print(tokenized_texts_train[0])\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"markdown","metadata":{},"source":["    Converting testing data to Byte Pair Encodings"]},{"cell_type":"code","execution_count":12,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:06:15.009438Z","iopub.status.busy":"2024-02-16T15:06:15.009022Z","iopub.status.idle":"2024-02-16T15:06:15.035527Z","shell.execute_reply":"2024-02-16T15:06:15.034580Z","shell.execute_reply.started":"2024-02-16T15:06:15.009406Z"},"trusted":true},"outputs":[{"data":{"application/vnd.jupyter.widget-view+json":{"model_id":"605f57e661084ba5a4e8da47aa1d0e2b","version_major":2,"version_minor":0},"text/plain":["  0%|          | 0/3 [00:00<?, ?it/s]"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Testing tokens for 1st essay:\n","['ĠAaa', 'Ġbbb', 'Ġccc', '.']\n"]}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Creating BPE for testing data\n","tokenized_texts_test = []\n","for text in tqdm(test['text'].tolist()):\n","    tokenized_texts_test.append(tokenizer.tokenize(text))\n","print(\"Testing tokens for 1st essay:\")\n","print(tokenized_texts_test[0])\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"markdown","metadata":{},"source":["# TF-IDF Vectorization with Custom Vocabulary\n","\n","1. **Function Definition**: A dummy function `dummy` is defined, which simply returns the input text.\n","2. **Vectorizer Initialization**: The `TfidfVectorizer` is initialized with specific parameters such as ngram_range, lowercase, sublinear_tf, and the use of the dummy function for tokenization and preprocessing.\n","3. **Fitting and Vocabulary Extraction**: The vectorizer is fitted on tokenized texts from the test dataset, and the resulting vocabulary is extracted.\n","4. **Vectorization with Custom Vocabulary**: Another `TfidfVectorizer` is initialized with the vocabulary extracted in the previous step, ensuring that the same vocabulary is used for both training and test data vectorization.\n","5. **Vectorization of Texts**: The training and test datasets are transformed into TF-IDF representations using the vectorizer.\n","6. **Memory Management**: The vectorizer object is deleted, and garbage collection is triggered to manage memory usage."]},{"cell_type":"code","execution_count":13,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:06:20.680310Z","iopub.status.busy":"2024-02-16T15:06:20.679250Z","iopub.status.idle":"2024-02-16T15:06:20.684664Z","shell.execute_reply":"2024-02-16T15:06:20.683608Z","shell.execute_reply.started":"2024-02-16T15:06:20.680272Z"},"trusted":true},"outputs":[],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","def dummy(text):\n","    return text\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":14,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:06:41.935676Z","iopub.status.busy":"2024-02-16T15:06:41.934886Z","iopub.status.idle":"2024-02-16T15:06:41.952449Z","shell.execute_reply":"2024-02-16T15:06:41.951341Z","shell.execute_reply.started":"2024-02-16T15:06:41.935643Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["{'ĠAaa Ġbbb Ġccc': 0, 'Ġbbb Ġccc .': 6, 'ĠAaa Ġbbb Ġccc .': 1, 'ĠBbb Ġccc Ġddd': 2, 'Ġccc Ġddd .': 7, 'ĠBbb Ġccc Ġddd .': 3, 'ĠCCC Ġddd Ġeee': 4, 'Ġddd Ġeee .': 8, 'ĠCCC Ġddd Ġeee .': 5}\n"]}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","vectorizer = TfidfVectorizer(ngram_range=(3, 5), lowercase=False, sublinear_tf=True, analyzer = 'word',\n","    tokenizer = dummy,\n","    preprocessor = dummy,\n","    token_pattern = None, strip_accents='unicode')\n","#<------------------------------------------------------------------------------------------------------------------------------------------>\n","vectorizer.fit(tokenized_texts_test)\n","# Getting vocab\n","vocab = vectorizer.vocabulary_\n","print(vocab)\n","vectorizer = TfidfVectorizer(ngram_range=(3, 5), lowercase=False, sublinear_tf=True, vocabulary=vocab,\n","                            analyzer = 'word',\n","                            tokenizer = dummy,\n","                            preprocessor = dummy,\n","                            token_pattern = None, strip_accents='unicode'\n","                            )\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":15,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:06:50.672574Z","iopub.status.busy":"2024-02-16T15:06:50.671733Z","iopub.status.idle":"2024-02-16T15:10:04.228443Z","shell.execute_reply":"2024-02-16T15:10:04.227502Z","shell.execute_reply.started":"2024-02-16T15:06:50.672542Z"},"trusted":true},"outputs":[{"data":{"text/plain":["44"]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Creating training and testing tf-idf representation\n","tf_train = vectorizer.fit_transform(tokenized_texts_train)\n","tf_test = vectorizer.transform(tokenized_texts_test)\n","#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Managing memory\n","del vectorizer\n","gc.collect()\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"markdown","metadata":{},"source":["    Training labels"]},{"cell_type":"code","execution_count":16,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:10:10.714580Z","iopub.status.busy":"2024-02-16T15:10:10.713922Z","iopub.status.idle":"2024-02-16T15:10:10.720704Z","shell.execute_reply":"2024-02-16T15:10:10.719679Z","shell.execute_reply.started":"2024-02-16T15:10:10.714547Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Training labels:\n","[0 0 0 ... 1 1 1]\n"]}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","y_train = train['label'].values\n","print(\"Training labels:\")\n","print(y_train)\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"markdown","metadata":{},"source":["# Ensemble Model Training and Prediction\n","\n","1. **Model Definition**: The `get_model()` function defines an ensemble of classifiers:\n","    - Multinomial Naive Bayes\n","    - Stochastic Gradient Descent\n","    - LightGBM\n","    - CatBoost.\n","2. **Initialization**: The ensemble model is initialized with specific parameters and weights assigned to each base classifier.\n","3. **Model Training**: If the number of texts in the test dataset is more than 5, the ensemble model is trained on the TF-IDF transformed training data (`tf_train`) along with labels (`y_train`).\n","4. **Prediction**: The trained model is used to predict probabilities for the test dataset (`tf_test`), and the predictions are stored in the 'generated' column of the submission dataframe (`sub`).\n","5. **Saving Predictions**: The submission dataframe with predictions is saved to a CSV file named 'submission.csv'."]},{"cell_type":"code","execution_count":17,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:10:14.563534Z","iopub.status.busy":"2024-02-16T15:10:14.562542Z","iopub.status.idle":"2024-02-16T15:10:14.975083Z","shell.execute_reply":"2024-02-16T15:10:14.973979Z","shell.execute_reply.started":"2024-02-16T15:10:14.563499Z"},"trusted":true},"outputs":[],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Importing libraries\n","from catboost import CatBoostClassifier\n","from sklearn.linear_model import LogisticRegression\n","#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Getting model\n","def get_model():\n","    clf = MultinomialNB(alpha=0.0225)\n","\n","    sgd_model = SGDClassifier(max_iter=9000, tol=1e-4, loss=\"modified_huber\", random_state=6743)\n","    p6={'n_iter': 3000,'verbose': -1,'objective': 'cross_entropy','metric': 'auc',\n","        'learning_rate': 0.00581909898961407, 'colsample_bytree': 0.78,\n","        'colsample_bynode': 0.8,\n","       }\n","    p6[\"random_state\"] = 6743\n","\n","    lgb=LGBMClassifier(**p6)\n","\n","    cat=CatBoostClassifier(iterations=3000,\n","                           verbose=0,\n","                           random_seed=6543,\n","                           learning_rate=0.005599066836106983,\n","                           subsample = 0.35,\n","                           allow_const_label=True,loss_function = 'CrossEntropy')\n","    \n","    #Setting weights\n","    weights = [0.055,0.31,0.23,0.80]\n"," \n","    ensemble = VotingClassifier(estimators=[('mnb',clf),\n","                                            ('sgd', sgd_model),\n","                                            ('lgb',lgb), \n","                                            ('cat', cat)\n","                                           ],\n","                                weights=weights, voting='soft', n_jobs=-1)\n","    return ensemble\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":18,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:10:20.360464Z","iopub.status.busy":"2024-02-16T15:10:20.359546Z","iopub.status.idle":"2024-02-16T15:10:20.378124Z","shell.execute_reply":"2024-02-16T15:10:20.377139Z","shell.execute_reply.started":"2024-02-16T15:10:20.360429Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["VotingClassifier(estimators=[('mnb', MultinomialNB(alpha=0.0225)),\n","                             ('sgd',\n","                              SGDClassifier(loss='modified_huber',\n","                                            max_iter=9000, random_state=6743,\n","                                            tol=0.0001)),\n","                             ('lgb',\n","                              LGBMClassifier(colsample_bynode=0.8,\n","                                             colsample_bytree=0.78,\n","                                             learning_rate=0.00581909898961407,\n","                                             metric='auc', n_iter=3000,\n","                                             objective='cross_entropy',\n","                                             random_state=6743, verbose=-1)),\n","                             ('cat',\n","                              <catboost.core.CatBoostClassifier object at 0x78c7add0f250>)],\n","                 n_jobs=-1, voting='soft', weights=[0.055, 0.31, 0.23, 0.8])\n"]}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","model = get_model()\n","print(model)\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":19,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:10:27.038790Z","iopub.status.busy":"2024-02-16T15:10:27.038389Z","iopub.status.idle":"2024-02-16T15:10:27.047910Z","shell.execute_reply":"2024-02-16T15:10:27.046988Z","shell.execute_reply.started":"2024-02-16T15:10:27.038759Z"},"trusted":true},"outputs":[],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","if len(test.text.values) <= 5:\n","    sub.to_csv('submission.csv', index=False)\n","else:\n","    model.fit(tf_train, y_train)\n","    final_preds = model.predict_proba(tf_test)[:,1]\n","    sub['generated'] = final_preds\n","    sub.to_csv('submission.csv', index=False)\n","    sub\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]},{"cell_type":"code","execution_count":20,"metadata":{"execution":{"iopub.execute_input":"2024-02-16T15:10:29.952095Z","iopub.status.busy":"2024-02-16T15:10:29.951150Z","iopub.status.idle":"2024-02-16T15:10:29.962955Z","shell.execute_reply":"2024-02-16T15:10:29.961882Z","shell.execute_reply.started":"2024-02-16T15:10:29.952034Z"},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>generated</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0000aaaa</td>\n","      <td>0.1</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1111bbbb</td>\n","      <td>0.9</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2222cccc</td>\n","      <td>0.4</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["         id  generated\n","0  0000aaaa        0.1\n","1  1111bbbb        0.9\n","2  2222cccc        0.4"]},"execution_count":20,"metadata":{},"output_type":"execute_result"}],"source":["#<------------------------------------------------------------------------------------------------------------------------------------------>\n","#Submission file\n","sub\n","#<------------------------------------------------------------------------------------------------------------------------------------------>"]}],"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"databundleVersionId":7516023,"sourceId":61542,"sourceType":"competition"},{"datasetId":4005256,"sourceId":6977472,"sourceType":"datasetVersion"}],"dockerImageVersionId":30648,"isGpuEnabled":true,"isInternetEnabled":false,"language":"python","sourceType":"notebook"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"}},"nbformat":4,"nbformat_minor":4}