HW3/HW3_Part2.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DHvhhpBU-QtI"
      },
      "source": [
        "# Homework 3 Part 2\n",
        "\n",
        "## Course Name: Deep Learning\n",
        "#### Lecturers: Dr. Beigy\n",
        "\n",
        "---\n",
        "\n",
        "#### Notebooks Supervised By: Zeinab Sadat Taghavi\n",
        "#### Notebooks Prepared By: Zahra Khoramnejad, Mehran Sarmadi, Zahra Rahimi\n",
        "\n",
        "**Contact**: Ask your questions in Quera\n",
        "\n",
        "---\n",
        "\n",
        "### Instructions:\n",
        "- Complete all exercises presented in this notebook.\n",
        "- Ensure you run each cell after you've entered your solution.\n",
        "- After completing the exercises, save the notebook and <font color='red'>follow the submission guidelines provided in the PDF.</font>\n",
        "\n",
        "\n",
        "---\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LG6yNYAmreI_"
      },
      "source": [
        "#Text Generation\n",
        "\n",
        "<p align='justify'>Text generation task involves generating new text based on a given input or a prompt. It is a natural language processing (NLP) task that aims to generate coherent and contextually relevant text.\n",
        "\n",
        "In text generation, a model is trained on a large corpus of text data and learns the patterns and structures of the language. This model can then be used to generate new text by sampling from the learned distribution of words or characters.\n",
        "\n",
        "Text generation has various applications, including chatbots, language translation, poetry generation, and content creation. It can be implemented using different techniques such as `recurrent neural networks (RNNs)`, `transformers`, and `Markov chains`.\n",
        "\n",
        "The goal of text generation is to produce text that is fluent, coherent, and contextually relevant. It requires a deep understanding of the language and the ability to generate text that follows grammatical rules and maintains semantic coherence.</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-u6fGIqx0LIz"
      },
      "source": [
        "##Charachter-level text generation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cKhboP-Y0TrV"
      },
      "source": [
        "One stage of the task of text generation is mapping, which can be at the word or character level. At this stage, a number is assigned to each word or character.\n",
        "\n",
        "In this exercise, we generate text at the character level. Because generating text at the word level, even though it leads to more meaningful outputs, requires a rich dataset with a high number of word repetitions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "00sNsfmXtW6K"
      },
      "source": [
        "We will implement models based on `recurrent networks` for text generation and compare the performance of different models. In the following, we will check the performance of the best models on different datasets and compare the results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jw8uJvD8vTyM"
      },
      "source": [
        "The steps of this exercise are as follows:\n",
        "1. Train RNN and LSTM\n",
        "2. FineTuning\n",
        "3. Experiment on different datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nb9egMrmAhXK"
      },
      "source": [
        "---\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBqL42w1vt1Q"
      },
      "source": [
        "#1. Train RNN and LSTM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dP0JQJIj2rL0"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x7cTJANKwpck",
        "outputId": "c9c4c1c7-0a05-442d-ac66-48224f432201"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "cuda\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from torch.distributions import Categorical\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import random\n",
        "import re\n",
        "import string\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hJSsyWUyv8KC"
      },
      "source": [
        "## Load data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oHGryhW7wbsZ"
      },
      "source": [
        "- We use the dataset of `Shakespeare's plays` as the main dataset for this exercise"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eUNd0YrFwW2E",
        "outputId": "0dccf369-b697-4772-b087-c76212145541"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-12-06 21:03:47--  https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1115394 (1.1M) [text/plain]\n",
            "Saving to: ‘data/input.txt’\n",
            "\n",
            "input.txt           100%[===================>]   1.06M  --.-KB/s    in 0.05s   \n",
            "\n",
            "2023-12-06 21:03:47 (22.4 MB/s) - ‘data/input.txt’ saved [1115394/1115394]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget \"https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\" -c -P {'data/'}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hYntrpS_Arse"
      },
      "source": [
        "- Load data in amout of 30kb for training models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "xYsndtIoxd0Z"
      },
      "outputs": [],
      "source": [
        "sh_data_file = \"./data/input.txt\"\n",
        "sh_data = open(sh_data_file, 'r').read(30000)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pj-xesSRwI9S"
      },
      "source": [
        "##Charachter mapping"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jl-LIH_Syl0M"
      },
      "source": [
        "- For better performance of the model, we limit the set of allowed characters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "KLiOHN8sCSe2"
      },
      "outputs": [],
      "source": [
        "chars = list(string.ascii_lowercase + '\\n' + ' ' + ':' + '.')\n",
        "vocab_size = len(chars)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "xvP9iW3FyT6g"
      },
      "outputs": [],
      "source": [
        "# Mapping of char-index\n",
        "char_to_ix = { ch:i for i,ch in enumerate(chars) }\n",
        "ix_to_char = { i:ch for i,ch in enumerate(chars) }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nh8Fo1WnwC9T"
      },
      "source": [
        "##Preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "vxGtjsgay0x9"
      },
      "outputs": [],
      "source": [
        "def remove_extraneous_characters(data, valid_char_list):\n",
        "    pattern = f\"[^{re.escape(''.join(valid_char_list))}]\"\n",
        "    return re.sub(pattern, '', data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "KscUSBpbyRNZ"
      },
      "outputs": [],
      "source": [
        "sh_data = remove_extraneous_characters(sh_data.lower(), chars)\n",
        "sh_data_size = len(sh_data)\n",
        "\n",
        "# Extract indexes of data characters\n",
        "sh_data = list(sh_data)\n",
        "for i, ch in enumerate(sh_data):\n",
        "    sh_data[i] = char_to_ix[ch]\n",
        "\n",
        "sh_data = torch.tensor(sh_data).to(device)\n",
        "sh_data = torch.unsqueeze(sh_data, dim=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Df7TxCxOzueM"
      },
      "source": [
        "##Modeling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rsj67WalA4fY"
      },
      "source": [
        "- In this part define RNN and LSTM model, according to the mentioned characteristics and function inputs.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kFj9nkzoz2Hs"
      },
      "source": [
        "###RNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "fgmKXfb1zuAJ"
      },
      "outputs": [],
      "source": [
        "class RNN(nn.Module):\n",
        "    def __init__(self, input_size, output_size, hidden_size=512, num_layers=3, dropout_enable=False):\n",
        "        super(RNN, self).__init__()\n",
        "        self.input_size = input_size\n",
        "        self.output_size = output_size\n",
        "        self.hidden_size = hidden_size\n",
        "        self.num_layers = num_layers\n",
        "        self.dropout_enable = dropout_enable\n",
        "        self.dropout = nn.Dropout(0.5)\n",
        "        self.hidden_state = None\n",
        "        ####################################\n",
        "        ######### Your code begins #########\n",
        "        ####################################\n",
        "        # Define self.rnn with model inputs\n",
        "        # Define self.decoder for decoding output character from last hidden state\n",
        "        # You can use torch.nn library\n",
        "\n",
        "        self.encoder = nn.Embedding(input_size, input_size)\n",
        "        self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True)\n",
        "        self.decoder = nn.Linear(hidden_size, output_size)\n",
        "\n",
        "        ####################################\n",
        "        ######### Your code ends ###########\n",
        "        ####################################\n",
        "\n",
        "\n",
        "    def forward(self, input_seq, hidden_state=None):\n",
        "        ####################################\n",
        "        ######### Your code begins #########\n",
        "        ####################################\n",
        "        # Implement forward part of model and save last hidden state on self.hidden_state\n",
        "        # print(f'{input_seq.size()=}')\n",
        "        x_encoded = self.encoder(input_seq)\n",
        "        # print(f'{x_encoded.size()=}')\n",
        "        output, hn = self.rnn(x_encoded, hidden_state)\n",
        "        # print(f'{output.size()=}')\n",
        "        self.hidden_state = hn.detach()\n",
        "\n",
        "        if self.dropout_enable:\n",
        "            output = self.dropout(output)\n",
        "        output = self.decoder(output)\n",
        "\n",
        "        ####################################\n",
        "        ######### Your code ends ###########\n",
        "        ####################################\n",
        "\n",
        "        return output\n",
        "\n",
        "    def save_model(self, path):\n",
        "        torch.save(self.state_dict(), path)\n",
        "\n",
        "    def load_model(self, path):\n",
        "        self.load_state_dict(torch.load(path))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xeoymmMx7RKR"
      },
      "source": [
        "###LSTM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "4PQDAnxc7Q26"
      },
      "outputs": [],
      "source": [
        "class LSTM(nn.Module):\n",
        "    def __init__(self, input_size, output_size, hidden_size=512, num_layers=3, dropout_enable=False):\n",
        "        super(LSTM, self).__init__()\n",
        "        self.input_size = input_size\n",
        "        self.output_size = output_size\n",
        "        self.hidden_size = hidden_size\n",
        "        self.num_layers = num_layers\n",
        "        self.dropout_enable = dropout_enable\n",
        "        self.dropout = nn.Dropout(0.5)\n",
        "        self.hidden_state = None\n",
        "\n",
        "        ####################################\n",
        "        ######### Your code begins #########\n",
        "        ####################################\n",
        "        # Define self.lstm with model inputs\n",
        "        # Define self.decoder for decoding output character from last hidden state\n",
        "        # You can use torch.nn library\n",
        "\n",
        "        self.encoder = nn.Embedding(input_size, input_size)\n",
        "        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)\n",
        "        self.decoder = nn.Linear(hidden_size, output_size)\n",
        "\n",
        "        ####################################\n",
        "        ######### Your code ends ###########\n",
        "        ####################################\n",
        "\n",
        "    def forward(self, input_seq, hidden_state=None):\n",
        "        ####################################\n",
        "        ######### Your code begins #########\n",
        "        ####################################\n",
        "        # Implement forward part of model and save last hidden state on self.hidden_state\n",
        "        x_encoded = self.encoder(input_seq)\n",
        "        output, hn = self.lstm(x_encoded, hidden_state)\n",
        "        self.hidden_state = (hn[0].detach(), hn[1].detach())\n",
        "\n",
        "        if self.dropout_enable:\n",
        "            output = self.dropout(output)\n",
        "        output = self.decoder(output)\n",
        "\n",
        "        ####################################\n",
        "        ######### Your code ends ###########\n",
        "        ####################################\n",
        "\n",
        "        return output\n",
        "\n",
        "    def save_model(self, path):\n",
        "        torch.save(self.state_dict(), path)\n",
        "\n",
        "    def load_model(self, path):\n",
        "        self.load_state_dict(torch.load(path))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y9XQWqoXwM4N"
      },
      "source": [
        "##Training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "ruPWrO7Z9JM2"
      },
      "outputs": [],
      "source": [
        "def print_sample_output(model, data, data_size, test_output_len = 200):\n",
        "    # Use this function to print sample that model generates from its current hidden state and random input character\n",
        "    # test_output_len is total num of characters in output test sequence\n",
        "\n",
        "    test_output = \"\"\n",
        "    data_ptr = 0\n",
        "\n",
        "    rand_index = np.random.randint(data_size-1)\n",
        "    input_seq = data[rand_index : rand_index+1]\n",
        "\n",
        "    model.hidden_state = None\n",
        "\n",
        "    while True:\n",
        "        output = model(input_seq, model.hidden_state)\n",
        "\n",
        "        output = F.softmax(torch.squeeze(output), dim=0)\n",
        "        dist = Categorical(output)\n",
        "        index = dist.sample().item()\n",
        "\n",
        "        test_output += ix_to_char[index]\n",
        "\n",
        "        input_seq[0][0] = index\n",
        "        data_ptr += 1\n",
        "\n",
        "        if data_ptr > test_output_len:\n",
        "            break\n",
        "\n",
        "    print(\"Train Sample +++++++++++++++++++++++++++++++++++++++++++++\")\n",
        "    print(test_output)\n",
        "    print(\"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WC337ho-XX-I"
      },
      "source": [
        "- For construction of each sample in the dataset, the output sequence is\n",
        "obtained from the shift of one character from the input sequence. For example, when sequence_length is 10 and our text is `Hello world`. The input sequence would be `Hello worl`, and the target sequence `ello world`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "aPpDSsZU7vWu"
      },
      "outputs": [],
      "source": [
        "def train_epoch(model, data, data_size, epoch, optimizer, seq_len=200):\n",
        "    # seq_length is length of training data sequence\n",
        "    model.train()\n",
        "    model = model.to(device)\n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "    total_loss = 0\n",
        "    sample_number = 0\n",
        "\n",
        "    ####################################\n",
        "    ######### Your code begins #########\n",
        "    ####################################\n",
        "    # Define training process for one epoch of input model\n",
        "    # At the end of every ten epochs, print current loss and a sample output of the model using print_sample_output function\n",
        "    # Feed all data sample to model by iterating over input data\n",
        "    model.hidden_state = None\n",
        "\n",
        "    for idx in range(seq_len, data_size - seq_len, 2*seq_len):\n",
        "        X_train = data[idx:idx+seq_len]\n",
        "        y_train = data[idx+1:idx+1+seq_len]\n",
        "\n",
        "        # Forward and loss\n",
        "        optimizer.zero_grad()\n",
        "        output = model(X_train, model.hidden_state)\n",
        "        train_loss = criterion(output.squeeze(), y_train.squeeze())\n",
        "        total_loss += train_loss.item()\n",
        "        sample_number += 1\n",
        "\n",
        "        # Backward and optimization\n",
        "        train_loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "    if epoch % 10 == 9:\n",
        "        with torch.no_grad():\n",
        "            model.eval()\n",
        "            print(f'Loss [{epoch=}] = {total_loss / sample_number}')\n",
        "            print_sample_output(model, data, data_size)\n",
        "            model.hidden_state = None\n",
        "\n",
        "    ####################################\n",
        "    ######### Your code ends ###########\n",
        "    ####################################\n",
        "\n",
        "    return total_loss / sample_number"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "1NT_mZuMf4do"
      },
      "outputs": [],
      "source": [
        "def train_rnn(data, data_size, model_save_file):\n",
        "    # RNN parameters\n",
        "    hidden_size = 512\n",
        "    num_layers = 6\n",
        "    lr = 0.002\n",
        "    epoch_num = 100\n",
        "    losses = [np.inf]\n",
        "\n",
        "    ####################################\n",
        "    ######### Your code begins #########\n",
        "    ####################################\n",
        "    # Define training process in the specified number of epochs for RNN model\n",
        "    # Use train_epoch function for train the model for one epoch\n",
        "    # Use Adam as optimizer\n",
        "    # Save best model in model_save_file address for next usage\n",
        "    rnn_model = RNN(vocab_size, vocab_size, hidden_size, num_layers, dropout_enable=False)\n",
        "    print(rnn_model)\n",
        "    optimizer = torch.optim.Adam(rnn_model.parameters(), lr)\n",
        "\n",
        "    data = data.to(device)\n",
        "    # data_oh = F.one_hot(data.squeeze(), vocab_size).to(torch.float32)\n",
        "\n",
        "    # dataset = torch.utils.data.TensorDataset(*prepare_dataset(data_oh, look_back=200))\n",
        "    # dataloader = torch.utils.data.DataLoader(dataset, batch_size=1)\n",
        "\n",
        "    for epoch in range(epoch_num):\n",
        "        print(f'{epoch=}')\n",
        "        loss = train_epoch(rnn_model, data, data_size, epoch, optimizer, seq_len=200)\n",
        "        if loss < losses[-1]:\n",
        "            rnn_model.save_model(model_save_file)\n",
        "            print(f'current model with {loss=} saved at {model_save_file}')\n",
        "\n",
        "        losses.append(loss)\n",
        "\n",
        "    ####################################\n",
        "    ######### Your code ends ###########\n",
        "    ####################################\n",
        "\n",
        "    return losses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "gd-wyFT4gerv"
      },
      "outputs": [],
      "source": [
        "def train_lstm(data, data_size, model_save_file):\n",
        "    # LSTM parameters\n",
        "    hidden_size = 512\n",
        "    num_layers = 3\n",
        "    lr = 0.002\n",
        "    epoch_num = 100\n",
        "    losses = [np.inf]\n",
        "\n",
        "    ####################################\n",
        "    ######### Your code begins #########\n",
        "    ####################################\n",
        "    # Define training process in the specified number of epochs for LSTM model\n",
        "    # Use train_epoch function for train the model for one epoch\n",
        "    # Use Adam as optimizer\n",
        "    # Save best model in model_save_file address for next usage\n",
        "    lstm_model = LSTM(vocab_size, vocab_size, hidden_size, num_layers, dropout_enable=False)\n",
        "    optimizer = torch.optim.Adam(lstm_model.parameters(), lr)\n",
        "\n",
        "    data = data.to(device)\n",
        "    # data_oh = F.one_hot(data.squeeze(), vocab_size).to(torch.float32)\n",
        "\n",
        "    # dataset = torch.utils.data.TensorDataset(*prepare_dataset(data_oh, look_back=200))\n",
        "    # dataloader = torch.utils.data.DataLoader(dataset, batch_size=1)\n",
        "\n",
        "    for epoch in range(epoch_num):\n",
        "        print(f'{epoch=}')\n",
        "        loss = train_epoch(lstm_model, data, data_size, epoch, optimizer, seq_len=200)\n",
        "        if loss < losses[-1]:\n",
        "            lstm_model.save_model(model_save_file)\n",
        "            print(f'current model with {loss=} saved at {model_save_file}')\n",
        "\n",
        "        losses.append(loss)\n",
        "\n",
        "    ####################################\n",
        "    ######### Your code ends ###########\n",
        "    ####################################\n",
        "\n",
        "    return losses"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KW4HTBwBEoHC"
      },
      "source": [
        "### RNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-mA-QGewD22i",
        "outputId": "75493d2f-6134-49fd-e016-5b5d18652f35"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "RNN(\n",
            "  (dropout): Dropout(p=0.5, inplace=False)\n",
            "  (encoder): Embedding(30, 30)\n",
            "  (rnn): RNN(30, 512, num_layers=6, batch_first=True)\n",
            "  (decoder): Linear(in_features=512, out_features=30, bias=True)\n",
            ")\n",
            "epoch=0\n",
            "current model with loss=2.8352030085192785 saved at ./model_sh_rnn.pth\n",
            "epoch=1\n",
            "current model with loss=2.660853925678465 saved at ./model_sh_rnn.pth\n",
            "epoch=2\n",
            "current model with loss=2.617702477508121 saved at ./model_sh_rnn.pth\n",
            "epoch=3\n",
            "current model with loss=2.59756773047977 saved at ./model_sh_rnn.pth\n",
            "epoch=4\n",
            "current model with loss=2.571855753660202 saved at ./model_sh_rnn.pth\n",
            "epoch=5\n",
            "current model with loss=2.5467909740077124 saved at ./model_sh_rnn.pth\n",
            "epoch=6\n",
            "current model with loss=2.5281242893801794 saved at ./model_sh_rnn.pth\n",
            "epoch=7\n",
            "current model with loss=2.5223184559080334 saved at ./model_sh_rnn.pth\n",
            "epoch=8\n",
            "current model with loss=2.5123447842068143 saved at ./model_sh_rnn.pth\n",
            "epoch=9\n",
            "Loss [epoch=9] = 2.498099704583486\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " is ar areruss o f ifey sof tofileser meinithaneres mad ili t tro t thid bdisieerlitinous avevenofs.\n",
            "mese thirbr. yefowethk t hkiut o siis: af thurfae ss chelirous s\n",
            "yimadet thererore amowe urero hary \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.498099704583486 saved at ./model_sh_rnn.pth\n",
            "epoch=10\n",
            "current model with loss=2.484671311246024 saved at ./model_sh_rnn.pth\n",
            "epoch=11\n",
            "current model with loss=2.4725699755880566 saved at ./model_sh_rnn.pth\n",
            "epoch=12\n",
            "current model with loss=2.462332288424174 saved at ./model_sh_rnn.pth\n",
            "epoch=13\n",
            "current model with loss=2.45271353258027 saved at ./model_sh_rnn.pth\n",
            "epoch=14\n",
            "current model with loss=2.4378358953528934 saved at ./model_sh_rnn.pth\n",
            "epoch=15\n",
            "current model with loss=2.4248329831494226 saved at ./model_sh_rnn.pth\n",
            "epoch=16\n",
            "current model with loss=2.419968134827084 saved at ./model_sh_rnn.pth\n",
            "epoch=17\n",
            "current model with loss=2.407211641470591 saved at ./model_sh_rnn.pth\n",
            "epoch=18\n",
            "current model with loss=2.387348006169001 saved at ./model_sh_rnn.pth\n",
            "epoch=19\n",
            "Loss [epoch=19] = 2.3831848171022205\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " te pe this:\n",
            "the h oriuiges:\n",
            "athous::\n",
            "\n",
            "\n",
            "\n",
            "ngir surthatr. omin ius:\n",
            "th.\n",
            "pa geyucherens iesm talarellynd shelus:\n",
            "ad sepopanga ados\n",
            "ty chetigoundigcblayomssther liurt te thean theeans: bak bdvis.\n",
            "ted thar \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.3831848171022205 saved at ./model_sh_rnn.pth\n",
            "epoch=20\n",
            "current model with loss=2.370649026499854 saved at ./model_sh_rnn.pth\n",
            "epoch=21\n",
            "current model with loss=2.366863032182058 saved at ./model_sh_rnn.pth\n",
            "epoch=22\n",
            "current model with loss=2.3551866677072315 saved at ./model_sh_rnn.pth\n",
            "epoch=23\n",
            "current model with loss=2.3350926604535847 saved at ./model_sh_rnn.pth\n",
            "epoch=24\n",
            "current model with loss=2.334799435403612 saved at ./model_sh_rnn.pth\n",
            "epoch=25\n",
            "current model with loss=2.3293947080771127 saved at ./model_sh_rnn.pth\n",
            "epoch=26\n",
            "current model with loss=2.312715662850274 saved at ./model_sh_rnn.pth\n",
            "epoch=27\n",
            "current model with loss=2.299125947886043 saved at ./model_sh_rnn.pth\n",
            "epoch=28\n",
            "epoch=29\n",
            "Loss [epoch=29] = 2.282990041706297\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " mat e fomas:\n",
            "tel ss tolsmesolthe thetesollou t bsel iananeex chem hary tome ayen harot eigeved theen\n",
            "m thas mold br r allue the themearthomerchusure ad\n",
            "fbmunithlasevetrird mbe inothe\n",
            "r f thercghemey s\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.282990041706297 saved at ./model_sh_rnn.pth\n",
            "epoch=30\n",
            "current model with loss=2.2673443108797073 saved at ./model_sh_rnn.pth\n",
            "epoch=31\n",
            "current model with loss=2.2481847289535732 saved at ./model_sh_rnn.pth\n",
            "epoch=32\n",
            "current model with loss=2.2386317435238094 saved at ./model_sh_rnn.pth\n",
            "epoch=33\n",
            "current model with loss=2.223937796221839 saved at ./model_sh_rnn.pth\n",
            "epoch=34\n",
            "current model with loss=2.2045543409056134 saved at ./model_sh_rnn.pth\n",
            "epoch=35\n",
            "current model with loss=2.189296164446407 saved at ./model_sh_rnn.pth\n",
            "epoch=36\n",
            "current model with loss=2.1699582984050116 saved at ./model_sh_rnn.pth\n",
            "epoch=37\n",
            "current model with loss=2.164934174882041 saved at ./model_sh_rnn.pth\n",
            "epoch=38\n",
            "current model with loss=2.15168060362339 saved at ./model_sh_rnn.pth\n",
            "epoch=39\n",
            "Loss [epoch=39] = 2.1374511983659534\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "argomamum oupr ai h iande thu da t br t stedd whenure ndelar\n",
            "\n",
            "thilius t che thefa i es thelupll alul buporthe is ifoofrsus has rwe bbt:\n",
            "bfibtout priapl lowafea o thelathinoulinouts mom\n",
            "ovalirde orvy be\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.1374511983659534 saved at ./model_sh_rnn.pth\n",
            "epoch=40\n",
            "current model with loss=2.1268373214536243 saved at ./model_sh_rnn.pth\n",
            "epoch=41\n",
            "current model with loss=2.114711339275042 saved at ./model_sh_rnn.pth\n",
            "epoch=42\n",
            "current model with loss=2.08397633002864 saved at ./model_sh_rnn.pth\n",
            "epoch=43\n",
            "current model with loss=2.0695427970753775 saved at ./model_sh_rnn.pth\n",
            "epoch=44\n",
            "current model with loss=2.0516132331556745 saved at ./model_sh_rnn.pth\n",
            "epoch=45\n",
            "current model with loss=2.035863048500485 saved at ./model_sh_rnn.pth\n",
            "epoch=46\n",
            "current model with loss=2.0049240903721914 saved at ./model_sh_rnn.pth\n",
            "epoch=47\n",
            "current model with loss=1.975260328915384 saved at ./model_sh_rnn.pth\n",
            "epoch=48\n",
            "current model with loss=1.9696572340197034 saved at ./model_sh_rnn.pth\n",
            "epoch=49\n",
            "Loss [epoch=49] = 1.9376034057802625\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "stho\n",
            "lnce tdyes brs:\n",
            "aleand al\n",
            "fididfeni our pelndiu bus er an lyila he he ometheallitheme thes:\n",
            "yotouvartsthe iragaretowhene t hever n\n",
            "ato yowhombreiut rs che io nghe tusyo aid t b a s arf po martes f\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=1.9376034057802625 saved at ./model_sh_rnn.pth\n",
            "epoch=50\n",
            "epoch=51\n",
            "current model with loss=1.918004497885704 saved at ./model_sh_rnn.pth\n",
            "epoch=52\n",
            "current model with loss=1.8836168895165126 saved at ./model_sh_rnn.pth\n",
            "epoch=53\n",
            "current model with loss=1.865894690155983 saved at ./model_sh_rnn.pth\n",
            "epoch=54\n",
            "current model with loss=1.838755899005466 saved at ./model_sh_rnn.pth\n",
            "epoch=55\n",
            "current model with loss=1.8197803414530225 saved at ./model_sh_rnn.pth\n",
            "epoch=56\n",
            "current model with loss=1.8009491480059094 saved at ./model_sh_rnn.pth\n",
            "epoch=57\n",
            "current model with loss=1.7630114687813654 saved at ./model_sh_rnn.pth\n",
            "epoch=58\n",
            "current model with loss=1.734729043311543 saved at ./model_sh_rnn.pth\n",
            "epoch=59\n",
            "Loss [epoch=59] = 1.6997449398040771\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "e:\n",
            "tennvednotes:\n",
            "at foll afine mid ss thoutheange ifret therd rg\n",
            "ame ir ous t ce d prfitheflewy l\n",
            "soll ay s bibe ian fioundayome he thati m ad.\n",
            "atharachs\n",
            "the i m\n",
            "tas e bo th thefiu hroraliuamerd opall \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=1.6997449398040771 saved at ./model_sh_rnn.pth\n",
            "epoch=60\n",
            "current model with loss=1.6761045025454626 saved at ./model_sh_rnn.pth\n",
            "epoch=61\n",
            "current model with loss=1.6549388435151842 saved at ./model_sh_rnn.pth\n",
            "epoch=62\n",
            "current model with loss=1.6368882093164656 saved at ./model_sh_rnn.pth\n",
            "epoch=63\n",
            "current model with loss=1.6078510582447052 saved at ./model_sh_rnn.pth\n",
            "epoch=64\n",
            "current model with loss=1.5951722396744623 saved at ./model_sh_rnn.pth\n",
            "epoch=65\n",
            "current model with loss=1.546227799521552 saved at ./model_sh_rnn.pth\n",
            "epoch=66\n",
            "current model with loss=1.5145586878061295 saved at ./model_sh_rnn.pth\n",
            "epoch=67\n",
            "current model with loss=1.4747761620415583 saved at ./model_sh_rnn.pth\n",
            "epoch=68\n",
            "current model with loss=1.4723377318845854 saved at ./model_sh_rnn.pth\n",
            "epoch=69\n",
            "Loss [epoch=69] = 1.427802287042141\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            ".k ow che che ckg\n",
            "he cie\n",
            "y the er\n",
            "\n",
            "ndasdyoueava:\n",
            "yoff babiswe ckid ifouthearitissee che e t besir moomouligas hima arers the be ouan\n",
            "alar\n",
            "ave ble ldinend ig rcugursendiue\n",
            "wakasheshil hoo phercaicorn m \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=1.427802287042141 saved at ./model_sh_rnn.pth\n",
            "epoch=70\n",
            "current model with loss=1.3964105968674023 saved at ./model_sh_rnn.pth\n",
            "epoch=71\n",
            "current model with loss=1.3691244671742122 saved at ./model_sh_rnn.pth\n",
            "epoch=72\n",
            "current model with loss=1.3364344007439084 saved at ./model_sh_rnn.pth\n",
            "epoch=73\n",
            "current model with loss=1.3127963145573933 saved at ./model_sh_rnn.pth\n",
            "epoch=74\n",
            "current model with loss=1.2827789030141301 saved at ./model_sh_rnn.pth\n",
            "epoch=75\n",
            "current model with loss=1.281608218120204 saved at ./model_sh_rnn.pth\n",
            "epoch=76\n",
            "current model with loss=1.2493000932865672 saved at ./model_sh_rnn.pth\n",
            "epoch=77\n",
            "current model with loss=1.204523916873667 saved at ./model_sh_rnn.pth\n",
            "epoch=78\n",
            "current model with loss=1.1742934981981914 saved at ./model_sh_rnn.pth\n",
            "epoch=79\n",
            "Loss [epoch=79] = 1.167004805472162\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "fur e ar s:\n",
            "awe bo s m rce ifiuri n m d sishen ben he this\n",
            "is ad ghelou siuf wefine ereeaseler be tir m o h tesiuear brid s:\n",
            "\n",
            "nk t f ay t the bedn ck\n",
            "aw casidyoul be the benchatiushen wiuli whe athaius\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=1.167004805472162 saved at ./model_sh_rnn.pth\n",
            "epoch=80\n",
            "current model with loss=1.129109828422467 saved at ./model_sh_rnn.pth\n",
            "epoch=81\n",
            "current model with loss=1.0967334401276376 saved at ./model_sh_rnn.pth\n",
            "epoch=82\n",
            "current model with loss=1.0806601233780384 saved at ./model_sh_rnn.pth\n",
            "epoch=83\n",
            "current model with loss=1.0295444826285045 saved at ./model_sh_rnn.pth\n",
            "epoch=84\n",
            "current model with loss=1.0132819914983378 saved at ./model_sh_rnn.pth\n",
            "epoch=85\n",
            "current model with loss=0.9738812872933017 saved at ./model_sh_rnn.pth\n",
            "epoch=86\n",
            "current model with loss=0.9700127231578032 saved at ./model_sh_rnn.pth\n",
            "epoch=87\n",
            "current model with loss=0.9365540424154865 saved at ./model_sh_rnn.pth\n",
            "epoch=88\n",
            "current model with loss=0.9097151503794723 saved at ./model_sh_rnn.pth\n",
            "epoch=89\n",
            "Loss [epoch=89] = 0.8836197699937556\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "ese prso wintheand ire anos o bsias\n",
            "st hendou mechisin: casiroman pe o bserthee o ts: hemadive o ve pind ressicok\n",
            "lly plisd ani ge woethend\n",
            "\n",
            "t ve wofes blnd oescome heabe bey shy ce iseve hea grde or p\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.8836197699937556 saved at ./model_sh_rnn.pth\n",
            "epoch=90\n",
            "current model with loss=0.8734603259298537 saved at ./model_sh_rnn.pth\n",
            "epoch=91\n",
            "current model with loss=0.8581640844543775 saved at ./model_sh_rnn.pth\n",
            "epoch=92\n",
            "current model with loss=0.8440428115427494 saved at ./model_sh_rnn.pth\n",
            "epoch=93\n",
            "current model with loss=0.8093114445606867 saved at ./model_sh_rnn.pth\n",
            "epoch=94\n",
            "epoch=95\n",
            "current model with loss=0.7966860394097037 saved at ./model_sh_rnn.pth\n",
            "epoch=96\n",
            "current model with loss=0.7708117067813873 saved at ./model_sh_rnn.pth\n",
            "epoch=97\n",
            "current model with loss=0.7408132499290837 saved at ./model_sh_rnn.pth\n",
            "epoch=98\n",
            "current model with loss=0.7378117599421077 saved at ./model_sh_rnn.pth\n",
            "epoch=99\n",
            "Loss [epoch=99] = 0.7449434089163939\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "eall ghe the choa t gh the cie of corur shal ar ciesthet thend i a:\n",
            "\n",
            "t walldid s:\n",
            "aniesticiusonir s a s f pen silt shiowalopon one uthio le pey o he the hiudinintinould be the he cu\n",
            "\n",
            "ne t ceabat be the\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n"
          ]
        }
      ],
      "source": [
        "rnn_sh_losses = train_rnn(sh_data, sh_data_size, './model_sh_rnn.pth')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0LNMgGhjEqZn"
      },
      "source": [
        "### LSTM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "oA6zVNt6Ep7S",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "094a597f-1111-4e62-d2dd-290c72296bf0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "epoch=0\n",
            "current model with loss=2.9531518618265786 saved at ./model_sh_lstm.pth\n",
            "epoch=1\n",
            "current model with loss=2.5973354710472956 saved at ./model_sh_lstm.pth\n",
            "epoch=2\n",
            "current model with loss=2.4870109293195934 saved at ./model_sh_lstm.pth\n",
            "epoch=3\n",
            "current model with loss=2.4346225692166223 saved at ./model_sh_lstm.pth\n",
            "epoch=4\n",
            "current model with loss=2.3962298035621643 saved at ./model_sh_lstm.pth\n",
            "epoch=5\n",
            "current model with loss=2.3590226107173495 saved at ./model_sh_lstm.pth\n",
            "epoch=6\n",
            "current model with loss=2.3120613992214203 saved at ./model_sh_lstm.pth\n",
            "epoch=7\n",
            "current model with loss=2.2520479030079312 saved at ./model_sh_lstm.pth\n",
            "epoch=8\n",
            "current model with loss=2.1725487262010574 saved at ./model_sh_lstm.pth\n",
            "epoch=9\n",
            "Loss [epoch=9] = 2.0798637800746493\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "vort cuser:\n",
            "met njsgle be an thatheatheat t ndepriue dreallaiurindi amig cr allans t geiugiusugr bus pr m he bot\n",
            "yo vom ssofoud lur ct if\n",
            "aangur iol bu se t w d ghesthofgllu t the the tithom\n",
            "amiue miur\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.0798637800746493 saved at ./model_sh_lstm.pth\n",
            "epoch=10\n",
            "current model with loss=1.955585617158148 saved at ./model_sh_lstm.pth\n",
            "epoch=11\n",
            "current model with loss=1.8072873420185513 saved at ./model_sh_lstm.pth\n",
            "epoch=12\n",
            "current model with loss=1.5791271643506155 saved at ./model_sh_lstm.pth\n",
            "epoch=13\n",
            "current model with loss=1.3435245255629222 saved at ./model_sh_lstm.pth\n",
            "epoch=14\n",
            "current model with loss=1.1331234731607966 saved at ./model_sh_lstm.pth\n",
            "epoch=15\n",
            "current model with loss=0.9559995763831668 saved at ./model_sh_lstm.pth\n",
            "epoch=16\n",
            "current model with loss=0.7945040737589201 saved at ./model_sh_lstm.pth\n",
            "epoch=17\n",
            "current model with loss=0.6369625959131453 saved at ./model_sh_lstm.pth\n",
            "epoch=18\n",
            "current model with loss=0.49796301705969703 saved at ./model_sh_lstm.pth\n",
            "epoch=19\n",
            "Loss [epoch=19] = 0.37485692443119156\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "lifonelerurupe founonond cthecik\n",
            "hiushiul ss th anoourthollururelyove finshinon:\n",
            "wa andsshetorcoun\n",
            "ar fanpint t\n",
            "s thes:\n",
            "\n",
            "\n",
            "\n",
            "toouthe cousthes be t fone dse shelo the lliarouthe toupe he be t fitheathan r\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.37485692443119156 saved at ./model_sh_lstm.pth\n",
            "epoch=20\n",
            "current model with loss=0.2622793987393379 saved at ./model_sh_lstm.pth\n",
            "epoch=21\n",
            "current model with loss=0.18090731422934267 saved at ./model_sh_lstm.pth\n",
            "epoch=22\n",
            "current model with loss=0.12023668384386434 saved at ./model_sh_lstm.pth\n",
            "epoch=23\n",
            "current model with loss=0.08509707262015177 saved at ./model_sh_lstm.pth\n",
            "epoch=24\n",
            "current model with loss=0.06138885066482342 saved at ./model_sh_lstm.pth\n",
            "epoch=25\n",
            "current model with loss=0.04773462218387673 saved at ./model_sh_lstm.pth\n",
            "epoch=26\n",
            "current model with loss=0.04228940576690042 saved at ./model_sh_lstm.pth\n",
            "epoch=27\n",
            "current model with loss=0.038781032422169424 saved at ./model_sh_lstm.pth\n",
            "epoch=28\n",
            "current model with loss=0.03793157176793708 saved at ./model_sh_lstm.pth\n",
            "epoch=29\n",
            "Loss [epoch=29] = 0.03618636118254573\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "swnt thay od\n",
            "whe hasirt tox.\n",
            "marckean an hicove toulinthould ulewoulshen ns ithind tes\n",
            "\n",
            "vee he tcomather:\n",
            "\n",
            "nchollun s:\n",
            "we he pr sty had i hendasit t touns com inct t d illl owint tomas t t t the ourece\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.03618636118254573 saved at ./model_sh_lstm.pth\n",
            "epoch=30\n",
            "epoch=31\n",
            "current model with loss=0.03560947172647704 saved at ./model_sh_lstm.pth\n",
            "epoch=32\n",
            "current model with loss=0.0343875144042411 saved at ./model_sh_lstm.pth\n",
            "epoch=33\n",
            "current model with loss=0.03413231292021616 saved at ./model_sh_lstm.pth\n",
            "epoch=34\n",
            "current model with loss=0.03410483115941235 saved at ./model_sh_lstm.pth\n",
            "epoch=35\n",
            "epoch=36\n",
            "current model with loss=0.034310479795547306 saved at ./model_sh_lstm.pth\n",
            "epoch=37\n",
            "current model with loss=0.03393361032309864 saved at ./model_sh_lstm.pth\n",
            "epoch=38\n",
            "current model with loss=0.03331328822969226 saved at ./model_sh_lstm.pth\n",
            "epoch=39\n",
            "Loss [epoch=39] = 0.03352600965445163\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "y d inothecogoun mouswir ar s:\n",
            "\n",
            "mansthes he tustorend t.\n",
            "wa anoth mncof s thay the ourts:\n",
            "\n",
            "n t tofan thay casendiushard we be cind hau s ile he prous tallyouthens t t founar s th y itico tes he pr stiu\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "epoch=40\n",
            "current model with loss=0.033176524849194616 saved at ./model_sh_lstm.pth\n",
            "epoch=41\n",
            "epoch=42\n",
            "current model with loss=0.03266179101531937 saved at ./model_sh_lstm.pth\n",
            "epoch=43\n",
            "current model with loss=0.03258644175285655 saved at ./model_sh_lstm.pth\n",
            "epoch=44\n",
            "current model with loss=0.03236824326652115 saved at ./model_sh_lstm.pth\n",
            "epoch=45\n",
            "epoch=46\n",
            "current model with loss=0.03239283348294622 saved at ./model_sh_lstm.pth\n",
            "epoch=47\n",
            "epoch=48\n",
            "current model with loss=0.03251631734100455 saved at ./model_sh_lstm.pth\n",
            "epoch=49\n",
            "Loss [epoch=49] = 0.03282455019749856\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "wathe ofon anop hi oulathe thise the sprif t\n",
            "the qe the omathe the sprd we the the elenounoul sinot ars:\n",
            "\n",
            "ndeand he wamould inthins:\n",
            "\n",
            "ndeearey stius\n",
            "\n",
            "nd hinyone thesounche s\n",
            "te pe thendatiushar sgis\n",
            "\n",
            "w\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "epoch=50\n",
            "current model with loss=0.03269894687279399 saved at ./model_sh_lstm.pth\n",
            "epoch=51\n",
            "epoch=52\n",
            "current model with loss=0.03288088141481341 saved at ./model_sh_lstm.pth\n",
            "epoch=53\n",
            "epoch=54\n",
            "current model with loss=0.03289387363272706 saved at ./model_sh_lstm.pth\n",
            "epoch=55\n",
            "current model with loss=0.03285521181613957 saved at ./model_sh_lstm.pth\n",
            "epoch=56\n",
            "epoch=57\n",
            "epoch=58\n",
            "current model with loss=0.03305300825053968 saved at ./model_sh_lstm.pth\n",
            "epoch=59\n",
            "Loss [epoch=59] = 0.03286541297267023\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "roull wafan ve t fo che cthilll liburenethe pr sears\n",
            "th\n",
            "cknd\n",
            "wouthes:\n",
            "\n",
            "ndeand t toul cigan at thars bun wie t be t pr youlintho tes therd he pr stan cterthalos\n",
            "\n",
            "vear the thally wn:\n",
            "wa anole minghis\n",
            "\n",
            "ve\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.03286541297267023 saved at ./model_sh_lstm.pth\n",
            "epoch=60\n",
            "current model with loss=0.03251172414376116 saved at ./model_sh_lstm.pth\n",
            "epoch=61\n",
            "epoch=62\n",
            "current model with loss=0.032587204258460484 saved at ./model_sh_lstm.pth\n",
            "epoch=63\n",
            "current model with loss=0.03251672102002582 saved at ./model_sh_lstm.pth\n",
            "epoch=64\n",
            "epoch=65\n",
            "current model with loss=0.03235365044404009 saved at ./model_sh_lstm.pth\n",
            "epoch=66\n",
            "epoch=67\n",
            "current model with loss=0.03239129125924794 saved at ./model_sh_lstm.pth\n",
            "epoch=68\n",
            "epoch=69\n",
            "Loss [epoch=69] = 0.032678976683099686\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "lly fad uthe tousinburd o twhe ch thendathe s if be lie fin as s:\n",
            "\n",
            "\n",
            "\n",
            "s an the of be lize lin:\n",
            "wewine\n",
            "t th ther:\n",
            "we ie\n",
            "thee t fo ter o the s oulathes:\n",
            "\n",
            "n wi thaliushar se memushet y memathe sthe the the\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "epoch=70\n",
            "epoch=71\n",
            "current model with loss=0.033144565680534775 saved at ./model_sh_lstm.pth\n",
            "epoch=72\n",
            "epoch=73\n",
            "current model with loss=0.03264779258501625 saved at ./model_sh_lstm.pth\n",
            "epoch=74\n",
            "epoch=75\n",
            "current model with loss=0.03295015858626963 saved at ./model_sh_lstm.pth\n",
            "epoch=76\n",
            "current model with loss=0.03277904567120357 saved at ./model_sh_lstm.pth\n",
            "epoch=77\n",
            "epoch=78\n",
            "epoch=79\n",
            "Loss [epoch=79] = 0.03319976400780433\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " memathe s t hishen whe s porus t ts the.\n",
            "\n",
            "wan he prus t t the ourns t t founole choof mous\n",
            "\n",
            "ve wi cagofe ve tous thes:\n",
            "\n",
            "ne\n",
            "yowe incis\n",
            "inthe. ntind t to the ounandiushare ato te he ouneandius the is de\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.03319976400780433 saved at ./model_sh_lstm.pth\n",
            "epoch=80\n",
            "epoch=81\n",
            "current model with loss=0.0326079808247515 saved at ./model_sh_lstm.pth\n",
            "epoch=82\n",
            "epoch=83\n",
            "epoch=84\n",
            "epoch=85\n",
            "current model with loss=0.03306906927779588 saved at ./model_sh_lstm.pth\n",
            "epoch=86\n",
            "epoch=87\n",
            "current model with loss=0.03262016968720774 saved at ./model_sh_lstm.pth\n",
            "epoch=88\n",
            "epoch=89\n",
            "Loss [epoch=89] = 0.03303582124731798\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "eswitorg oaicot ce inonus are is borey oule be the lofir at pean whe toulinthoul he prourd o hacot peano tiren oninor anof be cha ar s ote cinchurouf vesthe the im inoren s m iren arn s:\n",
            "we we inoll wn\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.03303582124731798 saved at ./model_sh_lstm.pth\n",
            "epoch=90\n",
            "epoch=91\n",
            "epoch=92\n",
            "current model with loss=1.6268432819181018 saved at ./model_sh_lstm.pth\n",
            "epoch=93\n",
            "current model with loss=1.045176920791467 saved at ./model_sh_lstm.pth\n",
            "epoch=94\n",
            "current model with loss=0.6534194370938672 saved at ./model_sh_lstm.pth\n",
            "epoch=95\n",
            "current model with loss=0.44307451736595893 saved at ./model_sh_lstm.pth\n",
            "epoch=96\n",
            "current model with loss=0.3176865049948295 saved at ./model_sh_lstm.pth\n",
            "epoch=97\n",
            "current model with loss=0.25633229470501345 saved at ./model_sh_lstm.pth\n",
            "epoch=98\n",
            "current model with loss=0.19492695139100155 saved at ./model_sh_lstm.pth\n",
            "epoch=99\n",
            "Loss [epoch=99] = 0.15682106816934216\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "ithar owa th ghat t wilius the.\n",
            "tirghoulanf towe ha iu t omans bo the he h iall:\n",
            "sh theloul thesthe\n",
            "lewallilvestol thes t t tr o the pod nus bur wiuse: ste t\n",
            "\n",
            "\n",
            "\n",
            "te bsth lfir atiuso\n",
            "\n",
            "\n",
            "am m\n",
            "\n",
            "nenf wie had\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.15682106816934216 saved at ./model_sh_lstm.pth\n"
          ]
        }
      ],
      "source": [
        "lstm_sh_losses = train_lstm(sh_data, sh_data_size, './model_sh_lstm.pth')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tUGxc-Z1wRiP"
      },
      "source": [
        "##Generating texts"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "43rEOvbopt3s"
      },
      "source": [
        "- A sample text to input the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {
        "id": "2gkZC4UagpoB"
      },
      "outputs": [],
      "source": [
        "input_sample_text = 'First Citizen:\\nYou are all resolved rather to die than to famish?\\n'\n",
        "\n",
        "def create_input_sample_dataset(input_sample_text):\n",
        "    input_sample = remove_extraneous_characters(input_sample_text.lower(), chars)\n",
        "    input_sample = list(input_sample)\n",
        "    for i, ch in enumerate(input_sample):\n",
        "        input_sample[i] = char_to_ix[ch]\n",
        "\n",
        "    input_sample = torch.tensor(input_sample).to(device)\n",
        "    input_sample = torch.unsqueeze(input_sample, dim=1)\n",
        "    return input_sample"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7N1xZocpFIf"
      },
      "source": [
        "- This function generates the output generated by the model for the input sample, and if the input sample text is not given, it samples a sequence of original data and gives it to the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {
        "id": "zTFXBqKqII_j"
      },
      "outputs": [],
      "source": [
        "def generate_text(model, data, data_size, input_sample_test = None, output_len=1000):\n",
        "    model.eval()\n",
        "    data_ptr = 0\n",
        "    test_output=\"\"\n",
        "\n",
        "    if input_sample_test is not None:\n",
        "        index = 0\n",
        "        seq_len = len(input_sample_test)\n",
        "        input_seq = input_sample_test[index : index + seq_len-1]\n",
        "    else:\n",
        "        # If input sample not declared, select an initial string from the data of 10 characters randomly\n",
        "        index = np.random.randint(data_size - 11)\n",
        "        seq_len = 10\n",
        "        input_seq = data[index : index + 9]\n",
        "\n",
        "    model.hidden_state = None\n",
        "    # Set last hidden state of model by feeding input sequence to model\n",
        "    output = model(input_seq.to(torch.long), model.hidden_state)\n",
        "\n",
        "    # Last charachter feed to model\n",
        "    if input_sample_test is not None:\n",
        "        input_seq = input_sample_test[index + seq_len-1 : index + seq_len]\n",
        "    else:\n",
        "        input_seq = data[index + seq_len-1 : index + seq_len]\n",
        "\n",
        "    model.hidden_state = None\n",
        "    while True:\n",
        "        output = model(input_seq.to(torch.long), model.hidden_state)\n",
        "\n",
        "        output = F.softmax(torch.squeeze(output), dim=0)\n",
        "        dist = Categorical(output)\n",
        "        index = dist.sample().item()\n",
        "\n",
        "        test_output += ix_to_char[index]\n",
        "        input_seq[0][0] = index\n",
        "        data_ptr += 1\n",
        "\n",
        "        if data_ptr  > output_len:\n",
        "            break\n",
        "\n",
        "    print(\"Eaxmple of generated text --------------------------------------------------------------------------\")\n",
        "    print(test_output)\n",
        "    print(\"----------------------------------------------------------------------------------------------------\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MatZYKCUFOSn"
      },
      "source": [
        "### RNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "-oT9TsOBFQ9k",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c87cbfee-51f5-42cb-c72c-a92b8dfaa407"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "best loss 0.7378117599421077\n",
            "Eaxmple of generated text --------------------------------------------------------------------------\n",
            "len:\n",
            "wa\n",
            "t cey ithe oufal anom o hiciche o f that pekde athe oefieenod ow ce cou noueakoulieanendse:\n",
            "\n",
            "ven o hali m al s:\n",
            "we bo the s f tesive the st hen s p the ousheyor wolest himage wofen\n",
            "the our whar adn bl oninarit ceys urcis\n",
            "aditasoustys t s\n",
            "\n",
            "and\n",
            "tyof mole be po the sthen anou w brof o heysede aselire at fousthen s at fouve he the ceasheyoul bey shady bperad this\n",
            "ir gla oura:\n",
            "\n",
            "ns he the aselinist hiryondarery bey o ma bo the st s t t l at fersh s at way shan pe yomouthen o haposil\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "s fiduli anoman s ala:\n",
            "adr ourveyoueast t fowaloula he he nd\n",
            "thy ciu athe st we cr\n",
            "fo o s\n",
            "\n",
            "ce che wn ofive gofe hemesivoupou n mole enou e oushetiustornou ar gione at foustys\n",
            "fomadyoushetoun at fom enghe st hend s\n",
            "\n",
            "t fowadoun be oureenghe\n",
            "t the sthil ld\n",
            "an anos t the ouin ghe che alouthes:\n",
            "is\n",
            "\n",
            "vendeli nd s: auso aurcie\n",
            "iperthenid s:\n",
            "anonthealfius s\n",
            "\n",
            "t t tlldsear sey s bo the st fur wa d sutlizeengons\n",
            "ty cace cos o he thayane at o the s\n",
            "ty co ch iafis\n",
            "m the hinfinavo sit s\n",
            "\n",
            "thithomalobrerse:\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "s fo\n",
            "----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "best_model_rnn =  RNN(vocab_size, vocab_size, 512, 6).to(device)\n",
        "best_model_rnn.load_model('./model_sh_rnn.pth')\n",
        "print(\"best loss\", min(rnn_sh_losses))\n",
        "generate_text(best_model_rnn, sh_data, sh_data_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fwr90H6eFPqR"
      },
      "source": [
        "### LSTM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "R2DUmeREFUkc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd037b51-7ad5-43c9-99ca-7e31351c8d92"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "best loss 0.03235365044404009\n",
            "Eaxmple of generated text --------------------------------------------------------------------------\n",
            "is\n",
            "\n",
            "t hen nste ir ghathar ges\n",
            "tiueandius t t foman hhary lipealare blerd llian:\n",
            "aken are m atofiued ansthe harst the hare minchur fous iser\n",
            "fomofirn whe hr atre yo ther we ldius th ge minchurerstyouerif re oulintors t t folest coce theld. fumagonce inods by: t the otherst f h moun d e wadinof thacerd ines:\n",
            "yone he se urell o\n",
            "wh whe ha bo t d f fiat th fbrayof t th mal oto hrcorid th gus\n",
            "then t leians fius:\n",
            "whee w m d bld ve t t e pathere he pr\n",
            "pr n:\n",
            "ceierd y ter arr f tha f athou anothamerst then wher i a ghe piatyouke ind h ghilou andare mrathe ld. wame atot\n",
            "\n",
            "thalomestheatius beandius the ld lens t t to al cenice t printht plircius t t t pred llictitice are oular se.\n",
            "th l lear thar ger al thace ctherd wathor t itherd llert thasere meverdid y gimache ld bufatrthathamig p be imashe \n",
            "theashe the pie y ms: shierdry toul st then n.\n",
            "l pe hir aty pas th lll oniar ghagoul shil keandius the thize tr mathou anothago afonsnd hinct t perurell owa burersusen: gof t ben k therd wartusest\n",
            "whe h herd \n",
            "----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "best_model_lstm =  LSTM(vocab_size, vocab_size, 512, 3).to(device)\n",
        "best_model_lstm.load_model('./model_sh_lstm.pth')\n",
        "print(\"best loss\", min(lstm_sh_losses))\n",
        "generate_text(best_model_lstm, sh_data, sh_data_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_thNtUicv2kd"
      },
      "source": [
        "## Plotting the losses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "f_If3o7grbO8"
      },
      "outputs": [],
      "source": [
        "def plot_losses(losses):\n",
        "    xpoints = np.array(range(len(losses)))\n",
        "    ypoints = np.array(losses)\n",
        "\n",
        "    plt.plot(xpoints, ypoints, color='blue',label='losses')\n",
        "    plt.xlabel(\"epoch\")\n",
        "    plt.ylabel(\"loss\")\n",
        "    plt.legend()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylDZpjFtJ6Mo"
      },
      "source": [
        "### RNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "lvQC5FNzJ4Q1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "outputId": "0ead8322-7904-44a0-fc9f-e5b58e4f7b1a"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABBPklEQVR4nO3dd3hUZcL+8e8kIYUyCS0QIIEA0puIYsACr3RpgqCIAkoRDD+qioC6KAtxdVFEENsKKiCKCCJFehHpVTpEWED6UhJCCZCc3x/PEs1KScIkZ8r9ua65mMycCXfOuy+5PecpDsuyLERERES8hJ/dAURERERcSeVGREREvIrKjYiIiHgVlRsRERHxKio3IiIi4lVUbkRERMSrqNyIiIiIVwmwO0BOS01N5ejRo+TLlw+Hw2F3HBEREckAy7I4f/48xYoVw8/v1tdmfK7cHD16lMjISLtjiIiISBYcPnyYEiVK3PIYnys3+fLlA8zJcTqdNqcRERGRjEhMTCQyMjLt9/it+Fy5uX4ryul0qtyIiIh4mIwMKdGAYhEREfEqKjciIiLiVVRuRERExKv43JgbERGRrEhJSeHq1at2x/BqgYGBt53mnREqNyIiIrdgWRbHjx/n3Llzdkfxen5+fkRHRxMYGHhH30flRkRE5BauF5vw8HBy586tBWCzyfVFdo8dO0ZUVNQdnWeVGxERkZtISUlJKzYFCxa0O47XK1y4MEePHuXatWvkypUry99HA4pFRERu4voYm9y5c9ucxDdcvx2VkpJyR99H5UZEROQ2dCsqZ7jqPKvciIiIiFdRuRERERGvonIjIiLiherVq0e/fv3sjmELlRsXOnUKdu+2O4WIiIhvU7lxkblzITwcOnSwO4mIiIhvU7lxkXLlzJ+7d8MdzmATERE3Zllw4ULOPywr65nPnj1Lp06dyJ8/P7lz56Zp06bs27cv7f2DBw/SokUL8ufPT548eahcuTJz585N+2zHjh0pXLgwISEh3HXXXUyYMCHts4cPH6Z9+/aEhYVRoEABWrVqxb///e+095ctW8Z9991Hnjx5CAsLo27duhw8eDDrP0wGaBE/F4mOhqAguHwZDh6E0qXtTiQiItnh4kXImzfn/96kJMiTJ2uf7dKlC/v27WPWrFk4nU4GDRpEs2bN2LlzJ7ly5SI2NpYrV66wYsUK8uTJw86dO8n73x/ytddeY+fOncybN49ChQoRHx/PpUuXALMOUOPGjYmJieHnn38mICCAv//97zRp0oRff/0VPz8/WrduTffu3fn666+5cuUK69aty/ap9So3LuLvDxUqwNatsHOnyo2IiLiH66Xml19+oU6dOgBMnjyZyMhIZs6cSbt27Th06BBt27alatWqAJT+0y+xQ4cOcffdd1OrVi0ASpUqlfbeN998Q2pqKp999llaYZkwYQJhYWEsW7aMWrVqkZCQQPPmzSlTpgwAFStWzPafWeXGhSpV+qPcNG9udxoREckOuXObqyh2/L1ZsWvXLgICAqhdu3baawULFqR8+fLs2rULgD59+tCrVy8WLFhAgwYNaNu2LdWqVQOgV69etG3blk2bNtGoUSNat26dVpK2bt1KfHw8+fLlS/d3Xr58md9++41GjRrRpUsXGjduTMOGDWnQoAHt27cnIiIiaz9MBmnMjQtdL6M7d9qbQ0REso/DYW4P5fQjO+/kdOvWjf379/PMM8+wbds2atWqxQcffABA06ZNOXjwIP379+fo0aM88sgjvPjiiwAkJSVxzz33sGXLlnSPvXv38tRTTwHmSs7q1aupU6cO33zzDeXKlWPNmjXZ98OgcuNSlSqZP/9bhEVERGxXsWJFrl27xtq1a9NeO336NHv27KHS9V9cQGRkJD179uT7779n4MCBfPrpp2nvFS5cmM6dOzNp0iRGjx7NJ598AkDNmjXZt28f4eHhlC1bNt0jNDQ07fN33303gwcPZtWqVVSpUoUpU6Zk68+scuNC1/83snPnnY1qFxERcZW77rqLVq1a0b17d1auXMnWrVt5+umnKV68OK1atQKgX79+zJ8/nwMHDrBp0yaWLl2aNjbm9ddf54cffiA+Pp4dO3Ywe/bstPc6duxIoUKFaNWqFT///DMHDhxg2bJl9OnTh99//50DBw4wePBgVq9ezcGDB1mwYAH79u3L9nE3KjcuVLYsBASYe7G//253GhEREWPChAncc889NG/enJiYGCzLYu7cueTKlQswu3DHxsZSsWJFmjRpQrly5fjwww8Bs1P34MGDqVatGg899BD+/v5MnToVMLulr1ixgqioKNq0aUPFihXp2rUrly9fxul0kjt3bnbv3k3btm0pV64cPXr0IDY2lueffz5bf16HZfnWNYbExERCQ0NJSEjA6XS6/PtXqmRuS/30EzRu7PJvLyIiOejy5cscOHCA6OhogoOD7Y7j9W51vjPz+1tXblxM425ERETspXLjYn8edyMiIiI5T+XGxVRuRERE7KVy42J/XuvGt0YziYh4Lx8bnmobV51nlRsXK1cO/Pzg7Fk4ccLuNCIicieuzya6ePGizUl8w5UrVwDw9/e/o++j7RdcLCTE7CsVH28GFRctanciERHJKn9/f8LCwjh58iRgpj5n96aPvio1NZVTp06RO3duAgLurJ6o3GSDSpVMudm5E+rXtzuNiIjciaL//a/U6wVHso+fnx9RUVF3XCBVbrJBxYowa5YGFYuIeAOHw0FERATh4eFcvXrV7jheLTAwED+/Ox8xo3KTDTRjSkTE+/j7+9/xWBDJGRpQnA20kJ+IiIh9VG6yQYUK5s8TJ+D0aXuziIiI+BqVm2yQNy9ERZnnunojIiKSs1RusonG3YiIiNhD5SabaNyNiIiIPVRusomu3IiIiNhD5Sab/HmPKREREck5KjfZ5Hq5+f13SEy0N4uIiIgvUbnJJvnzQ0SEeb57t71ZREREfInKTTa6Pu7m55/tzSEiIuJLVG6yUevW5s833oDDh22NIiIi4jNUbrJRr15w//1w/jw8/zxYlt2JREREvJ/KTTby94fPP4egIJg3D7780u5EIiIi3k/lJptVrAjDhpnn/frB0aN2phEREfF+Kjc54MUX4Z574Nw5c6tKt6dERESyj8pNDggIgAkTIFcumDULpk61O5GIiIj3UrnJIVWrwquvmuddu8JHH+kKjoiISHZQuclBr7wCTZrApUvm9lSrVnDqlN2pREREvIvKTQ4KDIQ5c2DUKPP8xx/NFZ158+xOJiIi4j1UbnKYnx8MGADr1kHlynDiBDRrZmZU6TaViIjInVO5sUn16rB+PfTubb5+4w3o2RNSUuzNJSIi4ulUbmwUEgIffADjx4PDAZ98Au3aweXLdicTERHxXCo3bqBnT5g2zYzDmTEDGjUya+KIiIhI5qncuIm2bWHBAnA6zS7iDz0Ev/9udyoRERHPo3LjRh5+GFasgKJFYds2iIkxf4qIiEjGqdy4merVYfVqsyfV77/DAw/AkiV2pxIREfEcKjduqFQpWLkSHnwQEhPNwn9TptidSkRExDOo3LipAgXMGJx27eDqVejYEQYN0kwqERGR21G5cWPBwWaTzQEDzNdvvw01asCqVbbGEhERcWsqN27Oz89s1zBjhhlovGePGYczYABcvGh3OhEREfejcuMhWreGnTuhc2ezTcN770Hp0vD44/DWW7BokdbGERERAXBYlm/taJSYmEhoaCgJCQk4nU6742TJvHnw/PNw+PBf36tdG556Ctq3N1d6REREvEFmfn/beuUmLi6Oe++9l3z58hEeHk7r1q3Zs2fPLT8zceJEHA5HukdwcHAOJXYPTZua21NLl8I778ATT0CZMua9tWuhb18oXhwaNoQvvoArV+zNKyIikpNsLTfLly8nNjaWNWvWsHDhQq5evUqjRo24cOHCLT/ndDo5duxY2uPgwYM5lNh9hIRAvXrw4otm0HF8PBw7BmPGwP33Q2qquVXVpQvcdRd8/LFKjoiI+Aa3ui116tQpwsPDWb58OQ899NANj5k4cSL9+vXjXBYHmHjDbamM2L/frI3z4Yem9ABERcHQoabwBAbaGk9ERCRTPOa21P9KSEgAoECBArc8LikpiZIlSxIZGUmrVq3YsWPHTY9NTk4mMTEx3cMXlC4Nr74Kv/0G779vxt8cOmTG6pQta167zQUyERERj+Q25SY1NZV+/fpRt25dqlSpctPjypcvz+eff84PP/zApEmTSE1NpU6dOvx+k10m4+LiCA0NTXtERkZm14/glkJCoE8fcyVn9GhTcg4fhn79oGRJGD4czpyxO6WIiIjruM1tqV69ejFv3jxWrlxJiRIlMvy5q1evUrFiRTp06MDw4cP/8n5ycjLJyclpXycmJhIZGen1t6Vu5vJlM8j47bdN4QHIk8fcqurdGypUsDWeiIjIDXncbanevXsze/Zsli5dmqliA5ArVy7uvvtu4uPjb/h+UFAQTqcz3cOXBQebW1N79pgxOdWqmdtT48aZzTobNYLZsyElxe6kIiIiWWNrubEsi969ezNjxgyWLFlCdHR0pr9HSkoK27ZtIyIiIhsSeq+AAOjQAbZsgYULoWVLcDjM8xYtzJidIUPMwoEiIiKexNZyExsby6RJk5gyZQr58uXj+PHjHD9+nEuXLqUd06lTJwYPHpz29ZtvvsmCBQvYv38/mzZt4umnn+bgwYN069bNjh/B4zkc0KAB/PCDGXz80kuQP78ZfBwXB5UrQ82aZguImwxrEhERcSu2lpvx48eTkJBAvXr1iIiISHt88803acccOnSIY9fnMgNnz56le/fuVKxYkWbNmpGYmMiqVauoVKmSHT+CV4mONmNxjh6FadOgVSvIlQs2bzbr6URGwoMPwgcf/DG9XERExN24zYDinOIr69y4yunT8O23MHky/PLLH687HGZ8zrBhZtFAERGR7ORxA4rFfRUsCL16wcqVZgr5e++ZMmNZMH8+xMSYTT23bbM7qYiIiKFyIxlWooRZH2f1ajM+59lnwc/PjNepXh06doTdu+1OKSIivk7lRrKkdGn4/HPYsQPatTNXcqZMMdPJW7UyV3p864aniIi4C5UbuSMVKpgxOZs2mVLjcMCsWWbgcZ068NVXZjr51at2JxUREV+hAcXiUnv2mGnjX34Jf1oYmly5oHx5qFLFjNFp29astSMiIpIRmfn9rXIj2eLECbPq8YIF5tZVUlL696OioG9f6NYN9H8GERG5HZWbW1C5yXmpqWam1fbtZjDyJ5/AqVPmPafTbAfx2muQL5+9OUVExH2p3NyCyo39Ll2CSZPg3Xf/mF1VurQZn1Onjr3ZRETEPWmdG3FrISHQvbu5XTVrFpQsaXYof/BBGDoUrlyxO6GIiHgylRuxjZ+f2aRz61bo1Mncvho50iwMOHs2xMfDtWt2pxQREU+jciO2Cw2FL74w+1kVKGCmlbdoAXfdZa7ylC8PbdqYqzypqXanFRERd6dyI27j8cfNNg7PPgtVq0JwsLlys3cvzJhh1tEpXx7Gjv3r7CsREZHrNKBY3FZqKhw5YsrNggVmltW5c+a9sDBo3hzKljWDkUuXNs+LFLEzsYiIZBfNlroFlRvPlZRkbl+9/z7s23fjYx580Gz02aYNBAXlbD4REck+Kje3oHLj+VJTYeFC2LzZzLL67Tfz58GDf+xnVbgwdO1qik5UlL15RUTkzqnc3ILKjfc6cgQ+/dQ8jh41rwUHm+0gevUy+16JiIhn0jo34pOKF4dhw+Df/4bp0+GBB+DyZYiNNbOvTpywO6GIiOQElRvxOrlymTE3y5eb8TlBQTBnjpmBNXu23elERCS76baUeL3t26FjR/j11z9e8/MDf3/zCAuD6tXh7rv/eJQtq9tYIiLuRGNubkHlxjddvmy2dnj/fUhJuf3xMTEwZgzUqpX92URE5PZUbm5B5ca3JSXBhQum4KSkmJlXx47Bli1m9tWWLWY7iORkc+Wma1cYMQLCw+1OLiLi21RubkHlRm7n6FEYNMjsXA5me4i//c3MuAoOtjebiIiv0mwpkTtQrBh89RWsXGnG3yQkwIABEB0N77wD58/bnVBERG5FV25EbiElBT7/HIYPh8OHzWv580OfPmafqz8vInjtmhnX07SpvZlFRLyRbkvdgsqNZMWVK+Y21Vtv3Xzrh+tiY+HttyF37pzJJiLiC1RubkHlRu5ESopZIHD8eDMY+fqmnaVLw9q18MEH5rgKFWDyZKhZ0968IiLeQuXmFlRuJDstWABdupgZWAEB8Oqr8NJLuoojInKnNKBYxCaNGsG2bdC2rRmDM2wYlCsHEyZkbH0dERG5cyo3Ii5WsCBMmwZTp0LJkmZDz+eeMzOv5s+3O52IiPdTuRHJBg4HPPEE7N5tpo+HhZkrOk2aQLt2cOqU3QlFRLyXyo1INgoOhhdfNNPFBwww43C++w4qV4YZM+xOJyLinVRuRHJAgQIwapSZUVWlirly06YNPP00nDljdzoREe+iciOSg2rWhA0bYPBgszP55MlQqpQpObNmmT2tRETkzqjciOSwoCAYORJWr4ZKlcx2DpMnQ6tWZoPOTp1g/Xq7U4qIeC6VGxGb3HefGWS8ahX06wfFi0NiotnX6r77oHVr876IiGSOyo2Ijfz8ICYG3nsPDh0ym3V26mRe/+EHqF4dOnSAvXvtTioi4jlUbkTchJ8f1K0LX3wB27dD+/ZgWWa9nCpVYMQIszCgiIjcmsqNiBuqWBG++QY2bza7jF+9arZyqFvXrJ0jIiI3p3Ij4sZq1IA5c+DLLyE0FNatMysdjx5tNu4UEZG/UrkRcXMOBzzzjLlV1bAhXL4M/ftD7dqwZInd6URE3I/KjYiHKFHC7E01fjzkzWvWy3nkEbOlw+bNdqcTEXEfKjciHsThgJ49IT4eevc22znMn28WB+zYEU6etDuhiIj9VG5EPFCRIvDBB2ZwcYcO5rUpU8yeVdOn25tNRMRuKjciHqxMGVNqNmyAatXgP/+Bxx83V3HOnrU7nYiIPVRuRLzAPfeYLRuGDjXr5Vy/ijNqFCxapNtVIuJbHJZlWXaHyEmJiYmEhoaSkJCA0+m0O46Iy61dC507w5496V8PDzdjc55/Hlq2NCVIRMRTZOb3t/55E/EytWub2VPvvANt2kDZsmYg8smT8NNP8Nhj5qrO55/DlSt2pxURcT1duRHxARcuwM6dMGMGfPghJCSY14sXh4EDzdWc3LntzSgiciu6ciMi6eTJA/feCyNHmg0633kHihWDI0dgwAAoXdps3nnxot1JRUTunMqNiI9xOuHFF2H/fvj0UyhVCk6c+KPkvPvuH1d2REQ8kcqNiI8KCoJu3WDvXvjssz9KzsCBULQoPPWUWSAwJcXupCIimaNyI+LjcuWCrl1Nyfn0U6hUyexf9fXXZmuHkiXNFPPDh+1OKiKSMSo3IgKYktOtm9mgc/16iI2FAgXMuJyRIyE62iwQuGIF+NY0BBHxNCo3IpKOwwG1asHYsXD0KHz7LdSrZ25PTZ8ODz8MNWrAmjV2JxURuTGVGxG5qaAgaNcOli6FX3+FHj0gJMQ8b9xYu5GLiHtSuRGRDKlaFT7+GH7/HR58EBITzZicffvsTiYikp7KjYhkSoEC8OOP5tbUyZPQsKEpPCIi7kLlRkQyLTTUbOVQtiwcPGhuUZ0+bXcqERFD5UZEsqRIEVi40Kx0vHMnNGpkZlmJiNhN5UZEsqxUKViwwNyq2rQJ7rvP7DiugcYiYieVGxG5I5Urw4YN0KkT+PmZ8Tg1a0LbtrB7t93pRMQXqdyIyB2LjoYvvjC3p556yqyV8/33UK0aDBsGycl2JxQRX6JyIyIuU748TJ5sVjlu1gyuXoU33oDq1WH5crvTiYivULkREZerVAlmz4ZvvjEDj/fsMascd+1qNucUEclOKjciki0cDmjf3oy7ef5589rnn5vp4yNGwMWL9uYTEe+lciMi2SosDD76CFauhHvvhaQkePVVKFfOjNNJTbU7oYh4G5UbEckRdeuazTanTIGSJc1u4126QNOmGnAsIq6lciMiOcbPDzp0MLeq3nkH8uQx6+R07Gh2HRcRcQWVGxHJccHB8OKLMHMmBAbC9OnwwgtgWXYnExFvYGu5iYuL49577yVfvnyEh4fTunVr9uzZc9vPTZs2jQoVKhAcHEzVqlWZO3duDqQVEVdr0MBMHXc44JNPzFgcEZE7ZWu5Wb58ObGxsaxZs4aFCxdy9epVGjVqxIULF276mVWrVtGhQwe6du3K5s2bad26Na1bt2b79u05mFxEXOXxx82AY4CRI2H0aFvjiIgXcFiW+1wIPnXqFOHh4SxfvpyHHnrohsc88cQTXLhwgdmzZ6e9dv/991OjRg0+uv4v5C0kJiYSGhpKQkICTqfTZdlF5M6MHAlDh5rnEyaYwcYiItdl5ve3W425SUhIAKBAgQI3PWb16tU0aNAg3WuNGzdm9erVNzw+OTmZxMTEdA8RcT+DB0P//uZ5165mAUARkaxwm3KTmppKv379qFu3LlWqVLnpccePH6dIkSLpXitSpAjHjx+/4fFxcXGEhoamPSIjI12aW0Rcw+GAUaOge3ez9s3TT5tNOEVEMsttyk1sbCzbt29n6tSpLv2+gwcPJiEhIe1x+PBhl35/EXEdhwPGjzdTw69dM+NxFi60O5WIeBq3KDe9e/dm9uzZLF26lBIlStzy2KJFi3LifzanOXHiBEWLFr3h8UFBQTidznQPEXFf/v4wcSK0bQtXrkCrVjBtGpw9a3cyEfEUtpYby7Lo3bs3M2bMYMmSJURHR9/2MzExMSxevDjdawsXLiQmJia7YopIDgsIMCsZN2sGly6ZPaoKFIDSpaFdO3P7KinJ7pQi4q5sLTexsbFMmjSJKVOmkC9fPo4fP87x48e5dOlS2jGdOnVi8ODBaV/37duXn376iVGjRrF7926GDRvGhg0b6N27tx0/gohkk8BA+O47s7hf6dLmtQMHzGsvvggtW2rbBhG5MVvLzfjx40lISKBevXpERESkPb750zSJQ4cOcezYsbSv69Spw5QpU/jkk0+oXr063333HTNnzrzlIGQR8UwhITBuHPz2G5w5A4sXwz/+AXnzwtKl8Oyz2nhTRP7Krda5yQla50bE8y1caG5ZXbsGL70Eb79tdyIRyW4eu86NiEhGNGwIn39unr/zDowZ88d7V6/CunXwr3/B6dP25BMRewXYHUBEJCueeQZ+/x2GDIF+/SA+3uw2vmoVXN/B5R//MFd5Spa0NaqI5DDdlhIRj2VZEBtr1sb5s/z5zYyrU6egRAlTcCpUsCejiLhGZn5/68qNiHgshwM++ABy54bDh+HBB+Ghh6BKFTh6FBo1gl27zOvz50PNmnYnFpGcoCs3IuK1/vMfaNIENm6EfPlg9mxTfkTE82hAsYgIUKgQLFkCDz8M589D48awfr3dqUQku6nciIhXczph3jxTbC5fNts5HD1qdyoRyU4qNyLi9UJC4NtvoVIlOHYMWrc22zqIiHdSuRERn+B0wo8/QsGC5tZU165mtpWIeB+VGxHxGaVLm72pAgLg66/hrbfsTiQi2UHlRkR8Sr16MHaseT5kiFnpWFdwRLyLyo2I+Jznn4fevc3zrl2haVPYv9/eTCLiOio3IuKT3nsP3ngDgoLMAn+VK0NcHFy5YncyEblTKjci4pMCAuD11+HXX+H//s9MEx8yBO65Bw4csDudiNwJlRsR8WnlysGiRfDVV1C4MGzfblYx3rPH7mQiklUqNyLi8xwOePpp2LIFKlY0u40/9JC5qiMinkflRkTkv4oVg+XLoUYNOHnSzKzSdg0inkflRkTkTwoXhqVL4f774exZeOQR87WIeI4slZsvvviCOXPmpH398ssvExYWRp06dTh48KDLwomI2CEsDBYuhPr1zYabjzwCAwdqywYRT5GlcjNy5EhCQkIAWL16NePGjePtt9+mUKFC9O/f36UBRUTskDcvzJkDXbqYRf7efdfcrlq1yu5kInI7WSo3hw8fpmzZsgDMnDmTtm3b0qNHD+Li4vj5559dGlBExC4hITBhAsyebcbj7N0LDzxgruJcvGh3OhG5mSyVm7x583L69GkAFixYQMOGDQEIDg7mkq7bioiXefRRM0W8c+f0V3FWrrQ7mYjcSJbKTcOGDenWrRvdunVj7969NGvWDIAdO3ZQqlQpV+YTEXEL+fPDxIlmZ/FixWDfPjNdvH9/XcURcTdZKjfjxo0jJiaGU6dOMX36dAoWLAjAxo0b6dChg0sDioi4k+bNzVWc62NxRo+GatU0FkfEnTgsy7f2w01MTCQ0NJSEhAScTqfdcUTEg82bB927w5EjkCcPbNwI5cvbnUrEO2Xm93eWrtz89NNPrPzTzeZx48ZRo0YNnnrqKc6ePZuVbyki4nGaNoUdO8ztqQsXoF07TRcXcQdZKjcvvfQSiYmJAGzbto2BAwfSrFkzDhw4wIABA1waUETEnYWGwtSpEB4O27ZB3752JxKRLJWbAwcOUKlSJQCmT59O8+bNGTlyJOPGjWPevHkuDSgi4u4iImDyZLNH1aefmuciYp8slZvAwEAu/nd6wKJFi2jUqBEABQoUSLuiIyLiSxo0gNdeM8+ff167iovYKUvl5oEHHmDAgAEMHz6cdevW8eijjwKwd+9eSpQo4dKAIiKe4vXXzWabGn8jYq8slZuxY8cSEBDAd999x/jx4ylevDgA8+bNo0mTJi4NKCLiKfz9YcqUP8bftGsHycl2pxLxPZoKLiLiYsuXQ5MmcPkyNGsG06dDcLDdqUQ8W2Z+fwdk9S9JSUlh5syZ7Nq1C4DKlSvTsmVL/P39s/otRUS8wsMPm/2oWrSAuXOhTRv4/nsVHJGckqUrN/Hx8TRr1owjR45Q/r8rVu3Zs4fIyEjmzJlDmTJlXB7UVXTlRkRyypIlZkXjS5fMlZwZM1RwRLIq2xfx69OnD2XKlOHw4cNs2rSJTZs2cejQIaKjo+nTp0+WQouIeJv/+z+YM8fsLv7TT+YKzrVrdqcS8X5ZunKTJ08e1qxZQ9WqVdO9vnXrVurWrUtSUpLLArqartyISE5btszsLH7xolnkb/RouxOJeJ5sv3ITFBTE+fPn//J6UlISgYGBWfmWIiJeq149mDTJPH//fZgwwdY4Il4vS+WmefPm9OjRg7Vr12JZFpZlsWbNGnr27EnLli1dnVFExOM99hgMG2ae9+wJq1fbGkfEq2Wp3IwZM4YyZcoQExNDcHAwwcHB1KlTh7JlyzJa11tFRG7otddMyblyxYy/OXLE7kQi3umO1rmJj49PmwpesWJFypYt67Jg2UVjbkTETufPQ506sH073HsvrFihGVQiGZGZ398ZLjeZ2e373XffzfCxOU3lRkTstn+/KTZnzpg9qb79FvLntzuViHvLlkX8Nm/enKHjHA5HRr+liIhPKl0avvvOrIGzaBHcfz/8+COUK2d3MhHvoO0XRERssmULtGwJhw9DWBhMm2au5IjIX2X7VHAREblzNWrA+vUQEwPnzplVjMeOBd/6T04R11O5ERGxUZEiZpuGZ56BlBT4f/8PnnoKEhPtTibiuVRuRERsFhwMX3wB//wn+PvD1Klwzz2QwaGOIvI/VG5ERNyAwwEDB8LPP0NUFMTHm4HGH36o21QimaVyIyLiRmJizBWbli3NYn+xsdCiBRw6ZHcyEc+hciMi4mYKFICZM+G99yBXLrOzeOXKMGaMGZcjIremciMi4oYcDujXz0wXr1sXkpLMjuJ16sCvv9qdTsS9qdyIiLixSpXMFg3jx4PTCevWQa1aMGuW3clE3JfKjYiIm/PzMzuJ79oFjz4KV6/C44/D3Ll2JxNxTyo3IiIeolgxMxanXTtTcNq0gQUL7E4l4n5UbkREPEhAAEyeDI89BsnJ0KoVLF5sdyoR96JyIyLiYXLlMgv9NW8Oly+bqeIzZ2o9HJHrVG5ERDxQYKDZWbxJE7h0yVzJqVYNPv/cFB4RX6ZyIyLioYKCYMYM6N8f8uSB7duha1coWRJGjoRr1+xOKGIPlRsREQ8WHAzvvgu//w5vvw0lSsDJkzB0KLz1lt3pROyhciMi4gXCwuCll2D/fnjnHfNaXJwpPSK+RuVGRMSL5MplNuCsWxcuXoRBg+xOJJLzVG5ERLyMw2H2oXI4YMoU+OUXuxOJ5CyVGxERL1SzphlcDNCnjzbcFN+iciMi4qVGjDD7UW3aBBMn2p1GJOeo3IiIeKnwcPjb38zzIUMgIcHePCI5ReVGRMSL9e4N5cub6eHDhtmdRiRnqNyIiHixwEB47z3zfPRomDTJ1jgiOULlRkTEyzVtalYxBnj2We0kLt5P5UZExAf885/w5JNmS4a2bWHjRrsTiWQflRsRER/g52dmTD3yCCQlQbNm8NtvdqcSyR62lpsVK1bQokULihUrhsPhYObMmbc8ftmyZTgcjr88jh8/njOBRUQ8WFAQfP891KhhBhg3aQKnTtmdSsT1bC03Fy5coHr16owbNy5Tn9uzZw/Hjh1Le4SHh2dTQhER7+J0wrx5EB0N8fHQqROkptqdSsS1Auz8y5s2bUrTpk0z/bnw8HDCwsJcH0hExAcULQo//gi1asFPP8EHH0DfvnanEnEdjxxzU6NGDSIiImjYsCG/3GbTlOTkZBITE9M9RER8XeXKMGqUef7yy/Drr/bmEXEljyo3ERERfPTRR0yfPp3p06cTGRlJvXr12LRp000/ExcXR2hoaNojMjIyBxOLiLivXr2geXO4cgU6dIBLl+xOJOIaDsuyLLtDADgcDmbMmEHr1q0z9bmHH36YqKgovvrqqxu+n5ycTHJyctrXiYmJREZGkpCQgNPpvJPIIiIe7+RJqFYNTpyA2FgYO9buRCI3lpiYSGhoaIZ+f3vUlZsbue+++4iPj7/p+0FBQTidznQPERExwsPhiy/M83HjYOpUsweVe/xnr0jWeHy52bJlCxEREXbHEBHxWI0bQ79+5nmHDhAWBiEhEBUFDz4Ia9famU4k82ydLZWUlJTuqsuBAwfYsmULBQoUICoqisGDB3PkyBG+/PJLAEaPHk10dDSVK1fm8uXLfPbZZyxZsoQFWktcROSOxMXB77+b2VNJSZCcDIcPm8fjj8POnZAvn90pRTLG1nKzYcMG6tevn/b1gAEDAOjcuTMTJ07k2LFjHDp0KO39K1euMHDgQI4cOULu3LmpVq0aixYtSvc9REQk84KDYdo08/ziRbO434kT5krO/v0wdCiMGWNvRpGMcpsBxTklMwOSRER83cKF0KgROBzwyy8QE2N3IvFVPjWgWEREsk/DhtC5sxlg3L27mTYu4u5UbkRE5JZGjYLChWHHDvjHP+xOI3J7KjciInJLBQvC+++b53//O+zebW8ekdtRuRERkdt68klo2tTclureHa5dszuRyM2p3IiIyG05HDB+POTJAytXwgsvaKE/cV8qNyIikiElS8JXX4GfH3z6KQwbZncikRtTuRERkQx77DGzTQPAm2/Chx/am0fkRlRuREQkU3r2/OOqTe/e8N13tsYR+QuVGxERybTXXzclx7KgY0dYvNjuRCJ/ULkREZFMczhg7Fho08bMoGreHObPtzuViKFyIyIiWeLvD5Mnm2Jz+TK0bAk//GB3KhGVGxERuQPBwTB9OrRrZ67gtG0LU6fanUp8ncqNiIjckcBAmDIFnnkGUlLgqadgwgS7U4kvU7kREZE7FhAAEyfC88+bQcbPPQfffGN3KvFVKjciIuISfn5mFePevc3XnTqZ1YxFcprKjYiIuIzDAaNHm8X+rlyBVq1gzx67U4mvUbkRERGX8veHSZOgdm04cwaaNYNTp+xOJb5E5UZERFwud26YNQtKl4b9+8008UuX7E4lvkLlRkREskV4OMydC/nzw5o15lZVYqLdqcQXqNyIiEi2KV/eLOwXHGxWMK5Tx1zJEclOKjciIpKtHnwQfv4ZihWDHTvgvvtg+XK7U4k3U7kREZFsV6sWrFtn/jx9Gho0gM8+szuVeCuVGxERyRHFi5srNk88AdeuQffu8M9/2p1KvJHKjYiI5JjcueHrr+G118zXL70EH39sbybxPio3IiKSoxwOePNNGDzYfN2rl1kXR8RVVG5ERMQWI0ZAbKzZi6pLF5g50+5E4i0C7A4gIiK+yeGAMWMgKQm++MKMxfn73yEiwty+ypPHjNOpUsXupOJpVG5ERMQ2fn5m1lRSEkyfDi+//NdjBg2Ct97K+WziuVRuRETEVgEBMGWKuU21dStcvAgXLpjH1q3wj39AiRJ/7DYucjsOy7Isu0PkpMTEREJDQ0lISMDpdNodR0REbmHECHj1VXML6/vvoXVruxOJXTLz+1sDikVExG0NGQI9ephBxx06mD2qRG5H5UZERNyWwwHjxsGjj8Lly9CiBezbZ3cqcXcqNyIi4tYCAmDqVLjnHvjPf0zRuXjR7lTizlRuRETE7eXNC3PmmIHF+/bBG2/YnUjcmcqNiIh4hCJFYPx483zUKNi82d484r5UbkRExGM0bw7t20NKitl4MyXF7kTijlRuRETEo7z/PoSFwcaNZoVjkf+lciMiIh6laFF45x3z/NVX4d//tjWOuCGVGxER8TjPPQcPPWRmTb3wglkHR+Q6lRsREfE4fn7wyScQGAjz5sGECXYnEneiciMiIh6pfHl4/XXzvEcP+PFHe/OI+1C5ERERjzV4MHTqZGZNtWsHS5bYnUjcgcqNiIh4LD8/+Ne/zIaaycnQsiWsXWt3KrGbyo2IiHi069szNGgAFy5A06bw6692pxI7qdyIiIjHCwqCmTMhJgbOnoX774cmTeDtt2HDBi3252tUbkRExCvkyQNz50Lt2nDpEsyfD4MGwb33QqFCMHGi3Qklp6jciIiI1wgLg1WrYOtWeO89MwbH6YRz58x2DRqP4xtUbkRExKv4+UG1atCvH/zwA5w+bWZSXbsGTzwBZ87YnVCym8qNiIh4tYAA+PRTKFMGDh6EZ5/VisbeTuVGRES8XmgofPutWdF41iwYPdruRJKdVG5ERMQn1KxpxuEAvPyyxt94M5UbERHxGb16Qfv2ZvxN+/Zw6pTdiSQ7qNyIiIjPcDjM+JuyZeHQITOb6uJFu1OJq6nciIiIT3E6zSab+fPDmjXQsaMW+fM2KjciIuJzKlQw08Svr2zcv79mUHkTlRsREfFJDz4IX35pnn/wwR+DjcXzqdyIiIjPat8e3nnHPB84ED7+WFdwvIHKjYiI+LSBA6F3b/O8Z0+z+eaqVfZmkjujciMiIj7N4TCL+o0YYTbfXLsW6taFJ5+Ef//b7nSSFSo3IiLi8/z9YcgQ2LcPnnvOFJ5vvoFKlWD9ervTSWap3IiIiPxXRAT861+waRPcfz9cugR9+2ocjqdRuREREfkfNWrA9OmQOzesXg3ffWd3IskMlRsREZEbKFYMXnrJPH/lFUhOtjePZJzKjYiIyE28+CIULQr798O4cXankYxSuREREbmJvHnh7383z4cPhzNn7M0jGaNyIyIicgtdukDVqnDu3B9FR9ybyo2IiMgt+PvDP/9pno8dC/Hx9uaR21O5ERERuY1GjaBxY7h6Fbp2hUOH7E4kt6JyIyIikgH//KfZRXzFCrOr+N/+Bhcu2J1KbsTWcrNixQpatGhBsWLFcDgczJw587afWbZsGTVr1iQoKIiyZcsyceLEbM8pIiJSpQqsWQMPPWQW93vzTShXDr76Sov8uRtby82FCxeoXr064zI4v+7AgQM8+uij1K9fny1bttCvXz+6devG/PnzszmpiIiIWdxv2TKYNg1KlYKjR6FTJxg50uZgko7DstyjbzocDmbMmEHr1q1vesygQYOYM2cO27dvT3vtySef5Ny5c/z0008Z+nsSExMJDQ0lISEBp9N5p7FFRMRHXb5sNtv8+9/NoOOffzY7ikv2yMzvb48ac7N69WoaNGiQ7rXGjRuzevXqm34mOTmZxMTEdA8REZE7FRxsbk099RSkpJg/ExLsTiXgYeXm+PHjFClSJN1rRYoUITExkUuXLt3wM3FxcYSGhqY9IiMjcyKqiIj4AIcDPvwQoqPh3/+Gnj01/sYdeFS5yYrBgweTkJCQ9jh8+LDdkURExIuEhsKUKebW1NSp8OWXdicSjyo3RYsW5cSJE+leO3HiBE6nk5CQkBt+JigoCKfTme4hIiLiSvffD2+8YZ7HxsK+ffbm8XUBdgfIjJiYGObOnZvutYULFxKjEVwiImKzV16BhQth+XIzsLhaNShTxjwqVIDmzSHAo37rei5bT3NSUhLxf1rH+sCBA2zZsoUCBQoQFRXF4MGDOXLkCF/+9xpfz549GTt2LC+//DLPPfccS5Ys4dtvv2XOnDl2/QgiIiKAuS01aZK5inPkCCxdah7XtWwJ339vjpPsZWu52bBhA/Xr10/7esCAAQB07tyZiRMncuzYMQ79aY3r6Oho5syZQ//+/Xn//fcpUaIEn332GY0bN87x7CIiIv+rRAnYswe2boXffvvjMW0azJoFgwb9sU+VZB+3Wecmp2idGxERyWlTp0KHDub5xx9Djx725vFEXrvOjYiIiCd68kmzJg7ACy/AokX25vF2KjciIiI54NVX4emnzYJ/jz8Ou3bZnch7qdyIiIjkAIcDPvsMHnjArGTcrBkcPGh3Ku+kciMiIpJDgoJgxgwoW9asaPzgg7B3r92pvI/KjYiISA4qVMjsLF6xIhw+bArO1q12p/IuKjciIiI5rHhxs9jf3XfDyZNQrx7cYg9oySSVGxERERsULmwW+atbF86dg4YNYfFiu1N5B5UbERERm4SGwvz50KgRXLhgBhl/953dqTyfyo2IiIiN8uQxqxc//jhcuQLt28Mnn9idyrOp3IiIiNgsKMisYvz882BZ5s8RI8xzyTyVGxERETfg7w/jx5vF/sD82bs3nDhhby5PpHIjIiLiJhwOGD4cRo82X3/4oZlZ1aSJ2XE8KcnWeB5D5UZERMTN9O1rFvurXdts1zB/PjzzDBQpArGxcPy43Qndm8qNiIiIG2rdGtasgX37YNgwuOsuuHjRXM0pUwZefx0SE+1O6Z5UbkRERNxY2bLwt7/Bnj2wZIm5mnPxorl9VaaMuYV1+bLdKd2Lyo2IiIgHcDigfn2zkvH06VCuHPznP9C/vyk5Y8bApUt2p3QPKjciIiIexOGANm1gxw6zHk5kJBw9asbplCkD77+vkqNyIyIi4oECAqB7dzMm56OPICoKjh2Dfv3Mlg6+PLNK5UZERMSDBQWZRf/27YOPPza7jm/eDJ06QWqq3ensoXIjIiLiBQIDoUcPs5VDYKCZSv7mm3ansofKjYiIiBeJiTFXcADeeMMMPvY1KjciIiJepksXM4sKzO2prVttjZPjAuwOICIiIq739tuwfTssXAjNm5tp5ImJcP68eZQrB4MGQdWqdid1PYdl+daeo4mJiYSGhpKQkIDT6bQ7joiISLY5exbuuw/i429+TJs2ZrXj6tVzLldWZOb3t8qNiIiIFzt40Gy6GRQE+fKB0wnBwTB1KkybBtdbwGOPwXvvQcmS9ua9GZWbW1C5ERERMXbsMNs4fPutKTkREfDTT1Ctmt3J/iozv781oFhERMRHVa5sruBs3w5VqphFAB96CH7+2e5kd0blRkRExMdVqgQrVsADD0BCAjRqZNbL8VQqNyIiIkL+/LBgAbRsaXYZf+wx+Pxzu1NljcqNiIiIABASYhb9e+45s3VD166eWXBUbkRERCRNQAB89tkfiwB2724GHHsSlRsRERFJx+GAUaPMXlWpqdCxI8yZY3eqjFO5ERERkb9wOODDD+Gpp+DaNWjbFpYuvfVnkpPNTCu7Z1tp+wURERG5IX9/mDgRLlyAH36AFi2gTx+IjITixc3j4kVYtsw8Vq0yg5EfeQQWLbIvt8qNiIiI3FSuXGYtnBYtTGGJi7v18eHhpvzYSeVGREREbik42Fy5+de/YPduOHLkjweY9XHq14d69aBCBXNLy04qNyIiInJbuXPD//t/dqfIGA0oFhEREa+iciMiIiJeReVGREREvIrKjYiIiHgVlRsRERHxKio3IiIi4lVUbkRERMSrqNyIiIiIV1G5EREREa+iciMiIiJeReVGREREvIrKjYiIiHgVlRsRERHxKio3IiIi4lUC7A6Q0yzLAiAxMdHmJCIiIpJR139vX/89fis+V27Onz8PQGRkpM1JREREJLPOnz9PaGjoLY9xWBmpQF4kNTWVo0ePki9fPhwOR5a/T2JiIpGRkRw+fBin0+nChPK/dK5zjs51ztL5zjk61zknu861ZVmcP3+eYsWK4ed361E1Pnflxs/PjxIlSrjs+zmdTv0/Sg7Ruc45Otc5S+c75+hc55zsONe3u2JznQYUi4iIiFdRuRERERGvonKTRUFBQfztb38jKCjI7iheT+c65+hc5yyd75yjc51z3OFc+9yAYhEREfFuunIjIiIiXkXlRkRERLyKyo2IiIh4FZUbERER8SoqN1k0btw4SpUqRXBwMLVr12bdunV2R/J4cXFx3HvvveTLl4/w8HBat27Nnj170h1z+fJlYmNjKViwIHnz5qVt27acOHHCpsTe4a233sLhcNCvX7+013SeXevIkSM8/fTTFCxYkJCQEKpWrcqGDRvS3rcsi9dff52IiAhCQkJo0KAB+/btszGxZ0pJSeG1114jOjqakJAQypQpw/Dhw9PtRaRznTUrVqygRYsWFCtWDIfDwcyZM9O9n5HzeubMGTp27IjT6SQsLIyuXbuSlJSUPYEtybSpU6dagYGB1ueff27t2LHD6t69uxUWFmadOHHC7mgerXHjxtaECROs7du3W1u2bLGaNWtmRUVFWUlJSWnH9OzZ04qMjLQWL15sbdiwwbr//vutOnXq2Jjas61bt84qVaqUVa1aNatv375pr+s8u86ZM2eskiVLWl26dLHWrl1r7d+/35o/f74VHx+fdsxbb71lhYaGWjNnzrS2bt1qtWzZ0oqOjrYuXbpkY3LPM2LECKtgwYLW7NmzrQMHDljTpk2z8ubNa73//vtpx+hcZ83cuXOtoUOHWt9//70FWDNmzEj3fkbOa5MmTazq1atba9assX7++WerbNmyVocOHbIlr8pNFtx3331WbGxs2tcpKSlWsWLFrLi4OBtTeZ+TJ09agLV8+XLLsizr3LlzVq5cuaxp06alHbNr1y4LsFavXm1XTI91/vx566677rIWLlxoPfzww2nlRufZtQYNGmQ98MADN30/NTXVKlq0qPXOO++kvXbu3DkrKCjI+vrrr3Miotd49NFHreeeey7da23atLE6duxoWZbOtav8b7nJyHnduXOnBVjr169PO2bevHmWw+Gwjhw54vKMui2VSVeuXGHjxo00aNAg7TU/Pz8aNGjA6tWrbUzmfRISEgAoUKAAABs3buTq1avpzn2FChWIiorSuc+C2NhYHn300XTnE3SeXW3WrFnUqlWLdu3aER4ezt13382nn36a9v6BAwc4fvx4uvMdGhpK7dq1db4zqU6dOixevJi9e/cCsHXrVlauXEnTpk0BnevskpHzunr1asLCwqhVq1baMQ0aNMDPz4+1a9e6PJPPbZx5p/7zn/+QkpJCkSJF0r1epEgRdu/ebVMq75Oamkq/fv2oW7cuVapUAeD48eMEBgYSFhaW7tgiRYpw/PhxG1J6rqlTp7Jp0ybWr1//l/d0nl1r//79jB8/ngEDBjBkyBDWr19Pnz59CAwMpHPnzmnn9Eb/puh8Z84rr7xCYmIiFSpUwN/fn5SUFEaMGEHHjh0BdK6zSUbO6/HjxwkPD0/3fkBAAAUKFMiWc69yI24pNjaW7du3s3LlSrujeJ3Dhw/Tt29fFi5cSHBwsN1xvF5qaiq1atVi5MiRANx9991s376djz76iM6dO9uczrt8++23TJ48mSlTplC5cmW2bNlCv379KFasmM61j9FtqUwqVKgQ/v7+f5k5cuLECYoWLWpTKu/Su3dvZs+ezdKlSylRokTa60WLFuXKlSucO3cu3fE695mzceNGTp48Sc2aNQkICCAgIIDly5czZswYAgICKFKkiM6zC0VERFCpUqV0r1WsWJFDhw4BpJ1T/Zty51566SVeeeUVnnzySapWrcozzzxD//79iYuLA3Sus0tGzmvRokU5efJkuvevXbvGmTNnsuXcq9xkUmBgIPfccw+LFy9Oey01NZXFixcTExNjYzLPZ1kWvXv3ZsaMGSxZsoTo6Oh0799zzz3kypUr3bnfs2cPhw4d0rnPhEceeYRt27axZcuWtEetWrXo2LFj2nOdZ9epW7fuX5Y02Lt3LyVLlgQgOjqaokWLpjvfiYmJrF27Vuc7ky5evIifX/pfa/7+/qSmpgI619klI+c1JiaGc+fOsXHjxrRjlixZQmpqKrVr13Z9KJcPUfYBU6dOtYKCgqyJEydaO3futHr06GGFhYVZx48ftzuaR+vVq5cVGhpqLVu2zDp27Fja4+LFi2nH9OzZ04qKirKWLFlibdiwwYqJibFiYmJsTO0d/jxbyrJ0nl1p3bp1VkBAgDVixAhr37591uTJk63cuXNbkyZNSjvmrbfessLCwqwffvjB+vXXX61WrVppenIWdO7c2SpevHjaVPDvv//eKlSokPXyyy+nHaNznTXnz5+3Nm/ebG3evNkCrHfffdfavHmzdfDgQcuyMnZemzRpYt19993W2rVrrZUrV1p33XWXpoK7mw8++MCKioqyAgMDrfvuu89as2aN3ZE8HnDDx4QJE9KOuXTpkvXCCy9Y+fPnt3Lnzm099thj1rFjx+wL7SX+t9zoPLvWjz/+aFWpUsUKCgqyKlSoYH3yySfp3k9NTbVee+01q0iRIlZQUJD1yCOPWHv27LEpredKTEy0+vbta0VFRVnBwcFW6dKlraFDh1rJyclpx+hcZ83SpUtv+O9z586dLcvK2Hk9ffq01aFDBytv3ryW0+m0nn32Wev8+fPZktdhWX9aulFERETEw2nMjYiIiHgVlRsRERHxKio3IiIi4lVUbkRERMSrqNyIiIiIV1G5EREREa+iciMiIiJeReVGREREvIrKjYj4vGXLluFwOP6yWaiIeCaVGxEREfEqKjciIiLiVVRuRMR2qampxMXFER0dTUhICNWrV+e7774D/rhlNGfOHKpVq0ZwcDD3338/27dvT/c9pk+fTuXKlQkKCqJUqVKMGjUq3fvJyckMGjSIyMhIgoKCKFu2LP/617/SHbNx40Zq1apF7ty5qVOnDnv27MneH1xEsoXKjYjYLi4uji+//JKPPvqIHTt20L9/f55++mmWL1+edsxLL73EqFGjWL9+PYULF6ZFixZcvXoVMKWkffv2PPnkk2zbto1hw4bx2muvMXHixLTPd+rUia+//poxY8awa9cuPv74Y/LmzZsux9ChQxk1ahQbNmwgICCA5557Lkd+fhFxLe0KLiK2Sk5OpkCBAixatIiYmJi017t168bFixfp0aMH9evXZ+rUqTzxxBMAnDlzhhIlSjBx4kTat29Px44dOXXqFAsWLEj7/Msvv8ycOXPYsWMHe/fupXz58ixcuJAGDRr8JcOyZcuoX78+ixYt4pFHHgFg7ty5PProo1y6dIng4OBsPgsi4kq6ciMitoqPj+fixYs0bNiQvHnzpj2+/PJLfvvtt7Tj/lx8ChQoQPny5dm1axcAu3btom7duum+b926ddm3bx8pKSls2bIFf39/Hn744VtmqVatWtrziIgIAE6ePHnHP6OI5KwAuwOIiG9LSkoCYM6cORQvXjzde0FBQekKTlaFhIRk6LhcuXKlPXc4HIAZDyQinkVXbkTEVpUqVSIoKIhDhw5RtmzZdI/IyMi049asWZP2/OzZs+zdu5eKFSsCULFiRX755Zd03/eXX36hXLly+Pv7U7VqVVJTU9ON4RER76UrNyJiq3z58vHiiy/Sv39/UlNTeeCBB0hISOCXX37B6XRSsmRJAN58800KFixIkSJFGDp0KIUKFaJ169YADBw4kHvvvZfhw4fzxBNPsHr1asaOHcuHH34IQKlSpejcuTPPPfccY8aMoXr16hw8eJCTJ0/Svn17u350EckmKjciYrvhw4dTuHBh4uLi2L9/P2FhYdSsWZMhQ4ak3RZ666236Nu3L/v27aNGjRr8+OOPBAYGAlCzZk2+/fZbXn/9dYYPH05ERARvvvkmXbp0Sfs7xo8fz5AhQ3jhhRc4ffo0UVFRDBkyxI4fV0SymWZLiYhbuz6T6ezZs4SFhdkdR0Q8gMbciIiIiFdRuRERERGvottSIiIi4lV05UZERES8isqNiIiIeBWVGxEREfEqKjciIiLiVVRuRERExKuo3IiIiIhXUbkRERERr6JyIyIiIl7l/wMPB/emhdwYeAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "plot_losses(rnn_sh_losses)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nbd_-EFInd-K"
      },
      "source": [
        "### LSTM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "g44i0NiKJ97c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "outputId": "940d50ea-13c5-43ef-80cc-ce09b568750b"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABEIklEQVR4nO3deXxU9b3/8fckIQmBLLJkAYKkbLKDIJCggCXIJku1SBEKLmBVqCC2WrTaVm+Nvb0oVSm4VNEfIhQFrIgKsoqGnSg7RCJBSIKyJKwBkvP743QmC0nIMjNnltfz8TiPc5g5c+YzB0Le813OsRmGYQgAAMBHBFhdAAAAgDMRbgAAgE8h3AAAAJ9CuAEAAD6FcAMAAHwK4QYAAPgUwg0AAPApQVYX4G6FhYU6duyYwsPDZbPZrC4HAABUgmEYOnPmjBo1aqSAgIrbZvwu3Bw7dkzx8fFWlwEAAKrhyJEjatKkSYX7+F24CQ8Pl2SenIiICIurAQAAlZGXl6f4+HjH7/GK+F24sXdFRUREEG4AAPAylRlSwoBiAADgUwg3AADApxBuAACAT/G7MTcAAFRHQUGBLl++bHUZPi04OPia07wrg3ADAEAFDMNQdna2Tp8+bXUpPi8gIEAJCQkKDg6u0XEsDTezZ8/W7Nmz9f3330uS2rVrp2eeeUaDBg0q9zWLFi3S008/re+//14tW7bU3/72Nw0ePNhNFQMA/I092ERHRyssLIwLwLqI/SK7WVlZatq0aY3Os6XhpkmTJnrhhRfUsmVLGYahd955R8OHD9eOHTvUrl27q/b/+uuvNXr0aKWkpOj222/X/PnzNWLECG3fvl3t27e34BMAAHxZQUGBI9jUr1/f6nJ8XsOGDXXs2DFduXJFtWrVqvZxbIZhGE6sq8bq1aunv//977r//vuvem7UqFE6d+6cli1b5nisZ8+e6ty5s+bMmVOp4+fl5SkyMlK5ublc5wYAUKGLFy8qIyNDzZo1U+3ata0ux+dduHBB33//vRISEhQaGlriuar8/vaY2VIFBQVasGCBzp07p8TExDL3SU1NVXJyconHBgwYoNTU1HKPm5+fr7y8vBILAABVQVeUezjrPFsebnbu3Km6desqJCREDz74oJYsWaK2bduWuW92drZiYmJKPBYTE6Ps7Oxyj5+SkqLIyEjHwn2lAADwbZaHm9atWystLU2bNm3SQw89pPHjx2vPnj1OO/706dOVm5vrWI4cOeK0YwMAAM9j+VTw4OBgtWjRQpLUtWtXbdmyRf/4xz/02muvXbVvbGyscnJySjyWk5Oj2NjYco8fEhKikJAQ5xYNAICH69u3rzp37qyZM2daXYrbWd5yU1phYaHy8/PLfC4xMVGrVq0q8djKlSvLHaPjToWFUk6OlJ5udSUAAPg3S1tupk+frkGDBqlp06Y6c+aM5s+fr7Vr1+rzzz+XJI0bN06NGzdWSkqKJGnKlCnq06ePZsyYoSFDhmjBggXaunWrXn/9dSs/hiRp5Upp4ECpQwfp22+trgYAAP9lacvN8ePHNW7cOLVu3Vr9+vXTli1b9Pnnn6t///6SpMzMTGVlZTn2T0pK0vz58/X666+rU6dO+uCDD7R06VKPuMaNfZwyQ3oAwLcZhnTunPuXmly45dSpUxo3bpyuu+46hYWFadCgQTp48KDj+cOHD2vo0KG67rrrVKdOHbVr107Lly93vHbMmDFq2LChateurZYtW+rtt992vPbIkSO66667FBUVpXr16mn48OGOi/NK0tq1a9W9e3fVqVNHUVFR6tWrlw4fPlz9D1MJlrbc/Otf/6rw+bVr11712MiRIzVy5EgXVVR9TZqY69OnpbNnpbp1LS0HAOAi589b83/82bNSnTrVe+0999yjgwcP6j//+Y8iIiL0xBNPaPDgwdqzZ49q1aqlSZMm6dKlS1q/fr3q1KmjPXv2qO5/P+TTTz+tPXv26NNPP1WDBg2Unp6uCxcuSJIuX76sAQMGKDExUV9++aWCgoL0P//zPxo4cKC+/fZbBQQEaMSIEZo4caLef/99Xbp0SZs3b3b51HrLBxT7iogIc8nLk374QbrhBqsrAgBAjlDz1VdfKSkpSZL03nvvKT4+XkuXLtXIkSOVmZmpO++8Ux06dJAk/exnP3O8PjMzU126dFG3bt0kSc2aNXM8t3DhQhUWFurNN990BJa3335bUVFRWrt2rbp166bc3Fzdfvvtat68uSSpTZs2Lv/MhBsnatJE2rOHcAMAviwszGxFseJ9q2Pv3r0KCgpSjx49HI/Vr19frVu31t69eyVJjzzyiB566CGtWLFCycnJuvPOO9WxY0dJ0kMPPaQ777xT27dv12233aYRI0Y4QtI333yj9PR0hYeHl3jPixcv6rvvvtNtt92me+65RwMGDFD//v2VnJysu+66S3FxcdX7MJXkcbOlvBnjbgDA99lsZveQuxdX9uRMmDBBhw4d0q9//Wvt3LlT3bp10yuvvCJJGjRokA4fPqxHH31Ux44dU79+/fS73/1OknT27Fl17dpVaWlpJZYDBw7o7rvvlmS25KSmpiopKUkLFy5Uq1attHHjRtd9GBFunMo+7uaHH6ytAwAAuzZt2ujKlSvatGmT47ETJ05o//79Je4IEB8frwcffFCLFy/WY489pjfeeMPxXMOGDTV+/HjNmzdPM2fOdMxSvvHGG3Xw4EFFR0erRYsWJZbIyEjH67t06aLp06fr66+/Vvv27TV//nyXfmbCjRPRcgMA8DQtW7bU8OHDNXHiRG3YsEHffPONxo4dq8aNG2v48OGSpKlTp+rzzz9XRkaGtm/frjVr1jjGxjzzzDP66KOPlJ6ert27d2vZsmWO58aMGaMGDRpo+PDh+vLLL5WRkaG1a9fqkUce0Q8//KCMjAxNnz5dqampOnz4sFasWKGDBw+6fNwNY26ciJYbAIAnevvttzVlyhTdfvvtunTpknr37q3ly5erVq1aksybV0+aNEk//PCDIiIiNHDgQL300kuSzDsJTJ8+Xd9//71q166tW265RQsWLJAkhYWFaf369XriiSd0xx136MyZM2rcuLH69euniIgIXbhwQfv27dM777yjEydOKC4uTpMmTdJvfvMbl35em2HUZOa896nKLdOrasUKacAAqX17aedOpx4aAGCBixcvKiMjQwkJCQoNDbW6HJ9X0fmuyu9vuqWciJYbAACsR7hxIvuYG/uF/AAAgPsRbpwoPNy8kJ9E6w0AAFYh3DgZM6YAwPf42fBUyzjrPBNunIxxNwDgO+yzic6fP29xJf7h0qVLkqTAwMAaHYep4E5Gyw0A+I7AwEBFRUXp+PHjksypz66+6aO/Kiws1I8//qiwsDAFBdUsnhBunIyWGwDwLbGxsZLkCDhwnYCAADVt2rTGAZJw42S03ACAb7HZbIqLi1N0dLQuX75sdTk+LTg4WAEBNR8xQ7hxMlpuAMA3BQYG1ngsCNyDAcVORssNAADWItw4mb3lJjdXOnPG2loAAPBHhBsnCw+X7Hd5p2sKAAD3I9y4AONuAACwDuHGBRh3AwCAdQg3LkDLDQAA1iHcuAAtNwAAWIdw4wK03AAAYB3CjQvQcgMAgHUINy5Ayw0AANYh3LgAF/IDAMA6hBsX4EJ+AABYh3DjIoy7AQDAGoQbF2HcDQAA1iDcuAgtNwAAWINw4yK03AAAYA3CjYvYW24INwAAuBfhxkXsLTd0SwEA4F6EGxeh5QYAAGsQblyEC/kBAGANwo2L1K0rRUWZ27TeAADgPoQbF2LcDQAA7ke4cSH7uJtDh6ytAwAAf0K4caEePcz1Rx9ZWwcAAP6EcONCd99trleskHJyrK0FAAB/QbhxoZYtzdabwkJp4UKrqwEAwD8Qblxs7FhzPW+etXUAAOAvCDcuNmqUFBgobdki7d9vdTUAAPg+wo2LNWwoDRxobr/3nrW1AADgDwg3blC8a8owrK0FAABfZ2m4SUlJ0U033aTw8HBFR0drxIgR2n+Nvpu5c+fKZrOVWEJDQ91UcfUMG2ZesTgjQ0pNtboaAAB8m6XhZt26dZo0aZI2btyolStX6vLly7rtttt07ty5Cl8XERGhrKwsx3L48GE3VVw9YWHSHXeY2wwsBgDAtYKsfPPPPvusxJ/nzp2r6Ohobdu2Tb179y73dTabTbGxsa4uz6nGjpXefdecEj5zphQcbHVFAAD4Jo8ac5ObmytJqlevXoX7nT17Vtdff73i4+M1fPhw7d69u9x98/PzlZeXV2Kxws9/LsXGSidPSqUyHQAAcCKPCTeFhYWaOnWqevXqpfbt25e7X+vWrfXWW2/po48+0rx581RYWKikpCT9UM6tt1NSUhQZGelY4u03fHKzwMCiKxbTNQUAgOvYDMMz5u889NBD+vTTT7VhwwY1sd9OuxIuX76sNm3aaPTo0Xruueeuej4/P1/5+fmOP+fl5Sk+Pl65ubmKiIhwSu2VtWOHdOONZtBZv15KSnLr2wMA4LXy8vIUGRlZqd/fHtFyM3nyZC1btkxr1qypUrCRpFq1aqlLly5KT08v8/mQkBBFRESUWKzSpYs0erRUUGBe3O+nnywrBQAAn2VpuDEMQ5MnT9aSJUu0evVqJSQkVPkYBQUF2rlzp+Li4lxQofO99prUqpX0ww/SuHHmfacAAIDzWBpuJk2apHnz5mn+/PkKDw9Xdna2srOzdeHCBcc+48aN0/Tp0x1/fvbZZ7VixQodOnRI27dv19ixY3X48GFNmDDBio9QZeHh0qJFUmio9Omn0t/+ZnVFAAD4FkvDzezZs5Wbm6u+ffsqLi7OsSwsdgvtzMxMZWVlOf586tQpTZw4UW3atNHgwYOVl5enr7/+Wm3btrXiI1RLx47Sq6+a23/8ozn+BgAAOIfHDCh2l6oMSHIlw5Duuce89k1cnDnYOCbGsnIAAPBoXjeg2B/ZbNI//ym1bStlZUnJyVJOjtVVAQDg/Qg3FqpTR1qyxGy52bVL6tNHOnrU6qoAAPBuhBuLtWpljrmJj5f27zcDTmam1VUBAOC9CDceoEULM+AkJEjffSf17i0dOmR1VQAAeCfCjYdo1kxat05q2VI6fNgMOPv2WV0VAADeh3DjQeLjzYDTtq059uaWW6Tt262uCgAA70K48TBxcWbA6drVvD3DrbdKGzZYXRUAAN6DcOOBGjSQVq82u6by8qTbbpM+/9zqqgAA8A6EGw8VEWHenmHQIOnCBWnoUOk//7G6KgAAPB/hxoOFhUlLl0p33SVdviz96lfStm1WVwUAgGcj3Hi44GDpvfekAQPMFpxhw7jQHwAAFSHceIGgIGnhQnMW1bFjZsA5d87qqgAA8EyEGy8RGSktW2YONt6+Xfr1r6XCQqurAgDA8xBuvEhCgjkGJzjYvCfVU09ZXREAAJ6HcONlevWS/vUvc/uFF6Tly62tBwAAT0O48UJjx0pTppjbDzwg5eZaWw8AAJ6EcOOlnn/evOHm0aPS735ndTUAAHgOwo2XCgsr6p56801p5Upr6wEAwFMQbrxY797S5Mnm9oQJ0pkz1tYDAIAnINx4uZQUqVkzKTNTeuIJq6sBAMB6hBsvV7eu2S0lSbNnS2vWWFsPAABWI9z4gH79zFlTkvTII1zcDwDg3wg3PuJvfzOvYrxrl7R4sdXVAABgHcKNj4iKKrr2zV/+QusNAMB/EW58yNSpUkSE2XqzZInV1QAAYA3CjQ+57jpabwAAINz4GHvrzc6d5k02AQDwN4QbH1OvnjljSqL1BgDgnwg3PujRR6XwcOnbb6WPPrK6GgCAqxiGlJdndRWeh3Djg2i9AQD/MG2aVL++ORQBRQg3PsreevPNN9Ly5VZXAwBwha1bpStXpC1brK7EsxBufFT9+tJvfmNuv/yytbUAAFzj8mVzffKktXV4GsKND3v4Yclmk1aulPbts7oaAICz2cPNiRPW1uFpCDc+LCFBGjrU3H71VWtrAQA4Hy03ZSPc+Dj7wOK5c6XcXEtLAQA4GeGmbIQbH/fzn0tt20rnzpkBBwDgO+iWKhvhxsfZbNJvf2tuv/oq08IBwJfQclM2wo0fGDtWioyU0tOlzz6zuhoAgLMQbspGuPEDdetK991nbr/yirW1AACch26pshFu/MSkSWYX1WefSfv3W10NAMAZ7OHm/Hnp4kVra/EkhBs/0by5NGSIuT1rlrW1AACcwx5uJOnUKevq8DSEGz8yebK5njev5A8EAMA7Ff+/nK6pIoQbP5KcLDVsaKb7deusrgYAUBOGUTLcMKi4COHGjwQGSsOGmdtLllhbCwCgZgoKSv6ZcFOEcONn7rjDXC9ZwjVvAMCblR5eQLdUEcKNn+nXTwoPl7KypM2bra4GAFBdpcMNLTdFCDd+JiSkaNbU4sXW1gIAqD7CTfksDTcpKSm66aabFB4erujoaI0YMUL7K3ERlkWLFumGG25QaGioOnTooOXLl7uhWt9RvGvKMKytBQBQPXRLlc/ScLNu3TpNmjRJGzdu1MqVK3X58mXddtttOnfuXLmv+frrrzV69Gjdf//92rFjh0aMGKERI0Zo165dbqzcuw0aZLbgpKdLnDYA8E603JTPZhie8939xx9/VHR0tNatW6fevXuXuc+oUaN07tw5LVu2zPFYz5491blzZ82ZM+ea75GXl6fIyEjl5uYqIiLCabV7m2HDpI8/lv78Z+lPf7K6GgBAVR06ZF6g1e7WW6XVq62rx9Wq8vvbo8bc5ObmSpLq1atX7j6pqalKTk4u8diAAQOUmppa5v75+fnKy8srsaBk1xQAwPvQLVU+jwk3hYWFmjp1qnr16qX27duXu192drZiYmJKPBYTE6Ps7Owy909JSVFkZKRjiY+Pd2rd3mroUPO6N998Y6Z/AIB3oVuqfB4TbiZNmqRdu3ZpwYIFTj3u9OnTlZub61iOHDni1ON7q/r1pT59zG1abwDA+xBuyucR4Wby5MlatmyZ1qxZoyZNmlS4b2xsrHJycko8lpOTo9jY2DL3DwkJUURERIkFJnvXFFPCAcD72MON/dcadwYvYmm4MQxDkydP1pIlS7R69WolJCRc8zWJiYlatWpVicdWrlypxMREV5Xps0aMMNepqeZF/QAA3sMebho0kAL++9uc1huTpeFm0qRJmjdvnubPn6/w8HBlZ2crOztbFy5ccOwzbtw4TZ8+3fHnKVOm6LPPPtOMGTO0b98+/fnPf9bWrVs12X7La1Ra48ZSjx7mtW4+/tjqagAAVWEPN8HBkn0eDuHGZGm4mT17tnJzc9W3b1/FxcU5loULFzr2yczMVFaxZoWkpCTNnz9fr7/+ujp16qQPPvhAS5curXAQMso3eLC5XrnS2joAAFVjDze1ahWFG2ZMmYKsfPPKXGJn7dq1Vz02cuRIjRw50gUV+Z/+/c3r3Kxebd5hNjDQ6ooAAJVRPNzUqWNu03Jj8ogBxbDOTTeZN9I8eVJKS7O6GgBAZRUPN/Xrm9uEGxPhxs8FBZlXtZTomgIAb0K3VPkIN1D//ub6iy+srQMAUHlXrphrWm6uRriB7Hez2LBBKjZRDQDgwcpquSHcmAg3UOvW5rTw/Hwz4AAAPB/dUuUj3EA2G11TAOBtGFBcPsINJBV1TRFuAMA70C1VPsINJEn9+pnrHTukn36ythYAwLXRLVU+wg0kSbGxUocO5q0YVq+2uhoAwLXQLVU+wg0c6JoCAO9RVsvNhQvMepUINyjGHm5WrjRbcAAAnqt4uImIKLp9Dq03hBsU07u3+UPy/ffSoUNWVwMAqEjxcGOzMai4OMINHOrWlRITzW26pgDAsxUPNxLhpjjCDUpg3A0AeIfywg0zpgg3KMV+Mb/Vqxl3AwCerHS4YcZUEcINSrjxRvNO4SdPSkePWl0NAKA8dEuVj3CDEoKDpVatzO2dO62tBQBQPrqlyke4wVU6dDDXu3ZZWwcAoHx0S5WPcIOrtG9vrmm5AQDPRbdU+Qg3uAotNwDg+eiWKh/hBlext9zs2SMVFFhbCwCgbHRLlY9wg6skJEhhYVJ+vpSebnU1AICy0C1VPsINrhIQILVrZ27TNQUAnqm8lhu6pQg3KId93A2DigHAM5XXcnPxIncGJ9ygTPZxN7TcAIBnKh1uwsPNi7BKdE0RblAmpoMDgGcrHW6K3xnc37umCDcok71bKj2d5k0A8ESlw43EoGI7wg3KFBNjDk4rLJT27bO6GgBAaYSb8hFuUCabjUHFAODJygo3zJgyEW5QLgYVA4DnouWmfIQblIuWGwDwXISb8hFuUC5abgDAc9EtVT7CDcplv0rxDz9Ip09bWgoAoBRabspHuEG5IiOlpk3NbVpvAMCzEG7KR7hBheiaAgDPRLdU+Qg3qBCDigHAM1XUckO4ASpAyw0AeJ7CQnORSoab664z16dOub8mT0K4QYWKt9wYhrW1AABM9lYbqeyWm4sXzcVfEW5QoRtukAIDzW8BWVlWVwMAkMoPN+HhUsB/f7P7c+sN4QYVCgmRWrUyt+maAgDPUF64CQgo6pry5xlThBtck33cDYOKAcAzFA83QUEln2PcDeEGlcCgYgDwLPZwExRk3ui4OK51U81w88477+iTTz5x/Pnxxx9XVFSUkpKSdPjwYacVB89gv1Lx7t3W1gEAMJU1DdyOlptqhpvnn39etWvXliSlpqZq1qxZ+t///V81aNBAjz76qFMLhPXs4WbPnqKphwAA6xBuKhZ07V2uduTIEbVo0UKStHTpUt1555164IEH1KtXL/Xt29eZ9cEDtGghBQdL585JmZlSs2ZWVwQA/q2icEO3VDVbburWrasT/7384YoVK9S/f39JUmhoqC5cuOC86uARgoKk1q3NbbqmAMB6tNxUrFrhpn///powYYImTJigAwcOaPDgwZKk3bt3q1kVvtavX79eQ4cOVaNGjWSz2bR06dIK91+7dq1sNttVS3Z2dnU+BqqAcTcA4DloualYtcLNrFmzlJiYqB9//FEffvih6v/3Tl3btm3T6NGjK32cc+fOqVOnTpo1a1aV3n///v3KyspyLNHR0VV6PaqOcAMAnqP4bKnSaLmp5pibqKgovfrqq1c9/pe//KVKxxk0aJAGDRpU5fePjo5WVFRUlV+H6iPcAIDnoFuqYtVqufnss8+0YcMGx59nzZqlzp076+6779YpN5zNzp07Ky4uTv3799dXX31V4b75+fnKy8srsaDq7OFm715mTAGA1eiWqli1ws3vf/97R0jYuXOnHnvsMQ0ePFgZGRmaNm2aUwssLi4uTnPmzNGHH36oDz/8UPHx8erbt6+2b99e7mtSUlIUGRnpWOLj411Wny9r3ty8FcP589L331tdDQD4N1puKlatbqmMjAy1bdtWkvThhx/q9ttv1/PPP6/t27c7Bhe7QuvWrdXaPm1HUlJSkr777ju99NJL+n//7/+V+Zrp06eXCFx5eXkEnGoIDDRvovnNN2bX1M9+ZnVFAOC/KttyYxhXX8HYH1Sr5SY4OFjnz5+XJH3xxRe67bbbJEn16tVze7dP9+7dlZ6eXu7zISEhioiIKLGgehh3AwCeoTItNwUF0tmz7qvJk1Sr5ebmm2/WtGnT1KtXL23evFkLFy6UJB04cEBNmjRxaoHXkpaWpri4OLe+p78i3ACAZ7hyxVyXFW5q1zaHEeTnm6034eHurc0TVCvcvPrqq3r44Yf1wQcfaPbs2WrcuLEk6dNPP9XAgQMrfZyzZ8+WaHXJyMhQWlqa6tWrp6ZNm2r69Ok6evSo3n33XUnSzJkzlZCQoHbt2unixYt68803tXr1aq1YsaI6HwNVRLgBAM9QUcuNzWa23mRnm+Nurr/evbV5gmqFm6ZNm2rZsmVXPf7SSy9V6Thbt27Vrbfe6vizfWzM+PHjNXfuXGVlZSkzM9Px/KVLl/TYY4/p6NGjCgsLU8eOHfXFF1+UOAZcp/iMqYICcxwOAMD9Kgo3Uslw44+qFW4kqaCgQEuXLtXevXslSe3atdOwYcMUWIXfeH379pVhGOU+P3fu3BJ/fvzxx/X4449Xq17UXEKCFBoqXbwoZWSY95wCALjftcKNv08Hr1a4SU9P1+DBg3X06FHH7KWUlBTFx8frk08+UfPmzZ1aJDxDYKDUpo20Y4fZNUW4AQBrVKblRvLflptqzZZ65JFH1Lx5cx05ckTbt2/X9u3blZmZqYSEBD3yyCPOrhEehHE3AGA9Wm4qVq2Wm3Xr1mnjxo2qZz97kurXr68XXnhBvXr1clpx8DyEGwCwHi03FatWy01ISIjOnDlz1eNnz55VcHBwjYuC5yLcAID1KttyQ7ipgttvv10PPPCANm3aJMMwZBiGNm7cqAcffFDDhg1zdo3wIPZws2+fOWMKAOB+lW258dduqWqFm5dfflnNmzdXYmKiQkNDFRoaqqSkJLVo0UIzZ850conwJM2aSWFh5sWhvvvO6moAwD/RLVWxao25iYqK0kcffaT09HTHVPA2bdqoBdNnfF5AgDljats2s2uqVSurKwIA/8OA4opVOtxc627fa9ascWy/+OKL1a8IHq9du6Jw84tfWF0NAPgfWm4qVulws2PHjkrtZ/PH24/6GQYVA4C1aLmpWKXDTfGWGfg3wg0AWKuyLTe5uf55u5xqDSiGf7OHm/37i37AAADuU9lwI5kBx98QblBlTZtKdetKly5JBw5YXQ0A+J9rhZtatcz/pyX/7Joi3KDKAgKkTp3M7bQ0S0sBAL90rXAj+fegYsINqqVLF3NdyXHmAAAnqky48edBxYQbVEvnzuaalhsAcD9abipGuEG1FA83hmFlJQDgf6rSckO4ASqpXTtzauGJE9LRo1ZXAwD+pSotN3RLAZUUGmrehkGiawoA3I1uqYoRblBt9q4pBhUDgHsxoLhihBtUm33GFC03AOBetNxUjHCDamPGFABYg5abihFuUG32C/kdOuSfl/cGAKvQclMxwg2qrX59KT7e3P72W2trAQB/wlTwihFuUCN0TQGA+zEVvGKEG9QIM6YAwP2qEm7On5fy811fkych3KBGmDEFAO5XmXATGSnZbOa2v3VNEW5QI/aWm927pUuXLC0FAPxGZcJNQIAUFWVuE26AKmjWTIqIMIPNvn1WVwMA/qEy4Uby30HFhBvUiM3GoGIAcLfKhht/HVRMuEGNMagYANyLlpuKEW5QY7TcAID7GIZ05Yq5TctN2Qg3qLHiM6YMw9JSAMDn2YONVPlwQ8sNUEVt25o/YKdPS5mZVlcDAL7N3iUlVb5bipYboIqCg82AI9E1BQCuVpVwQ8sNUAMMKgYA96hOyw3hBqgG+7ibrVutrQMAfJ093AQEmEtFGFAM1EDPnuZ640YGFQOAK1V2GrhEtxRQI126SCEh0okT0sGDVlcDAL6rKuGGAcVADQQHS926mdupqdbWAgC+rLotN/7Uqk64gdMkJprrr7+2tg4A8GXVabm5fFk6f951NXkawg2cxh5uaLkBANepSrgJCyvaz5+6pgg3cBp7uNm1S8rLs7YWAPBVVQk3Npt/Tgcn3MBp4uKkZs3Mft3Nm62uBgB8U1XCjeSf08EJN3AquqYAwLXs4SYoqHL7++N0cMINnIpBxQDgWlVtufHH6eCEGziVPdxs3CgVFlpbCwD4oqqGm/r1zfWJE66pxxMRbuBUnTpJtWubdwjfv9/qagDA91Q13MTGmuusLNfU44ksDTfr16/X0KFD1ahRI9lsNi1duvSar1m7dq1uvPFGhYSEqEWLFpo7d67L60Tl1aol3XSTuc24GwBwvitXzHVlw02jRub62DHX1OOJLA03586dU6dOnTRr1qxK7Z+RkaEhQ4bo1ltvVVpamqZOnaoJEybo888/d3GlqAoGFQOA61S15cYfw00lx1q7xqBBgzRo0KBK7z9nzhwlJCRoxowZkqQ2bdpow4YNeumllzRgwABXlYkqYlAxALgO4ebavGrMTWpqqpKTk0s8NmDAAKVW0ESQn5+vvLy8Egtcyx5u9uwxx94AAJynJuHGX+4v5VXhJjs7WzExMSUei4mJUV5eni5cuFDma1JSUhQZGelY4uPj3VGqX4uOlpo3N7c3bbK2FgDwNVUNN3Fx5vriRf/5wulV4aY6pk+frtzcXMdy5MgRq0vyC4y7AQDXqGq4CQ0tutaNv8yY8qpwExsbq5ycnBKP5eTkKCIiQrVr1y7zNSEhIYqIiCixwPUYdwMArlHVcCMVtd74y7gbrwo3iYmJWrVqVYnHVq5cqUT7b1J4jKQkc71pExfzAwBnqk648bdBxZaGm7NnzyotLU1paWmSzKneaWlpyszMlGR2KY0bN86x/4MPPqhDhw7p8ccf1759+/TPf/5T//73v/Xoo49aUT4q0L69VKeOeXfw3butrgYAfAfh5tosDTdbt25Vly5d1KVLF0nStGnT1KVLFz3zzDOSpKysLEfQkaSEhAR98sknWrlypTp16qQZM2bozTffZBq4BwoKKuqaWr/e2loAwJcQbq7N0uvc9O3bV0YF89LKuvpw3759tWPHDhdWBWfp00f64gtp3Tpp0iSrqwEA30C4uTavGnMD79Knj7let85/rq0AAK5GuLk2wg1cpnt3KSREOn5cOnDA6moAwDcQbq6NcAOXCQmRevY0t9ets7YWAPAVNZkKnpXlHy3phBu4VPGuKQBAzVUn3MTGmutLl6STJ51fk6ch3MClGHcDAM5VnXATEiI1aGBu+0PXFOEGLtWzp/kDePSolJFhdTUA4P2qE24k/xp3Q7iBS4WFmQOLJbqmAMAZCDfXRriBy/Xuba4JNwBQc4SbayPcwOUYVAwAzkO4uTbCDVwuKUkKDJS+/14qdjcNAEA1VDfc+NOdwQk3cLnwcKlrV3Ob+0wBQM3QcnNthBu4BeNuAMA5ahpusrKcW48nItzALRh3AwDO4YxwU1jo3Jo8DeEGbnHzzZLNJh086B/fGgDAVaobbmJizP+Hr1yRfvrJ+XV5EsIN3CIqSurc2dxm3A0AVF91w02tWlJ0tLnt6+NuCDdwG7qmAKDmqhtuJP8ZVEy4gdsQbgCg5moSbvxlOjjhBm7Tu7fZ37tnj3T8uNXVAIB3ouXm2gg3cJt69aSOHc3ttWstLQUAvBbh5toIN3CrW28114QbAKgews21EW7gVn37mmvCDQBUjzPCja9fkoNwA7e65RZz3M3evVJ2ttXVAID3oeXm2gg3cKt69aROncxtZk0BQNU5I9xkZ0sFBc6rydMQbuB29nE3a9ZYWwcAeKOahJvoaCkgwLz9gi/PWiXcwO0YdwMA1VNQIBmGuV2dcBMYaN6GQfLtrinCDdzOPu5m/37fH9QGAM5kb7WRqhduJP8Yd0O4gdtdd53UpYu5TesNAFQe4aZyCDewBF1TAFB1hJvKIdzAEoQbAKi64uEmMLB6x/CHa90QbmCJW24xR+wfOODb3x4AwJmKz5Sy2ap3DFpuABeJimLcDQBUVU2mgdv5w53BCTewDF1TAFA1zgg3tNwALsTF/ACgapwRbpo0MdfHj0vnz9e8Jk9EuIFlbr7ZHHeTni798IPV1QCA53NGuGnQwLxSsWFIu3Y5py5PQ7iBZSIjpa5dze1Vq6ytBQC8gTPCjc0mde5sbqel1bQiz0S4gaUGDDDXn35qbR0A4A2cEW4kwg3gUoMGmesVK6QrV6ytBQA8nbPDzY4dNTuOpyLcwFLdu5vTwk+dkjZvtroaAPBszg43335r3ozT1xBuYKmgIOm228xtuqYAoGL2cBMUVLPjtGol1a5tzpZKT695XZ6GcAPL2bumCDcAUDFntdwEBkodO5rbvjjuhnADyw0caK63bZNycqytBQA8mX1sYk3DjeTbg4oJN7BcbGzRrRg+/9zaWgDAkzmr5Uby7UHFhBt4BLqmAODaXBFuaLkBXKT4lHBfHLkPAM7gzHDToYN5Qb+cHCk7u+bH8ySEG3iEnj3NKeEnT0pbtlhdDQB4JmeGmzp1pNatzW1fa70h3MAjBAVJ/fub23RNAUDZnBluJN/tmiLcwGMw7gYAKka4qRyPCDezZs1Ss2bNFBoaqh49emhzBZeqnTt3rmw2W4klNDTUjdXCVez3mdq6VfrxR2trAQBP5Kpw42szpiwPNwsXLtS0adP0pz/9Sdu3b1enTp00YMAAHT9+vNzXREREKCsry7EcPnzYjRXDVRo1kjp1kgyDKeEAUBZXhZuDB6WzZ51zTE9gebh58cUXNXHiRN17771q27at5syZo7CwML311lvlvsZmsyk2NtaxxMTElLtvfn6+8vLySizwXHRNAUD5nB1uYmLMa40ZhrRzp3OO6QksDTeXLl3Stm3blJyc7HgsICBAycnJSk1NLfd1Z8+e1fXXX6/4+HgNHz5cu3fvLnfflJQURUZGOpb4+HinfgY41+DB5vrTT4t+iAEAJmeHG6noIqq+NO7G0nDz008/qaCg4KqWl5iYGGWXM+m+devWeuutt/TRRx9p3rx5KiwsVFJSkn744Ycy958+fbpyc3Mdy5EjR5z+OeA8SUlSw4bmXcLXrbO6GgDwLK4IN744qNjybqmqSkxM1Lhx49S5c2f16dNHixcvVsOGDfXaa6+VuX9ISIgiIiJKLPBcgYHSiBHm9ocfWloKAHgcV4YbXxpUbGm4adCggQIDA5VT6m6JOTk5io2NrdQxatWqpS5duijdF+/Z7qfuuMNcL1nC1YoBoDhXhpudO4tuzOntLA03wcHB6tq1q1atWuV4rLCwUKtWrVJiYmKljlFQUKCdO3cqLi7OVWXCzX7+cyky0rwk+MaNVlcDAJ7DFeGmRQvzasUXL0oHDjjvuFayvFtq2rRpeuONN/TOO+9o7969euihh3Tu3Dnde++9kqRx48Zp+vTpjv2fffZZrVixQocOHdL27ds1duxYHT58WBMmTLDqI8DJgoOloUPNbbqmAKCIK8JNQIB5GQ7Jd8bdWB5uRo0apf/7v//TM888o86dOystLU2fffaZY5BxZmamsrKyHPufOnVKEydOVJs2bTR48GDl5eXp66+/Vtu2ba36CHABe9fU4sXmFEUAgGvCjVTUNVXBNXS9is0w/OtXR15eniIjI5Wbm8vgYg92/rw5a+r8eWnbNunGG62uCACsN3q0tGCBNHOmNGWK84774YfSL38pJSRI331n3i3c01Tl97flLTdAWcLCii7ot3ixtbUAgKdwVcvNwIFSaKiUkSF9+61zj20Fwg08lr1rinE3AGByVbipU0e67TZze8kS5x7bCoQbeKzbbzcHF+/bJ+3da3U1AGA9V4UbSfrFL8z10qXOP7a7EW7gsSIiJPudOeiaAgDXhpuhQ80LqX7zjdk95c0IN/BodE0BQBFXhpv69aXevc1tb++aItzAow0fbl6DYccO7/8mAQA15cpwIxV1TRFuABdq0EDq08fcXrTI2loAwGquDjfDh5vrr74yrxLvrQg38Hi/+pW5fv99a+sAAKu5Otw0bSp17WpePPU//3HNe7gD4QYe7847zR/ktDRpzx6rqwEA67g63Ei+MWuKcAOPV7++eYEpidYbAP7NneHmiy+kvDzXvY8rEW7gFe6+21zPn8+9pgD4L3eEmzZtpFatpEuXpE8/dd37uBLhBl5h6FDzCpqHDvnOjd0AoKrcEW5sNu+fNUW4gVeoU0caMcLcnj/f0lIAwDLuCDdS0f+3y5dLZ8+69r1cgXADr2Hvmlq4ULpyxdpaAMAK7go33btLzZtLZ85ITz3l2vdyBcINvEb//ubg4pwcac0aq6sBAPdzV7gJCJD++U9z+5VXpNRU176fsxFu4DVq1ZLuusvcpmsKgD9yV7iRzLuEjxtnTuK4/34pP9/17+kshBt4FXvX1IcfShcuWFsLALibO8ONJL34ohQdLe3dKz3/vHve0xkIN/AqSUnmFTTPnDEHugGAvzAMqaDA3HZXuKlf3+yWkqSUFGnnTve8b00RbuBVAgKk0aPNbbqmAPgTe6uN5L5wI0kjR0rDhpnvP2FCUcDyZIQbeB1719THH0tHjlhbCwC4i1XhxmYzBxdHRJjXGXv5Zfe9d3URbuB1OnaU+vY1f9D/93+trgYA3MOqcCNJjRsX/X/7xz9KGRnuff+qItzAKz39tLl+4w0pO9vaWgDAHawMN5I0caLUu7d0/rz0wAOefSscwg280q23SomJ5tTE//s/q6sBANezh5vAQLOryN0CAswvlCEh5k0133nH/TVUFuEGXslmK2q9mT1b+ukna+sBAFdz9zTwsrRqJf3lL+b2tGnmRVU9EeEGXmvgQKlrV7OJ9KWXrK4GAFzLE8KNJD32mNSli3TqlPTb31pbS3kIN/BaNps5sE0yr8Nw6pS19QCAK3lKuAkKkv71L7N7bNEi6aOPrK2nLIQbeLVhw6QOHcyL+tkvNAUAvshTwo1kttz87nfm9sMPS8eOWVtPaYQbeLWAgKI71s6cKeXlWVoOALiMJ4UbSfrTn8wxOMeOSX36SJmZVldUhHADr/fLX0qtW5vdUjNmWF0NALiGp4Wb2rWlzz6TmjWT0tPNaeLffWd1VSbCDbxeYKD03HPmdkqKtGuXtfUAgCt4WriRpIQE6csvzRacw4fNgLNvn9VVEW7gI375y6J7n9x3n3TlitUVAYBzeWK4kaQmTaR166R27cwuqt69pW++sbYmwg18gs1mXu8mMlLassUcfwMAvsRTw40kxcZKa9eaA41//FEaMEA6d866egg38BmNGhWNuXn6aengQWvrAQBnunTJXAcFWVtHeRo0kFavlnr1Mmev1qljXS2EG/iU++6TkpOlixelCROkwkKrKwIA51i71lw3b25pGRWKipLWr5dGjrS2DsINfIrNJr3+uvmNYf16ac4cqysCgJorKJAWLDC3R4+2tpZrCfCAZOEBJQDOlZBgzpqSzItMffKJtfUAQE2tW2cO1r3uOmnQIKur8XyEG/ikSZOkIUOkCxek4cPNS4UDgLd67z1z/ctfSsHB1tbiDQg38EkBAdKSJdK4cWZz7oQJ0p//LBmG1ZUBQNVcvCh98IG5PWaMtbV4C8INfFatWtLcuUW3Z/jLX6SJE7kGDgDvsny5eWuZJk2kW26xuhrvQLiBT7PZpP/5H3NgcUCA2T3Vo4e0caPVlQFA5di7pEaP9ozBut6A0wS/8JvfmN1UkZHS9u1SYqJ0//3mxaYAwFOdPi0tW2Zu0yVVeYQb+I1hw6T9+6Xx480/v/WWeT+UmTOlEycsLQ0AyrR4sXnxvnbtpI4dra7GexBu4FdiYsxxOF99JXXubH4revRR89LhQ4ZI8+ZJZ85YXCQA/Je9S+ruu81udlSOzTD8a/5IXl6eIiMjlZubq4iICKvLgYUKCqQ33pBee01KSyt6PDRU6tDBvAqofUlIMINRTIx5nQn+kwHgakePSvHx5izPjAypWTOrK7JWVX5/e+gdKgDXCwyUHnzQXPbulRYulN5/XzpwwLz55pYtZb8uKEiKjpbq1jVnZAUHm+uQEPPKyMWXy5elkyelU6fM9dmzZjhq2NA8RsOG5uXKQ0OLlpAQszabzRw8aLMVLcUZhhnQCgvNxf41pfj+AQHmEhhYtA4KKrnYj2UYRbersL/OvpQV5gIDi45nP37xOovXULymwsKSdRev076v/Tn7Yj9e8WOVtY99P/va/pmKf7bS+xUUmDPo7OvCwpLnx/7ZKmI/fvHjFBRcfe5Lnwubrehx+z6l6y5dc/HPV3xd/O+x+P7297R/Bvs+pfcrfs6KL/bzVnz/ggKzq+Ty5aJ18c8SGFi0X/HzERRk/vsODjbXQUEl/w4LCsr+Oyr9b9peV/HXlX4vwyj6N2qvqfjnNgzzeEFB5s+vfbEf4/Jlc7ly5epzUvx82xX/+bJ//tKvK/7vqlYt8/2L11763+iiRebrevUi2FQVLTdAMYZhjsvZu1f67jtzSU+XMjOlnBwpN9fqCgH4m1mzpIcftroK63ldy82sWbP097//XdnZ2erUqZNeeeUVde/evdz9Fy1apKefflrff/+9WrZsqb/97W8aPHiwGyuGr7LZpBtuMJey5OdLx4+by/nzJb+5XrxoPnb2rHTunLkODpbq1TNba+rVM1tzTp0yZ2kdP26u8/LM1168aB7/4sWSrTFlfYM3jLJbV4p/K7Uv9m+Exb8h2r+RXr5s7l/Wt/virSv2Fhb7OSreUmH/pmz/1l1WDfbPYP82XbyVQrq6Nad4a4Z9n9LfgkvvU1arQ+nzU/xY9nXp1iz7t2n7Z7N/cy99/ku3ZgUEFH1rDwoqamEofv6Ln4fSrQ/Fl7Ja7Eq3OpReF6+n9Lko/ndYeil9zkq39pTVWhQQULLVslYt8/HSLYnFWzICA81zeemS+e/c/nNTvFWr9N93WX/v9vNTein9XqVbRQoKyv7sxVtpLl82X1urVlHriv3fROml9L+H4j9j9mtpFT+H9vNT/N+V/d968Ram0sdu1oxZUtVhebhZuHChpk2bpjlz5qhHjx6aOXOmBgwYoP379ys6Ovqq/b/++muNHj1aKSkpuv322zV//nyNGDFC27dvV/v27S34BPAnISFmH3h8vNWVAADKY3m3VI8ePXTTTTfp1VdflSQVFhYqPj5ev/3tb/WHP/zhqv1HjRqlc+fOaZl94r+knj17qnPnzppTiVtA0y0FAID3qcrvb0ungl+6dEnbtm1TcnKy47GAgAAlJycrNTW1zNekpqaW2F+SBgwYUO7++fn5ysvLK7EAAADfZWm4+emnn1RQUKCYmJgSj8fExCg7O7vM12RnZ1dp/5SUFEVGRjqWePoTAADwaT5/Eb/p06crNzfXsRw5csTqkgAAgAtZOqC4QYMGCgwMVE5OTonHc3JyFBsbW+ZrYmNjq7R/SEiIQkJCnFMwAADweJa23AQHB6tr165atWqV47HCwkKtWrVKiYmJZb4mMTGxxP6StHLlynL3BwAA/sXyqeDTpk3T+PHj1a1bN3Xv3l0zZ87UuXPndO+990qSxo0bp8aNGyslJUWSNGXKFPXp00czZszQkCFDtGDBAm3dulWvv/66lR8DAAB4CMvDzahRo/Tjjz/qmWeeUXZ2tjp37qzPPvvMMWg4MzNTAcWufZ6UlKT58+frj3/8o5588km1bNlSS5cu5Ro3AABAkgdc58bduM4NAADex2uucwMAAOBshBsAAOBTCDcAAMCnEG4AAIBPIdwAAACfYvlUcHezTw7jBpoAAHgP++/tykzy9rtwc+bMGUniBpoAAHihM2fOKDIyssJ9/O46N4WFhTp27JjCw8Nls9mqfZy8vDzFx8fryJEjXC/HxTjX7sO5di/Ot/twrt3HVefaMAydOXNGjRo1KnFx37L4XctNQECAmjRp4rTjRURE8IPiJpxr9+Fcuxfn23041+7jinN9rRYbOwYUAwAAn0K4AQAAPoVwU00hISH605/+pJCQEKtL8Xmca/fhXLsX59t9ONfu4wnn2u8GFAMAAN9Gyw0AAPAphBsAAOBTCDcAAMCnEG4AAIBPIdxU06xZs9SsWTOFhoaqR48e2rx5s9Uleb2UlBTddNNNCg8PV3R0tEaMGKH9+/eX2OfixYuaNGmS6tevr7p16+rOO+9UTk6ORRX7hhdeeEE2m01Tp051PMZ5dq6jR49q7Nixql+/vmrXrq0OHTpo69atjucNw9AzzzyjuLg41a5dW8nJyTp48KCFFXungoICPf3000pISFDt2rXVvHlzPffccyXuRcS5rp7169dr6NChatSokWw2m5YuXVri+cqc15MnT2rMmDGKiIhQVFSU7r//fp09e9Y1BRuosgULFhjBwcHGW2+9ZezevduYOHGiERUVZeTk5FhdmlcbMGCA8fbbbxu7du0y0tLSjMGDBxtNmzY1zp4969jnwQcfNOLj441Vq1YZW7duNXr27GkkJSVZWLV327x5s9GsWTOjY8eOxpQpUxyPc56d5+TJk8b1119v3HPPPcamTZuMQ4cOGZ9//rmRnp7u2OeFF14wIiMjjaVLlxrffPONMWzYMCMhIcG4cOGChZV7n7/+9a9G/fr1jWXLlhkZGRnGokWLjLp16xr/+Mc/HPtwrqtn+fLlxlNPPWUsXrzYkGQsWbKkxPOVOa8DBw40OnXqZGzcuNH48ssvjRYtWhijR492Sb2Em2ro3r27MWnSJMefCwoKjEaNGhkpKSkWVuV7jh8/bkgy1q1bZxiGYZw+fdqoVauWsWjRIsc+e/fuNSQZqampVpXptc6cOWO0bNnSWLlypdGnTx9HuOE8O9cTTzxh3HzzzeU+X1hYaMTGxhp///vfHY+dPn3aCAkJMd5//313lOgzhgwZYtx3330lHrvjjjuMMWPGGIbBuXaW0uGmMud1z549hiRjy5Ytjn0+/fRTw2azGUePHnV6jXRLVdGlS5e0bds2JScnOx4LCAhQcnKyUlNTLazM9+Tm5kqS6tWrJ0natm2bLl++XOLc33DDDWratCnnvhomTZqkIUOGlDifEufZ2f7zn/+oW7duGjlypKKjo9WlSxe98cYbjuczMjKUnZ1d4nxHRkaqR48enO8qSkpK0qpVq3TgwAFJ0jfffKMNGzZo0KBBkjjXrlKZ85qamqqoqCh169bNsU9ycrICAgK0adMmp9fkdzfOrKmffvpJBQUFiomJKfF4TEyM9u3bZ1FVvqewsFBTp05Vr1691L59e0lSdna2goODFRUVVWLfmJgYZWdnW1Cl91qwYIG2b9+uLVu2XPUc59m5Dh06pNmzZ2vatGl68skntWXLFj3yyCMKDg7W+PHjHee0rP9TON9V84c//EF5eXm64YYbFBgYqIKCAv31r3/VmDFjJIlz7SKVOa/Z2dmKjo4u8XxQUJDq1avnknNPuIFHmjRpknbt2qUNGzZYXYrPOXLkiKZMmaKVK1cqNDTU6nJ8XmFhobp166bnn39ektSlSxft2rVLc+bM0fjx4y2uzrf8+9//1nvvvaf58+erXbt2SktL09SpU9WoUSPOtZ+hW6qKGjRooMDAwKtmjuTk5Cg2NtaiqnzL5MmTtWzZMq1Zs0ZNmjRxPB4bG6tLly7p9OnTJfbn3FfNtm3bdPz4cd14440KCgpSUFCQ1q1bp5dffllBQUGKiYnhPDtRXFyc2rZtW+KxNm3aKDMzU5Ic55T/U2ru97//vf7whz/oV7/6lTp06KBf//rXevTRR5WSkiKJc+0qlTmvsbGxOn78eInnr1y5opMnT7rk3BNuqig4OFhdu3bVqlWrHI8VFhZq1apVSkxMtLAy72cYhiZPnqwlS5Zo9erVSkhIKPF8165dVatWrRLnfv/+/crMzOTcV0G/fv20c+dOpaWlOZZu3bppzJgxjm3Os/P06tXrqksaHDhwQNdff70kKSEhQbGxsSXOd15enjZt2sT5rqLz588rIKDkr7XAwEAVFhZK4ly7SmXOa2Jiok6fPq1t27Y59lm9erUKCwvVo0cP5xfl9CHKfmDBggVGSEiIMXfuXGPPnj3GAw88YERFRRnZ2dlWl+bVHnroISMyMtJYu3atkZWV5VjOnz/v2OfBBx80mjZtaqxevdrYunWrkZiYaCQmJlpYtW8oPlvKMDjPzrR582YjKCjI+Otf/2ocPHjQeO+994ywsDBj3rx5jn1eeOEFIyoqyvjoo4+Mb7/91hg+fDjTk6th/PjxRuPGjR1TwRcvXmw0aNDAePzxxx37cK6r58yZM8aOHTuMHTt2GJKMF1980dixY4dx+PBhwzAqd14HDhxodOnSxdi0aZOxYcMGo2XLlkwF9zSvvPKK0bRpUyM4ONjo3r27sXHjRqtL8nqSylzefvttxz4XLlwwHn74YeO6664zwsLCjF/84hdGVlaWdUX7iNLhhvPsXB9//LHRvn17IyQkxLjhhhuM119/vcTzhYWFxtNPP23ExMQYISEhRr9+/Yz9+/dbVK33ysvLM6ZMmWI0bdrUCA0NNX72s58ZTz31lJGfn+/Yh3NdPWvWrCnz/+fx48cbhlG583rixAlj9OjRRt26dY2IiAjj3nvvNc6cOeOSem2GUezSjQAAAF6OMTcAAMCnEG4AAIBPIdwAAACfQrgBAAA+hXADAAB8CuEGAAD4FMINAADwKYQbAADgUwg3APze2rVrZbPZrrpZKADvRLgBAAA+hXADAAB8CuEGgOUKCwuVkpKihIQE1a5dW506ddIHH3wgqajL6JNPPlHHjh0VGhqqnj17ateuXSWO8eGHH6pdu3YKCQlRs2bNNGPGjBLP5+fn64knnlB8fLxCQkLUokUL/etf/yqxz7Zt29StWzeFhYUpKSlJ+/fvd+0HB+AShBsAlktJSdG7776rOXPmaPfu3Xr00Uc1duxYrVu3zrHP73//e82YMUNbtmxRw4YNNXToUF2+fFmSGUruuusu/epXv9LOnTv15z//WU8//bTmzp3reP24ceP0/vvv6+WXX9bevXv12muvqW7duiXqeOqppzRjxgxt3bpVQUFBuu+++9zy+QE4F3cFB2Cp/Px81atXT1988YUSExMdj0+YMEHnz5/XAw88oFtvvVULFizQqFGjJEknT55UkyZNNHfuXN11110aM2aMfvzxR61YscLx+scff1yffPKJdu/erQMHDqh169ZauXKlkpOTr6ph7dq1uvXWW/XFF1+oX79+kqTly5dryJAhunDhgkJDQ118FgA4Ey03ACyVnp6u8+fPq3///qpbt65jeffdd/Xdd9859isefOrVq6fWrVtr7969kqS9e/eqV69eJY7bq1cvHTx4UAUFBUpLS1NgYKD69OlTYS0dO3Z0bMfFxUmSjh8/XuPPCMC9gqwuAIB/O3v2rCTpk08+UePGjUs8FxISUiLgVFft2rUrtV+tWrUc2zabTZI5HgiAd6HlBoCl2rZtq5CQEGVmZqpFixYllvj4eMd+GzdudGyfOnVKBw4cUJs2bSRJbdq00VdffVXiuF999ZVatWqlwMBAdejQQYWFhSXG8ADwXbTcALBUeHi4fve73+nRRx9VYWGhbr75ZuXm5uqrr75SRESErr/+eknSs88+q/r16ysmJkZPPfWUGjRooBEjRkiSHnvsMd1000167rnnNGrUKKWmpurVV1/VP//5T0lSs2bNNH78eN133316+eWX1alTJx0+fFjHjx/XXXfdZdVHB+AihBsAlnvuuefUsGFDpaSk6NChQ4qKitKNN96oJ5980tEt9MILL2jKlCk6ePCgOnfurI8//ljBwcGSpBtvvFH//ve/9cwzz+i5555TXFycnn32Wd1zzz2O95g9e7aefPJJPfzwwzpx4oSaNm2qJ5980oqPC8DFmC0FwKPZZzKdOnVKUVFRVpcDwAsw5gYAAPgUwg0AAPApdEsBAACfQssNAADwKYQbAADgUwg3AADApxBuAACATyHcAAAAn0K4AQAAPoVwAwAAfArhBgAA+JT/D6apFb+TgxpdAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "plot_losses(lstm_sh_losses)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nkoXQFMmKbU1"
      },
      "source": [
        "## Report"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "liCOCuR5KeFE"
      },
      "source": [
        "According to the texts generated from different models and the losses during the training process of the models, analyze what is the reason for the difference in the result of models.\n",
        "\n",
        "Which model works better and what do you think are the reasons?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_TMkOL1uK8AU"
      },
      "source": [
        "<font color='#73FF73'><b>Your answer : </b></font>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "LSTM result text is better than RNN. because the structure of the output is more alike to the Shakespeare plays. one of the potential reasons of the better performance of LSTM is the computational flexibility which is more in the LSTM and it can manage its memory cells more efficiently."
      ],
      "metadata": {
        "id": "-NcM0sBh96rg"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3UI0UnS9tzg"
      },
      "source": [
        "---\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UjJP-PvXLKVf"
      },
      "source": [
        "#2. FineTuning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqR62s6qMfc5"
      },
      "source": [
        "FineTuning is a technique used in neural network training where a pre-trained model is further trained on a new task or dataset. It allows us to leverage the knowledge and representations learned by a pre-trained model and adapt it to a specific task or domain.\n",
        "\n",
        "In this exercise, we first train the models with a `wikipedia` dataset that contains english texts, then we fine-tune this pre-trained model again with the Shakespeare play dataset to check the effect of this method on different models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hrBfhPcpOB-x"
      },
      "source": [
        "## Load Wikipedia dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "UiqBbrV4kSYN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c819d14-24c1-4a28-c349-29ff798087d5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-12-06 21:12:59--  https://s3.amazonaws.com/fast-ai-nlp/wikitext-2.tgz\n",
            "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.29.254, 52.216.114.205, 16.182.42.16, ...\n",
            "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.29.254|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 4070055 (3.9M) [application/x-tar]\n",
            "Saving to: ‘wikitext-2.tgz’\n",
            "\n",
            "wikitext-2.tgz      100%[===================>]   3.88M  19.1MB/s    in 0.2s    \n",
            "\n",
            "2023-12-06 21:13:00 (19.1 MB/s) - ‘wikitext-2.tgz’ saved [4070055/4070055]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://s3.amazonaws.com/fast-ai-nlp/wikitext-2.tgz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "ecXOywp5lv0D",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7f560a6d-e522-43cb-a5b9-c6127e8b48a2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "wikitext-2/\n",
            "wikitext-2/train.csv\n",
            "wikitext-2/test.csv\n"
          ]
        }
      ],
      "source": [
        "!tar -xvzf '/content/wikitext-2.tgz' -C '/content/data'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "c52RASUHpGM3"
      },
      "outputs": [],
      "source": [
        "!cat './data/wikitext-2/train.csv' | tr -d '\\n' > ./data/wikitext.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Sfwpof1OGOv"
      },
      "source": [
        "##Preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "cellView": "code",
        "id": "CniodW0Knkld"
      },
      "outputs": [],
      "source": [
        "def clean_wiki_data(data):\n",
        "    repl=''\n",
        "    data=re.sub('\\(', repl, data)\n",
        "    data=re.sub('\\)', repl, data)\n",
        "    for pattern in set(re.findall(\"=.*=\",data)):\n",
        "        data=re.sub(pattern, repl, data)\n",
        "    for pattern in set(re.findall(\"<unk>\",data)):\n",
        "        data=re.sub(pattern,repl,data)\n",
        "    for pattern in set(re.findall(r\"[^\\w ]\", data)):\n",
        "        repl=''\n",
        "        if pattern=='-':\n",
        "            repl=' '\n",
        "        if pattern!='.' and pattern!=\"\\'\":\n",
        "            data=re.sub(\"\\\\\"+pattern, repl, data)\n",
        "\n",
        "    return data\n",
        "\n",
        "def load_data(filepath):\n",
        "    f=open(filepath)\n",
        "    return f.read()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "N-BjAHGHpm4H"
      },
      "outputs": [],
      "source": [
        "wikidata=load_data(\"./data/wikitext.txt\")\n",
        "data=wikidata[:]\n",
        "data=clean_wiki_data(data)\n",
        "wikiPreprocessed_file = open(\"./data/wiki_preprocesed.txt\", \"w\")\n",
        "f = wikiPreprocessed_file.write(data)\n",
        "wikiPreprocessed_file.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MxILpQvZmFs8"
      },
      "source": [
        "- Load data in amount of 50kb for finetuning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "Q9AUJmRFPGqH"
      },
      "outputs": [],
      "source": [
        "wi_data_file = \"./data/wiki_preprocesed.txt\"\n",
        "wi_data = open(wi_data_file, 'r').read(50000)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "hdxMOch8e4mB"
      },
      "outputs": [],
      "source": [
        "wi_data = remove_extraneous_characters(wi_data.lower(), chars)\n",
        "wi_data_size = len(wi_data)\n",
        "\n",
        "wi_data = list(wi_data)\n",
        "for i, ch in enumerate(wi_data):\n",
        "    wi_data[i] = char_to_ix[ch]\n",
        "\n",
        "wi_data = torch.tensor(wi_data).to(device)\n",
        "wi_data = torch.unsqueeze(wi_data, dim=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Z9RXj7fOPFh"
      },
      "source": [
        "## Pre-training by wikipedia dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WQRRXlSDfVEI"
      },
      "source": [
        "### RNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "00m3G13vfcQ1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6259b979-8413-4671-e3d5-ab76fcef2fcc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "RNN(\n",
            "  (dropout): Dropout(p=0.5, inplace=False)\n",
            "  (encoder): Embedding(30, 30)\n",
            "  (rnn): RNN(30, 512, num_layers=6, batch_first=True)\n",
            "  (decoder): Linear(in_features=512, out_features=30, bias=True)\n",
            ")\n",
            "epoch=0\n",
            "current model with loss=2.7290302651827454 saved at ./model_wi_rnn.pth\n",
            "epoch=1\n",
            "current model with loss=2.6248034629665438 saved at ./model_wi_rnn.pth\n",
            "epoch=2\n",
            "current model with loss=2.5716143826969335 saved at ./model_wi_rnn.pth\n",
            "epoch=3\n",
            "current model with loss=2.5590264074137954 saved at ./model_wi_rnn.pth\n",
            "epoch=4\n",
            "current model with loss=2.545735462767179 saved at ./model_wi_rnn.pth\n",
            "epoch=5\n",
            "current model with loss=2.5204814047109885 saved at ./model_wi_rnn.pth\n",
            "epoch=6\n",
            "current model with loss=2.512947760644506 saved at ./model_wi_rnn.pth\n",
            "epoch=7\n",
            "current model with loss=2.4945776931575088 saved at ./model_wi_rnn.pth\n",
            "epoch=8\n",
            "current model with loss=2.4864187905045805 saved at ./model_wi_rnn.pth\n",
            "epoch=9\n",
            "Loss [epoch=9] = 2.480671847452883\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "eirein mas . . cof t ted . ver as . iotinicherorris  oyotlathe h    iatrt ler lfitondalatulilerl  in we oul dtooug  bre deenon esryocatioukres  . in povale tounn srein .inan rothacr co tres   dcpeviofh\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.480671847452883 saved at ./model_wi_rnn.pth\n",
            "epoch=10\n",
            "current model with loss=2.4726943559333927 saved at ./model_wi_rnn.pth\n",
            "epoch=11\n",
            "current model with loss=2.467470898002875 saved at ./model_wi_rnn.pth\n",
            "epoch=12\n",
            "current model with loss=2.4647690526774673 saved at ./model_wi_rnn.pth\n",
            "epoch=13\n",
            "current model with loss=2.4614837169647217 saved at ./model_wi_rnn.pth\n",
            "epoch=14\n",
            "epoch=15\n",
            "current model with loss=2.460257145225025 saved at ./model_wi_rnn.pth\n",
            "epoch=16\n",
            "current model with loss=2.452102586871288 saved at ./model_wi_rnn.pth\n",
            "epoch=17\n",
            "current model with loss=2.4486564417354395 saved at ./model_wi_rnn.pth\n",
            "epoch=18\n",
            "current model with loss=2.4429728554897623 saved at ./model_wi_rnn.pth\n",
            "epoch=19\n",
            "Loss [epoch=19] = 2.4382849071846633\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "onxhes mlk walen storo mrinc theur .com  e mit  wenthat m  m tt ite  hek     r eniiglbous meral ddinther thees ye .  ay im uned rt  d . mumin gat frsin herind ug .   . m   fiered . teslmh y . mcave n  \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.4382849071846633 saved at ./model_wi_rnn.pth\n",
            "epoch=20\n",
            "current model with loss=2.4335148764438315 saved at ./model_wi_rnn.pth\n",
            "epoch=21\n",
            "current model with loss=2.4302387862909036 saved at ./model_wi_rnn.pth\n",
            "epoch=22\n",
            "epoch=23\n",
            "current model with loss=2.4323081188514584 saved at ./model_wi_rnn.pth\n",
            "epoch=24\n",
            "current model with loss=2.4296783111134514 saved at ./model_wi_rnn.pth\n",
            "epoch=25\n",
            "current model with loss=2.426776925071341 saved at ./model_wi_rnn.pth\n",
            "epoch=26\n",
            "current model with loss=2.419494456932193 saved at ./model_wi_rnn.pth\n",
            "epoch=27\n",
            "current model with loss=2.4104420943338365 saved at ./model_wi_rnn.pth\n",
            "epoch=28\n",
            "current model with loss=2.405701043175869 saved at ./model_wi_rnn.pth\n",
            "epoch=29\n",
            "Loss [epoch=29] = 2.400565723903844\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "mphethaceclorlintwhrin emon th six nth in  oun  br  omoryones ntan iofr    wsoubonn  nd  mndin   ine d dile in merngal  eop  fintrandigrevald . s lomrl carthrl  s tininalo odiere  carrengh t of ed cy e\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.400565723903844 saved at ./model_wi_rnn.pth\n",
            "epoch=30\n",
            "current model with loss=2.3984204354833385 saved at ./model_wi_rnn.pth\n",
            "epoch=31\n",
            "current model with loss=2.3888978938587377 saved at ./model_wi_rnn.pth\n",
            "epoch=32\n",
            "current model with loss=2.3852797316723184 saved at ./model_wi_rnn.pth\n",
            "epoch=33\n",
            "current model with loss=2.3832359255337323 saved at ./model_wi_rnn.pth\n",
            "epoch=34\n",
            "current model with loss=2.377720365758802 saved at ./model_wi_rnn.pth\n",
            "epoch=35\n",
            "current model with loss=2.3750864150094206 saved at ./model_wi_rnn.pth\n",
            "epoch=36\n",
            "current model with loss=2.3656232474280183 saved at ./model_wi_rnn.pth\n",
            "epoch=37\n",
            "current model with loss=2.3589895905041303 saved at ./model_wi_rnn.pth\n",
            "epoch=38\n",
            "current model with loss=2.3562653494662924 saved at ./model_wi_rnn.pth\n",
            "epoch=39\n",
            "Loss [epoch=39] = 2.350537468175419\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "ked ie .unde  soulnd acoriie ale shedove   adseali m  w terhd tesfrstatbejedindigovonond  indvne  m g ndirind ismagloindiomy  irt  iconi t omon swilat fietind thed thetr geseitasterom sioustofaledity  \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.350537468175419 saved at ./model_wi_rnn.pth\n",
            "epoch=40\n",
            "current model with loss=2.3412155440596285 saved at ./model_wi_rnn.pth\n",
            "epoch=41\n",
            "current model with loss=2.3386915531314787 saved at ./model_wi_rnn.pth\n",
            "epoch=42\n",
            "current model with loss=2.3365562841540477 saved at ./model_wi_rnn.pth\n",
            "epoch=43\n",
            "current model with loss=2.333213720165315 saved at ./model_wi_rnn.pth\n",
            "epoch=44\n",
            "current model with loss=2.3218542591470186 saved at ./model_wi_rnn.pth\n",
            "epoch=45\n",
            "current model with loss=2.3202905068632034 saved at ./model_wi_rnn.pth\n",
            "epoch=46\n",
            "current model with loss=2.307840487996086 saved at ./model_wi_rnn.pth\n",
            "epoch=47\n",
            "epoch=48\n",
            "current model with loss=2.2994309272922453 saved at ./model_wi_rnn.pth\n",
            "epoch=49\n",
            "Loss [epoch=49] = 2.2923771084332074\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "ead oereth  ted   wn inin k . . ugby  athticod . onpexrom wan sicothead ic ss ththan  s tsendom    olin kewi by cgiof onthtivoear nourintsproty ingosocheme  sousun itoof thesindire witamihachetind    s\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.2923771084332074 saved at ./model_wi_rnn.pth\n",
            "epoch=50\n",
            "current model with loss=2.281422486070727 saved at ./model_wi_rnn.pth\n",
            "epoch=51\n",
            "current model with loss=2.277028883089785 saved at ./model_wi_rnn.pth\n",
            "epoch=52\n",
            "current model with loss=2.2697574427870455 saved at ./model_wi_rnn.pth\n",
            "epoch=53\n",
            "current model with loss=2.2681958323619407 saved at ./model_wi_rnn.pth\n",
            "epoch=54\n",
            "current model with loss=2.2637841633108797 saved at ./model_wi_rnn.pth\n",
            "epoch=55\n",
            "current model with loss=2.2586622804891867 saved at ./model_wi_rnn.pth\n",
            "epoch=56\n",
            "current model with loss=2.2553846562495 saved at ./model_wi_rnn.pth\n",
            "epoch=57\n",
            "current model with loss=2.249412685144143 saved at ./model_wi_rnn.pth\n",
            "epoch=58\n",
            "current model with loss=2.2413281866761503 saved at ./model_wi_rnn.pth\n",
            "epoch=59\n",
            "Loss [epoch=59] = 2.240586308182263\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "r wis  leli in  moletherdane  richak pon he fedaf demonaterlliun de n min ted m igninie tma twretyy lode herd t  d smy  chbononsthied sthediedind .ornthemosiodioredioutere zeeryt couabauk..tcafrinrecre\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.240586308182263 saved at ./model_wi_rnn.pth\n",
            "epoch=60\n",
            "current model with loss=2.2306986871312877 saved at ./model_wi_rnn.pth\n",
            "epoch=61\n",
            "current model with loss=2.2269052081420773 saved at ./model_wi_rnn.pth\n",
            "epoch=62\n",
            "current model with loss=2.2180110988069752 saved at ./model_wi_rnn.pth\n",
            "epoch=63\n",
            "current model with loss=2.203646656919698 saved at ./model_wi_rnn.pth\n",
            "epoch=64\n",
            "epoch=65\n",
            "current model with loss=2.197559778807593 saved at ./model_wi_rnn.pth\n",
            "epoch=66\n",
            "current model with loss=2.18884747536456 saved at ./model_wi_rnn.pth\n",
            "epoch=67\n",
            "current model with loss=2.1829634396756283 saved at ./model_wi_rnn.pth\n",
            "epoch=68\n",
            "current model with loss=2.1786892736544377 saved at ./model_wi_rnn.pth\n",
            "epoch=69\n",
            "Loss [epoch=69] = 2.17772010213039\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "s  in .ortr ound . theniphal t tbon   e m  me   oused ctwhie . . omieporatostilavede rson lefiser inmode hoffon cond rd tis an trakilfon by on gindis tspebaltherne i ctathaced on  anrtorthrethe  oc  ty\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.17772010213039 saved at ./model_wi_rnn.pth\n",
            "epoch=70\n",
            "current model with loss=2.1678937732196246 saved at ./model_wi_rnn.pth\n",
            "epoch=71\n",
            "current model with loss=2.1580116211390887 saved at ./model_wi_rnn.pth\n",
            "epoch=72\n",
            "current model with loss=2.1552991358960263 saved at ./model_wi_rnn.pth\n",
            "epoch=73\n",
            "current model with loss=2.14928622030821 saved at ./model_wi_rnn.pth\n",
            "epoch=74\n",
            "epoch=75\n",
            "current model with loss=2.1340951851156893 saved at ./model_wi_rnn.pth\n",
            "epoch=76\n",
            "current model with loss=2.128554944132195 saved at ./model_wi_rnn.pth\n",
            "epoch=77\n",
            "current model with loss=2.114602078179844 saved at ./model_wi_rnn.pth\n",
            "epoch=78\n",
            "current model with loss=2.11129543722653 saved at ./model_wi_rnn.pth\n",
            "epoch=79\n",
            "Loss [epoch=79] = 2.0968618529741883\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " asoup m. wentern   getin nuriomarotindsousspe  . reaye igintoninoby  thes sutafos tedtwhrtonge den .   .r nthes tonsthrre w lm  mbless.siewe bexref  l mend  tast r . isly mon t hes   mor   rumn m and \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.0968618529741883 saved at ./model_wi_rnn.pth\n",
            "epoch=80\n",
            "current model with loss=2.094268458788512 saved at ./model_wi_rnn.pth\n",
            "epoch=81\n",
            "epoch=82\n",
            "current model with loss=2.079660716604014 saved at ./model_wi_rnn.pth\n",
            "epoch=83\n",
            "current model with loss=2.068406156829146 saved at ./model_wi_rnn.pth\n",
            "epoch=84\n",
            "current model with loss=2.064988371778707 saved at ./model_wi_rnn.pth\n",
            "epoch=85\n",
            "current model with loss=2.055394944597463 saved at ./model_wi_rnn.pth\n",
            "epoch=86\n",
            "current model with loss=2.052206368719945 saved at ./model_wi_rnn.pth\n",
            "epoch=87\n",
            "current model with loss=2.0450528684209606 saved at ./model_wi_rnn.pth\n",
            "epoch=88\n",
            "current model with loss=2.0348775806974193 saved at ./model_wi_rnn.pth\n",
            "epoch=89\n",
            "Loss [epoch=89] = 2.0287839711689557\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "em .i ories liliccathxomaf  t  in diesolerfierecliceetorubulion f  me t rof  ie   fe n wom checliredingloand hond t esorod anemile   lll es lteng th  te  in ouse dull. od t d  ommi  oninodam lyhe   e  \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.0287839711689557 saved at ./model_wi_rnn.pth\n",
            "epoch=90\n",
            "current model with loss=2.012025881009024 saved at ./model_wi_rnn.pth\n",
            "epoch=91\n",
            "current model with loss=2.0041925555369895 saved at ./model_wi_rnn.pth\n",
            "epoch=92\n",
            "current model with loss=1.9916191101074219 saved at ./model_wi_rnn.pth\n",
            "epoch=93\n",
            "current model with loss=1.9771868127291319 saved at ./model_wi_rnn.pth\n",
            "epoch=94\n",
            "epoch=95\n",
            "current model with loss=1.9754237141765532 saved at ./model_wi_rnn.pth\n",
            "epoch=96\n",
            "current model with loss=1.9621130638435238 saved at ./model_wi_rnn.pth\n",
            "epoch=97\n",
            "current model with loss=1.9536192593027333 saved at ./model_wi_rnn.pth\n",
            "epoch=98\n",
            "current model with loss=1.9400435060751242 saved at ./model_wi_rnn.pth\n",
            "epoch=99\n",
            "Loss [epoch=99] = 1.9316932783752192\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "  thentochare ricud . hed rel  tederofond  dodis omift lome    ssindilandstitorued mev d s of ddign henfiercouboomarty s  wilorereayeshom  pe    om  g we  on of    o cod    oind ondoin den wof me  cent\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=1.9316932783752192 saved at ./model_wi_rnn.pth\n"
          ]
        }
      ],
      "source": [
        "rnn_wi_losses = train_rnn(wi_data, wi_data_size, './model_wi_rnn.pth')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iI8hukxjfc0g"
      },
      "source": [
        "### LSTM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "b5KmatWPhUYB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c95924a3-9699-455a-b1ef-57a6646a3f01"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "epoch=0\n",
            "current model with loss=2.765177064254636 saved at ./model_wi_lstm.pth\n",
            "epoch=1\n",
            "current model with loss=2.4869486464828743 saved at ./model_wi_lstm.pth\n",
            "epoch=2\n",
            "current model with loss=2.4286629821433396 saved at ./model_wi_lstm.pth\n",
            "epoch=3\n",
            "current model with loss=2.3936734375406483 saved at ./model_wi_lstm.pth\n",
            "epoch=4\n",
            "current model with loss=2.370305915347865 saved at ./model_wi_lstm.pth\n",
            "epoch=5\n",
            "current model with loss=2.3500055856392033 saved at ./model_wi_lstm.pth\n",
            "epoch=6\n",
            "current model with loss=2.324465608987652 saved at ./model_wi_lstm.pth\n",
            "epoch=7\n",
            "current model with loss=2.2888159341499454 saved at ./model_wi_lstm.pth\n",
            "epoch=8\n",
            "current model with loss=2.2492011433742087 saved at ./model_wi_lstm.pth\n",
            "epoch=9\n",
            "Loss [epoch=9] = 2.1894182748481876\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "d . ck precanoprind d founa ltie od twh finexoonbs meve eefthedatralan is   wakioes  bar fangathe  fcantrenolle groth thenddisond ondeconghome wed . urecu reshofieison th wro fragig edetar allo itutite\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=2.1894182748481876 saved at ./model_wi_lstm.pth\n",
            "epoch=10\n",
            "current model with loss=2.10457169814188 saved at ./model_wi_lstm.pth\n",
            "epoch=11\n",
            "current model with loss=2.008090181428878 saved at ./model_wi_lstm.pth\n",
            "epoch=12\n",
            "current model with loss=1.876010856667503 saved at ./model_wi_lstm.pth\n",
            "epoch=13\n",
            "current model with loss=1.730841052336771 saved at ./model_wi_lstm.pth\n",
            "epoch=14\n",
            "current model with loss=1.5796662334535942 saved at ./model_wi_lstm.pth\n",
            "epoch=15\n",
            "current model with loss=1.4360001634378903 saved at ./model_wi_lstm.pth\n",
            "epoch=16\n",
            "current model with loss=1.3108098213789894 saved at ./model_wi_lstm.pth\n",
            "epoch=17\n",
            "current model with loss=1.1633115923795543 saved at ./model_wi_lstm.pth\n",
            "epoch=18\n",
            "current model with loss=1.019600795429261 saved at ./model_wi_lstm.pth\n",
            "epoch=19\n",
            "Loss [epoch=19] = 0.8613079385679276\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "s  fedstit ckes tsonyond d  d ted tthat  tect wectinonot o it meashe  wonche  wn sthamy ethashericeth ckeralfaveep cth ug . cofinnecotrus sond athtomoct ffof as  ty t byuan .  ashrthahicke bythasoy bel\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.8613079385679276 saved at ./model_wi_lstm.pth\n",
            "epoch=20\n",
            "current model with loss=0.7483405373135551 saved at ./model_wi_lstm.pth\n",
            "epoch=21\n",
            "current model with loss=0.6608106078671627 saved at ./model_wi_lstm.pth\n",
            "epoch=22\n",
            "current model with loss=0.5740681505594097 saved at ./model_wi_lstm.pth\n",
            "epoch=23\n",
            "current model with loss=0.48296512052661084 saved at ./model_wi_lstm.pth\n",
            "epoch=24\n",
            "current model with loss=0.43003263849703993 saved at ./model_wi_lstm.pth\n",
            "epoch=25\n",
            "current model with loss=0.3828401455625159 saved at ./model_wi_lstm.pth\n",
            "epoch=26\n",
            "current model with loss=0.3604932360228945 saved at ./model_wi_lstm.pth\n",
            "epoch=27\n",
            "current model with loss=0.3107540652644439 saved at ./model_wi_lstm.pth\n",
            "epoch=28\n",
            "current model with loss=0.2905741084794529 saved at ./model_wi_lstm.pth\n",
            "epoch=29\n",
            "Loss [epoch=29] = 0.2749326958275232\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "erdd twitonin th  y tr tumbrtorungun d oy n  ced  grembe wofod  woacst fre atongrues cesedenconit  le  thewe ele  geped comariapord che cro aroce sstesecct vectavelothe wese s t tug wey ty e he  ts ing\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.2749326958275232 saved at ./model_wi_lstm.pth\n",
            "epoch=30\n",
            "current model with loss=0.25360334664583206 saved at ./model_wi_lstm.pth\n",
            "epoch=31\n",
            "epoch=32\n",
            "current model with loss=0.25049279672933406 saved at ./model_wi_lstm.pth\n",
            "epoch=33\n",
            "current model with loss=0.23233873403218927 saved at ./model_wi_lstm.pth\n",
            "epoch=34\n",
            "current model with loss=0.2161269503050163 saved at ./model_wi_lstm.pth\n",
            "epoch=35\n",
            "current model with loss=0.21589474990719654 saved at ./model_wi_lstm.pth\n",
            "epoch=36\n",
            "current model with loss=0.21400395188419546 saved at ./model_wi_lstm.pth\n",
            "epoch=37\n",
            "current model with loss=0.2125113666301868 saved at ./model_wi_lstm.pth\n",
            "epoch=38\n",
            "epoch=39\n",
            "Loss [epoch=39] = 0.20721065143092735\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "speve sffith t wn tomo w crth  s formand an    tse l heltid  thathe  we .  . f gratont cke ibutstesthe d erviepitingnat that   tt  lis  tickepitrind aromak bel     edad m thengr  anjan ly  tly wathat s\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.20721065143092735 saved at ./model_wi_lstm.pth\n",
            "epoch=40\n",
            "current model with loss=0.19773120378128817 saved at ./model_wi_lstm.pth\n",
            "epoch=41\n",
            "epoch=42\n",
            "current model with loss=0.19547242494147332 saved at ./model_wi_lstm.pth\n",
            "epoch=43\n",
            "epoch=44\n",
            "current model with loss=0.19788507772151565 saved at ./model_wi_lstm.pth\n",
            "epoch=45\n",
            "epoch=46\n",
            "current model with loss=0.2152176832200074 saved at ./model_wi_lstm.pth\n",
            "epoch=47\n",
            "current model with loss=0.21051399073884136 saved at ./model_wi_lstm.pth\n",
            "epoch=48\n",
            "current model with loss=0.20044963580907368 saved at ./model_wi_lstm.pth\n",
            "epoch=49\n",
            "Loss [epoch=49] = 0.19523943569816526\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "twirle t afofin . titeve borora hal  hrag t  heiched radethact  tr ion thas t as thathistrmerillonuns sumomaninilunadroclathed ch t fes by wnit tratherenthan     shry te   worthedsf r f  pr . th  g ato\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.19523943569816526 saved at ./model_wi_lstm.pth\n",
            "epoch=50\n",
            "epoch=51\n",
            "epoch=52\n",
            "epoch=53\n",
            "current model with loss=0.19169826842233784 saved at ./model_wi_lstm.pth\n",
            "epoch=54\n",
            "current model with loss=0.190155492881771 saved at ./model_wi_lstm.pth\n",
            "epoch=55\n",
            "current model with loss=0.18933159473245262 saved at ./model_wi_lstm.pth\n",
            "epoch=56\n",
            "epoch=57\n",
            "epoch=58\n",
            "current model with loss=0.19043876102469007 saved at ./model_wi_lstm.pth\n",
            "epoch=59\n",
            "Loss [epoch=59] = 0.1935167076890586\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " ang n d   mign twh memace  .  wato itond athe  we hede wn bes tisiggumeen t mis pet    mpo . . lt th al  tenth tuinth ad s     fatethe  wedisils  gomallfind t akis omanin ar wapreed aver thend . theno\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "epoch=60\n",
            "current model with loss=0.1832600284795292 saved at ./model_wi_lstm.pth\n",
            "epoch=61\n",
            "epoch=62\n",
            "current model with loss=0.18299303264891514 saved at ./model_wi_lstm.pth\n",
            "epoch=63\n",
            "current model with loss=0.17810538069146578 saved at ./model_wi_lstm.pth\n",
            "epoch=64\n",
            "epoch=65\n",
            "current model with loss=0.18322324557382552 saved at ./model_wi_lstm.pth\n",
            "epoch=66\n",
            "epoch=67\n",
            "epoch=68\n",
            "epoch=69\n",
            "Loss [epoch=69] = 0.20131313312249105\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "al  ini ithef ine al  lunitiere  wofinmiche athatuig bed m   re rof aredeag uthe  ndararick fixeaco wind jar   alorexte  iede mawhale ay avingname  ngava  are ay erabus awatalinjed abconin ted  thamint\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.20131313312249105 saved at ./model_wi_lstm.pth\n",
            "epoch=70\n",
            "epoch=71\n",
            "current model with loss=0.2032407499605515 saved at ./model_wi_lstm.pth\n",
            "epoch=72\n",
            "epoch=73\n",
            "current model with loss=0.19797743325594996 saved at ./model_wi_lstm.pth\n",
            "epoch=74\n",
            "epoch=75\n",
            "current model with loss=0.19868018389603154 saved at ./model_wi_lstm.pth\n",
            "epoch=76\n",
            "epoch=77\n",
            "epoch=78\n",
            "current model with loss=0.1876064661096354 saved at ./model_wi_lstm.pth\n",
            "epoch=79\n",
            "Loss [epoch=79] = 0.18281645016347775\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "rthe  wnier ass .  touaviensthast agutor ratind as heamet d . condis sioreas  thelen tithe  wes by wintn t th cal c .  th artis be sue ped testhame tmanin  tthastoro   orbe bororareareroorthache cont b\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.18281645016347775 saved at ./model_wi_lstm.pth\n",
            "epoch=80\n",
            "current model with loss=0.1806296102214055 saved at ./model_wi_lstm.pth\n",
            "epoch=81\n",
            "epoch=82\n",
            "epoch=83\n",
            "epoch=84\n",
            "epoch=85\n",
            "epoch=86\n",
            "current model with loss=0.19372972186471596 saved at ./model_wi_lstm.pth\n",
            "epoch=87\n",
            "current model with loss=0.1926289153453268 saved at ./model_wi_lstm.pth\n",
            "epoch=88\n",
            "epoch=89\n",
            "Loss [epoch=89] = 0.18386403868188622\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "n s ton  ithe  ng funjan e  wahed m bonted m tha  oreveas  thathame mevin ed luaie tofindinted  fo   al al cothe  bue  le  he th al .  fore a o i i m esthay e    wimer ades gt  liguthe   . s    a heled\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.18386403868188622 saved at ./model_wi_lstm.pth\n",
            "epoch=90\n",
            "epoch=91\n",
            "epoch=92\n",
            "epoch=93\n",
            "epoch=94\n",
            "current model with loss=0.19175492007224287 saved at ./model_wi_lstm.pth\n",
            "epoch=95\n",
            "current model with loss=0.1888907629935468 saved at ./model_wi_lstm.pth\n",
            "epoch=96\n",
            "epoch=97\n",
            "current model with loss=0.18610823013987698 saved at ./model_wi_lstm.pth\n",
            "epoch=98\n",
            "epoch=99\n",
            "Loss [epoch=99] = 0.19177386158558188\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "o ed acofin d umay .  thoin.  . th al . f brie onchast t fininofonis  he th arthe sthas ted o he  ysa   ed tere ed  ulercoritus than      t sitixodiest so a a  med our ched o gan ed . . th trag twk .  \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "current model with loss=0.19177386158558188 saved at ./model_wi_lstm.pth\n"
          ]
        }
      ],
      "source": [
        "lstm_wi_losses = train_lstm(wi_data, wi_data_size, './model_wi_lstm.pth')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4EDmKGTOTaQ"
      },
      "source": [
        "## Finetuning by Shakespeare"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J99viDjUqxUz"
      },
      "source": [
        "- Define the following functions to use the previous model as a pre-trained model and fine-tunes it using Shakespeare's plays dataset with lower learning rate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "FKa2cGcVjRje"
      },
      "outputs": [],
      "source": [
        "def finetune_rnn(data, data_size, model_save_file, model_pretrained_path):\n",
        "    # RNN parameters\n",
        "    hidden_size = 512\n",
        "    num_layers = 6\n",
        "    lr = 0.001\n",
        "    epoch_num = 100\n",
        "    losses = []\n",
        "\n",
        "    ####################################\n",
        "    ######### Your code begins #########\n",
        "    ####################################\n",
        "    # In this section finetune the model that trained by wikipedia dataset with Shakespeare plays dataset\n",
        "    model = RNN(input_size=vocab_size, output_size=vocab_size, hidden_size=hidden_size, num_layers=num_layers, dropout_enable=False)\n",
        "    model = model.to(device)\n",
        "\n",
        "    # Load the pre-trained weights\n",
        "    model.load_model(model_pretrained_path)\n",
        "\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
        "\n",
        "    for epoch in range(epoch_num):\n",
        "        loss = train_epoch(model, data, data_size, epoch, optimizer)\n",
        "        losses.append(loss)\n",
        "\n",
        "    # Save the fine-tuned model\n",
        "    model.save_model(model_save_file)\n",
        "    ####################################\n",
        "    ######### Your code ends ###########\n",
        "    ####################################\n",
        "\n",
        "    return losses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "ijG0lP75lHfm"
      },
      "outputs": [],
      "source": [
        "def finetune_lstm(data, data_size, model_save_file, model_pretrained_path):\n",
        "    # LSTM parameters\n",
        "    hidden_size = 512\n",
        "    num_layers = 3\n",
        "    lr = 0.001\n",
        "    epoch_num = 100\n",
        "    losses = []\n",
        "\n",
        "    ####################################\n",
        "    ######### Your code begins #########\n",
        "    ####################################\n",
        "    # In this section finetune the model that trained by wikipedia dataset with Shakespeare plays dataset\n",
        "    # Create LSTM model\n",
        "    model = LSTM(input_size=vocab_size, output_size=vocab_size, hidden_size=hidden_size, num_layers=num_layers, dropout_enable=False)\n",
        "    model = model.to(device)\n",
        "\n",
        "    # Load the pre-trained weights\n",
        "    model.load_model(model_pretrained_path)\n",
        "\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
        "\n",
        "    for epoch in range(epoch_num):\n",
        "        loss = train_epoch(model, data, data_size, epoch, optimizer)\n",
        "        losses.append(loss)\n",
        "\n",
        "    # Save the fine-tuned model\n",
        "    model.save_model(model_save_file)\n",
        "    ####################################\n",
        "    ######### Your code ends ###########\n",
        "    ####################################\n",
        "\n",
        "    return losses"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wwqLT0A0mSuo"
      },
      "source": [
        "### RNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "DixZCVqZmV8I",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "776e1baa-4826-4f40-a908-d03dcb5b418e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loss [epoch=9] = 2.1707068847285376\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "e chigocopeu avoworo thowe e iss mmid d se th urs y otnupageeere bemcien wawiun athaudeanonomed\n",
            " coverenessth fof te ped ce rt le amee asenir ad. she berchillout irnthesthatithe anor mse broranthearhen\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=19] = 1.804780489868588\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "noreiuthe d mmen wina d\n",
            "m\n",
            "br\n",
            "s ans f amaniotinirme: coe d inrct s by gh f \n",
            "tis\n",
            "fachareyess ak ppe rand\n",
            "prons nd cia me iuranik \n",
            "ld tomalld o ethobyothens whalyende e\n",
            "ad my gomeco chthee hind ar s\n",
            "ionin\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=29] = 1.5068895932700899\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "llidethatiul te gimale at athationof\n",
            "pry te thernse g\n",
            "imathal oeano me pf ck se t le a: ces b:\n",
            "red\n",
            "pe nedgouel lg ct y thay t towerne amatoug cacene mad bbrert ga bemoo codonousuthetheref\n",
            "llirsour t th\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=39] = 1.2284692699710529\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "eanthcolapge. whe f therare ye co p i anoue imatonist tiu and. atanisay the then t orino:\n",
            "setoutousuawhe ione ours owefthe he t largiuf bothan pescid thie\n",
            "veeressoue peschiusody mus i f ak th s cas tha\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=49] = 1.0040710353189044\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "are itr gerd adom md peathe r comblfenche f ise ff g ised at\n",
            "a:d\n",
            "y lace tealanendsthe thy g brtheeavoulthe henen be ine ofr:\n",
            "one tustysigeres t:\n",
            "ak the the  m othince imed\n",
            "pinthiynde bld ai ghe t\n",
            "the t\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=59] = 0.7370631098747253\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " mevs\n",
            "turd s:\n",
            "fomeman fven thayouruthe hrestus aithy hen ran r y ce\n",
            "bevan he our sers\n",
            "t \n",
            "f m t the iov\n",
            "len:ouene athildebuangir thatiubhen n s:imatheplt tiueuenv in t see te in ouss pr\n",
            "\n",
            "mane trsthe the\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=69] = 0.5880693602893088\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "of the the sthea me thindimorgone tsous t e himanillinthend\n",
            "aby of ticherouara:ore ercitorare\n",
            "mandaneche\n",
            "finod ths s\n",
            "iss cohe ionimou nd gn che the tistoroun\n",
            "t st s tha me de t:\n",
            "ffot a arcik the our s:\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=79] = 0.46238038688898087\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "lashing r\n",
            "ire\n",
            "\n",
            "fomed\n",
            "\n",
            "atohecircid\n",
            "thy\n",
            "\n",
            "cangi f a:e s:\n",
            "\n",
            "r wiup mrione t d s:\n",
            "puli ay adiarius t thest t\n",
            "\n",
            "dercodorecis hememat me thio\n",
            "cr\n",
            "athed winswnt ansty\n",
            "ad w\n",
            "man peate yoeth tusout t thay w\n",
            "d tistom\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=89] = 0.37353859862519634\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "ganaly\n",
            "he ce\n",
            "we chathertyonengy t cthe reronourerd\n",
            "fus anof wust\n",
            "achakhit f thayolsthouls fa ar s p tilye arnewe t fo imatha  ma iu anomnsherar t werding where ous se fimathepld: co chetiuso mimatha ar\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=99] = 0.3340444134341346\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " pre ffof cers\n",
            "mema iupecogan ch t fote hed\n",
            "cotho tss ted th arengome tory memcthy d t anoulucothin thioared m\n",
            "t whe the the the imathe stiusoven tercir t anustunn wierchiule ffo mafilolo iu\n",
            "mins man p\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n"
          ]
        }
      ],
      "source": [
        "rnn_sh_finetune_losses = finetune_rnn(sh_data, sh_data_size, './model_sh_finetune_rnn.pth', './model_wi_rnn.pth')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "By2oc0qlmUeq"
      },
      "source": [
        "### LSTM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "m5tlC-O6nJdv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fc746ec4-d75b-4bde-91ce-fefc8428840f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loss [epoch=9] = 1.0464529560671911\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "sul\n",
            "miuve acabbuma murumato ouls\n",
            "\n",
            "we hil whilllllllavisoullr. f brr siuse erisonds thicethicarin h go uniun\n",
            " malth: sus:\n",
            "\n",
            "us i he figiuifren athe lele the veve m the tor m yor yofi wholed m buryey keth\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=19] = 0.28446573784781826\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "iol bollfi h.\n",
            "ld witi o iere the me ucoos owern m\n",
            ":\n",
            ".\n",
            ":\n",
            "dindear waf aue\n",
            "n lfomst f mommatoronenoselased lden we te te:\n",
            "elod wif arit.\n",
            "ar th latheson\n",
            "f ao p.\n",
            "fie\n",
            "swhno hen athey winviff y wn d\n",
            "swhe meva\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=29] = 0.12201122670537895\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "t petothinenl m iume cie m n be d\n",
            "y tinghinvimer be s oomy d re oto y bliethouse.\n",
            "nconis yo omomas: abr fiad shen athe otive fo whir mns t th t th ed bthir m\n",
            "athes t wauranis outit de fir pborestor cas\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=39] = 0.08677754401125842\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            ". wherin.\n",
            "this:\n",
            "\n",
            "goratro ters t wavake\n",
            "rvo thendinoulase be m the torse h pplt mbenof lis yole mey hinof d oner hingil s ayof ms sple al s f m rn thes t th hes\n",
            "oryou mavetamer anksutte banoorthes ithou\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=49] = 0.08809941484489375\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " murese. ato machar ansevebe bon\n",
            " an out anibe theere owh llimollfim lfimy.\n",
            "thareyone yorcie outcotik herig t be tha:\n",
            "\n",
            "gea . hadulvearenacie hinces lsinye aucoulinouto f th fousey t theinchizercie hin:\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=59] = 0.08663441799581051\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            " th t tire ar he fiat he fik\n",
            "we: hor yoled tha histyofanasheianol ily outiur s:y s s: m nd tha hountinin\n",
            "\n",
            "m anofi mene tourer\n",
            "\n",
            "at d wse finds.\n",
            "dilds t tis reld molest inoulie\n",
            "\n",
            "wone hinoune \n",
            "\n",
            "winthed ur\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=69] = 0.06823586008977145\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "t t t\n",
            "th thund ioraciscothe pe thas: tomor he fia ipsthen t s ots all or\n",
            "\n",
            "\n",
            "\n",
            "t\n",
            "\n",
            "fin\n",
            "e hendir hi s os nt blber nd:s he t:\n",
            " m t s.\n",
            "te thef thed w w fuss ty iee s ou mendils t wnis blyonirithenl wan. buli \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=79] = 0.07920581467139225\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "e t heatouren he poour\n",
            "\n",
            "soule mendilarelllior\n",
            "\n",
            "wn tilsh f bunus t i\n",
            "ore cky boule onermia is siferthal yorithus t thor pre hiner vey ser haeerayope inwindeves\n",
            "fro theat t th thes:\n",
            "y d t s sthe plin: ld\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=89] = 0.06766472465824336\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "limou wheneri they chize hinsthomare a:oow anot wispeizs: f i rs si\n",
            "\n",
            "\n",
            "n outius nd wef pier:\n",
            "l one be t d rell gofe s althery\n",
            "n t theag ver ie\n",
            "y wal honld wilish onen ounid ots ds:e t ats.\n",
            "we terig hino\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=99] = 0.05855425226036459\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "ndiuse h one unit st\n",
            "winof\n",
            "iueyoulens ch therd thinghed ble m nd tha therd ind uene\n",
            "he the our he fomathee hee loneve r\n",
            "mir ckis byotolest lsourinthd ul pr shefrbe mine alenans rt iu\n",
            "ithoy be the tor h\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n"
          ]
        }
      ],
      "source": [
        "lstm_sh_finetune_losses = finetune_lstm(sh_data, sh_data_size, './model_sh_finetune_lstm.pth', './model_wi_lstm.pth')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPr9aM-COlGM"
      },
      "source": [
        "## Plotting Losses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "seWKCPaCnQ2N"
      },
      "outputs": [],
      "source": [
        "def plot_losses_together(losses1, losses2):\n",
        "    xpoints = np.array(range(len(losses1)))\n",
        "    ypoints1 = np.array(losses1)\n",
        "    ypoints2 = np.array(losses2)\n",
        "\n",
        "    plt.plot(xpoints, ypoints1, color='blue',label='base_losses' )\n",
        "    plt.plot(xpoints, ypoints2, color='red',label='finetune_losses' )\n",
        "    plt.xlabel(\"epoch\")\n",
        "    plt.ylabel(\"loss\")\n",
        "    plt.legend()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pl3ao3hXnne2"
      },
      "source": [
        "### RNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "oCiLrTAahpTr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 452
        },
        "outputId": "a9382be3-f4e2-4eec-b65a-c25ceb38800f"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGzCAYAAADT4Tb9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABkkklEQVR4nO3deXiMZxcG8Huyh2yCLEgIIWLfCVpUiJ1qUbSWWqr2aqulpVQrWtVFq+gm2lq+omjtscQau9iXIIRKRJHNEpF5vj9Ok0iRJszknczcv+t6L5mZd2ZO3m++zsmznKNTSikQERERmQkrrQMgIiIiMiQmN0RERGRWmNwQERGRWWFyQ0RERGaFyQ0RERGZFSY3REREZFaY3BAREZFZYXJDREREZoXJDREREZkVJjdERERkVmy0fPPZs2dj9uzZuHDhAgCgatWqmDhxItq2bfvI88PCwtC/f/8c99nb2+Pu3bt5fk+9Xo8rV67A2dkZOp3uiWMnIiKigqOUQkpKCkqVKgUrq9zHZjRNbsqUKYNp06ahYsWKUEph/vz56Ny5Mw4dOoSqVas+8jkuLi44ffp01u38JihXrlyBj4/PU8VNRERE2rh06RLKlCmT6zmaJjcdO3bMcfvjjz/G7NmzsXv37scmNzqdDl5eXk/8ns7OzgDk4ri4uDzx6xAREVHBSU5Oho+PT9b3eG40TW4elJGRgSVLluDWrVsICgp67HmpqakoW7Ys9Ho96tSpg6lTpz42EQKAtLQ0pKWlZd1OSUkBICNATG6IiIgKl7zM2Gi+oPjo0aNwcnKCvb09hgwZguXLl6NKlSqPPDcgIAA//fQTVq5ciV9//RV6vR6NGzfG5cuXH/v6oaGhcHV1zTo4JUVERGTedEoppWUA9+7dQ2xsLJKSkrB06VL88MMP2Lp162MTnAelp6cjMDAQPXv2xJQpUx55zr9HbjKHtZKSkjhyQ0REVEgkJyfD1dU1T9/fmk9L2dnZwd/fHwBQt25d7Nu3D1999RXmzp37n8+1tbVF7dq1cfbs2ceeY29vD3t7e4PFS0RERKZN8+Tm3/R6fY6RltxkZGTg6NGjaNeunZGjIiKybBkZGUhPT9c6DDJzdnZ2/7nNOy80TW7GjRuHtm3bwtfXFykpKVi4cCEiIiKwfv16AECfPn1QunRphIaGAgA+/PBDNGrUCP7+/khMTMT06dNx8eJFDBw4UMtfg4jIbCmlEB8fj8TERK1DIQtgZWUFPz8/2NnZPdXraJrcJCQkoE+fPoiLi4Orqytq1KiB9evXo1WrVgCA2NjYHBnczZs3MWjQIMTHx6NYsWKoW7cudu3alaf1OURElH+ZiY2HhweKFCnC4qdkNJlFduPi4uDr6/tUnzXNFxQXtPwsSCIismQZGRk4c+YMPDw8ULx4ca3DIQuQlJSEK1euwN/fH7a2tjkey8/3t+ZbwYmIyDRlrrEpUqSIxpGQpcicjsrIyHiq12FyQ0REueJUFBUUQ33WmNwQERGRWWFyQ0RERGaFyQ0REZmd5s2bY/To0VqH8Uj9+vVDly5dtA7DrDG5MZSMDCAuDjh3TutIiIiILBqTG0PZsgUoVQro3FnrSIiIiCwakxtD8fSUf69e1TYOIiIjUgq4dUubI79V2e7fv4/hw4fD1dUVJUqUwIQJE5BZ2u2XX35BvXr14OzsDC8vL/Tq1QsJCQlZz7158yZ69+6NkiVLwtHRERUrVsS8efOyHr906RK6d+8ONzc3uLu7o3Pnzrhw4cITXdO0tDSMHDkSHh4ecHBwQNOmTbFv3748xXLv3j0MHz4c3t7ecHBwQNmyZbOq+gNAYmIiBg4ciJIlS8LFxQXPPfccDh8+nPX44cOH0aJFCzg7O8PFxQV169bF/v37n+j3MCUm11uq0MpMbq5fB+7fB2x4aYnI/Ny+DTg5afPeqalA0aJ5P3/+/PkYMGAA9u7di/3792Pw4MHw9fXFoEGDkJ6ejilTpiAgIAAJCQkYM2YM+vXrhzVr1gAAJkyYgBMnTmDt2rUoUaIEzp49izt37gCQ+j8hISEICgrC9u3bYWNjg48++ght2rTBkSNH8t06YOzYsVi2bBnmz5+PsmXL4tNPP0VISAjOnj0Ld3f3XGOZOXMm/vjjD/z222/w9fXFpUuXcOnSpazX7tatGxwdHbF27Vq4urpi7ty5aNmyJc6cOQN3d3f07t0btWvXxuzZs2FtbY2oqKiHiucVSsrCJCUlKQAqKSnJsC98/75SVlZKAUpduWLY1yYi0sCdO3fUiRMn1J07d7LuS02V/8xpcaSm5j32Zs2aqcDAQKXX67Pue+edd1RgYOAjz9+3b58CoFJSUpRSSnXs2FH179//kef+8ssvKiAgIMdrp6WlKUdHR7V+/fr/jK1v376qc+fOSimlUlNTla2trVqwYEHW4/fu3VOlSpVSn3766X/GMmLECPXcc8/liCXT9u3blYuLi7p7926O+ytUqKDmzp2rlFLK2dlZhYWF/WfMBeVRn7lM+fn+5rSUoVhbAyVLys+cmiIiM1WkiIygaHHkt1Byo0aNchSFCwoKQnR0NDIyMnDgwAF07NgRvr6+cHZ2RrNmzQBIT0MAeP3117F48WLUqlULY8eOxa5du7Je5/Dhwzh79iycnZ3h5OQEJycnuLu74+7duziXz00l586dQ3p6Opo0aZJ1n62tLRo0aICTJ0/+Zyz9+vVDVFQUAgICMHLkSGzYsCFHnKmpqShevHhWnE5OToiJicmKc8yYMRg4cCCCg4Mxbdq0fMdvqjh3YkienpLYMLkhIjOl0+VvasgU3b17FyEhIQgJCcGCBQtQsmRJxMbGIiQkBPfu3QMAtG3bFhcvXsSaNWsQHh6Oli1bYtiwYfjss8+QmpqKunXrYsGCBQ+9dsnMP3INKLdY6tSpg5iYGKxduxYbN25E9+7dERwcjKVLlyI1NRXe3t6IiIh46DXd3NwAAJMmTUKvXr2wevVqrF27Fh988AEWL16M559/3uC/R4EyxrCSKTPatJRSSrVqJWOnJjTER0T0pHKbIjB1zZo1U1WqVMlx37vvvqsCAwPV/v37FQAVGxub9dgvv/yiAKhDhw498vXmzJmjnJ2dlVJKfffdd6pYsWJP/D3y72kpOzu7h6alSpcuraZPn/6fsfzbunXrFAB1/fp1tWHDBmVtba1iYmLyHNtLL72kOnbsmOfzDY3TUqaIO6aIiExGbGwsxowZg9OnT2PRokX4+uuvMWrUKPj6+sLOzg5ff/01zp8/jz/++ANTpkzJ8dyJEydi5cqVOHv2LI4fP45Vq1YhMDAQANC7d2+UKFECnTt3xvbt2xETE4OIiAiMHDkSly9fzleMRYsWxeuvv463334b69atw4kTJzBo0CDcvn0bAwYM+M9YPv/8cyxatAinTp3CmTNnsGTJEnh5ecHNzQ3BwcEICgpCly5dsGHDBly4cAG7du3Ce++9h/379+POnTsYPnw4IiIicPHiRezcuRP79u3Leu3CjNNShsTkhojIZPTp0wd37txBgwYNYG1tjVGjRmHw4MHQ6XQICwvD+PHjMXPmTNSpUwefffYZOnXqlPVcOzs7jBs3DhcuXICjoyOeeeYZLF68GIB0Sd+2bRveeecddO3aFSkpKShdujRatmwJFxeXfMc5bdo06PV6vPLKK0hJSUG9evWwfv16FCtW7D9jcXZ2xqefforo6GhYW1ujfv36WLNmDaysZOxizZo1eO+999C/f39cu3YNXl5eePbZZ+Hp6Qlra2tcv34dffr0wdWrV1GiRAl07doVkydPftpLrzmdUvmtHFC4JScnw9XVFUlJSU/0IczV9OnA2LFA797Ar78a9rWJiArY3bt3ERMTAz8/Pzg4OGgdDlmA3D5z+fn+5rSUIXHkhoiISHNMbgyJyQ0RkcV7cNv1v4/t27drHZ5F4JobQ2JyQ0Rk8aKioh77WOnSpQsuEAvG5MaQMpObv/+WLuHW1trGQ0REBc7f31/rECwep6UMqWRJqXCl10uCQ0RERAWOyY0h2dgAJUrIz5yaIiIi0gSTG0PjuhsiIiJNMbkxNCY3REREmmJyY2hMboiIiDTF5MbQmNwQEWlOKYXBgwfD3d0dOp0Obm5uGD16tNZhFYgLFy5Ap9PluiXd3HEruKExuSEi0ty6desQFhaGiIgIlC9fHlZWVnB0dDToe/Tr1w+JiYlYsWKFQV+Xnh6TG0NjckNEpLlz587B29sbjRs31joU0gCnpQyNyQ0RmTOlgFu3tDny2Oe5X79+GDFiBGJjY6HT6VCuXDk0b948x7RUuXLlMHXqVLz66qtwdnaGr68vvvvuuxyvc+nSJXTv3h1ubm5wd3dH586dceHCBQDApEmTMH/+fKxcuRI6nQ46nQ4RERGIiIiATqdDYmJi1utERUVBp9NlPTcsLAxubm5Yv349AgMD4eTkhDZt2iAuLi7H+//www8IDAyEg4MDKleujG+//Tbf/3Nl2rp1Kxo0aAB7e3t4e3vj3Xffxf3797MeX7p0KapXrw5HR0cUL14cwcHBuHXrFgAgIiICDRo0QNGiReHm5oYmTZrg4sWLWc9duXIl6tSpAwcHB5QvXx6TJ0/Oem2lFCZNmgRfX1/Y29ujVKlSGDly5BP/HnmmLExSUpICoJKSkozzBgcOKAUo5e1tnNcnIiogd+7cUSdOnFB37tzJvjM1Vf4bp8WRmpqnuBMTE9WHH36oypQpo+Li4lRCQoJq1qyZGjVqVNY5ZcuWVe7u7mrWrFkqOjpahYaGKisrK3Xq1CmllFL37t1TgYGB6tVXX1VHjhxRJ06cUL169VIBAQEqLS1NpaSkqO7du6s2bdqouLg4FRcXp9LS0tSWLVsUAHXz5s2s9zp06JACoGJiYpRSSs2bN0/Z2tqq4OBgtW/fPnXgwAEVGBioevXqlfWcX3/9VXl7e6tly5ap8+fPq2XLlil3d3cVFhb2n79/TEyMAqAOHTqklFLq8uXLqkiRImro0KHq5MmTavny5apEiRLqgw8+UEopdeXKFWVjY6M+//xzFRMTo44cOaJmzZqlUlJSVHp6unJ1dVVvvfWWOnv2rDpx4oQKCwtTFy9eVEoptW3bNuXi4qLCwsLUuXPn1IYNG1S5cuXUpEmTlFJKLVmyRLm4uKg1a9aoixcvqj179qjvvvvusbE/8jP3j/x8fzO5MbTLl+X/hNbWSmVkGOc9iIgKQGFNbpRS6osvvlBly5bNuv2o5Obll1/Ouq3X65WHh4eaPXu2UkqpX375RQUEBCi9Xp91TlpamnJ0dFTr169XSinVt29f1blz5xzvm9fkBoA6e/Zs1jmzZs1Snp6eWbcrVKigFi5cmOO1p0yZooKCgv7zd/93cjN+/PiHfpdZs2YpJycnlZGRoQ4cOKAAqAsXLjz0WtevX1cAVERExCPfq2XLlmrq1Kk57vvll1+U9z9/4M+YMUNVqlRJ3bt37z/jVspwyQ3X3Biah4f8m5EB3LiRXbGYiMgcFCkCpKZq994GVKNGjayfdTodvLy8kJCQAAA4fPgwzp49C2dn5xzPuXv3Ls6dO/fU712kSBFUqFAh67a3t3fWe9+6dQvnzp3DgAEDMGjQoKxz7t+/D1dX13y/18mTJxEUFASdTpd1X5MmTZCamorLly+jZs2aaNmyJapXr46QkBC0bt0aL774IooVKwZ3d3f069cPISEhaNWqFYKDg9G9e3d4e3sDkOu0c+dOfPzxx1mvnZGRgbt37+L27dvo1q0bvvzyS5QvXx5t2rRBu3bt0LFjR9jYGDf9YHJjaLa2gLu7JDZXrzK5ISLzotMBRYtqHYVB2Nra5rit0+mg1+sBAKmpqahbty4WLFjw0PNKliz52Ne0spKlrOqB9UHp6el5eu/M56T+kzx+//33aNiwYY7zrI3QkNna2hrh4eHYtWsXNmzYgK+//hrvvfce9uzZAz8/P8ybNw8jR47EunXr8L///Q/vv/8+wsPD0ahRI6SmpmLy5Mno2rXrQ6/r4OAAHx8fnD59Ghs3bkR4eDiGDh2K6dOnY+vWrQ9dA0PigmJj4KJiIqJCrU6dOoiOjoaHhwf8/f1zHJmjJ3Z2dsjIyMjxvMzE58HFwfmtN+Pp6YlSpUrh/PnzD723n59fvn+XwMBAREZG5ki4du7cCWdnZ5QpUwaAJFdNmjTB5MmTcejQIdjZ2WH58uVZ59euXRvjxo3Drl27UK1aNSxcuBCAXKfTp08/FKe/v39Woufo6IiOHTti5syZiIiIQGRkJI4ePZrv3yM/mNwYA5MbIqJCrXfv3ihRogQ6d+6M7du3IyYmBhERERg5ciQuX74MQHZcHTlyBKdPn8bff/+N9PR0+Pv7w8fHB5MmTUJ0dDRWr16NGTNm5Pv9J0+ejNDQUMycORNnzpzB0aNHMW/ePHz++ef5fq2hQ4fi0qVLGDFiBE6dOoWVK1figw8+wJgxY2BlZYU9e/Zg6tSp2L9/P2JjY/H777/j2rVrCAwMRExMDMaNG4fIyEhcvHgRGzZsQHR0NAIDAwEAEydOxM8//4zJkyfj+PHjOHnyJBYvXoz3338fgOwM+/HHH3Hs2DGcP38ev/76KxwdHVG2bNl8/x75weTGGJjcEBEVakWKFMG2bdvg6+uLrl27IjAwEAMGDMDdu3fh4uICABg0aBACAgJQr149lCxZEjt37oStrS0WLVqEU6dOoUaNGvjkk0/w0Ucf5fv9Bw4ciB9++AHz5s1D9erV0axZM4SFhT3RyE3p0qWxZs0a7N27FzVr1sSQIUMwYMCArATExcUF27ZtQ7t27VCpUiW8//77mDFjBtq2bYsiRYrg1KlTeOGFF1CpUiUMHjwYw4YNw2uvvQYACAkJwapVq7BhwwbUr18fjRo1whdffJGVvLi5ueH7779HkyZNUKNGDWzcuBF//vknihcvnu/fIz906sFxKguQnJwMV1dXJCUlZX1ADW7UKGDmTOCdd4Bp04zzHkRERnb37l3ExMTAz88PDg4OWodDFiC3z1x+vr85cmMMHLkhIiLSDJMbY2ByQ0RERjJ16lQ4OTk98mjbtq3W4ZkEbgU3BiY3RERkJEOGDEH37t0f+Zihm4MWVkxujIHJDRERGYm7uzvc3d21DsOkcVrKGDKTm4SEPDd6IyIyVZmF7YiMzVB7nDhyYwyZLRjS04GbN6ViMRFRIWNnZwcrKytcuXIFJUuWhJ2dXY4S/kSGpJTCtWvXoNPpnrp6MZMbY3BwAFxdgaQkmZpickNEhZCVlRX8/PwQFxeHK1euaB0OWQCdTocyZco8dZsJTZOb2bNnY/bs2bhw4QIAoGrVqpg4cWKuq72XLFmCCRMm4MKFC6hYsSI++eQTtGvXroAizgdPz+zk5p9KjkREhY2dnR18fX1x//79h1oNEBmara2tQfpnaZrclClTBtOmTUPFihWhlML8+fPRuXNnHDp0CFWrVn3o/F27dqFnz54IDQ1Fhw4dsHDhQnTp0gUHDx5EtWrVNPgNcuHpCZw5w0XFRFToZU4TGLPRIZEhmVyFYnd3d0yfPh0DBgx46LEePXrg1q1bWLVqVdZ9jRo1Qq1atTBnzpw8vX6BVCgGgG7dgKVLga++AkaONN77EBERWYBCWaE4IyMDixcvxq1btxAUFPTIcyIjIxEcHJzjvpCQEERGRj72ddPS0pCcnJzjKBDcDk5ERKQJzZObo0ePwsnJCfb29hgyZAiWL1+OKlWqPPLc+Ph4eGYmDf/w9PREfHz8Y18/NDQUrq6uWYePj49B438sJjdERESa0Dy5CQgIQFRUFPbs2YPXX38dffv2xYkTJwz2+uPGjUNSUlLWcenSJYO9dq6Y3BAREWlC863gdnZ28Pf3BwDUrVsX+/btw1dffYW5c+c+dK6Xlxeu/itZuHr1Kry8vB77+vb29rC3tzds0HnB5IaIiEgTmo/c/Jter0daWtojHwsKCsKmTZty3BceHv7YNTqaYnJDRESkCU1HbsaNG4e2bdvC19cXKSkpWLhwISIiIrB+/XoAQJ8+fVC6dGmEhoYCAEaNGoVmzZphxowZaN++PRYvXoz9+/fju+++0/LXeLTM0aSrV6UFA6t6EhERFQhNk5uEhAT06dMHcXFxcHV1RY0aNbB+/Xq0atUKABAbGwsrq+zBpcaNG2PhwoV4//33MX78eFSsWBErVqwwvRo3QPbITVoakJwsFYuJiIjI6Eyuzo2xFVidGwBwcQFSUoDTp4FKlYz7XkRERGasUNa5MUtcd0NERFTgmNwYE5MbIiKiAsfkxpi8veXfgqqtQ0RERExujKpGDfn3wAFt4yAiIrIgTG6MqUED+XfvXm3jICIisiBMboypfn35NzoauHFD21iIiIgsBJMbY3J3B/5pLYH9+7WNhYiIyEIwuTE2Tk0REREVKCY3xsbkhoiIqEAxuTGw+/f/dceDyY1lFYMmIiLSBJMbA9m2DaheHWjX7l8P1KoF2NhIIT/WuyEiIjI6JjcG4u4OHDsG7NwJpKc/8ICjY3a9G05NERERGR2TGwOpUgUoVgy4fRuIivrXg5lTU/v2FXRYREREFofJjYFYWQFNmsjP27f/60EuKiYiIiowTG4MqGlT+XfHjn89kFnMb/9+ICOjQGMiIiKyNExuDOiZZ+TfHTv+tTEqMBAoWhRITQVOndIkNiIiIkvB5MaA6tYF7O2Ba9eAM2ceeMDaGqhXT37m1BQREZFRMbkxIHt7oGFD+ZnrboiIiLTB5MbAHrvuhskNERFRgWByY2CZ624eO3Jz5Ahw506BxkRERGRJmNwYWFAQoNMB588DV6488ICPD+DpKf0ZHiqEQ0RERIbC5MbAXF2BmjXl5xxTUzodp6aIiIgKAJMbI+C6GyIiIu0wuTGC/1x3s2dPgcZDRERkSZjcGEHmyM2RI0BS0gMPNGggfRrOnQMuXtQkNiIiInPH5MYISpUCypcH9HogMvKBB9zcshtQrVqlRWhERERmj8mNkTx23U3HjvLvn38WaDxERESWgsmNkTx23U1mcrNlC5CSUqAxERERWQImN0aSOXKzdy+QlvbAAwEBQIUKwL17wMaNmsRGRERkzpjcGElAAFCiBHD3LrB//wMP6HScmiIiIjIiJjdGotMBzZvLz8OGATdvPvBgZnKzerWsOiYiIiKDYXJjRKGh0nHh8GGgbdsHltg0bQq4uAAJCSzoR0REZGBMbozI3x8IDwfc3aVuX4cOwO3bAOzsgDZt5CRuCSciIjIoJjdGVr06sH69DNRs2wZ07frPAmOuuyEiIjIKJjcFoF49WV5TpIgkOu3aAUdKt5VqxUeOsFoxERGRATG5KSBNmwIrVwL29sDmzUDN54rjuFtjeZBTU0RERAbD5KYABQcDhw4BvXrJoM38GzI1te+DVThyROPgiIiIzASTmwIWGAgsWACcPAnYdO4AAKhxfTOeqZ2KsWOBW7c0DpCIiKiQY3KjkUqVgKnLA5HuWx72uIfn9OGYPh2oWhVYs0br6IiIiAovJjda0ulg+0JnAMDshmHw9ZW1xe3bA926AZcvaxwfERFRIcTkRmuDBwMAvPb+iZMrz+CttwBra2DpUpnC+vxzID1d4xiJiIgKESY3WqtcWar7KYUic7/A9OnAgQNAUBCQmgq8+SZQpw6wY4fWgRIRERUOTG5MwZtvyr9hYcDff6NmTUlmfvwRKF4cOHYMeOYZoEsXuV8pLYMlIiIybUxuTEGzZjI8c/cuMHs2ANkq/uqrwOnTMnOl00mdnGeekVGdpUuBjAyN4yYiIjJBOqUsaxwgOTkZrq6uSEpKgouLi9bhZFu4EOjdG/DwkFXFDg45Hj51CvjiC2D+/H/aNwAoWRKoUkV6WGUedesC5cpJMkRERGQu8vP9renITWhoKOrXrw9nZ2d4eHigS5cuOH36dK7PCQsLg06ny3E4/CsRKJS6dQPKlJFO4QsWPPRw5crA3LlAbCwwcaJMV127BmzdKtNX48bJS5QvD/j6Sp40dy5w5owGvwsREZGGNE1utm7dimHDhmH37t0IDw9Heno6WrdujVv/UcnOxcUFcXFxWcdFc+jNZGsLjBolP3/++WMX1nh4AJMnyzbxPXskD5o8GXjlFaB+fcDGRh5buBAYMgQICABq1ACmTgXOny/A34eIiEgjJjUtde3aNXh4eGDr1q149tlnH3lOWFgYRo8ejcTExDy9ZlpaGtIy53Egw1o+Pj6mNy0FAElJgI8PkJIilfzats33S9y+DezeLR3It22TBcgPbiVv0ADo0QN44QWgbFkDxk5ERGREhWZa6t+SkpIAAO7u7rmel5qairJly8LHxwedO3fG8ePHH3tuaGgoXF1dsw4fHx+DxmxQrq7AwIHy84wZT/QSRYoAzz0HTJokDTqvXgV++EH6WllZAXv3yuascuVkpGfaNCA62mC/ARERkeZMZuRGr9ejU6dOSExMxI5cirpERkYiOjoaNWrUQFJSEj777DNs27YNx48fR5kyZR46v1CN3ACymLhCBdkKFRkJNGpksJeOj5ddVkuXAtu3A3p99mMBAVIZuUMH6WBua2uwtyUiInpq+Rm5MZnk5vXXX8fatWuxY8eORyYpj5Oeno7AwED07NkTU6ZM+c/zTXa31INefRWYNw9o1QrYsMEob5GQAKxYIYnOli3A/fvZj7m4AC1byg715s2B6tVl1IeIiEgrhS65GT58OFauXIlt27bBz88v38/v1q0bbGxssGjRov88t1AkNzEx0lnz/n3ZDvWY9UeGkpQkOdTq1bLU59q1nI8XKyYhPPMM0KSJlOSxszNqSERERDkUmjU3SikMHz4cy5cvx+bNm58oscnIyMDRo0fh7e1thAg14ueXvfbm/feNXpLY1VW2kYeFydTVnj1AaCjQpg3g5ATcvCkFBN96SwoIurrKqM6HHwJnzxo1NCIionzTdORm6NChWLhwIVauXImAgICs+11dXeHo6AgA6NOnD0qXLo3Q0FAAwIcffohGjRrB398fiYmJmD59OlasWIEDBw6gSpUq//mehWLkBpD93P7+UrFvwwaZotLA/fvAwYMygLRzp+y+un495zlBQcDLL8surOLFNQmTiIjMXKEZuZk9ezaSkpLQvHlzeHt7Zx3/+9//ss6JjY1FXFxc1u2bN29i0KBBCAwMRLt27ZCcnIxdu3blKbEpVMqUkUI1ADBhgmYNpWxsZPv422/LGp1r14CTJ4HvvgNat5a1OJGRwLBhgJeXjPbMnSsjQERERFowiTU3BanQjNwAkiGULw/cuQP8+adsZTIxcXHAokXAr78Chw5l36/TyUavkJDsqsm+vpKzcScWERHlV6FbUFyQClVyAwDvvAN8+ilQuzawf79Jb1s6dQpYvlxGePbuffQ5VlYyEtS5sxyVK7MPFhER/TcmN7kodMnN33/LAuPUVGDJEuDFF7WOKE8uX5ZFyPv3A5cuSU+s2Njspp+ZKlYEunYFhg+XUR0iIqJHYXKTi0KX3ADSKXPKFElyjh2TMsSFkFKS6KxdK4nPpk3AvXvymK0t0K8f8O67Mo1FRET0ICY3uSiUyU1qKhAYKMMh48cDH3+sdUQGkZICrFsHfPstEBEh91lbAz17Su+r0qWBUqUAT09Z2ExERJaLyU0uCmVyA8hClueflyGOqCjAzHaH7dghOdu6dQ8/ptNJotO6NdCpk/TJKlq04GMkIiLtMLnJRaFNbgBZgfvHH1IqeOtWs1yJe+AAMHOmLE6+ckV2Y2Vk5DzHwUHaQwQFyfZzT085Mkd6iIjI/DC5yUWhTm4uXpQRm9u3pfdUv35aR2R0er3U1jl6FFi1StbqXLjw+PMrVJBaO23bSl8sjvAQEZkHJje5KNTJDQBMnw6MHSulgE+ftriSwEoBx49LohMdDVy9KuWArl59eJTH3l4SnXffNWhzdSIi0gCTm1wU+uQmPV06Vx47Jt3Df/xR64hMRkqKdDhfu1aOixezHwsOlkLPRu5BSkRERsLkJheFPrkBgF27pD03IN/mzZtrGo4pUgo4cQL4/HPg55+lRxYANG0KVKsmxQR1OvnXyUl22fv5yTZ0Hx9WUSYiMjVMbnJhFskNIH2n5s6Vb+MjR7i4JBcXLgCffAL89FN2XZ3cWFvLguUhQ4COHbkNnYjIFDC5yYXZJDfJyUDVqlL75o03ZIiCcnX5MvC//wG3bslC5cwjMRGIick+7t7Nfk6pUsCAAcDAgdIbi4iItMHkJhdmk9wAsrCkXTuZX9mxA2jcWOuICj29Hjh3Tjaj/fgjkJAg9+t0cnm7dZN2ET4+OZ+nlDzX2rrgYyYisgRMbnJhVskNINvB588HAgKkuJ+Dg9YRmY1796R24pw5srTpQY0aAR4eslMr8wCAV16R3Vn+/gUeLhGRWWNykwuzS25u3JDpqfh4+VYNDdU6IrN0+TLw++/A0qUySJbb/2usrIBevYBx48yukDQRkWaY3OTC7JIbQCrbdekicyK7dwP16mkdkVm7cgVYs0amoTw9pUqyl5c0BZ06VWYLAZnK6thRduy3a8cdWERET4PJTS7MMrkBpNvk4sVA9erSw4DfpJo5cECSnN9/z77PwwN4+WWgf3/Zik5ERPmTn+9vqwKKiYxt5kypVnz0qFQxJs3UrQssWwacPAm89ZaM7iQkyIa26tVlm/mGDblPbRER0ZNjcmMuSpYEvvxSfv7wQ2nNQJqqXFnyzEuXpN9p5szh5s1ASIgkQYsXZxcYJCIiw+C0lDlRShZ3rFsnncMjImR1K5mMixdlBOeHH6T/KSB5adOmUnS6aVOgdm3Azk7bOImITA3X3OTCrJMbQL49q1aVSnWzZ0uZXTI5168D33wDfP21/PwgR0fJUV9+Wbqb29trEyMRkSlhcpMLs09uAFl/M2oU4OwsCz9Kl9Y6InqMtDRg/37ZXr5zp7QNezDZcXOTwoF9+0oRQZ1Os1CJiDTF5CYXFpHcZGTIHMeePUCnTlKJjt+KhYJSUotx4UJg0SLgr7+yHwsKklJGHTpwtpGILA93S1k6a2tZ1GFrKytZ2Xeq0NDpZM3N9Okyw7h5s2wft7cHIiOBzp1lx9XPP+etCSgRkSVicmOuqlXLrlb81luS7FChYm0NtGgh3cwvXJBRGxcX4MQJmaYqXRoYOVLq6ljW+CsRUe6Y3JizMWOAsWPl58GDpSU2FUpeXpKrxsYC06YB3t7A33/LguR69WQ0Z8YMIClJ60iJiLTHNTfmTinZMfXdd4CNjbRqaNdO66joKd2/D4SHS8/UFStkYTIgIzuvvw6MHi0JERGRueCC4lxYXHIDyALjl1+WinEODsD69cCzz2odFRlIYiLw22/AV1/JlBUga3T69we6dgXKlgV8fdkwnogKNyY3ubDI5AYA0tOB558HVq+WLeKbNgH162sdFRmQXg+sWiXTV7t3P/y4lxfg7y+1c154AQgIKPgYiYieFJObXFhscgMAd+7IlFREBFCsGLBlC1CzptZRkYEpBWzfLkUCjx+XXVe3bj18XpUqMrLz4otAjRqsFkBEpo3JTS4sOrkBgJQUaWwUGSl1/7duBQIDtY6KjEgp4MYN2XF14ACwfLkM3KWnZ58TGAi89JIclSppFioR0WMxucmFxSc3gCzSaNkSOHhQtt1s2ybzFWQxEhNlCmvZMmDt2uwFyYDU2WnbFnjuOamK7OioWZhERFmY3OSCyc0/rl8HmjcHjh2T1abbt8u/ZHGSkmQT3aJFsgMrIyP7MTs7SXCCg4H27WUWk9NXRKQFJje5YHLzgPh4oFkz4MwZoGJFSXA8PbWOijT0998yorNli0xdPdj+AZDCge3aSQuItm2lCDYRUUFgcpMLJjf/cvky0LSprDqtXl0WG7u7ax0VmQClgLNnJclZuxbYuBG4fTv78QYNpC5kuXKahUhEFoS9pSjvypSRby0vL+DoUfmzPCVF66jIBOh0MqA3ZIhMW12/LknO8OGAqyuwd6+sz1mxQutIiYhyYnJDspg4PFxGbPbske6Md+5oHRWZGAcHoE0bafkQFSUjN4mJUj5p9Gg28iQi08HkhkS1asC6dVLgb8sWqfLGBIceo1w5WaL15pty+6uvgLp1pbbO1auahkZExOSGHlC/vqwmdXSU+YeQEHZipMeyswM++wz44w8Z9Dt2DBgxAihVSj468+cDqalaR0lElojJDeX07LMyguPiIn+aN2/OP8UpVx07AqdOAV98IVNVej2wYQPQr58s6XrzTSAmRusoiciScLcUPVpUlPz5nZCQvSaH22IoD86elZo5P/8sPwOAlZUs5Ro1SvJn1sohovzibil6erVqATt3SkJz9izQpEl2y2miXPj7AxMmAKdPS5/W1q1lNGf5chkIDAiQ5p7/rqFDRGQoTG7o8fz9JcGpWhW4cgVo0UI6MRLlgZWVVBZYv14+Nq+9Bjg5AdHRwPjxUhC7fXvp/kFEZEhMbih3pUpJc83atWWKqkULWTlKlA9VqgBz5gBxccC8ecAzz8hozpo1MpozeXLOtg9ERE9D0+QmNDQU9evXh7OzMzw8PNClSxecPn36P5+3ZMkSVK5cGQ4ODqhevTrWrFlTANFasOLFpdBfnTrAtWuS4Bw9qnVUVAg5OclC423bZNqqb1+phDxpkizx4tp1IjIETZObrVu3YtiwYdi9ezfCw8ORnp6O1q1b49atW499zq5du9CzZ08MGDAAhw4dQpcuXdClSxcc42iCcbm7S4JTt640IGrRAjh8WOuoqBCrVAkICwN++QUoWlTaPNSqJWWWiIiehkntlrp27Ro8PDywdetWPPvss488p0ePHrh16xZWrVqVdV+jRo1Qq1YtzJkz56Hz09LSkJaWlnU7OTkZPj4+3C31pBITZYXovn1AsWLAb79Jy2iip3DqFNCtm8x4WlnJmpwPPgBsbLSOjIhMRaHdLZX0T8E491waN0ZGRiL4X1+mISEhiIyMfOT5oaGhcHV1zTp8fHwMF7AlcnOTIiZBQcDNmzKX8MUXMrdA9IQqV5bOHwMGyFqcjz6SLeOsj0NET8Jkkhu9Xo/Ro0ejSZMmqFat2mPPi4+Ph6enZ477PD09ER8f/8jzx40bh6SkpKzj0qVLBo3bIrm5AZs3y+IJvR4YM0YWT7BdAz2FIkWAH34AFi+WxpyRkTJNtXCh1pERUWFjMoO+w4YNw7Fjx7Bjxw6Dvq69vT3s7e0N+poE6aL400+yi2rMGFk4cfKkFDMpU0br6KgQ69EDaNQI6N1bKhH07g3MnAkEBgIVKsgRGAjUrMligET0aCYxcjN8+HCsWrUKW7ZsQZn/+GL08vLC1X9tqbh69Sq8vLyMGSI9ik4HjBwp01Tu7sD+/bLgeOtWrSOjQq5sWSAiQnZRWVnJlFVYmBQH7NVLcuru3YGUFI0DJSKTpGlyo5TC8OHDsXz5cmzevBl+fn7/+ZygoCBs2rQpx33h4eEICgoyVpj0X557ThKbmjWlFk7LltImmutw6CnY2Mii4tOnZWpqyhSZCX3mGcDWFli6VHpZnTypdaREZGo03S01dOhQLFy4ECtXrkRAQEDW/a6urnB0dAQA9OnTB6VLl0ZoaCgA2QrerFkzTJs2De3bt8fixYsxdepUHDx4MNe1OpnYW8qIbt8GBg3KXiTRuzfw3XeymILIgCIjZXfVX3/JNvKffpKRHKWkmPbx41IwMCQE4KAukXnIz/e3psmN7jET5vPmzUO/fv0AAM2bN0e5cuUQFhaW9fiSJUvw/vvv48KFC6hYsSI+/fRTtGvXLk/vyeTGyJSSUZu33pKSs7VqSYOhUqW0jozMTEIC8NJL2XVxateW3VWJidnnuLkBM2YA/ftzfQ5RYVdokhstMLkpIFu3yp/SCQnSfHPDBqBiRa2jIjNz/z7w/vvAJ59k32dtLW3RAJnSAmSm9LvvgPLlCz5GIjIMo9e5mT9/PlavXp11e+zYsXBzc0Pjxo1x8eLFJ3lJMjfNmgG7d8u3zIULQNOmwMGDWkdFZsbGBpg2Ddi7F1i0SIpm37olRQGPHQM+/VQ29m3aBFSrxpJMRJbiiZKbqVOnZq2JiYyMxKxZs/Dpp5+iRIkSeOONNwwaIBVifn7Ajh3ZTTebN2dtfTKK+vVliqpGDSCz8oONDfD229IGrUULKcM0ZozstmJJJiLz9kTJzaVLl+D/z7jvihUr8MILL2Dw4MEIDQ3F9u3bDRogFXKenpLQNG8u+3bbtJFtLkQFxN9fRm5mzZKEZ/FiGViMi9M6MiIylidKbpycnHD9+nUAwIYNG9CqVSsAgIODA+7wTyL6N1dXYO1a4PnngXv3ZJvLlCmcH6ACo9MBQ4cC4eFSkmnfPhnt4UwpkXl6ouSmVatWGDhwIAYOHIgzZ85k7VQ6fvw4ypUrZ8j4yFw4OEiTzZEj5fbEiTKPcPu2tnGRRWneXNbnBAbKNvKmTYHPPwf+aWtHRGbiiZKbWbNmISgoCNeuXcOyZctQvHhxAMCBAwfQs2dPgwZIZsTGRraJf/+9VGH77TepyHb5staRkQWpUEHq5ISEyNqbN9+UjiEjRmTvriKiwo1bwUkb27YBL7wA/P23VFlbsEAqHRMVkPv3gR9/lHz7wSrH7doBX37JygVEpsboW8HXrVuXo8HlrFmzUKtWLfTq1Qs3b958kpckS/Pss7LwoXp1ID4eCA4Gxo0D0tO1jowshI0N8NprUs04PBzo2FHW5qxZI7uuZsyQOpREVPg8UXLz9ttvIzk5GQBw9OhRvPnmm2jXrh1iYmIwZswYgwZIZqxcOZkfGDhQFhdPmyaLIM6d0zoysiA6neTWf/wh01LBwcDdu1Jku2lT9q4iKoyeKLmJiYlBlSpVAADLli1Dhw4dMHXqVMyaNQtr1641aIBk5ooWlTU4S5ZIrfy9e6UuzuLFWkdGFqhiRSmm/d13gLOz1KGsVQsYPx64elXr6Igor54oubGzs8Ptf3a5bNy4Ea1btwYAuLu7Z43oEOXLiy9KedmmTaUeTs+eUnHt/n2tIyMLo9NJ/9fjx4G2baV6QWgoULYsMGQIEB2tdYRE9F+eKLlp2rQpxowZgylTpmDv3r1o3749AODMmTMoU6aMQQMkC+LrKwX/xo2T2198AbRuDVy7pm1cZJF8fKTn6/LlQMOGQFoaMHcuEBAga+H37NE6QiJ6nCdKbr755hvY2Nhg6dKlmD17NkqXLg0AWLt2Ldq0aWPQAMnC2NgAU6cCy5YBTk6S7NStCxw4oHVkZIF0OqBLF1katm0b0KGDLA/7/XegUSNZF//nn4Ber3WkRPQgbgUn03XihHyzREdLw6CFC4GuXbWOiizc8ePAZ59J9YLMzX2VKwMff8yPJ5Ex5ef7+4mTm4yMDKxYsQIn/9lKULVqVXTq1AnW1tZP8nIFhslNIZOUBLz8MrBqFWBlBfz0E9C3r9ZREeGvv4CZM4E5c4DMpYbTp8suKyIyPKMnN2fPnkW7du3w119/ISAgAABw+vRp+Pj4YPXq1ahQocKTRV4AmNwUQhkZwODBktgA8o0yYoS2MRH9IzkZmDBBPpYA8M47sgBZp9M2LiJzY/QifiNHjkSFChVw6dIlHDx4EAcPHkRsbCz8/PwwMrN3EJGhWFvLdvHRo+X2yJHARx+x8SaZBBcXqXL8ySdy+5NPZLcVN/oRaeeJRm6KFi2K3bt3o3r16jnuP3z4MJo0aYLU1FSDBWhoHLkpxJQCPvwQmDRJbo8eLfMANjZaRkWU5YcfpOqxXg88/7ysy3F01DoqIvNg9JEbe3t7pKSkPHR/amoq7OzsnuQlif6bTgd88IFsEQekAVBwMBAXp2lYRJkGDpR6lHZ2soW8Xj1u9CPSwhMlNx06dMDgwYOxZ88eKKWglMLu3bsxZMgQdOrUydAxEuU0erR8gzg7A1u3SkXjiAitoyICIDum1q8HPD1lw1+jRjLgyLZpRAXniZKbmTNnokKFCggKCoKDgwMcHBzQuHFj+Pv748svvzRwiESP8OKLwP790njz6lWgZUtZxcmCI2QCmjcHjh2Tj+n9+zLg2LgxcOqU1pERWYanqnNz9uzZrK3ggYGB8Pf3N1hgxsI1N2bm9m1g2DAgLExuN2kie3OrVdM0LCJAloktWiQf0cREaaW2aJF0ICei/DHKVvD8dPv+/PPP83xuQWNyY6Z++kl2Ud26JQuMx4wBJk6UbxMijf31F9CnD7B5sywdmzFDZle5XZwo74yS3LRo0SJPb67T6bB58+Y8nasFJjdm7NIlYNQoWckJSKfDWbOAf3qfEWkpPV3KM82dK7dfew34+mvA1lbbuIgKiwKpUFxYMbmxAH/+CQwfDsTGyu1Bg2SHFUdxSGNKyUfxrbfk51atgMWLAXd3rSMjMn1G3wpOZNI6dpRtKmPGyLj/99/Ljqp9+7SOjCycTicfyxUrJNcODwcqVpQBRhb9IzIcJjdknooWlYUNGzcCpUtL883GjaW7YUaG1tGRhevUCdi+HahaFbhxQwYaa9WSZIeInh6TGzJvzz0HHDkCdO8ufxq//z4QEgL8/bfWkZGFq10biIoCvv0WKF5cuo23bi0Dj4cPax0dUeHG5IbMn7u7LGyYP19GdDZtAurX5zcIac7GBnj9dRlYHD1abq9aJaM43bsD/1TaIKJ8YnJDlkGnk724kZFA+fLAhQtAUJAkPUQaK1ZMFhofPw689JJ8XJcskXJNr7zC4n9E+cXkhixL9eqysDgkBLhzB+jZU7au3LundWREqFRJivwdPiyNN/V64NdfgcBAoG1baetgWftbiZ4MkxuyPO7uwOrVwLvvyu0ZM4AGDWRtDpEJqF4d+P136TDSpYuM5KxbB7RpI4uQf/iB6+KJcsPkhiyTtbX0olq2TFZzHj4sLZw//ph7cslk1K0rNSmjo6U+pZOTrMMZNAgIDs4u5UREOTG5IcvWtassdOjSRUrIvv++bBk/flzryIiyVKgAfPklcPkyMH26rIuPiABq1OCyMaJHYXJD5OkpcwC//AK4ucmanNq1gffek8acRCbC1VWWiEVFAQ0bAklJsmzs5ZeB69e1jo7IdDC5IQJkUcPLLwPHjkmhkfR0YOpU2a6ybp3W0RHl4O8vRQA/+ACwsgIWLAC8vICWLYGZM2UzIJElY3JD9KDSpYE//pCFDmXKADExsk2lRw/+aUwmxdYWmDQJ2LFD6uLcvy9dx0eNAvz8ZI18dLTWURJpg8kN0aN06SIrN8eMkcXHv/0m3yDbt2sdGVEOQUHAoUPA2bOy8e/ZZ2U0Z98+oEMHae9AZGmY3BA9jpOTfFvs3SsFSC5fBpo3Bz76iPtwyeRUqCC5+NatMi3l4wOcOQN06yazrESWhMkN0X+pUwc4cEAqHOv1wIQJUgQwPl7ryIgeycdH2jg4OclU1fDhLP5HloXJDVFeODlJb6qwMKBIEelPVbu2tHMgMkE1aki1Y50O+O472UpOZCmY3BDlR9++MopTrZqM3DRrBnz/vdZRET1Shw4yswoAb74J/PmntvEQFRQmN0T5VbmyjNi8+KIsZhg8GBgyhP2pyCSNHi0fUaWkZuVXX3GKiswfkxuiJ+HkJDuopk6Vcf+5c4EWLbj3lkyOTgd88410G79/X5KdHj2A5GStIyMyHk2Tm23btqFjx44oVaoUdDodVqxYkev5ERER0Ol0Dx3xXNhJWtDpgHHjpAmnqyuwaxdQpYoUGmFNHDIhtrbAwoUyamNjAyxZAtSvLzUricyRpsnNrVu3ULNmTcyaNStfzzt9+jTi4uKyDg8PDyNFSJQHbdvKOpz27eVP45kzpYTs558DaWlaR0cEQHLxkSOBbdukPuWZM1Loj+twyBxpmty0bdsWH330EZ5//vl8Pc/DwwNeXl5Zh5UVZ9dIYxUqyN7b8HDZppKYKCs4a9YE9uzROjqiLEFBwMGDQKtWwJ07sg5nyRKtoyIyrEKZFdSqVQve3t5o1aoVdu7cmeu5aWlpSE5OznEQGU1wsHxz/PCDNOQ8fVq6jL//Phcck8koWRJYswbo1UsGG196Cfj1V62jIjKcQpXceHt7Y86cOVi2bBmWLVsGHx8fNG/eHAcPHnzsc0JDQ+Hq6pp1+Pj4FGDEZJGsrYEBA4ATJ6Rls14PfPyxzAEcOaJ1dEQAZO3Nzz/LR1WvlxqVrGpA5kKnlGlsCtTpdFi+fDm6dOmSr+c1a9YMvr6++OWXXx75eFpaGtIeWPeQnJwMHx8fJCUlwcXF5WlCJsqbJUuA11+XRca2tvJtMmYMULGi1pERQa8HRowAvv1Wbo8YIUvGXF3l8PICGjaUNTtEWkpOToarq2uevr8L1cjNozRo0ABnz5597OP29vZwcXHJcRAVqG7dZFtKx45SF2fOHCAgQBY77NqldXRk4aysZKv4m2/K7a+/lg1//foBzz8va3S6dWM7NSpcCn1yExUVBW9vb63DIMqdlxewciWwZYvsqlIKWL4caNIEeO45IDZW6wjJgul0wPTpMk01aBDQvbu0T2vUSAYbly0DXnuNxf+o8LDR8s1TU1NzjLrExMQgKioK7u7u8PX1xbhx4/DXX3/h559/BgB8+eWX8PPzQ9WqVXH37l388MMP2Lx5MzZs2KDVr0CUdzqddBVv3lzW48yYIas4t2yRPlVhYTK6Q6QBnQ545RU5HvT77zJy8+OPQPHiwCefaBMfUX5oOnKzf/9+1K5dG7Vr1wYAjBkzBrVr18bEiRMBAHFxcYh94C/ae/fu4c0330T16tXRrFkzHD58GBs3bkTLli01iZ/oiVWpIt8WJ08C9eoBN24AnTrJ3AB3VZEJ6dpVGm8CwKefykFk6kxmQXFByc+CJKICkZYGvPtudtvmBg2knGyFCpqGRfSg6dOBsWPl5++/BwYO1DYesjwWtaCYqNCztwe++AJYsQJwcwP27gWqV5dvk/v3tY6OCADw9tvAO+/Iz4MGAePH8+NJpovJDZGp6NwZiIqSBcZ37sifyQ0aSFFAIhMQGgq88Ub2z61aAWztR6aIyQ2RKSlbFti4EfjpJ6BYMeDQIUlwxo5lnyrSnE4nLdMWLwacnICICFkLv3Wr1pER5cTkhsjU6HRA//6y2LhHDykwMn26tHGIjtY6OiL06AHs3w9UrSojN889B0yaxLXwZDqY3BCZKk9P+RN55UrZg3vwIFCnjiw2JtJYQID0hO3TR6ocT54M1K8vg41EWmNyQ2TqOnWStTjPPgukpgK9e0sLh1u3tI6MLFzRolKeadEiyb+PHJEEZ8IEzqKStpjcEBUGZcoAmzcDH3wg9fJ/+knq47ARJ2lMp5Ou4idOAC++KLOoH30kH89Tp7SOjiwVkxuiwsLaWhY2bNoElCol3xwNGkivKssqV0UmyMNDesT+9htQsqS0U2vQQCocEBU0JjdEhU3z5sDhw9KjKi1NOo536wYkJmodGRG6dQOOHgWeeQZISZHmmxMmsPEmFSwmN0SFUYkSwJ9/Sn+qzM6GtWpJnyoijXl6ygDjyJFy+6OPpG3azZvaxkWWg8kNUWGl0wFjxgA7dwLlywMXL8qe3GHDZOExkYZsbYGvvgJ++QVwcADWrgUCA6WlGkdxyNiY3BAVdvXry26q116T299+C9SowVEcMgkvvwzs2gVUqgRcvSo9qerXB7Zv1zoyMmdMbojMgbOzLCwODwd8fYGYGBnFGTOGDYBIc7VryzqcGTMAV1ephfPss0D37pLwEBkakxsicxIcLNtUMkdxvvgCaNeOix1Ic3Z2kmtHR8vH08pKdlfVrAls2KB1dGRumNwQmZvMUZxly4AiRWQ0p1Ej4PRprSMjQsmS8vE8eBCoVk1GbkJCpOM42zeQoeiUsqwCGcnJyXB1dUVSUhJcXFy0DofIuKKipNt4bKzMB/zvf/JNQmQC7twB3npLlokBshZn4kRpymlvL0eJEjLTSpSf728mN0TmLiEB6NpVdlVZWQEffgi8+64UBSQyAcuXS0eRx82eDhoEfP21JDtkufLz/c1pKSJz5+EhRUdefVU6HL7/PtCmjbRzJjIBzz8vdSm7d5dyTYGBUt2gVCmpePD991IUMDZW60ipsODIDZElmT8fGDoUuH1bkp4FC2QRMpGJWr8e6NULuHFDpqgWLwZattQ6KtICR26I6NH69gX275eVnAkJQOvWwJtvcjcVmayQEODAAdlO/vff8pGdMUPrqMjUMbkhsjSBgcDevcDgwdJw8/PPZQ7gk09kRIfIxJQrJ0vG+veXmdW33pIqB0SPw+SGyBI5OgJz50p/qqpVpenmu+8C/v6yT5eF/8jEODpK64aPPpLbY8YAP/+sbUxkupjcEFmyDh1kJefPP8ufx3Fx0mW8USO5n8iE6HTA+PHAG2/I7Vdflfyc6N+Y3BBZOmtr4JVXgFOngJkzgWLFZJFDvXrAhAlAWprWERJl0emAzz4D+vSRBpzdu7NPFT2MyQ0RCXt7YMQI4MQJqYtz/77MAdSuLZ0PiUyElRXwww9Ax47A3bvy7549WkdFpoTJDRHl5OUlrRuWLgU8PYGTJ4EmTYAePYCzZ7WOjggAYGsrBbefeQZISgKaNpVdVHq91pGRKWByQ0SP9sILMorz6qsyF/Dbb7LTasQI2UZOpDFHR2DVKvmo3r8vu6g6dODHk5jcEFFu3N1li0pUFNC2rXyDfPMNUKGC1MO3rBqgZIJcXKS7+Jw5gIMDsHatdBr//Xfg+nWtoyOtsEIxEeXdli3A2LFSCBCQCmvz5gHe3trGRQTg2DGZPT1xIvu+UqWA6tUl4Rk8WPJyKpxYoZiIjKNFC1m5+c038mfy+vXyzbFihdaREaFaNWDfPmD0aKlsAABXrsjH9NNPgYYNpX4lmT8mN0SUP1ZWwLBhsl28Vi0Z+3/+eWnrfOOG1tGRhStSRKoXx8TIQuNdu6ReZd268lF97jkgPFzrKMnYmNwQ0ZOpUgXYvVumqXQ64KefgIAAmabilhUyAS4uQFCQTEdFRACtWgG3bgHt28s6HTJfTG6I6MnZ20tPqogISXb+/lt2Vz3zjCxCJjIRTk5SzbhbNyA9XdbmzJmjdVRkLExuiOjpPfusJDPTpwNFi8pcQN26wMiR0reKyATY2wOLFgFDhshGv9dfl4/ovXtaR0aGxuSGiAzD1lYKjZw6JTXx9XrZLh4QAMyfz6kqMgnW1sC33wKTJsntr7+WdThxcZqGRQbG5IaIDKtMGSkdGx4uiU1CAtCvH6eqyGTodMAHHwArV8q6nJ07gTp15F8yD0xuiMg4goOBI0dkTU7mVFW9esCsWVpHRgQA6NRJSjZVqwbExwPNmwPvvgucP691ZPS0mNwQkfHY2cluqlOnpBlnRgYwfDgwdKis6iTSWMWKsunvpZekAPcnn0ihv5YtZX3O3btaR0hPgskNERlfmTLSiHPaNJkTmD1b2jmwLg6ZgKJFgYULpV9sSIh8RDdvBnr1AsqWlZJOVLgwuSGigqHTAe+8I9WMixYFNm0CGjWSZkBXrrBPFWlKp5PBxXXrpADgBx9ITp6QAHTpAly9qnWElB/sLUVEBe/IEaBjRyA2Nvs+NzeplVO7tnQeDwjQLDwiQCocN2wInD4NNG0q+bidndZRWS72liIi01ajhjQB6tsXqFRJWjokJsqi41mzJMnp3x+4cEHrSMmCubpm76jasUN6VlHhwOSGiLTh4QGEhcmfxbduyTbxhQtlC4teL49VqiQLkFmEhDQSECAfy8ylYt9/r3VElBdMbohIew4OQM2aQM+e8qfy7t2ylTw9XUZyAgKAmTNltxVRAWvfHpgyRX4eNgzYtk3beOi/aZrcbNu2DR07dkSpUqWg0+mwYsWK/3xOREQE6tSpA3t7e/j7+yMsLMzocRJRAWvYUIoAbtkCNGgApKQAo0bJAuSDB7WOjizQ+PHAiy9Kvt2smVQ1XrQISEvTOjJ6FE2Tm1u3bqFmzZqYlceiXjExMWjfvj1atGiBqKgojB49GgMHDsT69euNHCkRaaJ5cyAyUjocurpKxbX69SXR2b0buHNH6wjJQuh00vC+a1f5ecsW2SpeqhTw5ptsoWZqTGa3lE6nw/Lly9GlS5fHnvPOO+9g9erVOHbsWNZ9L730EhITE7Fu3bo8vQ93SxEVUvHxwJgx8udyJmtroGpVqXzctavMHxAZ2cWLwE8/yXH5stxXqxawYQNQsqSmoZk1s90tFRkZieDg4Bz3hYSEIDIy8rHPSUtLQ3Jyco6DiAohLy9Z2bl+PdChA+DpKWtwjhyRb5kOHaQjIpGRlS0LTJ4sm/n++EPWxkdFyUDjlSsaB0cACllyEx8fD09Pzxz3eXp6Ijk5GXceMzwdGhoKV1fXrMPHx6cgQiUiY2ndGvjzT9lBdemSFAXs21ceGzaM21mowFhbS7mmbduA0qWBEyeAZ5+VkR3SVqFKbp7EuHHjkJSUlHVcunRJ65CIyBB0Oikh27mzLIYYM0buf+012UZOVEACAoDt2wE/P+DcOeCZZ4BDh4B797SOzHIVquTGy8sLV/9VA/vq1atwcXGBo6PjI59jb28PFxeXHAcRmRmdDvjsM6lsrBTw6qvAr79qHRVZED8/GcEJCJABxTp1AHt7oHhxWRbWti2wdavWUVqOQpXcBAUFYdOmTTnuCw8PR1BQkEYREZHJ0OmAr74ChgyRBKdvX+ll9WCLByIjKlNGEpznngNsbOS+GzdkumrdOuk0/s03bKNWEDRNblJTUxEVFYWoqCgAstU7KioKsf/8x2jcuHHo06dP1vlDhgzB+fPnMXbsWJw6dQrffvstfvvtN7zxxhtahE9Epkank6J/AwdKleNPP5U/qV94AYiI4LcKGZ2Hh/SgSksDrl0Djh6Vkk29esn69xEjgEGDWB/H6JSGtmzZogA8dPTt21cppVTfvn1Vs2bNHnpOrVq1lJ2dnSpfvryaN29evt4zKSlJAVBJSUmG+SWIyPTo9UqtWKHUc88pJSmNHLVqKbVxo9bRkQXS65WaPl0pKyv5KAYFKRUXp3VUhUt+vr9Nps5NQWGdGyILc+yYzAX88gtw+7bc17kzMH06ULGitrGRxVm/HnjpJSn65+0t9Sk7ddI6qsLBbOvcEBHlW7Vq8g0SGytzAtbW0r+qalXg7beBpCStIyQLEhIC7N0rje/j4iTP7tlTprDIcJjcEJFlKF5cmm8ePQq0aSNNgj77TL5l8tDXjshQKlaUTiLvvCO59uLFQGCg1Ki0rLkU42FyQ0SWJTAQWLsWWLNGvmWuXAGef166IsbFaR0dWQhHR2DaNGDPHqBGDeD6daB3bym0zSKAT4/JDRFZprZtgcOHpd2zjQ2wbJkkPt9/LzutiApA3boyijNlCmBnJzl31arAl1/K7ip6MkxuiMhyOToCH3+c3W08KQkYPBho2BDYuVPr6MhC2NoC778v/ameeQa4dQt44w2gUSP5aHKqKv+Y3BAR1awJREYCX3wBuLjIN0rTprLSk0UAqYAEBko5pu++A1xds3NuX1/g5ZdlUPHMGSY7ecGt4ERED0pIkD+jf/hBvkUcHYFJk4C33gKs+PcgFYy4OODNN4GlS2Xt+4MCAqQA9yuvSFVkS5Gf728mN0REj3LoEDB6tNTTB2Sl588/A8WKaRoWWZbbt2VQcetWOXbvzm7IqdNJS4c+fYAuXQBnZ01DNTomN7lgckNEeaaUjOCMGCH18v38ZOFx7dpaR0YWKjlZRnPmz8/OuwHAwUHy7x49gPbtZcDR3DC5yQWTGyLKt4MHZat4TIy0ev7mG2DAAPnTmUgjMTEymLhgARAdnX2/kxMwdKislc9s4GkOmNzkgskNET2Rmzdl/H/VKrndqpXs161SRdOwiJSSnVaLFwP/+192nZyQELnt6qppeAbD9gtERIZWrJi0bQgNlYIk4eFSfe2NN6RREJFGdDqZKf3kExnN+e03oEgR6WPVuDFw/rzWERY8JjdERHllZQW8+y5w4oQ0BcrIkNGbSpWAr75ikkOa0+mAbt2AHTuA0qXlo9qwIbB9u3xcU1Olj1VsbHYfWXPE5IaIKL8qVJB+VOvXS3GSa9dkZ1WpUkD//rK9xbJm/MnE1K4tDTrr1gX+/ht49llZf+PsDHh4AGXLAiVKAN27A7//Dty5o3XEhsXkhojoSbVuLS0cZs+W7uN37gBhYTIXULNmzu0sRAWsVCn5CPbo8fBjNjbycV2yBHjhBUl4XnlFRnrMAZMbIqKnYWsLDBkCHDkC7NoF9Osn+3CPHgVatACmTmWvKtJMkSKy0DghQQYYU1OB+/elVs7+/cDbb0sF5NRU4NdfgVq1gAkTgLt3tY786XC3FBGRod28KdNUP/8st9u0AX75ReYBiEyMXi/FAadNA/78U+7z9wfmzgWee07b2B7E3VJERFoqVkymp378UaqrrVsniyDYjJNMkJWVzKSuXCk1KkuVAs6elerH/ftL4cDChskNEZEx6HTAq68Ce/bIbqrLl4FmzWQrOaepyATpdEDXrrLuZtgwuR0Wlr04uTBhckNEZEw1asjihl69ZC/u+PFSXS0+XuvIiB7J1VWKcG/fLruqzp8HmjQBPv0073m51gteuOaGiKggKAXMmwcMHy7bVDw8ZB1O69ZaR0b0WImJwODBsqsKkMLcAwcC7u5A8eJy3LsHHD8ua+iPHZOjRg1g4ULDxsL2C7lgckNEmjp5UvbmHj0qt4cOBT76iN3GyWQpBfz0k/SPzWs9nIAA4NQpw8bB5CYXTG6ISHN37gBjxgBz5sjtkiWldn7fvrK6k8gEnTwpzTgvXQKuXwdu3JB/dTppsVatGlC9uvxbrRrg42PY92dykwsmN0RkMiIiZOVmZuW0Ro2kyIiTk9zW6QBrayk+UqSIVlESPVZmBqHTGf+9mNzkgskNEZmU9HRg5kxg0iSppPYobm6yJ3foUClAQmSBmNzkgskNEZmkv/4C3ntPto4D2X8SJyYCV69mnxcSAowcCbRtWzB/LhOZCCY3uWByQ0SFil4vRQBnzQLWrs1Oerp1kzU77u7axkdUQFihmIjIXFhZAe3aAatXA9HRwBtvSNfDJUtk9eaGDVpHSGRymNwQERUWFSoAn38OREbKXtsrV2SaasQI4PZtraMjMhlMboiICpt69YCDB2WnFSDlZMuVA8aNAy5e1DQ0IlPA5IaIqDAqUkSSmrVrAV9f4No1aetcvjzQqROwZg2QlqZ1lESaYHJDRFSYtWkDnDsH/P47EBwsC5D//BNo316qHoeEAJ99BkRFsWEnWQzuliIiMienTwOzZwOLF+fcQg4AVasCX30FtGypTWxET4G7pYiILFVAAPDll0BcnPSv+uILGcUpWlS6GwYHAy+8AFy4oHWkREbDkRsiIktw8ybwwQfAt98CGRmAg4P0twoMlK3lmUdAgNxHZGJYxC8XTG6IyKIdPSoVjiMiHn9O8+ayvbxTJ0l4iEwAk5tcMLkhIounFLBsGfDLL9KhPCMDuH9fdlft3y+3AdmF9frrQKtWsl7HwUHbuMmiMbnJBZMbIqJcXL4sC5K/+w74++/s+62tgcqVgZo1gWefBfr2ZbJDBYrJTS6Y3BAR5cHdu7LjasEC4NAh4Pr1nI+XLi2NPgcMAOzstImRLAqTm1wwuSEiyielpGv54cNSGfn774FLl+QxX19gwgQZybG11TZOMmtMbnLB5IaI6CmlpQE//AB8/LFsOQcADw/g5ZeBV1+V9TlEBsY6N0REZDz29tLX6tw5YMYMwMsLSEiQpp7VqgENG8qanTt3tI6ULBSTGyIiejKOjlIrJzYW+OMPoEsX2Tq+dy/w2muAnx8wfTqQkqJ1pGRhmNwQEdHTsbUFOnYEli+XtTkzZshanKtXgbFjgbJlgUmTHl6UTGQkTG6IiMhwPDxkNOfsWSAsTCoe37wJTJ4sSc5bbwFXrmgdJZk5k0huZs2ahXLlysHBwQENGzbE3r17H3tuWFgYdDpdjsOBtRaIiEyLra3soDp+HPjtN6BWLeDWLRnV8fMDhgyRNTtERqB5cvO///0PY8aMwQcffICDBw+iZs2aCAkJQUJCwmOf4+Ligri4uKzj4sWLBRgxERHlmbU10K2bbCFfswZo2hS4dw+YOxfw9wfKlwd69ZJu5bt3A4mJsvWc6ClovhW8YcOGqF+/Pr755hsAgF6vh4+PD0aMGIF33333ofPDwsIwevRoJCYm5un109LSkJaWlnU7OTkZPj4+3ApORKSV7duBqVOBdese/bi9vUxveXkBJUrIKJCVVfZRqRIwaBBQrlyBhk3ays9WcE07ot27dw8HDhzAuHHjsu6zsrJCcHAwIiMjH/u81NRUlC1bFnq9HnXq1MHUqVNR9TF1FUJDQzF58mSDx05ERE/omWeAtWtllGbfPmDPHhm12bsXuHZN6uhcupRdKPBRpk0DOnQAhg8HgoMBna7AwifTp+nIzZUrV1C6dGns2rULQUFBWfePHTsWW7duxZ49ex56TmRkJKKjo1GjRg0kJSXhs88+w7Zt23D8+HGUKVPmofM5ckNEVIjcuSO7rK5eBeLjZYdVRoZMVen1MqW1ahUQHp79nIAA2Y3VoweTHDNWaEZunkRQUFCORKhx48YIDAzE3LlzMWXKlIfOt7e3h729fUGGSERET8rRUaabcptyGjkSOHUK+PZb2ZF1+jTQs6fc/vprae5JFk3TBcUlSpSAtbU1rl69muP+q1evwsvLK0+vYWtri9q1a+Ps2bPGCJGIiExR5crAzJlSV+fDDyUp2r4dqFMHeP114NgxYPNmSX4mTwaGDgV++glIStI6cioAmiY3dnZ2qFu3LjZt2pR1n16vx6ZNm3KMzuQmIyMDR48ehbe3t7HCJCIiU+XsLI07T52SaSm9HpgzB6heHWjZEujfX6asZs+WDuaenrJ7a8UKWdtDZknzaakxY8agb9++qFevHho0aIAvv/wSt27dQv/+/QEAffr0QenSpREaGgoA+PDDD9GoUSP4+/sjMTER06dPx8WLFzFw4EAtfw0iItKSry+weLGM2rz9NnDiBODjI/f7+gJubsDq1cDJk8DSpXI4OAAVKsiW9MyjenWgdm2gSBGtfyN6CponNz169MC1a9cwceJExMfHo1atWli3bh08PT0BALGxsbCyyh5gunnzJgYNGoT4+HgUK1YMdevWxa5du1ClShWtfgUiIjIVzZrJrqtH+fRTICoKWLAAWLRIKiUfPy7Hg6yspLN5vXpA/fpSm6dqVbmfCgXN69wUtPystiYiIjOl1wPnz0uV5LNn5ThzRooNxsc/fH6xYrKF/dlngSpVgJIlpQZPyZIyysNdWkaXn+9vJjdEREQPunIF2L9fjl27gMhI4Pbtx59ftKgUFgwMzD6cnIC//5bj+nVpPdGkCdCmDae8nhCTm1wwuSEionxJT5cRnW3bgB07gNhYSVoyCw7mR5EiQNu2wAsvAO3bA/weyjMmN7lgckNERAahFJCaKiM9p07JYuXMIy1Npq0yD0B6a124kP18JydpIDpmDMAdv/+JyU0umNwQEZEmlAIOHQKWLQOWLAGio+V+e3vZsv7229JIlB6JyU0umNwQEZHmlJL+Wh9/LOt6AOmg3rgx0Ly5HEFBUpwwIUGmxQ4elN1eiYnA3bvZh62tLHZu2VJ2i5npdxuTm1wwuSEiIpOhVHaX9PXrcz5mZwcULw7ExeX99aytgYYNgdKlpU/XnTuSANnZAa+9BnTvXmh3djG5yQWTGyIiMknnzgFbtgAREfLvlStyv04nu7Hq1JECg97eUoAw87hxQ1pNbNwor5Gb+vWl3k/z5sb+bQyOyU0umNwQEZHJU0oSlWvXgGrVpM1EXly4IMlRSopMaTk6SgJ07Bjw2WeyABoA2rWTthT16hWakRwmN7lgckNERBYpIUGajM6dC9y/L/eVLQt06SJH06aSHO3YIcfOnUBGBtC6tWxfb9FC0xo9TG5yweSGiIgsWnS0jNosXy5rcjLZ2kpNn8ext5fprGrVpG9X5uHiAty8KdNjmf+WKAH07GnQsJnc5ILJDREREaTqcni4dEj/80+ppGxnl91Pq2lTaVOxdq0cFy/m/bUbN5aRHwPKz/e35o0ziYiISANFigCdO8tx/7701ypXTtboPKhTJ1kDdPKkLFyOiQEuXZIjNlbW8bi7Zx/FikmjUQ0xuSEiIrJ0NjZA5cqPf1ynk4ahVaoUXExPgf3biYiIyKwwuSEiIiKzwuSGiIiIzAqTGyIiIjIrTG6IiIjIrDC5ISIiIrPC5IaIiIjMCpMbIiIiMitMboiIiMisMLkhIiIis8LkhoiIiMwKkxsiIiIyK0xuiIiIyKwwuSEiIiKzYqN1AAVNKQUASE5O1jgSIiIiyqvM7+3M7/HcWFxyk5KSAgDw8fHROBIiIiLKr5SUFLi6uuZ6jk7lJQUyI3q9HleuXIGzszN0Op1BXzs5ORk+Pj64dOkSXFxcDPralBOvdcHhtS44vNYFh9e64BjqWiulkJKSglKlSsHKKvdVNRY3cmNlZYUyZcoY9T1cXFz4f5YCwmtdcHitCw6vdcHhtS44hrjW/zVik4kLiomIiMisMLkhIiIis8LkxoDs7e3xwQcfwN7eXutQzB6vdcHhtS44vNYFh9e64GhxrS1uQTERERGZN47cEBERkVlhckNERERmhckNERERmRUmN0RERGRWmNwYyKxZs1CuXDk4ODigYcOG2Lt3r9YhFXqhoaGoX78+nJ2d4eHhgS5duuD06dM5zrl79y6GDRuG4sWLw8nJCS+88AKuXr2qUcTmY9q0adDpdBg9enTWfbzWhvPXX3/h5ZdfRvHixeHo6Ijq1atj//79WY8rpTBx4kR4e3vD0dERwcHBiI6O1jDiwikjIwMTJkyAn58fHB0dUaFCBUyZMiVHbyJe6ye3bds2dOzYEaVKlYJOp8OKFStyPJ6Xa3vjxg307t0bLi4ucHNzw4ABA5Camvr0wSl6aosXL1Z2dnbqp59+UsePH1eDBg1Sbm5u6urVq1qHVqiFhISoefPmqWPHjqmoqCjVrl075evrq1JTU7POGTJkiPLx8VGbNm1S+/fvV40aNVKNGzfWMOrCb+/evapcuXKqRo0aatSoUVn381obxo0bN1TZsmVVv3791J49e9T58+fV+vXr1dmzZ7POmTZtmnJ1dVUrVqxQhw8fVp06dVJ+fn7qzp07GkZe+Hz88ceqePHiatWqVSomJkYtWbJEOTk5qa+++irrHF7rJ7dmzRr13nvvqd9//10BUMuXL8/xeF6ubZs2bVTNmjXV7t271fbt25W/v7/q2bPnU8fG5MYAGjRooIYNG5Z1OyMjQ5UqVUqFhoZqGJX5SUhIUADU1q1blVJKJSYmKltbW7VkyZKsc06ePKkAqMjISK3CLNRSUlJUxYoVVXh4uGrWrFlWcsNrbTjvvPOOatq06WMf1+v1ysvLS02fPj3rvsTERGVvb68WLVpUECGajfbt26tXX301x31du3ZVvXv3VkrxWhvSv5ObvFzbEydOKABq3759WeesXbtW6XQ69ddffz1VPJyWekr37t3DgQMHEBwcnHWflZUVgoODERkZqWFk5icpKQkA4O7uDgA4cOAA0tPTc1z7ypUrw9fXl9f+CQ0bNgzt27fPcU0BXmtD+uOPP1CvXj1069YNHh4eqF27Nr7//vusx2NiYhAfH5/jWru6uqJhw4a81vnUuHFjbNq0CWfOnAEAHD58GDt27EDbtm0B8FobU16ubWRkJNzc3FCvXr2sc4KDg2FlZYU9e/Y81ftbXONMQ/v777+RkZEBT0/PHPd7enri1KlTGkVlfvR6PUaPHo0mTZqgWrVqAID4+HjY2dnBzc0tx7menp6Ij4/XIMrCbfHixTh48CD27dv30GO81oZz/vx5zJ49G2PGjMH48eOxb98+jBw5EnZ2dujbt2/W9XzUf1N4rfPn3XffRXJyMipXrgxra2tkZGTg448/Ru/evQGA19qI8nJt4+Pj4eHhkeNxGxsbuLu7P/X1Z3JDhcKwYcNw7Ngx7NixQ+tQzNKlS5cwatQohIeHw8HBQetwzJper0e9evUwdepUAEDt2rVx7NgxzJkzB3379tU4OvPy22+/YcGCBVi4cCGqVq2KqKgojB49GqVKleK1NnOclnpKJUqUgLW19UO7Rq5evQovLy+NojIvw4cPx6pVq7BlyxaUKVMm634vLy/cu3cPiYmJOc7ntc+/AwcOICEhAXXq1IGNjQ1sbGywdetWzJw5EzY2NvD09OS1NhBvb29UqVIlx32BgYGIjY0FgKzryf+mPL23334b7777Ll566SVUr14dr7zyCt544w2EhoYC4LU2prxcWy8vLyQkJOR4/P79+7hx48ZTX38mN0/Jzs4OdevWxaZNm7Lu0+v12LRpE4KCgjSMrPBTSmH48OFYvnw5Nm/eDD8/vxyP161bF7a2tjmu/enTpxEbG8trn08tW7bE0aNHERUVlXXUq1cPvXv3zvqZ19owmjRp8lBJgzNnzqBs2bIAAD8/P3h5eeW41snJydizZw+vdT7dvn0bVlY5v+asra2h1+sB8FobU16ubVBQEBITE3HgwIGsczZv3gy9Xo+GDRs+XQBPtRyZlFKyFdze3l6FhYWpEydOqMGDBys3NzcVHx+vdWiF2uuvv65cXV1VRESEiouLyzpu376ddc6QIUOUr6+v2rx5s9q/f78KCgpSQUFBGkZtPh7cLaUUr7Wh7N27V9nY2KiPP/5YRUdHqwULFqgiRYqoX3/9NeucadOmKTc3N7Vy5Up15MgR1blzZ25PfgJ9+/ZVpUuXztoK/vvvv6sSJUqosWPHZp3Da/3kUlJS1KFDh9ShQ4cUAPX555+rQ4cOqYsXLyql8nZt27Rpo2rXrq327NmjduzYoSpWrMit4Kbk66+/Vr6+vsrOzk41aNBA7d69W+uQCj0AjzzmzZuXdc6dO3fU0KFDVbFixVSRIkXU888/r+Li4rQL2oz8O7nhtTacP//8U1WrVk3Z29urypUrq++++y7H43q9Xk2YMEF5enoqe3t71bJlS3X69GmNoi28kpOT1ahRo5Svr69ycHBQ5cuXV++9955KS0vLOofX+slt2bLlkf+N7tu3r1Iqb9f2+vXrqmfPnsrJyUm5uLio/v37q5SUlKeOTafUA6UaiYiIiAo5rrkhIiIis8LkhoiIiMwKkxsiIiIyK0xuiIiIyKwwuSEiIiKzwuSGiIiIzAqTGyIiIjIrTG6IiIjIrDC5ISKLFxERAZ1O91BjUCIqnJjcEBERkVlhckNERERmhckNEWlOr9cjNDQUfn5+cHR0RM2aNbF06VIA2VNGq1evRo0aNeDg4IBGjRrh2LFjOV5j2bJlqFq1Kuzt7VGuXDnMmDEjx+NpaWl455134OPjA3t7e/j7++PHH3/Mcc6BAwdQr149FClSBI0bN8bp06eN+4sTkVEwuSEizYWGhuLnn3/GnDlzcPz4cbzxxht4+eWXsXXr1qxz3n77bcyYMQP79u1DyZIl0bFjR6SnpwOQpKR79+546aWXcPToUUyaNAkTJkxAWFhY1vP79OmDRYsWYebMmTh58iTmzp0LJyenHHG89957mDFjBvbv3w8bGxu8+uqrBfL7E5FhsSs4EWkqLS0N7u7u2LhxI4KCgrLuHzhwIG7fvo3BgwejRYsWWLx4MXr06AEAuHHjBsqUKYOwsDB0794dvXv3xrVr17Bhw4as548dOxarV6/G8ePHcebMGQQEBCA8PBzBwcEPxRAREYEWLVpg48aNaNmyJQBgzZo1aN++Pe7cuQMHBwcjXwUiMiSO3BCRps6ePYvbt2+jVatWcHJyyjp+/vlnnDt3Luu8BxMfd3d3BAQE4OTJkwCAkydPokmTJjlet0mTJoiOjkZGRgaioqJgbW2NZs2a5RpLjRo1sn729vYGACQkJDz170hEBctG6wCIyLKlpqYCAFavXo3SpUvneMze3j5HgvOkHB0d83Sera1t1s86nQ6ArAciosKFIzdEpKkqVarA3t4esbGx8Pf3z3H4+Phknbd79+6sn2/evIkzZ84gMDAQABAYGIidO3fmeN2dO3eiUqVKsLa2RvXq1aHX63Os4SEi88WRGyLSlLOzM9566y288cYb0Ov1aNq0KZKSkrBz5064uLigbNmyAIAPP/wQxYsXh6enJ9577z2UKFECXbp0AQC8+eabqF+/PqZMmYIePXogMjIS33zzDb799lsAQLly5dC3b1+8+uqrmDlzJmrWrImLFy8iISEB3bt31+pXJyIjYXJDRJqbMmUKSpYsidDQUJw/fx5ubm6oU6cOxo8fnzUtNG3aNIwaNQrR0dGoVasW/vzzT9jZ2QEA6tSpg99++w0TJ07ElClT4O3tjQ8//BD9+vXLeo/Zs2dj/PjxGDp0KK5fvw5fX1+MHz9ei1+XiIyMu6WIyKRl7mS6efMm3NzctA6HiAoBrrkhIiIis8LkhoiIiMwKp6WIiIjIrHDkhoiIiMwKkxsiIiIyK0xuiIiIyKwwuSEiIiKzwuSGiIiIzAqTGyIiIjIrTG6IiIjIrDC5ISIiIrPyf8CjdLpEEAwHAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "plot_losses_together(rnn_sh_losses[1:], rnn_sh_finetune_losses)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGI_uZZhnpdF"
      },
      "source": [
        "### LSTM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "liBF6jxUnPpO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "outputId": "db357da3-e739-4847-cfad-2f3c4bce63e0"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGwCAYAAACHJU4LAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABPvklEQVR4nO3dd3wT5eMH8M8lbdOWLgqUttBCZZYpyLAUBWUjCKLyE0GGCCIgS1EBQZBRHChfhogLEBFQEVRkiGgBkb0UqWUVWoUCMjroTp7fHw9JmtKRtkkubT/v1+ted00ud0+Oknz6rFOEEAJERERETkijdgGIiIiICsKgQkRERE6LQYWIiIicFoMKEREROS0GFSIiInJaDCpERETktBhUiIiIyGm5qF2A0jAYDLh06RK8vb2hKIraxSEiIiIrCCGQkpKC4OBgaDSF15mU6aBy6dIlhISEqF0MIiIiKoGEhATUrFmz0H3KdFDx9vYGIN+oj4+PyqUhIiIiayQnJyMkJMT0PV6YMh1UjM09Pj4+DCpERERljDXdNtiZloiIiJwWgwoRERE5LQYVIiIiclpluo8KEREVn16vR3Z2ttrFoHLM1dUVWq3WJsdiUCEiqiCEEEhMTMStW7fULgpVAH5+fggMDCz1PGcMKkREFYQxpAQEBMDT05MTZZJdCCGQlpaGq1evAgCCgoJKdTwGFSKiCkCv15tCSpUqVdQuDpVzHh4eAICrV68iICCgVM1A7ExLRFQBGPukeHp6qlwSqiiMv2ul7Q/FoEJEVIGwuYccxVa/awwqRERE5LQYVIiIiMhpMagQEZFT69ixIyZMmKB2MfI1dOhQ9O3bV+1ilGsc9ZOftDTgv/8AFxcgOFjt0hAREVVYrFHJz4YNQK1awNChapeEiIioQmNQyc+d8d9IT1e3HEREdiQEcPu2OosQxStrTk4Oxo4dC19fX1StWhXTp0+HuHOQ1atXo1WrVvD29kZgYCCefvpp02RjAHDz5k0MHDgQ1apVg4eHB+rVq4cVK1aYnk9ISED//v3h5+cHf39/9OnTBxcuXCjRNc3MzMS4ceMQEBAAd3d3tG/fHocOHbKqLFlZWRg7diyCgoLg7u6OWrVqISoqyvTaW7du4bnnnkO1atXg4+ODhx9+GCdOnDA9f+LECTz00EPw9vaGj48P7rvvPhw+fLhE78OZsOknP+7ucp2RoW45iIjsKC0N8PJS59ypqUClStbvv2rVKgwfPhwHDx7E4cOHMXLkSISGhmLEiBHIzs7G7Nmz0aBBA1y9ehWTJk3C0KFDsWXLFgDA9OnTcerUKWzduhVVq1bF2bNnkX7nD9Hs7Gx069YNERER2LNnD1xcXDBnzhx0794df/zxB9zc3Ir1vl555RVs2LABq1atQq1atfD222+jW7duOHv2LPz9/Qsty6JFi/D999/jq6++QmhoKBISEpCQkGA69pNPPgkPDw9s3boVvr6+WL58OTp16oTTp0/D398fAwcORIsWLbBs2TJotVocP34crq6uxSq/UxJlWFJSkgAgkpKSbHvgHTuEAIRo0sS2xyUiUkl6ero4deqUSE9PNz2Wmio/6tRYUlOtL3uHDh1EeHi4MBgMpsdeffVVER4enu/+hw4dEgBESkqKEEKI3r17i2HDhuW77+rVq0WDBg0sjp2ZmSk8PDzE9u3biyzbkCFDRJ8+fYQQQqSmpgpXV1exZs0a0/NZWVkiODhYvP3220WW5cUXXxQPP/ywRVmM9uzZI3x8fERGRobF43Xq1BHLly8XQgjh7e0tVq5cWWSZHSW/3zmj4nx/s0YlP2z6IaIKwNNT1myode7iuP/++y0mEIuIiMCCBQug1+tx/PhxzJw5EydOnMDNmzdhMBgAAPHx8WjUqBFeeOEFPP744zh69Ci6du2Kvn37ol27dgBkc8nZs2fh7e1tcb6MjAycO3euWGU8d+4csrOzERkZaXrM1dUVbdq0QUxMDAAUWpahQ4eiS5cuaNCgAbp3745evXqha9eupnKmpqbedfuD9PR0UzknTZqE5557DqtXr0bnzp3x5JNPok6dOsV6D86IQSU/bPohogpAUYrX/OKMMjIy0K1bN3Tr1g1r1qxBtWrVEB8fj27duiErKwsA0KNHD1y8eBFbtmzBjh070KlTJ4wZMwbvvvsuUlNTcd9992HNmjV3HbtatWo2L29hZWnZsiXi4uKwdetW/Pzzz+jfvz86d+6Mb775BqmpqQgKCkJ0dPRdx/Tz8wMAzJw5E08//TR+/PFHbN26FW+88QbWrVuHxx57zObvw6HsUd3jKHZr+vnrL1k36e9v2+MSEamksGp4Z9ehQwfRqFEji8dee+01ER4eLg4fPiwAiPj4eNNzq1evFgDEsWPH8j3ehx9+KLy9vYUQQnz00UeicuXKJf4eydv04+bmdlfTT40aNcQ777xTZFny2rZtmwAgrl+/Ln766Seh1WpFXFyc1WV76qmnRO/eva3e39Zs1fTDUT/5YY0KEZFTiY+Px6RJkxAbG4u1a9di8eLFGD9+PEJDQ+Hm5obFixfj/Pnz+P777zF79myL186YMQPfffcdzp49i7/++gubN29GeHg4AGDgwIGoWrUq+vTpgz179iAuLg7R0dEYN24c/vnnn2KVsVKlSnjhhRcwefJkbNu2DadOncKIESOQlpaG4cOHF1mW9957D2vXrsXff/+N06dP4+uvv0ZgYCD8/PzQuXNnREREoG/fvvjpp59w4cIF/P7775g2bRoOHz6M9PR0jB07FtHR0bh48SL27t2LQ4cOmY5dlrHpJz+5+6gIIetHiYhINYMHD0Z6ejratGkDrVaL8ePHY+TIkVAUBStXrsTUqVOxaNEitGzZEu+++y4effRR02vd3NwwZcoUXLhwAR4eHnjggQewbt06APIOv7t378arr76Kfv36ISUlBTVq1ECnTp3g4+NT7HLOnz8fBoMBzzzzDFJSUtCqVSts374dlStXLrIs3t7eePvtt3HmzBlotVq0bt0aW7ZsgUYj6xS2bNmCadOmYdiwYbh27RoCAwPx4IMPonr16tBqtbh+/ToGDx6MK1euoGrVqujXrx9mzZpV2kuvOkWI4o5mdx7Jycnw9fVFUlJSiX6hCnTzJuDvL7czM4FiDk8jInI2GRkZiIuLQ1hYGNyNtcZEdlTY71xxvr/Z9JOf3BeUzT9ERESqYVDJT+6gwiHKREQVlpeXV4HLnj171C5ehcA+KvlRFECnk80+rFEhIqqwjh8/XuBzNWrUcFxBKjAGlYJ4eMigwhoVIqIKq27dumoXocJj009BOESZiIhIdQwqBTEOUWZQISIiUg2DSkGMNSps+iEiIlKN6kHl33//xaBBg1ClShV4eHigadOmOHz4sNrFYtMPERGRE1C1M+3NmzcRGRmJhx56CFu3bkW1atVw5swZ0wx+quIdlImIiFSnao3KW2+9hZCQEKxYsQJt2rRBWFgYunbt6hy3pWaNChGRUxBCYOTIkfD394eiKPDz88OECRPULpZDXLhwAYqiFDpMurxTtUbl+++/R7du3fDkk09i165dqFGjBkaPHo0RI0bku39mZiYyMzNNPycnJ9uvcKxRISJyCtu2bcPKlSsRHR2Ne+65BxqNBh7Gz2gbGTp0KG7duoVNmzbZ9LhUeqrWqJw/fx7Lli1DvXr1sH37drzwwgsYN24cVq1ale/+UVFR8PX1NS0hISH2KxxrVIiInMK5c+cQFBSEdu3aITAwEAEBAfD29la7WOQgqgYVg8GAli1bYt68eWjRogVGjhyJESNG4MMPP8x3/ylTpiApKcm0JCQk2K9wDCpEVN4JAdy+rc5i5f1whw4dihdffBHx8fFQFAW1a9dGx44dLZp+ateujXnz5uHZZ5+Ft7c3QkND8dFHH1kcJyEhAf3794efnx/8/f3Rp08fXLhwAQAwc+ZMrFq1Ct999x0URYGiKIiOjkZ0dDQURcGtW7dMxzl+/DgURTG9duXKlfDz88P27dsRHh4OLy8vdO/eHZcvX7Y4/yeffILw8HC4u7ujYcOG+OCDD4r9z2W0a9cutGnTBjqdDkFBQXjttdeQk5Njev6bb75B06ZN4eHhgSpVqqBz5864ffs2ACA6Ohpt2rRBpUqV4Ofnh8jISFy8eNH02u+++w4tW7aEu7s77rnnHsyaNct0bCEEZs6cidDQUOh0OgQHB2PcuHElfh/WUjWoBAUFoVGjRhaPhYeHIz4+Pt/9dTodfHx8LBa7YdMPEZV3aWmAl5c6S1qaVUX83//+hzfffBM1a9bE5cuXcejQoXz3W7BgAVq1aoVjx45h9OjReOGFFxAbGwsAyM7ORrdu3eDt7Y09e/Zg7969pkCRlZWFl19+Gf379zcFjMuXL6Ndu3bFuIxpePfdd7F69Wrs3r0b8fHxePnll03Pr1mzBjNmzMDcuXMRExODefPmYfr06QW2HhTm33//Rc+ePdG6dWucOHECy5Ytw6effoo5c+YAAC5fvowBAwbg2WefRUxMDKKjo9GvXz8IIZCTk4O+ffuiQ4cO+OOPP7Bv3z6MHDkSiqIAAPbs2YPBgwdj/PjxOHXqFJYvX46VK1di7ty5AIANGzbg/fffx/Lly3HmzBls2rQJTZs2LfZ7KDahogEDBoj27dtbPDZhwgQRERFh1euTkpIEAJGUlGT7wo0dKwQgxLRptj82EZGDpaeni1OnTon09HTzg6mp8nNOjSU11eqyv//++6JWrVqmnzt06CDGjx9v+rlWrVpi0KBBpp8NBoMICAgQy5YtE0IIsXr1atGgQQNhMBhM+2RmZgoPDw+xfft2IYQQQ4YMEX369LE476+//ioAiJs3b5oeO3bsmAAg4uLihBBCrFixQgAQZ8+eNe2zdOlSUb16ddPPderUEV9++aXFsWfPnm3Vd11cXJwAII4dOyaEEGLq1Kl3vZelS5cKLy8vodfrxZEjRwQAceHChbuOdf36dQFAREdH53uuTp06iXnz5lk8tnr1ahEUFCSEEGLBggWifv36Iisrq8hyC1HA79wdxfn+VrUz7cSJE9GuXTvMmzcP/fv3x8GDB/HRRx/dVWWnCtaoEFF55+kJpKaqd24batasmWlbURQEBgbi6tWrAIATJ07g7Nmzd/VrycjIwLlz50p9bk9PT4vRqkFBQaZz3759G+fOncPw4cMtBork5OTA19e32OeKiYlBRESEqRYEACIjI5Gamop//vkHzZs3R6dOndC0aVN069YNXbt2xRNPPIHKlSvD398fQ4cORbdu3dClSxd07twZ/fv3R1BQEAB5nfbu3WuqQQEAvV6PjIwMpKWl4cknn8TChQtxzz33oHv37ujZsyd69+4NFxf7RglVg0rr1q2xceNGTJkyBW+++SbCwsKwcOFCDBw4UM1iSeyjQkTlnaIAlSqpXQqbcHV1tfhZURQYDAYAQGpqKu677z6sWbPmrtdVq1atwGNqNLJ3hMjVnyY7O9uqcxtfk3onCH788cdo27atxX5arbbAc5eUVqvFjh078Pvvv+Onn37C4sWLMW3aNBw4cABhYWFYsWIFxo0bh23btmH9+vV4/fXXsWPHDtx///1ITU3FrFmz0K9fv7uO6+7ujpCQEMTGxuLnn3/Gjh07MHr0aLzzzjvYtWvXXdfAllS/e3KvXr3Qq1cvtYtxN97rh4ioXGjZsiXWr1+PgICAAvs2urm5Qa/XWzxmDDGXL182TURa3PlMqlevjuDgYJw/f94mf4SHh4djw4YNEEKYalX27t0Lb29v1KxZE4AMSpGRkYiMjMSMGTNQq1YtbNy4EZMmTQIAtGjRAi1atMCUKVMQERGBL7/8Evfffz9atmyJ2NjYQu8Y7eHhgd69e6N3794YM2YMGjZsiD///BMtW7Ys9XsriOpBxWnxXj9EROXCwIED8c4776BPnz6mjrkXL17Et99+i1deeQU1a9ZE7dq1sX37dsTGxqJKlSrw9fVF3bp1ERISgpkzZ2Lu3Lk4ffo0FixYUOzzz5o1C+PGjYOvry+6d++OzMxMHD58GDdv3jSFB2uNHj0aCxcuxIsvvoixY8ciNjYWb7zxBiZNmgSNRoMDBw5g586d6Nq1KwICAnDgwAFcu3YN4eHhiIuLw0cffYRHH30UwcHBiI2NxZkzZzB48GAAwIwZM9CrVy+EhobiiSeegEajwYkTJ3Dy5EnMmTMHK1euhF6vR9u2beHp6YkvvvgCHh4eqFWrVrGvSXGofq8fp8WmHyKicsHT0xO7d+9GaGgo+vXrh/DwcAwfPhwZGRmmGpYRI0agQYMGaNWqFapVq4a9e/fC1dUVa9euxd9//41mzZrhrbfeMo2uKY7nnnsOn3zyCVasWIGmTZuiQ4cOWLlyJcLCwop9rBo1amDLli04ePAgmjdvjlGjRmH48OF4/fXXAQA+Pj7YvXs3evbsifr16+P111/HggUL0KNHD3h6euLvv//G448/jvr162PkyJEYM2YMnn/+eQBAt27dsHnzZvz0009o3bo17r//frz//vumIOLn54ePP/4YkZGRaNasGX7++Wf88MMPqFKlSrHfR3EoInfjWxmTnJwMX19fJCUl2X6o8sqVwLBhQPfuwNattj02EZGDZWRkIC4uDmFhYXA3/iFGZEeF/c4V5/ubNSoFYY0KERGR6hhUCsLhyUREZGfz5s2Dl5dXvkuPHj3ULp5TYGfagrBGhYiI7GzUqFHo379/vs/Z+saLZRWDSkEYVIiIyM78/f3h7++vdjGcGpt+CsKmHyIqh4yToBHZm61+11ijUhDWqBBROeLm5gaNRoNLly6hWrVqcHNzs5iGnchWhBDIysrCtWvXoNFo4ObmVqrjMagUhDUqRFSOaDQahIWF4fLly7h06ZLaxaEKwNPTE6GhoaZbEZQUg0pBWKNCROWMm5sbQkNDkZOTc9d08US2pNVq4eLiYpNaOwaVghhrVLKzAb0esMPNo4iIHE1RFLi6utr1JnJEtsTOtAXJPYsea1WIiIhUwaBSEAYVIiIi1TGoFMTFRS4AO9QSERGphEGlMOxQS0REpCoGlcIYO9QyqBAREamCQaUwxhoVNv0QERGpgkGlMGz6ISIiUhWDSmE4Oy0REZGqGFQKwxoVIiIiVTGoFIY1KkRERKpiUCkMa1SIiIhUxaBSGA5PJiIiUhWDSmE4PJmIiEhVDCqFYdMPERGRqhhUCsPOtERERKpiUCkMa1SIiIhUxaBSGHamJSIiUhWDSmHYmZaIiEhVDCqFYY0KERGRqhhUCsMaFSIiIlUxqBSGnWmJiIhUxaBSGA5PJiIiUhWDSmFYo0JERKQqBpXCsDMtERGRqhhUCsPOtERERKpiUCkMm36IiIhUxaBSGHamJSIiUhWDSmFYo0JERKQqBpXCsDMtERGRqhhUCpO7M60Q6paFiIioAlI1qMycOROKolgsDRs2VLNIlow1KgCQlaVeOYiIiCooF7UL0LhxY/z888+mn11cVC+SmbFGBZC1KjqdemUhIiKqgFRPBS4uLggMDFS7GPlzcwMURTb7sJ8KERGRw6neR+XMmTMIDg7GPffcg4EDByI+Pr7AfTMzM5GcnGyx2JWicNI3IiIiFakaVNq2bYuVK1di27ZtWLZsGeLi4vDAAw8gJSUl3/2joqLg6+trWkJCQuxfSA5RJiIiUo0ihPMMZ7l16xZq1aqF9957D8OHD7/r+czMTGRmZpp+Tk5ORkhICJKSkuDj42OfQtWoAVy6BBw9CrRoYZ9zEBERVSDJycnw9fW16vtb9T4qufn5+aF+/fo4e/Zsvs/rdDroHN2hlU0/REREqlG9j0puqampOHfuHIKCgtQuihmbfoiIiFSjalB5+eWXsWvXLly4cAG///47HnvsMWi1WgwYMEDNYlni/X6IiIhUo2rTzz///IMBAwbg+vXrqFatGtq3b4/9+/ejWrVqahbLEmtUiIiIVKNqUFm3bp2ap7cO7/dDRESkGqfqo+KU2JmWiIhINQwqRWGNChERkWoYVIrCGhUiIiLVMKgUhZ1piYiIVMOgUhQOTyYiIlINg0pRWKNCRESkGgaVorAzLRERkWoYVIrCzrRERESqYVApCmtUiIiIVMOgUhTWqBAREamGQaUo7ExLRESkGgaVorDph4iISDUMKkVh0w8REZFqGFSKwhoVIiIi1TCoFIU1KkRERKphUCkKO9MSERGphkGlKLzXDxERkWoYVIrCGhUiIiLVMKgUhZ1piYiIVMOgUhRjjUpOjlyIiIjIYRhUimKsUQFYq0JERORgDCpF0enM2+xQS0RE5FAMKkXRagFXV7nNGhUiIiKHYlCxBjvUEhERqYJBxRqcnZaIiEgVDCrWYI0KERGRKhhUrMEaFSIiIlUwqFiDs9MSERGpgkHFGrzfDxERkSoYVKzBGhUiIiJVMKhYg51piYiIVMGgYg12piUiIlIFg4o1WKNCRESkCgYVa7BGhYiISBUMKtZgZ1oiIiJVMKhYg00/REREqmBQsQabfoiIiFTBoGIN1qgQERGpgkHFGqxRISIiUgWDijXYmZaIiEgVDCrW4L1+iIiIVMGgYg3WqBAREanCaYLK/PnzoSgKJkyYoHZR7sbOtERERKpwiqBy6NAhLF++HM2aNVO7KPljZ1oiIiJVqB5UUlNTMXDgQHz88ceoXLmy2sXJH2tUiIiIVKF6UBkzZgweeeQRdO7cuch9MzMzkZycbLE4BGtUiIiIVOGi5snXrVuHo0eP4tChQ1btHxUVhVmzZtm5VPlgZ1oiIiJVqFajkpCQgPHjx2PNmjVwNwaBIkyZMgVJSUmmJSEhwc6lvINNP0RERKpQrUblyJEjuHr1Klq2bGl6TK/XY/fu3ViyZAkyMzOh1WotXqPT6aDT6RxdVDb9EBERqUS1oNKpUyf8+eefFo8NGzYMDRs2xKuvvnpXSFGVsUYlMxMQAlAUdctDRERUQagWVLy9vdGkSROLxypVqoQqVarc9bjqcjdNZWSYgwsRERHZleqjfsqE3MGE/VSIiIgcRtVRP3lFR0erXYT8ubgAGg1gMDCoEBERORBrVKyhKOxQS0REpAIGFWtxiDIREZHDMahYizUqREREDsegYi3WqBARETkcg4q1WKNCRETkcAwq1vL0lOvUVHXLQUREVIEwqFgrIECur15VtxxEREQVCIOKtQID5frKFXXLQUREVIEwqFirenW5TkxUtxxEREQVCIOKtVijQkRE5HAMKtZijQoREZHDMahYy1ijwqBCRETkMAwq1mLTDxERkcMxqFjL2PSTmgrcvq1uWYiIiCoIBhVreXubp9FnrQoREZFDMKhYS1HYoZaIiMjBGFSKg/1UiIiIHIpBpThYo0JERORQDCrFwRoVIiIih2JQKQ7WqBARETkUg0pxsEaFiIjIoRhUioOz0xIRETkUg0pxGJt+WKNCRETkEAwqxZG7RkUIdctCRERUATCoFIexRiU9HUhJUbcsREREFQCDSnFUqgR4ecltNv8QERHZXYmCyqpVq/Djjz+afn7llVfg5+eHdu3a4eLFizYrnFPiEGUiIiKHKVFQmTdvHjzu3KBv3759WLp0Kd5++21UrVoVEydOtGkBnQ6HKBMRETmMS0lelJCQgLp16wIANm3ahMcffxwjR45EZGQkOnbsaMvyOR/WqBARETlMiWpUvLy8cP36dQDATz/9hC5dugAA3N3dkZ6ebrvSOSPWqBARETlMiWpUunTpgueeew4tWrTA6dOn0bNnTwDAX3/9hdq1a9uyfM6Hk74RERE5TIlqVJYuXYqIiAhcu3YNGzZsQJUqVQAAR44cwYABA2xaQKfDSd+IiIgcpkQ1Kn5+fliyZMldj8+aNavUBXJ6rFEhIiJymBLVqGzbtg2//fab6eelS5fi3nvvxdNPP42bN2/arHBOiTUqREREDlOioDJ58mQkJycDAP7880+89NJL6NmzJ+Li4jBp0iSbFtDpcBp9IiIihylR009cXBwaNWoEANiwYQN69eqFefPm4ejRo6aOteWWsUYlKwtISgL8/FQtDhERUXlWohoVNzc3pKWlAQB+/vlndO3aFQDg7+9vqmkpt9zdAV9fuc1+KkRERHZVohqV9u3bY9KkSYiMjMTBgwexfv16AMDp06dRs2ZNmxbQKVWvLmtTEhOBhg3VLg0REVG5VaIalSVLlsDFxQXffPMNli1bhho1agAAtm7diu7du9u0gE6Jk74RERE5RIlqVEJDQ7F58+a7Hn///fdLXaAygdPoExEROUSJggoA6PV6bNq0CTExMQCAxo0b49FHH4VWq7VZ4ZwWa1SIiIgcokRB5ezZs+jZsyf+/fdfNGjQAAAQFRWFkJAQ/Pjjj6hTp45NC+l0OOkbERGRQ5Soj8q4ceNQp04dJCQk4OjRozh69Cji4+MRFhaGcePGWX2cZcuWoVmzZvDx8YGPjw8iIiKwdevWkhTJsTjpGxERkUOUqEZl165d2L9/P/z9/U2PValSBfPnz0dkZKTVx6lZsybmz5+PevXqQQiBVatWoU+fPjh27BgaN25ckqI5BmtUiIiIHKJEQUWn0yElJeWux1NTU+Hm5mb1cXr37m3x89y5c7Fs2TLs37/fuYMKa1SIiIgcokRNP7169cLIkSNx4MABCCEghMD+/fsxatQoPProoyUqiF6vx7p163D79m1ERETku09mZiaSk5MtFlXk7kxrMKhTBiIiogqgREFl0aJFqFOnDiIiIuDu7g53d3e0a9cOdevWxcKFC4t1rD///BNeXl7Q6XQYNWoUNm7caJqeP6+oqCj4+vqalpCQkJIUv/QCAuQ6Jwco7zdhJCIiUpEiRMnvrHf27FnT8OTw8HDUrVu32MfIyspCfHw8kpKS8M033+CTTz7Brl278g0rmZmZyMzMNP2cnJyMkJAQJCUlwcfHp6Rv4y7R0cCSJUCzZsCMGQXsVKUKcOMGcPIk4MzNVERERE4mOTkZvr6+Vn1/Wx1UinNX5Pfee8/qffPq3Lkz6tSpg+XLlxe5b3HeaHGsXw889RTQsiVw5EgBOzVqBMTEADt3Ag8/bLNzExERlXfF+f62ujPtsWPHrNpPURRrD5kvg8FgUWuihgcekOvjx+UtfYz3ILQQGCiDCkf+EBER2Y3VQeXXX3+1+cmnTJmCHj16IDQ0FCkpKfjyyy8RHR2N7du32/xcxREcDNStC5w9C/z+O9CjRz47cYgyERGR3ZV4Cn1buHr1KgYPHozLly/D19cXzZo1w/bt29GlSxc1iwUAePBBGVR27y4gqHCIMhERkd2pGlQ+/fRTNU9fqAcfBD77TAaVfLFGhYiIyO5KNDy5InjwQbk+dAhIS8tnB9aoEBER2R2DSgFq1wZq1gSys4EDB/LZwVijcvmyI4tFRERUoTCoFEBRzLUq+Tb/1K4t1+fOcXZaIiIiO2FQKUShQaVuXUCnA27fBi5ccGSxiIiIKgwGlUIY51PZtw/IysrzpIsLEB4ut0+edGi5iIiIKgoGlUKEhwNVqwLp6QXMUNukiVwzqBAREdkFg0ohFMVcq5Jv8w+DChERkV0xqBTB2E9lz558njQGlT//dFh5iIiIKhIGlSIYg8pvvwF6fZ4njUHl77/z6cRCREREpcWgUoTmzQFvb3lzwrsqTkJD5ZM5OcCZM6qUj4iIqDxjUCmCVgu0by+37+qnoijsp0JERGRHDCpWKHQ+FQYVIiIiu2FQsULuoCJEnifZoZaIiMhuGFSs0KoV4O4OXLuWTx5hjQoREZHdMKhYwc0N6NxZbg8fDmRk5HqyaVO5Pn9eTqdPRERENsOgYqUlSwB/f+DwYWD8+FxPVKsGBATINqGYGNXKR0REVB4xqFipVi3gyy/lQJ+PPgI++yzXk2z+ISIisgsGlWLo1g148025PXo0cPTonSfYoZaIiMguGFSKaepUoFcvIDMTePxx4MYNsEaFiIjIThhUikmjAT7/HLjnHuDCBaBrV+CM+50OtQwqRERENsWgUgKVKwPffgv4+ABHjgCthzSST1y6dKeKhYiIiGyBQaWEmjcHTp0C+vcHkoQPLqAWAGDnor9gMKhcOCIionKCQaUUatQA1q8Htm8H4jxlP5UNs/5EkybAxx8D6ekqF5CIiKiMY1Cxga5dgQdekEGlpetJxMQAI0fKmyu/8QZw5YrKBSQiIiqjGFRsxKWF7FA7tPVJvPeenHflv//kcObQUGDIkFzDmYmIiMgqDCq2cmeIskvMSUycIHD2rGwWatsWyMqSI4Xuuw944AHg668BvV7l8hIREZUBDCq20qABoNUCN28Cly/DxUV2tN2/Xy5PPw24uAC//SYfv/deYOvWfO7GTERERCYMKrbi7g7Uqye388xQ27YtsGYNcPEiMH26HN588iTQs6e82eGRIyqUl4iIqAxgULGlZs3k+tChfJ8ODpZ9Vs6dAyZPBnQ64JdfgFatgMGDOQULERFRXgwqttSxo1z/8kuhu1WuDLz9NhAbCzzzjLzR4erVQNOmwI4d9i8mERFRWcGgYksPPyzXv/9u1SQqtWrJTrb798suLpcuyaHO48dzDhYiIiKAQcW26teXs8BlZgJ791r9sjZt5NDlMWPkz4sWAS1b8mbMREREDCq2pChAp05yu4jmn7w8PYElS+RIoMBA4O+/5VDmYuQdIiKicodBxdaMzT87d5bo5d27y5qUyEggKQno0gXYssWG5SMiIipDGFRszVijcvgwcOtWiQ5RtSrw009y+HJ6OtCnD/Dll7YrIhERUVnBoGJrNWvKvioGA7B7d4kP4+kJbNokJ4rLyQEGDQKWLrVdMYmIyLmsWgWMGCE/88mMQcUeStn8Y+TqKoctjx0rZ7AdOxb48EMblI+IiJzOG28An3wC7NmjdkmcC4OKPZSwQ21+NBo5CmjKFPnz6NHAunWlPiwRETmZ5GS5/vtvdcvhbBhU7OGhh+T65EngypVSH05RgLlzZUgRQk4St3VrqQ9LREROJC1NrhlULDGo2EOVKvKug4BNalUAGVYWLwYGDJDtl48/zqHLRETlhV4vp+ACGFTyYlCxFxs2/xhpNLKzlXE00COPACdO2OzwRESkktu3zdsMKpYYVOzFRh1q83J1Bb7+GmjfXs6z0q8fkJJi01MQEZGD5Q4q8fGWP1d0qgaVqKgotG7dGt7e3ggICEDfvn0RGxurZpFs58EHARcXIC5OLjbk6Qn88AMQGgqcPw+89JJND09ERA6WN5icPq1OOZyRqkFl165dGDNmDPbv348dO3YgOzsbXbt2xe3yECW9vIC2beW2DZt/jPz8ZDOQogAffwxs3mzzUxARkYMYO9IalZe/2W1B1aCybds2DB06FI0bN0bz5s2xcuVKxMfH48iRI2oWy3bs1Pxj1LEjMHGi3B4+HLh2zS6nISIiO8v79zn7qZg5VR+VpKQkAIC/v3++z2dmZiI5OdlicWrGDrU7d8qZau1g7lygcWPg6lXg+efl8GUiIipbGFQK5jRBxWAwYMKECYiMjESTJk3y3ScqKgq+vr6mJSQkxMGlLKaICNlGc/Wq3cYSu7vL2WtdXYGNG4HPP7fLaYiIyI4YVArmNEFlzJgxOHnyJNYVMu3qlClTkJSUZFoSEhIcWMIScHOTdxQEgK++sttpWrQAZs2S2y++CDj7ZSEiIkvGoBIaKtexsXariC9znCKojB07Fps3b8avv/6KmjVrFrifTqeDj4+PxeL0+veX6w0b5Iw+dvLKK7ICJyUFmDzZbqchIiI7MAaVJk3k37gZGXKYMqkcVIQQGDt2LDZu3IhffvkFYWFhahbHPjp3ls0/ly/bdSpZrRb44AM5Cmj9et7UioioLDGO+vHxAerVk9ts/pFUDSpjxozBF198gS+//BLe3t5ITExEYmIi0tPT1SyWbbm5AX37yu2vv7brqe69V94iHADGjbNrBQ4REdmQsUbF0xNo2FBuc4iypGpQWbZsGZKSktCxY0cEBQWZlvXr16tZLNt78km5/uYbu6eHOXMAX1/g+HHgs8/seioiIrIRY1CpVMkcVFijIqne9JPfMnToUDWLZXvG5p/ERLvfSbBaNWDmTLk9dSpw65ZdT0dERDbAoFIwp+hMW+45sPkHAMaMAcLDgf/+A9580+6nIyKiUsodVBo0kNsMKhKDiqM4sPnH1RV4/325vXgxEBNj19MREVEp5RdUEhNZKw4wqDiOA5t/AKBbN6B3byAnB3j1VbufjoiISsE46sfTU478CQ6WP7NDLYOK4+Ru/rHj5G+5vfOOXG/ezEngiIicWe4aFYAjf3JjUHEkB03+ZtSgAdChg7z/D6fWJyJyXgUFFfZTYVBxrE6dzM0/v/3mkFMOGybXK1fyhoVERM6KQaVgDCqO5OYGPPaY3F6xwiGnfOIJwMsLOHvWIV1jiIioBPIGFY78MWNQcbSRI+V67VrgyhW7n65SJfOAIwdlIyIiKiZjZ9q8NSpnzwLZ2eqUyVkwqDja/fcDbdsCWVnAsmUOOaWx+eerr+6+lTgREakv9xT6AFCzptzOzgbi4tQrlzNgUFHDxIlyvWyZvEWmnbVvD9SpA6Smyn68RETkPIS4u+lHo2HzjxGDihr69ZNx+epVYN06u59OUQDjXQnY/ENE5FyysswDQY1BBeAQZSMGFTW4ugJjx8rt9993yHCcwYNlYImOZjUiEZEzyd0kn19QYY0KqWPECNkA+ccfMj3YWWioHB0NAKtW2f10RERkJWNQcXWVixGbfiQGFbX4+wNDhsjthQsdcsrcc6oYDA45JRERFSHviB8jY1A5fdqx5XE2DCpqGj9ern/4QY5Bs7O+feU9JC5edEglDhERWSHviB+junXl+r//gBs3HFsmZ8KgoqYGDYCePWUflUWL7H46T0/gqafk9urVdj8dERFZIe+IHyMvL6BGDbl95oxjy+RMGFTUNmGCXH/2mUPu5/3MM3L9zTfm6kYiIlJPQUEFAOrVk2sGFVJP585AkybyN/WTT+x+ushIICxMzqmyaZPdT0dEREUoLKjUry/XFbmfCoOK2hTFPAHcokV2nytZUcy1KryjMhGR+hhUCseg4gyefhoICAASEhwydawxqOzYAVy+bPfTERFRIQoa9QMwqAAMKs7B3R0YM0ZuL1hg9wng6tYFIiLkEOW1a+16KiIiKkJBo34Ay6DigLlBnRKDirN44QVApwMOHwb27rX76QYPlms2/xARqauwpp+wMECrlftU1BpwBhVnUa2auU3mvffsfrr+/QE3N+DECTk5LhERqaOwoOLmJsMKUHGbfxhUnIlxqPKmTcC5c3Y9lb8/0KuX3OacKkRE6iksqADsp8Kg4kwaNwa6d3fYBHDGCpw1a8x37iQiIscqrDMtwLlUGFSczaRJcv3pp3afAK5nT1mzcvkysHOnXU9FREQFYI1K4RhUnE3uCeA+/NCup3Jz45T6RERqK2zUD8CgwqDibBQFmDxZbr/1lrwblR0ZR/9s2OCQGfyJiCgPa2tUzp0DcnIcUyZnwqDijAYOBO69VyaHmTPteqo2bWQFTno6a1WIiNRQVFCpWVNOt5WdDVy86LhyOQsGFWek1ZqHKH/4IXDqlN1OpShyChcAWLas4k4oRESklqKCikZj7lBbEZt/GFSc1UMPAX36yOE4L79s11MNGiT/g8TEALt32/VURESUR1GjfoCK3U+FQcWZvfMO4OoKbN0KbN9ut9P4+MjWJsDu/XeJiCiPompUAAYVclb16gFjx8rtSZPs2otq1Ci53rABuHLFbqchIqI8ihr1A7Dph5zZ9OlAlSqyn8rHH9vtNC1aAG3bys5an31mt9MQEVEuBkPxmn4q4qRvDCrOrnJl88ifGTOApCS7ncrYqXb5cs5US0TkCOnp5m1rgkp8vOVrKgIGlbLg+eeBhg3lnCrz5tntNP37y1x08aJdu8QQEdEdxmYfoPCmn6pVAT8/OTLTzreCczoMKmWBqyvw7rtye+FCIC7OLqfx8ACGDZPby5bZ5RRERJSLsdnHw0MOQy6IolTcDrUMKmVFz55Ap05AVhYwZYrdTvP883L9448Vc2IhIiJHsmbEjxGDCjk3RQEWLJDr9euBffvscpr69WUeEgJYutQupyAiojusGfFjxKBCzq95c3PbzKRJdptGduJEuf7gA+D6dbucgoiIULwalYo6RJlBpayZPVv+Ru/fD3z1lV1O0bOnHK58+zbwv//Z5RRERAQ2/ViDQaWsCQ4GXnlFbr/2GpCRYfNTKArw+utye9Eiu46IJiKq0KyZQ8XIWKNy7VrFutu9qkFl9+7d6N27N4KDg6EoCjZt2qRmccqOl14CatQALlyQo4DsoG9foFEjGVKWLLHLKYiIKrzi1Kh4ewNBQXK7Ik38pmpQuX37Npo3b46l7LVZPJUqmedTmTXLLndX1miAadPk9vvvA6mpNj8FEVGFV5ygAsgptQDgjz/sUx5npGpQ6dGjB+bMmYPHHntMzWKUTc88A3TvLpt+Bg6Uw5Zt7P/+T1Y1Xr/OmxUSEdlDcUb9AECbNnJ94IB9yuOMylQflczMTCQnJ1ssFZaiyJvyVKkCHD9unmbfhrRa85Qt775b8aZtJiKyt+LWqNx/v1zbaYYKp1SmgkpUVBR8fX1NS0hIiNpFUldQEPDRR3L7rbeA336z+SkGDQJq1ZJ3VP7kE5sfnoioQitpUPnrL6Ci/K1epoLKlClTkJSUZFoSEhLULpL6+vUDhgyRt+B85hmb/+a6usrBRYDMQnYYZEREVGEVZ9QPAAQGArVry2m0Dh2yW7GcSpkKKjqdDj4+PhYLQY4hrlVLjgKaMMHmhx82DKhZE/j3X94DiIjIlopbowKYa1X277d9eZxRmQoqVAAfH2D1atlvZcUKYONGmx5epzN3gZk7t+JUNxIR2VtxO9MCDCoOlZqaiuPHj+P48eMAgLi4OBw/fhzx8fFqFqtseuAB80RwI0cCiYk2PfyQIXJY3PXr8pZDRERUeqWtUbHTnVSciqpB5fDhw2jRogVatGgBAJg0aRJatGiBGTNmqFmssmvWLKBZM+C//4ARI2z6G+ziImtTABlUrlyx2aGJiCqskgSVFi1kTfd//wHnztmnXM5E1aDSsWNHCCHuWlauXKlmscounQ744gvAzQ3YvNnmw3Qeewxo3Vr+xzKGFiIiKrmSBBU3N6BlS7ldEZp/2EelvGna1Dxr7cSJNo3bigLMny+3P/wQOH/eZocmIqqQijvqx6gi9VNhUCmPJk4EOnaUUf2ZZ4CcHJsd+uGHga5dgexs4I03bHZYIqIKqSQ1KkDFmviNQaU80miAlSvlaKB9+8zVIDYSFSXXa9ZUrPtNEBHZWklG/QBARIRcnzhhrpUprxhUyqtatYDFi+X2G28A27fb7NAtW8r7AAlhngyOiIiKr6Q1KjVrAsHBgF4PHDli+3I5EwaV8uyZZ4Dhw+WstU89ZdP7gs+ZI0cCbd0K/PKLzQ5LRFRh5OSY7ydb3KCiKBWnnwqDSnmmKMDSpbKO8NYtoE8fm83WVrcu8MILcvuVV2QWIiIi6+VusiluUAEqTj8VBpXyTqcDNmwAatQAYmLkXQZtlCqmTwe8vWW14/r1NjkkEVGFYWz20WjkR3VxGfup7NtXvid+Y1CpCIKC5LT6Oh3www82G65TrRrw6qtye+pUIDPTJoclIqoQcvdPUZTiv75lS9kEn5gIlOd79DKoVBStW5sngJszB/j4Y5scduJE2aHrwgXggw9sckgiogqhpCN+jDw9gebN5XZ57qfCoFKRDBpkrgIZORJYsqTUh/T0BN58U27PmSO7whARUdFKOuInt4rQT4VBpaKJigJeflluv/gi8M47pT7kkCFA48bAjRs2n7KFiKjcskVQMfZT+f330pfHWTGoVDSKArz9tuwJC8ghO7Nnl6onlosL8NZbcnvhQiAurvTFJCIq70o6fX5uHTrIj/WDB206A4VTYVCpiBRFttcY7yw4Y4bsDVuKsNKzp5xePzMTGDu2fPdAJyKyBVvUqNSsKT9/AXkPtvKIQaUimzoVeO89uT1/vkwYJRy6rCiyM62bG7BlC/DttzYsJxFROWSLoAKY57RasQJITy/dsZwRg0pFN3EisGyZOWkMGVLimxg2aGDuqztunM3mliMiKpdKO+rHqHt3edeUmzeBr78ufbmcDYMKAaNGAV98AWi1cv3EE0BGRokONXUqUKcOcOmSbFEiIqL82apGRauVAzkB+XdnecOgQtLTT5snhfvuO6BXLyAlpdiHcXc3/0dZvBg4etTG5SQiKidsFVQAeVs3V1c5n8rx46U/njNhUCGz3r2BbdsALy9g50457q0E3ci7dAEGDJDdXZ5/Xt7dk4iILNli1I9R9epAv35yu7zVqjCokKWOHeXtkIOCgL/+kjPa/vhjsQ/z3nuAry9w+DBnrCUiyo8ta1QAc6faNWvKVx9BBhW6W+vW8k6DkZFAUpKsaZk9u1gjggID5dxyAPDaa8Dp03YqKxFRGWXroPLgg0B4uDzu6tW2OaYzYFCh/AUFyZqVF16Qk6LMmCEDy/nzVh/i+eeBTp1k9eYzzwDZ2XYsLxFRGWOrUT9GiiLHRgByTpXyMp8VgwoVzM1Nttt8+ql5gpTwcGDyZKtu6qPRACtXAn5+ctZE4/xyRERk+xoVABg8WAafkyeBXbtsd1w1MahQ0Z59Vg7f6doVyMoC3n0XqFtX3tSwiGqSmjXNHbvmzCnfd/gkIioOW3amNfLzkzXYADB6dPmYAI5BhazTuLEcEbRlC9CoEXD9uryp4b33yhFChXjqKTn6Wa+XN3BOTXVMkYmInJk9alQAWXsdGAjExADTptn22GpgUCHrKQrQowdw4oSsJqlaFTh1CujcGXj8ceDChQJfunQpEBICnDsHTJrkuCITETkrewWVKlWATz6R2++/D0RH2/b4jsagQsXn4iJ7bJ0+LWtVtFp5c5/wcOCNN8z/+3Lx8wNWrZJZ5+OPgQ0bHF9sIiJnYq+gAgCPPAKMGCG3hw4t28OVGVSo5CpXBhYtAo4dk/OvZGTIuzLXrSvTSJ57Bj30kOyHC8huL+X1luRERNaw9aifvBYsAMLCgIsX5W3dyioGFSq9pk3lUOavvwbuuQdITJQ3nmjeHNi82WKM3Jw5wAMPyHT/xBPlo6MXEVFxCWHfGhUA8PY212R/9hnwww/2OY+9MaiQbSiKTB6nTgELFwL+/nK7d28ZZObNA86fh6srsG4dEBAA/PEHMHas2gUnInK8rCzzHJr2CiqA/MPwpZfk9qBBwNq19juXvShClN0pYZKTk+Hr64ukpCT4+PioXRzK7dYtYP584H//s7wTc9u2wIAB+C24Pzo8FQSDQSb9YcNUKykRkcPduCE7vQJylgcXF/udKyMD6NYN2L1b/jxokBzgoObXZnG+vxlUyL5u3ZIdbdeulc1Dxj8hFAUXanfEvLinsEXXDz8eqIrmzVUtKRGRwyQkAKGhci7NzEz7ny8nRza9G++GEhYGfPEF0K6d/c+dn+J8f7Pph+zLz0/2nN2xA/j3X1nDEhEBCIHacb/iIzyPC5mBuNG2By5HrbRqxlsiorLO3v1T8nJxAWbOBPbsAWrXBuLi5L2BZs26a9yD02GNCqnjwgXgq6+Qs2Y9XP44anrY4OoGTc8esr9LRISM/RrmaSIqX44cAVq1AmrUAP75x7HnTkqS/QO/+EL+HBkpt2vXdlwZ2PRDZcqN/aexrt96dLi8Do1xyvJJHx85+22LFuYlPBxwdVWlrEREtrB7N9ChA1C/PhAbq04ZvvxS3nc2OVl+1H7wATBwoGPOzaBCZU5Skpyg6Nbekxjsug6jwrbD5+Kf+TfeurkBTZoALVvKP0natJE/M7wQURmxbZuc6LtFC3krNbVcuCA71+7dK39+4gng9ddh9z6DDCpUJt2+DTz2mOzOotMBi9/LxnORMVCOH5OTyh07Bhw/nv8Ui+7u8n/8vfcC9eqZl7AwGWyIiJzIhg0yFLRvL/uNqCknR84g8eab8p5sAPDww3KSuJ497dP6zqBCZVZmJjBgALBxo/y5d295z4qAgDs7GAyyF9jx47KR99AhuSQl5X9ARZFjAKtWNS/Vq8sbD4WGmtc1ash0RETkAJ9/DgwZIocNb9umdmmkY8eAt9+Wc3caA0v9+nIelpEjbXsuBhUq0wwGOWfclClyUqSAAGDFCpnsC3zB2bPAwYPAX3/J7TNn5GK8j7o1AgKAmjXlUqsW0LChvFN0o0ZAtWoy9BAR2cD8+fIzrl8/57v3WXw8sHixvBNKUpK85+w339j2HAwqVC788Yfs2HXypPx50CBg/HjZLcUqQgBXrwLXrgH//WdeLl+WkxjEx5vXuSely4+/vwwynp6Ah4dce3oCXl5ynurcS6VKcvHykoufn1wqV5Y91rRa83ENBlnO3I8RUbkWEwO0bi2bu9991zxzrLNJSZF/JEZEyPLaEoMKlRsZGfKvjoULzY+1aiVv3vzUUzaag0AIOU3kP/+Yl3Pn5KdJTAxw/rzF/YpKRVFkp1+93ly3CsjHvLzMAcfb2xxuKleW297e5n0qVZI/+/hYLm5ucsIErVYuQgA3b8rlxg25dnGRwatyZbn29ZX75eSYy2VcDAbz2sVFLq6u5nMIYblkZMharLQ0+SmcmWm+dsa1VivLaVxcXc3lNS63bgGXLpmXa9fMrzUuer28WVR6ujxvVpYMjz4+8j0Zr0d2tnwuO1u+Ry8v+b6Ni4+PbPZzc5NrV1fZKG98T4A8V2amPI9xcXExh1FvbxlgWetGRUhNlf3/Y2LkjVp/+sm+s9I6KwYVKncOHACWLAG++kp+5wDy+6VbN7l07Sq7m9hFejpw+rT88kxLkz8bv4hTUiyX27flJ5FxnZIi605v3uQdGMs7jcYcHo1BydVVhiPjotfLkJU78BkXYwBzcTHX2hnX7u7y+Ioi1xqNDFXu7nIfDw/5s1Zrfl6rlc/7+JjLVamS+TiAXBsM5hBnXOc+hlYr98kd0ow1kLnLJIQ5zGVmFrzk/j+Uni6Pbwzfnp4yMCYny/9vxiU7W14v46LVytfevm1esrMtr6ExELu6WoZiRTGXWVHuDsmuruZr6+4ut729zX80VK4s/32Nr8/9b5J3yf3vq9FAQMHAgcDatQLBgQJHjwhUD66YtakMKlRu/fefrIpcvlxWeuQWHi5vwNWokexeEh4uu5s4zXxxmZnyQzcry/KDUVHkh27ugJOUZPlBffOmfDxvCEpJkR/qycnmqS7z4+tr/pDNyTHXsFjTh8f4QZy7Bqgwrq7mpjGdzvxhbqTXW9ZyZGVZ1uIA8rU1agDBwXIx9hEy1vDo9fLD3/hF7e4uv4jS0szXIzlZXnPjF5TxCy41Vb5345KcLMtgXApiDBDGL6+cHPO/CZEVhKJAyfuV6+1t2dnfx+fumjljaMwdgvLu4+pq/t00Bi3j773x/0BOjvmPrNu35eeOMZzq9eYpanMfQ6eTY5WfeMKm16LMBZWlS5finXfeQWJiIpo3b47FixejTZs2Rb6OQaXiMhhkLcv27bLq9MAB822EcvPwkN9xxm4ifn7yO9v4PWr8Iy73H7S5/xDK+3889x+UOTny/7HxWLn/qM39R3Lu7+m8a6O8LSi55X5N7iX355VGA2iEHkpONhR9DkSOHoo+B1AU6Cv5QHF1sXit6dhZmdCkJkPRyjetuGihcdVaXow7L1AgAL0eGn22PI9BDwPkX4kC8sBC5246l7HMBb2v3O/NtI9ByH/I3OfNu08hn1hCmLv95LqtlMV1KrR1RgggK8t0HoNQYBB33p+Li8UxTMcxGKDJSIPmdgq0t5PlkpoEl9tJgF4P4eICoZG/TEKjlf8uOTnyGubILwnFoIdiuPNFYdBDk5kBJT0NSnoaNBlpUDIzjBdIftEZ9FCyMqFkZkBJT4eSmQ4lKxMwGKAYDPJ5vR5KRjqU2ynQpCZDk1ZIkM19Ce7UjuT9QhUuLjC4uUPo3GFwlQFUEXfOB7mvcNNBuOlgcNWZti0Xdwh3DwgPTxjcPSDcPeTvVNptKGm3oaTfhpKVBb2XL/TefsjxkotwdYNGn23xuyfcPaDXecLgUQkGz0qAiys0Qm+xiCwZhEVWFpB5J4jm/kUyGOQ11d/5/3JnrcnOhCYrA8qdtTY1GZrkm9Am3YA2+ab8P2N837mOd1cIKS8GDJCzw9lQmQoq69evx+DBg/Hhhx+ibdu2WLhwIb7++mvExsYiwDQmNX8MKmR086a85+GxY+auJWfOOP89LIgcRQM9PJAODQx3oqVc9NAiBy7Ihity4ALAmMAENDBAAwMEFOhRATtSFJu8phoYoIUeWujhghzTooEBPR/R4NPPFCiaOyn+1i1ZVWzs9J+ScvdhDQbzYuwzBlj2/8rONjevGZvfsrPNtZZZWfKvp9x/oXl63t0EaWzCy73cdx8wdKhNr1SZCipt27ZF69atsWTJEgCAwWBASEgIXnzxRbz22muFvpZBhQqTnQ1cvChr93O3oiQlWfb3TEu784dsrs8AY3N97iVvS4NWe3ff0exscw2qcV1YbYlR3tqSvPIeI2/NgbHseWtf8r7WYCj4+LmPk9/zuY9l3M5du5C79iT3vvm9t7z75e02kN9586uZylvG/GrGc7/3/Grd8vs3ydsdJG/NUO7vibzvN79rnXed+5x5z298r3nfZ97jW9SMFXA98vt9yK9VIXeXlPx+b/JeW8CyFc74/ybfGqc8Zcq7zvv7UVANWN7f+/yuad7v87wtJkX9DuUuZ+7/X3mvY97fKeP/vaLef9u2wLp1snWnoivO97eqETkrKwtHjhzBlClTTI9pNBp07twZ+/btu2v/zMxMZOaaUj05vxlKie5wdQXq1lW7FEREVBqqdjP877//oNfrUb16dYvHq1evjsTExLv2j4qKgq+vr2kJsdswDyIiInIGzjIewipTpkxBUlKSaUlISFC7SERERGRHqjb9VK1aFVqtFleuXLF4/MqVKwgMDLxrf51OBx3vx0JERFRhqFqj4ubmhvvuuw87d+40PWYwGLBz505ERESoWDIiIiJyBqqPN5s0aRKGDBmCVq1aoU2bNli4cCFu376NYcOGqV00IiIiUpnqQeX//u//cO3aNcyYMQOJiYm49957sW3btrs62BIREVHFo/o8KqXBeVSIiIjKnuJ8f5epUT9ERERUsTCoEBERkdNiUCEiIiKnxaBCRERETotBhYiIiJwWgwoRERE5LQYVIiIiclqqT/hWGsYpYJKTk1UuCREREVnL+L1tzVRuZTqopKSkAABCQkJULgkREREVV0pKCnx9fQvdp0zPTGswGHDp0iV4e3tDURSbHjs5ORkhISFISEjgrLd2xmvtOLzWjsNr7Ti81o5jq2sthEBKSgqCg4Oh0RTeC6VM16hoNBrUrFnTrufw8fHhL76D8Fo7Dq+14/BaOw6vtePY4loXVZNixM60RERE5LQYVIiIiMhpMagUQKfT4Y033oBOp1O7KOUer7Xj8Fo7Dq+14/BaO44a17pMd6YlIiKi8o01KkREROS0GFSIiIjIaTGoEBERkdNiUCEiIiKnxaCSj6VLl6J27dpwd3dH27ZtcfDgQbWLVOZFRUWhdevW8Pb2RkBAAPr27YvY2FiLfTIyMjBmzBhUqVIFXl5eePzxx3HlyhWVSlx+zJ8/H4qiYMKECabHeK1t599//8WgQYNQpUoVeHh4oGnTpjh8+LDpeSEEZsyYgaCgIHh4eKBz5844c+aMiiUum/R6PaZPn46wsDB4eHigTp06mD17tsW9YnitS2737t3o3bs3goODoSgKNm3aZPG8Ndf2xo0bGDhwIHx8fODn54fhw4cjNTW19IUTZGHdunXCzc1NfPbZZ+Kvv/4SI0aMEH5+fuLKlStqF61M69atm1ixYoU4efKkOH78uOjZs6cIDQ0Vqamppn1GjRolQkJCxM6dO8Xhw4fF/fffL9q1a6diqcu+gwcPitq1a4tmzZqJ8ePHmx7ntbaNGzduiFq1aomhQ4eKAwcOiPPnz4vt27eLs2fPmvaZP3++8PX1FZs2bRInTpwQjz76qAgLCxPp6ekqlrzsmTt3rqhSpYrYvHmziIuLE19//bXw8vIS//vf/0z78FqX3JYtW8S0adPEt99+KwCIjRs3WjxvzbXt3r27aN68udi/f7/Ys2ePqFu3rhgwYECpy8agkkebNm3EmDFjTD/r9XoRHBwsoqKiVCxV+XP16lUBQOzatUsIIcStW7eEq6ur+Prrr037xMTECABi3759ahWzTEtJSRH16tUTO3bsEB06dDAFFV5r23n11VdF+/btC3zeYDCIwMBA8c4775geu3XrltDpdGLt2rWOKGK58cgjj4hnn33W4rF+/fqJgQMHCiF4rW0pb1Cx5tqeOnVKABCHDh0y7bN161ahKIr4999/S1UeNv3kkpWVhSNHjqBz586mxzQaDTp37ox9+/apWLLyJykpCQDg7+8PADhy5Aiys7Mtrn3Dhg0RGhrKa19CY8aMwSOPPGJxTQFea1v6/vvv0apVKzz55JMICAhAixYt8PHHH5uej4uLQ2JiosW19vX1Rdu2bXmti6ldu3bYuXMnTp8+DQA4ceIEfvvtN/To0QMAr7U9WXNt9+3bBz8/P7Rq1cq0T+fOnaHRaHDgwIFSnb9M35TQ1v777z/o9XpUr17d4vHq1avj77//VqlU5Y/BYMCECRMQGRmJJk2aAAASExPh5uYGPz8/i32rV6+OxMREFUpZtq1btw5Hjx7FoUOH7nqO19p2zp8/j2XLlmHSpEmYOnUqDh06hHHjxsHNzQ1DhgwxXc/8PlN4rYvntddeQ3JyMho2bAitVgu9Xo+5c+di4MCBAMBrbUfWXNvExEQEBARYPO/i4gJ/f/9SX38GFXK4MWPG4OTJk/jtt9/ULkq5lJCQgPHjx2PHjh1wd3dXuzjlmsFgQKtWrTBv3jwAQIsWLXDy5El8+OGHGDJkiMqlK1+++uorrFmzBl9++SUaN26M48ePY8KECQgODua1LufY9JNL1apVodVq7xr9cOXKFQQGBqpUqvJl7Nix2Lx5M3799VfUrFnT9HhgYCCysrJw69Yti/157YvvyJEjuHr1Klq2bAkXFxe4uLhg165dWLRoEVxcXFC9enVeaxsJCgpCo0aNLB4LDw9HfHw8AJiuJz9TSm/y5Ml47bXX8NRTT6Fp06Z45plnMHHiRERFRQHgtbYna65tYGAgrl69avF8Tk4Obty4Uerrz6CSi5ubG+677z7s3LnT9JjBYMDOnTsRERGhYsnKPiEExo4di40bN+KXX35BWFiYxfP33XcfXF1dLa59bGws4uPjee2LqVOnTvjzzz9x/Phx09KqVSsMHDjQtM1rbRuRkZF3DbM/ffo0atWqBQAICwtDYGCgxbVOTk7GgQMHeK2LKS0tDRqN5VeWVquFwWAAwGttT9Zc24iICNy6dQtHjhwx7fPLL7/AYDCgbdu2pStAqbrilkPr1q0TOp1OrFy5Upw6dUqMHDlS+Pn5icTERLWLVqa98MILwtfXV0RHR4vLly+blrS0NNM+o0aNEqGhoeKXX34Rhw8fFhERESIiIkLFUpcfuUf9CMFrbSsHDx4ULi4uYu7cueLMmTNizZo1wtPTU3zxxRemfebPny/8/PzEd999J/744w/Rp08fDpktgSFDhogaNWqYhid/++23omrVquKVV14x7cNrXXIpKSni2LFj4tixYwKAeO+998SxY8fExYsXhRDWXdvu3buLFi1aiAMHDojffvtN1KtXj8OT7WXx4sUiNDRUuLm5iTZt2oj9+/erXaQyD0C+y4oVK0z7pKeni9GjR4vKlSsLT09P8dhjj4nLly+rV+hyJG9Q4bW2nR9++EE0adJE6HQ60bBhQ/HRRx9ZPG8wGMT06dNF9erVhU6nE506dRKxsbEqlbbsSk5OFuPHjxehoaHC3d1d3HPPPWLatGkiMzPTtA+vdcn9+uuv+X5GDxkyRAhh3bW9fv26GDBggPDy8hI+Pj5i2LBhIiUlpdRlU4TINa0fERERkRNhHxUiIiJyWgwqRERE5LQYVIiIiMhpMagQERGR02JQISIiIqfFoEJEREROi0GFiIiInBaDChERETktBhUiKleio6OhKMpdN10korKJQYWIiIicFoMKEREROS0GFSKyKYPBgKioKISFhcHDwwPNmzfHN998A8DcLPPjjz+iWbNmcHd3x/3334+TJ09aHGPDhg1o3LgxdDodateujQULFlg8n5mZiVdffRUhISHQ6XSoW7cuPv30U4t9jhw5glatWsHT0xPt2rVDbGysfd84EdkFgwoR2VRUVBQ+//xzfPjhh/jrr78wceJEDBo0CLt27TLtM3nyZCxYsACHDh1CtWrV0Lt3b2RnZwOQAaN///546qmn8Oeff2LmzJmYPn06Vq5caXr94MGDsXbtWixatAgxMTFYvnw5vLy8LMoxbdo0LFiwAIcPH4aLiwueffZZh7x/IrKxUt9/mYjojoyMDOHp6Sl+//13i8eHDx8uBgwYYLqV/Lp160zPXb9+XXh4eIj169cLIYR4+umnRZcuXSxeP3nyZNGoUSMhhBCxsbECgNixY0e+ZTCe4+effzY99uOPPwoAIj093Sbvk4gchzUqRGQzZ8+eRVpaGrp06QIvLy/T8vnnn+PcuXOm/SIiIkzb/v7+aNCgAWJiYgAAMTExiIyMtDhuZGQkzpw5A71ej+PHj0Or1aJDhw6FlqVZs2am7aCgIADA1atXS/0eicixXNQuABGVH6mpqQCAH3/8ETVq1LB4TqfTWYSVkvLw8LBqP1dXV9O2oigAZP8ZIipbWKNCRDbTqFEj6HQ6xMfHo27duhZLSEiIab/9+/ebtm/evInTp08jPDwcABAeHo69e/daHHfv3r2oX78+tFotmjZtCoPBYNHnhYjKL9aoEJHNeHt74+WXX8bEiRNhMBjQvn17JCUlYe/evfDx8UGtWrUAAG+++SaqVKmC6tWrY9q0aahatSr69u0LAHjppZfQunVrzJ49G//3f/+Hffv2YcmSJfjggw8AALVr18aQIUPw7LPPYtGiRWjevDkuXryIq1evon///mq9dSKyEwYVIrKp2bNno1q1aoiKisL58+fh5+eHli1bYurUqaaml/nz52P8+PE4c+YM7r33Xvzwww9wc3MDALRs2RJfffUVZsyYgdmzZyMoKAhvvvkmhg4dajrHsmXLMHXqVIwePRrXr19HaGgopk6dqsbbJSI7U4QQQu1CEFHFEB0djYceegg3b96En5+f2sUhojKAfVSIiIjIaTGoEBERkdNi0w8RERE5LdaoEBERkdNiUCEiIiKnxaBCRERETotBhYiIiJwWgwoRERE5LQYVIiIicloMKkREROS0GFSIiIjIaf0/fWsksstYPZsAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "plot_losses_together(lstm_sh_losses[1:], lstm_sh_finetune_losses)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_3bCGDXxOvGv"
      },
      "source": [
        "## Report"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SOiRxwHGnnag"
      },
      "source": [
        "As you can see, fine-tuning has an effect in improving the training of the main model.\n",
        "\n",
        "By analyzing the obtained results, state the advantage of finetuning after pre-training the model by public dataset, and compare its performance in different models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NAEoxdCInnW-"
      },
      "source": [
        "<font color='#73FF73'><b>Your answer : </b></font>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6mVIAlPZLjaw"
      },
      "source": [
        "----\n",
        "----"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L3RR1kMxnnUf"
      },
      "source": [
        "#3. Experiment on different datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P7BcgGRrqBqz"
      },
      "source": [
        "In the previous section, you saw the performance results of the text generation model using the Shakespeare plays dataset. In the following, you will check the results of the LSTM model on the dialogues of the `Friends series`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wa54a7knq4N6"
      },
      "source": [
        "## Load dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "id": "u2XhR0uprKNF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d31d7309-9031-4a68-d4e6-2f4a6f9684ab"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-12-06 21:26:34--  https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-09-08/friends.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 5383844 (5.1M) [text/plain]\n",
            "Saving to: ‘./data/Friends.csv’\n",
            "\n",
            "./data/Friends.csv  100%[===================>]   5.13M  --.-KB/s    in 0.08s   \n",
            "\n",
            "2023-12-06 21:26:35 (66.2 MB/s) - ‘./data/Friends.csv’ saved [5383844/5383844]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-09-08/friends.csv -O ./data/Friends.csv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0X_z5mS3q7EN"
      },
      "source": [
        "## preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "id": "7IbK2Ean-l1w",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "3c82aa17-3d39-4e23-f25f-f80ad6562cff"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text   speaker\n",
              "0  There's nothing to tell! He's just some guy I ...    Monica\n",
              "1  C'mon, you're going out with the guy! There's ...      Joey\n",
              "2  All right Joey, be nice. So does he have a hum...  Chandler\n",
              "3                           Wait, does he eat chalk?    Phoebe\n",
              "4                         (They all stare, bemused.)     Scene"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c1b59d1d-7650-4bb6-9b5f-668dc3493d0b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>speaker</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>There's nothing to tell! He's just some guy I ...</td>\n",
              "      <td>Monica</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>C'mon, you're going out with the guy! There's ...</td>\n",
              "      <td>Joey</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>All right Joey, be nice. So does he have a hum...</td>\n",
              "      <td>Chandler</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Wait, does he eat chalk?</td>\n",
              "      <td>Phoebe</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(They all stare, bemused.)</td>\n",
              "      <td>Scene</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c1b59d1d-7650-4bb6-9b5f-668dc3493d0b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c1b59d1d-7650-4bb6-9b5f-668dc3493d0b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c1b59d1d-7650-4bb6-9b5f-668dc3493d0b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a79441d5-9c35-4362-80a7-c5c6f1dd0de2\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a79441d5-9c35-4362-80a7-c5c6f1dd0de2')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a79441d5-9c35-4362-80a7-c5c6f1dd0de2 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ],
      "source": [
        "friends = pd.read_csv('./data/Friends.csv')\n",
        "friends = friends.dropna()\n",
        "friends = friends[friends['speaker'].str.contains('SCENE')==False]\n",
        "friends['speaker'] = friends['speaker'].apply(lambda sp: sp.lower().capitalize().split(' ')[0])\n",
        "friends_texts = friends.drop(['episode','season','scene','utterance'], axis='columns')\n",
        "friends_texts.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "id": "mEaZR1jS-q_W"
      },
      "outputs": [],
      "source": [
        "f = open(\"./data/fiends.txt\", \"w\")\n",
        "for i,row in friends_texts.iterrows():\n",
        "    f.write(row['speaker'] + ':\\n' + row['text'] + '\\n\\n')\n",
        "\n",
        "f.close()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {
        "id": "1SPBHV5u-62e"
      },
      "outputs": [],
      "source": [
        "fr_data_file = \"./data/fiends.txt\"\n",
        "fr_data = open(fr_data_file, 'r').read(30000)\n",
        "fr_data = remove_extraneous_characters(fr_data.lower(), chars)\n",
        "fr_data_size = len(fr_data)\n",
        "\n",
        "fr_data = list(fr_data)\n",
        "for i, ch in enumerate(fr_data):\n",
        "    fr_data[i] = char_to_ix[ch]\n",
        "\n",
        "fr_data = torch.tensor(fr_data).to(device)\n",
        "fr_data = torch.unsqueeze(fr_data, dim=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6drMR6Gtq_jj"
      },
      "source": [
        "## Train finetuned LSTM by friends dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "id": "J7ruLz2r_O3r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0542561c-28a2-4d89-e192-bf06728df22c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loss [epoch=9] = 1.0821877558466415\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "eves aveveok. f at wet lofinothed abutabo. t a:ist...\n",
            ":\n",
            "wng..\n",
            "nicef as th whe gotohisd:\n",
            "\n",
            " gut be toer jurexcaves atss:\n",
            "\n",
            "fowighooup tork anet the yofedet blecho tong an\n",
            "r ah yor kate.. ano ay.. l: ane b\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=19] = 0.27898472201236535\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "wes l me a:\n",
            "t \n",
            "y.\n",
            "\n",
            "\n",
            ". t f oeoher ng pathe s cerni w \n",
            "nd t theas:\n",
            "\n",
            "\n",
            "\n",
            "iomo who g cot be ly t s to we wo anose l:\n",
            "\n",
            ":\n",
            "\n",
            "y hap ss:\n",
            "curand\n",
            "mowangtan marhe pot t.\n",
            "\n",
            "uringungar\n",
            "gros: t blore wau nn\n",
            "il:\n",
            "\n",
            "howherou\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=29] = 0.12478361766732915\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "kak th r lo le s jum ichedyondl owa:\n",
            "rere d ifer w pr lou a:\n",
            "t w blvanot lonod t aloeore callou thas anth siss js wes me avebe s ht moy t s to incthesnour\n",
            " onoutidayoor: i theay cadono boonthealss t be\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=39] = 0.09040747567171782\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "sey\n",
            "uthi fe cane cangephout bothhan st w d toichet blotanewhes on  t t wandout yonout t whame indu\n",
            "\n",
            "\n",
            "\n",
            "im t tera issous chedoutof de:\n",
            "d stt he otowi me tss fim ngont nknit lyothiy\n",
            "\n",
            "\n",
            "o wacaca:\n",
            "do o ke to\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=49] = 0.08537082723013951\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "heag otor\n",
            "ndy\n",
            "f d st t cavee tan he:\n",
            "pan ind\n",
            "\n",
            "\n",
            "jonas l:inout k g t...\n",
            "\n",
            "pptha:\n",
            "\n",
            "\n",
            "rr:w womeas i baseyokey:\n",
            "\n",
            "\n",
            "res:\n",
            "\n",
            "\n",
            "is:\n",
            "\n",
            "\n",
            "r anebet y atuthewn is han al on..\n",
            "cury contrey\n",
            " mowhebercauri ndan y ondourorere\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=59] = 0.09813134154488504\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "i may\n",
            "\n",
            "odea he to t pl l:\n",
            "\n",
            "\n",
            "ra:\n",
            "jul:jut mon\n",
            "\n",
            "\n",
            "\n",
            "m \n",
            "i y whes:\n",
            "f ereshanid th uros lis:\n",
            "\n",
            "os hat rkin tha auros i nicalabas\n",
            "\n",
            "canengere:\n",
            "in:\n",
            "\n",
            "mploreres\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "ngeloe y t wive l g bi m.\n",
            "menge pandomouto.\n",
            "\n",
            "\n",
            "\n",
            "jac\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=69] = 0.07576250169478671\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "ndo got hererireng a wat gha:\n",
            "\n",
            "rean whelet yod mad pppreyothimoullorea and.. ne so r butnerlore:\n",
            "lesindyaro.\n",
            "mo s...\n",
            "ce:\n",
            "\n",
            "\n",
            "res wontim..\n",
            "pl:\n",
            "\n",
            "\n",
            "m mmorandi youso ke pat...\n",
            "ce:\n",
            "t wind..\n",
            "chetsche\n",
            "ls.nit s i\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=79] = 0.06472494046557957\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "th n meap au at wheas ha los:momporloreach nduney\n",
            "s f mommascot pp u m chostrellust t sig kngndoew mabauha  athasout athll:\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "rerg hat lllll:\n",
            "\n",
            "\n",
            "\n",
            "c ead thth turanth t. kindfow heal s i gerillll.caloo \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=89] = 0.07534110834690887\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "eas y ghe ts.. au wheseow bu y o wanth mad tou me.. thengin mop chth athothe pan patothithe gord vore d ctowe recatssst t at rat w whay ini ineacheanghe and\n",
            "juss..\n",
            "chedes ythere\n",
            "nmerupatour:\n",
            "\n",
            "\n",
            "rr.. au \n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
            "Loss [epoch=99] = 0.07347499369315698\n",
            "Train Sample +++++++++++++++++++++++++++++++++++++++++++++\n",
            "wetherka:\n",
            "a got itom.. ass...\n",
            "cereleyoutshh s d at ra yor d cheanggandout yo ni whosjulit rat lpa:\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "indyme\n",
            "mo indaban jumorers:y itheal ryous:\n",
            "y ithere iceplelou ah g thimooreys osh m thifu ke pouto\n",
            "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n"
          ]
        }
      ],
      "source": [
        "lstm_fr_finetune_losses = finetune_lstm(fr_data, fr_data_size, './model_fr_lstm.pth', './model_wi_lstm.pth')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "62xwUDiPrEDq"
      },
      "source": [
        "## Generating texts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "id": "26S9Um9Jnm9T",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fb4271d7-2c5f-4c7d-b42d-815de26f0fe5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "best loss 0.062849813455742\n",
            "Eaxmple of generated text --------------------------------------------------------------------------\n",
            "alabooeeto ronig yom y\n",
            "\n",
            "im f werays:\n",
            "\n",
            "\n",
            "s pa h.\n",
            "\n",
            "t peutha:\n",
            "an m:\n",
            "\n",
            "s ke. me au d al a:\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "purom th kyoh acar osthrineisow hat w t at t ppeve ghereei dl tu t ppenout bigngout theruut ynd t poongracen wh\n",
            "wherl:\n",
            "\n",
            "\n",
            "ny i sso jut g w p is han mmernghor che me... thean yo bave ive.. ghs pr kal y t r kn\n",
            "stheronisowat t at r g moalel:\n",
            "ou ai a:\n",
            "\n",
            "\n",
            "jundoel and\n",
            "\n",
            "\n",
            "d\n",
            "\n",
            "oe imu wheshresi icaweng n outind\n",
            "au t rnonind.\n",
            "ure indd t pube ha tam li le a:\n",
            "\n",
            "\n",
            "\n",
            "jup t.\n",
            "y t hetichend t au whel:\n",
            "\n",
            "m momas:\n",
            "\n",
            "\n",
            "\n",
            "rerki\n",
            "\n",
            "\n",
            "ig y pp bllechednowassst y isf inan yousouticheas jug kngucar yo butonorerery atondneng cheto jupeverthre gof keyed:\n",
            "sce preli gea:\n",
            "wesesey\n",
            "wat.\n",
            "\n",
            "ou \n",
            "\n",
            "\n",
            "yos m neeamous o h i o jusss a anoutom thi aralllllllllloolegsther yo yo bu ten sss aro.\n",
            "\n",
            "ppech cheriseyoda ki wholit ct le.. cani s i arendabascald cheres\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "g momorg pper y cheng we y wn osthrevedeeet t f mandoutok..\n",
            "\n",
            "no s hes ig..\n",
            "chet bbe shedle pis:\n",
            "me.. chet b au osondle ime y imorerele\n",
            "rrs th amu y tsathe t ve f s kinth cherinth lyourey:\n",
            "ut y th\n",
            "----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "best_model_lstm =  LSTM(vocab_size, vocab_size, 512, 3).to(device)\n",
        "best_model_lstm.load_model('./model_fr_lstm.pth')\n",
        "print(\"best loss\", min(lstm_fr_finetune_losses))\n",
        "generate_text(best_model_lstm, fr_data, fr_data_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z7s2pjFjR-l4"
      },
      "source": [
        "- As you can see, the LSTM network has been able to learn the features of different datasets in terms of sentence length and writing style and use it in text generation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BGd20eltW3C"
      },
      "source": [
        "## The output of finetuned models on different datasets on the input sample"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rp_UEVqAQuvy"
      },
      "source": [
        "- In this section, you can see the result of the text generated by models with a sample input text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {
        "id": "NS_NqaJruKQe"
      },
      "outputs": [],
      "source": [
        "input_sample_text = \"Hello, have a nice day.\\n\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {
        "id": "mWgLQM0G2r7D",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0dc4f139-1fd9-4697-90c9-32a6d5c89ee0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Eaxmple of generated text --------------------------------------------------------------------------\n",
            "\n",
            "cks ithere.\n",
            " ut lel:\n",
            "\n",
            "y y pris:\n",
            "\n",
            "moplpe ney\n",
            "\n",
            "youroul:\n",
            "\n",
            "mp ghe ly cheameroumeabusonowasss js tes:\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "best_model_lstm =  LSTM(vocab_size, vocab_size, 512, 3).to(device)\n",
        "best_model_lstm.load_model('./model_fr_lstm.pth')\n",
        "generate_text(best_model_lstm, fr_data, fr_data_size, create_input_sample_dataset(input_sample_text),100)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {
        "id": "gmFUUJCUtLKm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "85a732db-80a2-4f16-f8de-198defda0d80"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Eaxmple of generated text --------------------------------------------------------------------------\n",
            "\n",
            "meankelelloouthear n n tomor hinof y\n",
            "n\n",
            "are outilo andi t llinththrali oor hine thashen athe urvie si\n",
            "----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "best_model_lstm =  LSTM(vocab_size, vocab_size, 512, 3).to(device)\n",
        "best_model_lstm.load_model('./model_sh_finetune_lstm.pth')\n",
        "generate_text(best_model_lstm, sh_data, sh_data_size, create_input_sample_dataset(input_sample_text),100)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kT-dTrk_RQyE"
      },
      "source": [
        "## Report"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vNJdGJN1RLpz"
      },
      "source": [
        "According to the sample input and output produced by the fine-tuned model with the Shakespeare dataset and the Friends dataset, which output is more meaningful and what is the reason for this difference?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k4nBCaS2RWN_"
      },
      "source": [
        "<font color='#73FF73'><b>Your answer : </b></font>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tq_1cBKGRX8i"
      },
      "source": [
        "----\n",
        "----"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}