GPT_Transformer.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/januverma/transformers-stuff/blob/main/GPT_Transformer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Transformer model has emerged as one of the most powerful neural network architecture in past couple of years. These models have enabled the development of large language models (LLMs) aka foundation models like GPT-3 and BERT. Ealrier this month, ChatGPT took the world by storm which is a GPT model trained to generate text in response to a human provided prompt. There is hardly any area of deep learning which is not revolutionized by the transformer models. It is worthwhile to understand various components of the transformer architecture. In this notebook, we will implement a transformer encoder model to build our own GPT model using PyTorch."
      ],
      "metadata": {
        "id": "oFYeiwvXeByv"
      },
      "id": "oFYeiwvXeByv"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's start with importing relevant python libraries."
      ],
      "metadata": {
        "id": "qUPx4hHqnDHP"
      },
      "id": "qUPx4hHqnDHP"
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "972284fc",
      "metadata": {
        "id": "972284fc"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "import numpy as np\n",
        "import math\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "89dcf3da",
      "metadata": {
        "id": "89dcf3da"
      },
      "source": [
        "At a high-level, a transformer encoder layer consists of multi-head self-attention layer and a point-wise feed-forward block.  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "64c419e0",
      "metadata": {
        "id": "64c419e0"
      },
      "source": [
        "## Multi-head self-attention block\n",
        "\n",
        "Self-attention provides a mechanism for tokens at different positions in the sequence to interact with each other and computes scores which quantify the relative importance of the tokens for each other. The output of the attention module is a vector representation of each token which is defined as the weighted sum of the input token embeddings of all the tokens in the sequence. The weighting is provided by the attention scores. \n",
        "\n",
        "In general, attention can be defined for a given query and a set of key-value pairs where query, keys and values are all vectors. The output of the attention is vector which is the weighted sum of the value vectors where the weight for each value  depends on the compatibility of the corresponding key with the query. In the orginal paper, the authors adopted scaled dot-product attention which uses dot-products of the keys and the query  to compute attention weights.Let $K, Q, V$ be matrices of Keys, Queries and Values, then the attention score is computed as \n",
        "$$Attention(Q, K, V) = softmax(\\frac{QK^T}{\\sqrt{d}}) V$$ \n",
        "\n",
        "Following the original paper, we use multi-headed attention i.e. rather than only computing the attention once, the multi-head mechanism runs through the scaled dot-product attention multiple times in parallel. The output of each head is then concatenated and linearly transformed to produce single embedding for each token. Multi-headed attention has evolved to be a standard in attention models since it allows the model to jointly attend to information from different representation sub-spaces at different positions. \n",
        "$$MultiHead(Q, K, V) = Concat(Head_1, Head_2, \\ldots Head_h)$$\n",
        "$$Head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)$$\n",
        "where the projections are parameter matrices. \n",
        "\n",
        "In this tutorial, we will use PyTorch implementation of multi-head attention which comes as [`MultiheadAttention`](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html) layer. An advantage of using `MultiHeadAttention` layer is that it has support for adding masks which makes the architecture more flexible in terms on where to apply attention. There is a `key_mask` to choose tokens in the key to ignore (useful when e.g. padding is applied to sequences), and `attention_mask` which allows selection of positions to ignore (useful e.g. in autoregressive models where it is not okay to look forward or decoder models) while calculating the attention scores."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "id": "38612993",
      "metadata": {
        "id": "38612993"
      },
      "outputs": [],
      "source": [
        "class MultiHeadSelfAttention(nn.Module):\n",
        "    '''\n",
        "    Implements MHSA using the PyTorch MultiheadAttention Layer.\n",
        "    '''\n",
        "    def __init__(self, hidden_dim, num_heads, dropout):\n",
        "        '''\n",
        "        Arguments:\n",
        "            hidden_dim: Dimension of the output of the self-attention.\n",
        "            num_heads: Number of heads for the multi-head attention. \n",
        "            dropout: Dropout probability for the self-attention. If `0.0` then no dropout will be used.\n",
        "            \n",
        "        Returns:\n",
        "            A tensor of shape `num_tokens x hidden_size` containing output of the MHSA for each token.\n",
        "        '''\n",
        "        super().__init__()\n",
        "        if hidden_dim % num_heads != 0:\n",
        "            print('The hidden size {} is not a multiple of the number of heads {}'.format(hidden_dim, num_heads))\n",
        "        self.attention_layer = nn.MultiheadAttention(hidden_dim, num_heads, dropout=dropout, batch_first=True)\n",
        "    def forward(self, x, key_padding_mask=None, attention_mask=None):\n",
        "        '''\n",
        "        Arguments:\n",
        "            x: Tensor containing input token embeddings.\n",
        "            key_padding_mask: Mask indicating which elements within the input sequence to be considered as padding and ignored for the computation of self-attention scores.  \n",
        "            attention_mask: Mask indicating which relative positions are allowed to attend.  \n",
        "        '''\n",
        "        return self.attention_layer(query=x, key=x, value=x, key_padding_mask=key_padding_mask, attn_mask=attention_mask)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "51f054f3",
      "metadata": {
        "id": "51f054f3"
      },
      "source": [
        "Let's explore the working of the `MultiHeadAttention` layer a bit. Consider a random tensor called `key` which can be thought of as representing a sequence of length 5 where each element in the sequence is encoded with a 8-dimensional embedding. For example, key could be representing a sentence of 5 words where each word has 8-dimensional word embedding.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "id": "c13b87c8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c13b87c8",
        "outputId": "e852fe7e-8d16-4e31-8338-5e0aab8a4029"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 1.3666,  0.3002,  3.2257, -2.1703, -1.0573,  2.2189,  1.4424,\n",
              "          -0.7223],\n",
              "         [-0.5133, -0.2807, -0.2915,  0.3097,  2.2854, -0.4527,  2.0117,\n",
              "          -0.4278],\n",
              "         [ 1.2703,  0.4770, -0.3417,  1.5140,  1.0205, -0.0381,  0.8618,\n",
              "           1.5436],\n",
              "         [-0.5828,  1.4535,  0.6763, -0.7810, -0.0103, -0.9812,  0.4252,\n",
              "          -0.4465],\n",
              "         [-1.5193, -0.2806,  0.4602, -1.5507, -0.7799, -0.5019,  0.8581,\n",
              "           1.3342]]])"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ],
      "source": [
        "num_batches = 1\n",
        "seq_len = 5\n",
        "embed_dim = 8\n",
        "key = torch.randn(num_batches,seq_len,embed_dim)\n",
        "key"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "id": "851be9a8",
      "metadata": {
        "id": "851be9a8"
      },
      "outputs": [],
      "source": [
        "attention_hidden_size = embed_dim\n",
        "num_heads = 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "id": "744ffe70",
      "metadata": {
        "id": "744ffe70"
      },
      "outputs": [],
      "source": [
        "mhsa = MultiHeadSelfAttention(attention_hidden_size, num_heads, dropout=0.1)\n",
        "attention_output, attention_weights = mhsa(key)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cf22b493",
      "metadata": {
        "id": "cf22b493"
      },
      "source": [
        "The output of the self-attention operator produces a vector output embedding for each token in the sequence of dimension `attention_hidden_size` which are fed to the subsequent (attention or otherwise) layers. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "id": "2adeda7e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2adeda7e",
        "outputId": "12b53f23-41f8-4403-babf-e79d9010e2c1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 0.4408, -0.4671,  0.0352, -0.3119,  0.0832, -0.3746, -0.2145,\n",
              "          -0.2026],\n",
              "         [ 0.6960, -0.4267,  0.1504, -0.3524,  0.1847, -0.2938, -0.3021,\n",
              "          -0.4950],\n",
              "         [ 0.7830, -0.3058,  0.1768, -0.2435,  0.2469, -0.1573, -0.3238,\n",
              "          -0.6513],\n",
              "         [ 0.4041, -0.1362,  0.1217, -0.2706,  0.1599, -0.2396, -0.1533,\n",
              "          -0.3026],\n",
              "         [ 0.5041, -0.2543,  0.1512, -0.3539,  0.2130, -0.2330, -0.2164,\n",
              "          -0.4725]]], grad_fn=<TransposeBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ],
      "source": [
        "attention_output"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c5157856",
      "metadata": {
        "id": "c5157856"
      },
      "source": [
        "In addition, the `MultiHeadAttention` produces a tensor called `attention_weights` which contains the pair-wise attention-scores for the tokens i.e. their relative importance to each other. We get a `seq_length x seq_length` matrix."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "id": "4788d5af",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4788d5af",
        "outputId": "ebd69e59-df90-4dbe-c39b-c739a42bf23e"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[0.0857, 0.2108, 0.5293, 0.1330, 0.1523],\n",
              "         [0.2534, 0.2171, 0.3006, 0.1062, 0.2339],\n",
              "         [0.3358, 0.0000, 0.1740, 0.1888, 0.2020],\n",
              "         [0.1373, 0.0000, 0.2353, 0.2688, 0.2135],\n",
              "         [0.1969, 0.2445, 0.0000, 0.1988, 0.1941]]], grad_fn=<DivBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ],
      "source": [
        "attention_weights"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Plotting the attention weights on a grid gives high-level understanding of correlation between the tokens. In language translation tasks, such a heatmap provides a coarse word level translation or at least which words in the source sequence are most relevant for a word in the target sequence. "
      ],
      "metadata": {
        "id": "gHxDEWsW_o_p"
      },
      "id": "gHxDEWsW_o_p"
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "id": "fd6f8702",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "fd6f8702",
        "outputId": "c6382fe2-edb0-4275-da15-5949886b0052"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPUAAAD4CAYAAAA0L6C7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAJfUlEQVR4nO3dX4hchR3F8XPcxppWRYwblWw0PqRCEKqwDULeAkL8g0KftOiTECoVIgRE32qfi0hBaIOKBUURtFSsRUKNlRSr2Wi0xihEUYyVJiYEk6Y1xJw+7Dykksnemcydu/Pj+4GF3Z3lziHsN3dmdrnrJAJQxzldDwAwWkQNFEPUQDFEDRRD1EAx32vjoJcsuzCrrljexqFH79iBrhcM5LPPjnQ9YSBX/mim6wnN5UTXCxr79IvD+urQMZ/utlaiXnXFcu3466/bOPTo7fhd1wsG8vN7/tL1hIH89k+bu57Q3H8Pdr2gsZ/89LG+t/HwGyiGqIFiiBoohqiBYogaKIaogWKIGiiGqIFiiBoohqiBYogaKIaogWKIGiiGqIFiiBoohqiBYogaKKZR1LY32P7I9l7bD7Q9CsDwFoza9pSkRyXdKGmNpDtsr2l7GIDhNDlTr5W0N8knSY5LelbSbe3OAjCsJlGvkPT5KR/v633u/9jeaHvO9tyBg1+Pah+AAY3shbIkW5LMJpmdXnbhqA4LYEBNov5C0spTPp7pfQ7AItQk6h2SVtu+yva5km6X9GK7swAMa8GL+Sc5YfteSa9ImpL0RJLdrS8DMJRGf6EjycuSXm55C4AR4DfKgGKIGiiGqIFiiBoohqiBYogaKIaogWKIGiiGqIFiiBoohqiBYogaKIaogWKIGiiGqIFiiBooptFFEgZ2+J/SH37ZyqFH7rLlXS8YyMN3Xdb1hMEsvbjrBc299VzXC5o71v+KvZypgWKIGiiGqIFiiBoohqiBYogaKIaogWKIGiiGqIFiiBoohqiBYogaKIaogWKIGiiGqIFiiBoohqiBYhaM2vYTtvfbfn8cgwCcnSZn6iclbWh5B4ARWTDqJK9LOjSGLQBGgOfUQDEji9r2RttztucOHDkxqsMCGNDIok6yJclsktnpC9q58jCAhfHwGyimyY+0npH0hqSrbe+zfXf7swAMa8HHyUnuGMcQAKPBw2+gGKIGiiFqoBiiBoohaqAYogaKIWqgGKIGiiFqoBiiBoohaqAYogaKIWqgGKIGiiFqoBiiBopp5WJiJ44d14Gdn7dx6JFb/puXu54wkB9c+quuJwzm4wn6912+rOsFzS3pny5naqAYogaKIWqgGKIGiiFqoBiiBoohaqAYogaKIWqgGKIGiiFqoBiiBoohaqAYogaKIWqgGKIGiiFqoBiiBopZMGrbK21vs/2B7d22N41jGIDhNLlG2QlJm5O8bfsCSTttb03yQcvbAAxhwTN1ki+TvN17/4ikPZJWtD0MwHAGek5te5Wk6yS9eZrbNtqesz138D8ZzToAA2scte3zJT0v6b4kX3/39iRbkswmmV221KPcCGAAjaK2vUTzQT+d5IV2JwE4G01e/bakxyXtSfJw+5MAnI0mZ+p1ku6StN72rt7bTS3vAjCkBX+klWS7JJ4kAxOC3ygDiiFqoBiiBoohaqAYogaKIWqgGKIGiiFqoBiiBoohaqAYogaKIWqgGKIGiiFqoBiiBoohaqCYJtf9Hvygl16u6c33tXFofPyPrhcM5OS/DnY9obFzVl/Z9YTmTva/Yi9naqAYogaKIWqgGKIGiiFqoBiiBoohaqAYogaKIWqgGKIGiiFqoBiiBoohaqAYogaKIWqgGKIGiiFqoJgFo7Z9nu23bL9re7fth8YxDMBwmlzO6BtJ65Mctb1E0nbbf07y95a3ARjCglEniaSjvQ+X9N76XyAJQKcaPae2PWV7l6T9krYmebPdWQCG1SjqJN8muVbSjKS1tq/57tfY3mh7zvbcgUP/HvVOAA0N9Op3ksOStknacJrbtiSZTTI7ffEPR7UPwICavPo9bfui3vtLJd0g6cO2hwEYTpNXvy+X9HvbU5r/T+C5JC+1OwvAsJq8+v2epOvGsAXACPAbZUAxRA0UQ9RAMUQNFEPUQDFEDRRD1EAxRA0UQ9RAMUQNFEPUQDFEDRRD1EAxRA0UQ9RAMUQNFNPkyieDO35U2re9lUOPWv72bNcTBuKf/bHrCQM5Z/s9XU9oburcrhc0NzXV9ybO1EAxRA0UQ9RAMUQNFEPUQDFEDRRD1EAxRA0UQ9RAMUQNFEPUQDFEDRRD1EAxRA0UQ9RAMUQNFEPUQDFEDRTTOGrbU7bfsf1Sm4MAnJ1BztSbJO1pawiA0WgUte0ZSTdLeqzdOQDOVtMz9SOS7pd0st8X2N5oe8723IHD34xkHIDBLRi17Vsk7U+y80xfl2RLktkks9MXfX9kAwEMpsmZep2kW21/KulZSettP9XqKgBDWzDqJA8mmUmyStLtkl5NcmfrywAMhZ9TA8UM9Gd3krwm6bVWlgAYCc7UQDFEDRRD1EAxRA0UQ9RAMUQNFEPUQDFEDRRD1EAxRA0UQ9RAMUQNFEPUQDFEDRRD1EAxRA0U4ySjP6h9QNJnIz7sJZK+GvEx2zRJeydpqzRZe9vaemWS6dPd0ErUbbA9l2S26x1NTdLeSdoqTdbeLrby8BsohqiBYiYp6i1dDxjQJO2dpK3SZO0d+9aJeU4NoJlJOlMDaICogWImImrbG2x/ZHuv7Qe63nMmtp+wvd/2+11vWYjtlba32f7A9m7bm7re1I/t82y/Zfvd3taHut7UhO0p2+/Yfmlc97noo7Y9JelRSTdKWiPpDttrul11Rk9K2tD1iIZOSNqcZI2k6yX9YhH/234jaX2SH0u6VtIG29d3vKmJTZL2jPMOF33UktZK2pvkkyTHNf+XN2/reFNfSV6XdKjrHU0k+TLJ2733j2j+m29Ft6tOL/OO9j5c0ntb1K/y2p6RdLOkx8Z5v5MQ9QpJn5/y8T4t0m+8SWZ7laTrJL3Z7ZL+eg9ld0naL2lrkkW7tecRSfdLOjnOO52EqNEy2+dLel7SfUm+7npPP0m+TXKtpBlJa21f0/WmfmzfIml/kp3jvu9JiPoLSStP+Xim9zmMgO0lmg/66SQvdL2niSSHJW3T4n7tYp2kW21/qvmnjOttPzWOO56EqHdIWm37Ktvnav4P37/Y8aYSbFvS45L2JHm46z1nYnva9kW995dKukHSh92u6i/Jg0lmkqzS/Pfsq0nuHMd9L/qok5yQdK+kVzT/Qs5zSXZ3u6o/289IekPS1bb32b67601nsE7SXZo/i+zqvd3U9ag+Lpe0zfZ7mv+PfmuSsf2YaJLwa6JAMYv+TA1gMEQNFEPUQDFEDRRD1EAxRA0UQ9RAMf8DFLwCrJblWyQAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "attention_weights = attention_weights.squeeze()\n",
        "plt.imshow(attention_weights.detach(), cmap='YlOrBr')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Feed-forward block\n",
        "\n",
        "The output vector for each token in the sequence is fed to a neural network block consisting of feed-forward layers. "
      ],
      "metadata": {
        "id": "lNlDLuzRb3Mu"
      },
      "id": "lNlDLuzRb3Mu"
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "id": "3ef31f77",
      "metadata": {
        "id": "3ef31f77"
      },
      "outputs": [],
      "source": [
        "class FeedForward(nn.Module):\n",
        "    '''\n",
        "    Implements the feed-forward component of the transfomer model.\n",
        "    '''\n",
        "    def __init__(self, input_dim, hidden_dim, dropout=0.0):\n",
        "        '''\n",
        "        Arguments:\n",
        "            input_dim: Dimension of the token embedding, output of the MHSA layer.\n",
        "            hidden_dim: Hidden size of the Transformer that this feed-forward layer is part of.\n",
        "            dropout: Dropout probability to use for the projected activations. If `0.0` then no dropout will be used.\n",
        "        Returns:\n",
        "            A tensor of shape `num_tokens x hidden_dim` containing projections for each token.\n",
        "        '''\n",
        "        super().__init__()\n",
        "        self.activation = nn.ReLU()\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.layer_1 = nn.Linear(input_dim, hidden_dim)\n",
        "        self.layer_2 = nn.Linear(hidden_dim, input_dim)\n",
        "    def forward(self, x):\n",
        "        x = self.layer_1(x)\n",
        "        x = self.activation(x)\n",
        "        x = self.dropout(x)\n",
        "        x = self.layer_2(x)\n",
        "        return x\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "id": "e2d56368",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e2d56368",
        "outputId": "1eee2885-4c54-4017-b444-164197e7990a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 0.1737, -0.2318,  0.2350,  0.1964,  0.2446, -0.0070,  0.0463,\n",
              "          -0.2898],\n",
              "         [ 0.1651, -0.2303,  0.2282,  0.2182,  0.2526, -0.0022,  0.0394,\n",
              "          -0.2952],\n",
              "         [ 0.1665, -0.2345,  0.2275,  0.2146,  0.2520, -0.0010,  0.0427,\n",
              "          -0.2961],\n",
              "         [ 0.1863, -0.2666,  0.2316,  0.1678,  0.2380,  0.0014,  0.0725,\n",
              "          -0.2945],\n",
              "         [ 0.1715, -0.2526,  0.2236,  0.2008,  0.2501,  0.0049,  0.0569,\n",
              "          -0.3008]]], grad_fn=<ViewBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ],
      "source": [
        "ffn = FeedForward(embed_dim, hidden_dim=5)\n",
        "key = ffn(attention_output)\n",
        "key"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c3c01da0",
      "metadata": {
        "id": "c3c01da0"
      },
      "source": [
        "## Naive Transformer Layer\n",
        "\n",
        "We are now ready to put together our first version of a transformer encoder layer as a sequential model comprising of multi-head self attention layer and a point-wise feed-forward layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "id": "fd3816c1",
      "metadata": {
        "id": "fd3816c1"
      },
      "outputs": [],
      "source": [
        "class TransformerLayer(nn.Module):\n",
        "    '''\n",
        "    Implements a transformer layer as a stack of MHSA and feed-forward blocks.\n",
        "\n",
        "    Arguments:\n",
        "      hidden_dim: Hidden dimension of the transformer layers.\n",
        "      num_heads: Number of attention heads.  \n",
        "      attn_dropout: Dropout for MHSA layers. \n",
        "        ffn_dropout: Dropout for feed-forward layers.\n",
        "\n",
        "    Returns:\n",
        "        A tensor containing attention scores for each token. \n",
        "        attn_weights: A tensor of shape `num_tokens x num_tokens` containing the attention weights.\n",
        "    '''\n",
        "    def __init__(self, hidden_dim, num_heads, attn_dropout=0.0, ffn_dropout=0.0):\n",
        "        super().__init__()\n",
        "        self.attn_layer = MultiHeadSelfAttention(hidden_dim, num_heads, dropout=attn_dropout)\n",
        "        self.ffn_layer = FeedForward(hidden_dim, hidden_dim, dropout=ffn_dropout)\n",
        "    def forward(self, x, key_padding_mask=None, attention_mask=None):\n",
        "        attn_out, attn_weights = self.attn_layer(x, key_padding_mask, attention_mask)\n",
        "        out = self.ffn_layer(attn_out)\n",
        "        return out, attn_weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "id": "d0a442fa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d0a442fa",
        "outputId": "52280eb6-7242-4ea6-89e8-01f06f0f89b3"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 0.0452,  0.2929,  0.0269,  0.0169,  0.3146,  0.3038, -0.0533,\n",
              "           0.1584],\n",
              "         [ 0.0452,  0.2929,  0.0269,  0.0169,  0.3146,  0.3038, -0.0533,\n",
              "           0.1584],\n",
              "         [ 0.0452,  0.2929,  0.0269,  0.0169,  0.3146,  0.3038, -0.0533,\n",
              "           0.1584],\n",
              "         [ 0.0452,  0.2929,  0.0269,  0.0169,  0.3146,  0.3038, -0.0533,\n",
              "           0.1584],\n",
              "         [ 0.0452,  0.2929,  0.0269,  0.0169,  0.3146,  0.3038, -0.0533,\n",
              "           0.1584]]], grad_fn=<ViewBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ],
      "source": [
        "transfomer_layer = TransformerLayer(hidden_dim=embed_dim, num_heads=1)\n",
        "out, weights = transfomer_layer(key)\n",
        "out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "id": "7198fca5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7198fca5",
        "outputId": "95b9f906-bf40-46a3-a1c5-e784d2e8e177"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[0.2002, 0.1998, 0.1998, 0.2004, 0.1998],\n",
              "         [0.2002, 0.1998, 0.1998, 0.2004, 0.1998],\n",
              "         [0.2002, 0.1998, 0.1998, 0.2004, 0.1998],\n",
              "         [0.2002, 0.1998, 0.1998, 0.2004, 0.1998],\n",
              "         [0.2002, 0.1998, 0.1998, 0.2004, 0.1998]]], grad_fn=<DivBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 64
        }
      ],
      "source": [
        "weights"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "33d6fad4",
      "metadata": {
        "id": "33d6fad4"
      },
      "source": [
        "## Strategies for stable transformer models\n",
        "\n",
        "In the original paper, Attention is all you need, the authors adopted some techniques to achieve stability in the training process and allow for these models to learn complex relations. "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c7f444f7",
      "metadata": {
        "id": "c7f444f7"
      },
      "source": [
        "- ### Residual Connections\n",
        " Both the self-attention and feed-forward layers are equipped with residual connections around them. Residual connections work by adding the input of a layer to the output of the layer, thus providing an alternate path for the information to flow through the network. Residual connections were first used in ResNets to train very deep convolutional networks, and have since become standard in deep neural networks to provide a stable training. \n",
        " \n",
        " Mathematically, residual connections can simply be described as addition. \n",
        "$$X → X + MHSA(X)$$\n",
        "$$X → FFN(X)$$\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "id": "89d28cef",
      "metadata": {
        "id": "89d28cef"
      },
      "outputs": [],
      "source": [
        "class TransformerLayer(nn.Module):\n",
        "    '''\n",
        "    '''\n",
        "    def __init__(self, embed_dim, num_heads, hidden_dim, attn_dropout=0.0, ffn_dropout=0.0):\n",
        "        super().__init__()\n",
        "        self.attn_layer = MultiHeadSelfAttention(embed_dim, num_heads, dropout=attn_dropout)\n",
        "        self.ffn_layer = FeedForward(embed_dim, hidden_dim, dropout=ffn_dropout)\n",
        "    def forward(self, x, key_padding_mask=None, attention_mask=None):\n",
        "        attn_out, attn_weights = self.attn_layer(x, key_padding_mask, attention_mask)\n",
        "        x = x + attn_out\n",
        "        ffn_out = self.ffn_layer(x)\n",
        "        out = x + ffn_out\n",
        "        return out, attn_weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "id": "fa536de9",
      "metadata": {
        "id": "fa536de9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0dba90d5-9012-4e4a-d41a-5ad963d3dbeb"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 0.2494, -0.6382,  0.1762, -0.0437,  0.4116, -0.2612,  0.1003,\n",
              "          -0.1834],\n",
              "         [ 0.2402, -0.6386,  0.1717, -0.0222,  0.4177, -0.2570,  0.0897,\n",
              "          -0.1875],\n",
              "         [ 0.2415, -0.6429,  0.1715, -0.0260,  0.4168, -0.2558,  0.0923,\n",
              "          -0.1883],\n",
              "         [ 0.2620, -0.6742,  0.1768, -0.0739,  0.4030, -0.2531,  0.1223,\n",
              "          -0.1875],\n",
              "         [ 0.2463, -0.6622,  0.1701, -0.0407,  0.4137, -0.2503,  0.1030,\n",
              "          -0.1922]]], grad_fn=<AddBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 66
        }
      ],
      "source": [
        "transfomer_layer = TransformerLayer(embed_dim=embed_dim, num_heads=1, hidden_dim=embed_dim)\n",
        "out, weights = transfomer_layer(key)\n",
        "out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 67,
      "id": "c11533a3",
      "metadata": {
        "id": "c11533a3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dae48f19-b067-444a-98b2-4f82f6257bc6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[0.2002, 0.2002, 0.2002, 0.1996, 0.1998],\n",
              "         [0.2002, 0.2002, 0.2002, 0.1996, 0.1998],\n",
              "         [0.2002, 0.2002, 0.2002, 0.1996, 0.1998],\n",
              "         [0.2002, 0.2002, 0.2002, 0.1996, 0.1998],\n",
              "         [0.2002, 0.2002, 0.2002, 0.1996, 0.1998]]], grad_fn=<DivBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 67
        }
      ],
      "source": [
        "weights"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6969b964",
      "metadata": {
        "id": "6969b964"
      },
      "source": [
        "- ### Layer Normalization\n",
        "After each residual connection, a [Layer Normalization](https://arxiv.org/abs/1607.06450) has been applied to the output. This stablizes the training by decreasing the chances that the values start to go to extremes. Layer normalization achieves this by centering all the values relative to the mean. \n",
        "Mathematically, \n",
        "\n",
        "$$y = \\frac{x - \\mathrm{E}[x]}{ \\sqrt{\\mathrm{Var}[x] + \\epsilon}} * \\gamma + \\beta$$\n",
        "\n",
        "where $\\gamma$ and $\\beta$ are learnable affine parameters. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 68,
      "id": "d5439541",
      "metadata": {
        "id": "d5439541"
      },
      "outputs": [],
      "source": [
        "class TransformerLayerNorm(nn.Module):\n",
        "    '''\n",
        "    Implements LayerNorm for self-attention and feed-forward networks.\n",
        "\n",
        "    Arguments:\n",
        "        input_dim: Input dimension.\n",
        "    \n",
        "    Returns:\n",
        "        A normalized tensor of the same dimension as the input. \n",
        "    '''\n",
        "    def __init__(self, input_dim):\n",
        "        super().__init__()\n",
        "        self.layer_norm = nn.LayerNorm(input_dim)\n",
        "    def forward(self, x):\n",
        "        x = x.to(self.layer_norm.weight.dtype)\n",
        "        return self.layer_norm(x) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 69,
      "id": "e975048c",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e975048c",
        "outputId": "26444410-158f-405d-89a1-04ba4300073b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 0.6523, -1.4180,  0.9652,  0.7684,  1.0143, -0.2703,  0.0021,\n",
              "          -1.7140],\n",
              "         [ 0.5956, -1.3976,  0.9134,  0.8630,  1.0364, -0.2480, -0.0383,\n",
              "          -1.7245],\n",
              "         [ 0.6035, -1.4128,  0.9104,  0.8457,  1.0336, -0.2388, -0.0189,\n",
              "          -1.7227],\n",
              "         [ 0.7186, -1.5383,  0.9447,  0.6266,  0.9765, -0.2028,  0.1517,\n",
              "          -1.6771],\n",
              "         [ 0.6316, -1.4749,  0.8906,  0.7772,  1.0225, -0.1955,  0.0628,\n",
              "          -1.7143]]], grad_fn=<NativeLayerNormBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ],
      "source": [
        "layer_norm = TransformerLayerNorm(embed_dim)\n",
        "key = layer_norm(key)\n",
        "key"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c41c9c1e",
      "metadata": {
        "id": "c41c9c1e"
      },
      "source": [
        "This allows us to put together a transformer layer as a sequential model\n",
        "$$X → X + MHSA(X)$$\n",
        "$$X → LayerNorm(X)$$\n",
        "$$X → FFN(X)$$\n",
        "$$X → LayerNorm(X)$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 70,
      "id": "bc2f73fb",
      "metadata": {
        "id": "bc2f73fb"
      },
      "outputs": [],
      "source": [
        "class TransformerLayer(nn.Module):\n",
        "    '''\n",
        "    A transformer layer which is a sequential model consisting of self-attention, layer norm, residual connection, feed-forward projection, layer norm, residual connection. \n",
        "    \n",
        "    Arguments:\n",
        "        hidden_dim: Hidden dimension transformer layers.  \n",
        "        num_heads: Number of attention heads. \n",
        "        attn_dropout: Dropout for MHSA layers. \n",
        "        ffn_dropout: Dropout for feed-forward layers.\n",
        "    Returns:\n",
        "        A tensor containing attention scores for each token. \n",
        "        attn_weights: A tensor of shape `num_tokens x num_tokens` containing the attention weights. \n",
        "    '''\n",
        "    def __init__(self, hidden_dim, num_heads, attn_dropout=0.0, ffn_dropout=0.0):\n",
        "        super().__init__()\n",
        "        self.attn_layer = MultiHeadSelfAttention(hidden_dim, num_heads, dropout=attn_dropout)\n",
        "        self.ffn_layer = FeedForward(hidden_dim, hidden_dim, dropout=ffn_dropout)\n",
        "        self.layer_norm = TransformerLayerNorm(hidden_dim)\n",
        "    def forward(self, x, key_padding_mask=None, attention_mask=None):\n",
        "        attn_out, attn_weights = self.attn_layer(x, key_padding_mask, attention_mask)\n",
        "        x = self.layer_norm(x + attn_out)\n",
        "        ffn_out = self.ffn_layer(x)\n",
        "        x = self.layer_norm(x + ffn_out)\n",
        "        return x, attn_weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 71,
      "id": "de79594f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "de79594f",
        "outputId": "11c19ed2-01c4-4a56-cb1a-4fd3ff73f456"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[-0.0717, -1.3824,  1.0876,  0.1094,  1.7286,  0.1873, -0.3345,\n",
              "          -1.3244],\n",
              "         [-0.1299, -1.3645,  1.0411,  0.1988,  1.7466,  0.2046, -0.3658,\n",
              "          -1.3309],\n",
              "         [-0.1194, -1.3775,  1.0383,  0.1796,  1.7440,  0.2159, -0.3521,\n",
              "          -1.3288],\n",
              "         [ 0.0144, -1.4843,  1.0664, -0.0423,  1.6944,  0.2670, -0.2255,\n",
              "          -1.2901],\n",
              "         [-0.0799, -1.4297,  1.0192,  0.1028,  1.7331,  0.2673, -0.2943,\n",
              "          -1.3185]]], grad_fn=<NativeLayerNormBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 71
        }
      ],
      "source": [
        "transfomer_layer = TransformerLayer(hidden_dim=embed_dim, num_heads=1)\n",
        "out, weights = transfomer_layer(key)\n",
        "out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 72,
      "id": "b3cf78a3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b3cf78a3",
        "outputId": "99bb7fd6-c1e6-40e9-9181-b569e4d9a51e"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[0.1999, 0.2029, 0.2022, 0.1953, 0.1996],\n",
              "         [0.1998, 0.2033, 0.2025, 0.1947, 0.1997],\n",
              "         [0.1998, 0.2033, 0.2025, 0.1947, 0.1997],\n",
              "         [0.2000, 0.2026, 0.2020, 0.1958, 0.1996],\n",
              "         [0.1999, 0.2032, 0.2024, 0.1948, 0.1997]]], grad_fn=<DivBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ],
      "source": [
        "weights"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "76610706",
      "metadata": {
        "id": "76610706"
      },
      "source": [
        "#### Transformer Encoder\n",
        "Finally, a transformer encoder is a stacked block of transformer layers each with their own layer normalization and residual connection. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 75,
      "id": "e2beb243",
      "metadata": {
        "id": "e2beb243"
      },
      "outputs": [],
      "source": [
        "class TransformerEncoder(nn.Module):\n",
        "    '''\n",
        "    Transformer Encoder which is composed for a stack of TransformerLayers. \n",
        "\n",
        "    Arguments:\n",
        "        num_layers: Number of Transformer layers in the encoder. \n",
        "        hidden_dim: Hidden dimension of the transformer layers.  \n",
        "        num_heads: Number of heads. \n",
        "        attn_dropout: Dropout for MHSA layers. \n",
        "        ffn_dropout: Dropout for feed-forward layers.\n",
        "    '''\n",
        "    def __init__(self, num_layers, hidden_dim, num_heads, attn_dropout=0.0, ffn_dropout=0.0):\n",
        "        super().__init__()\n",
        "        self.layers = nn.ModuleList([TransformerLayer(hidden_dim, num_heads, attn_dropout, ffn_dropout) for _ in range(num_layers)])\n",
        "        self.attn_weights = []\n",
        "    def forward(self, x, key_padding_mask=None, attention_mask=None):\n",
        "        for layer in self.layers:\n",
        "            x, weights = layer(x, key_padding_mask, attention_mask)\n",
        "            self.attn_weights.append(weights)\n",
        "        return x\n",
        "    def get_attention_weights(self):\n",
        "        if len(self.attn_weights) != 0:\n",
        "            return self.attn_weights\n",
        "        else:\n",
        "            print(\"The model hasn't been training yet\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 76,
      "id": "1a74ab28",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1a74ab28",
        "outputId": "4ef710da-a767-4e51-a039-d81638b74ae5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 0.4970, -0.6345,  0.4218,  1.0914,  1.2542, -1.1688,  0.2631,\n",
              "          -1.7242],\n",
              "         [ 0.4404, -0.5929,  0.3503,  1.1595,  1.2712, -1.1390,  0.2465,\n",
              "          -1.7359],\n",
              "         [ 0.4536, -0.6145,  0.3589,  1.1506,  1.2662, -1.1340,  0.2542,\n",
              "          -1.7349],\n",
              "         [ 0.5825, -0.7815,  0.4754,  1.0250,  1.2125, -1.1399,  0.3278,\n",
              "          -1.7019],\n",
              "         [ 0.4975, -0.6914,  0.3781,  1.1236,  1.2443, -1.1137,  0.2922,\n",
              "          -1.7306]]], grad_fn=<NativeLayerNormBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ],
      "source": [
        "encoder = TransformerEncoder(num_layers=5, num_heads=2, hidden_dim=embed_dim)\n",
        "attn_out = encoder(key)\n",
        "attn_out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "id": "92034a41",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "92034a41",
        "outputId": "d9666755-24d1-4a86-e845-8de67827d7bf"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[tensor([[[0.1999, 0.1993, 0.1995, 0.2012, 0.2001],\n",
              "          [0.1997, 0.1995, 0.1996, 0.2009, 0.2003],\n",
              "          [0.1997, 0.1995, 0.1996, 0.2009, 0.2003],\n",
              "          [0.2001, 0.1992, 0.1994, 0.2013, 0.1999],\n",
              "          [0.1998, 0.1994, 0.1996, 0.2009, 0.2002]]], grad_fn=<DivBackward0>),\n",
              " tensor([[[0.2004, 0.2054, 0.2040, 0.1909, 0.1994],\n",
              "          [0.2004, 0.2055, 0.2040, 0.1907, 0.1993],\n",
              "          [0.2004, 0.2054, 0.2040, 0.1909, 0.1993],\n",
              "          [0.2004, 0.2045, 0.2033, 0.1923, 0.1995],\n",
              "          [0.2004, 0.2049, 0.2036, 0.1917, 0.1994]]], grad_fn=<DivBackward0>),\n",
              " tensor([[[0.2000, 0.2021, 0.2014, 0.1971, 0.1995],\n",
              "          [0.1999, 0.2023, 0.2015, 0.1967, 0.1996],\n",
              "          [0.1999, 0.2023, 0.2015, 0.1967, 0.1996],\n",
              "          [0.2002, 0.2021, 0.2013, 0.1971, 0.1993],\n",
              "          [0.2000, 0.2023, 0.2015, 0.1967, 0.1995]]], grad_fn=<DivBackward0>),\n",
              " tensor([[[0.1978, 0.2036, 0.2027, 0.1947, 0.2012],\n",
              "          [0.1978, 0.2035, 0.2026, 0.1950, 0.2012],\n",
              "          [0.1978, 0.2035, 0.2026, 0.1949, 0.2012],\n",
              "          [0.1982, 0.2038, 0.2028, 0.1944, 0.2009],\n",
              "          [0.1980, 0.2036, 0.2026, 0.1948, 0.2010]]], grad_fn=<DivBackward0>),\n",
              " tensor([[[0.1992, 0.1998, 0.1999, 0.2005, 0.2005],\n",
              "          [0.1992, 0.1998, 0.1999, 0.2006, 0.2006],\n",
              "          [0.1992, 0.1997, 0.1998, 0.2007, 0.2006],\n",
              "          [0.1994, 0.1995, 0.1997, 0.2010, 0.2003],\n",
              "          [0.1993, 0.1995, 0.1997, 0.2010, 0.2005]]], grad_fn=<DivBackward0>)]"
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ],
      "source": [
        "encoder.get_attention_weights()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "encoder"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bRF9Dbt3Babo",
        "outputId": "bc3af7a4-0c93-4bd5-e909-6fc59c264c5e"
      },
      "id": "bRF9Dbt3Babo",
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TransformerEncoder(\n",
              "  (layers): ModuleList(\n",
              "    (0): TransformerLayer(\n",
              "      (attn_layer): MultiHeadSelfAttention(\n",
              "        (attention_layer): MultiheadAttention(\n",
              "          (out_proj): NonDynamicallyQuantizableLinear(in_features=8, out_features=8, bias=True)\n",
              "        )\n",
              "      )\n",
              "      (ffn_layer): FeedForward(\n",
              "        (activation): ReLU()\n",
              "        (dropout): Dropout(p=0.0, inplace=False)\n",
              "        (layer_1): Linear(in_features=8, out_features=8, bias=True)\n",
              "        (layer_2): Linear(in_features=8, out_features=8, bias=True)\n",
              "      )\n",
              "      (layer_norm): TransformerLayerNorm(\n",
              "        (layer_norm): LayerNorm((8,), eps=1e-05, elementwise_affine=True)\n",
              "      )\n",
              "    )\n",
              "    (1): TransformerLayer(\n",
              "      (attn_layer): MultiHeadSelfAttention(\n",
              "        (attention_layer): MultiheadAttention(\n",
              "          (out_proj): NonDynamicallyQuantizableLinear(in_features=8, out_features=8, bias=True)\n",
              "        )\n",
              "      )\n",
              "      (ffn_layer): FeedForward(\n",
              "        (activation): ReLU()\n",
              "        (dropout): Dropout(p=0.0, inplace=False)\n",
              "        (layer_1): Linear(in_features=8, out_features=8, bias=True)\n",
              "        (layer_2): Linear(in_features=8, out_features=8, bias=True)\n",
              "      )\n",
              "      (layer_norm): TransformerLayerNorm(\n",
              "        (layer_norm): LayerNorm((8,), eps=1e-05, elementwise_affine=True)\n",
              "      )\n",
              "    )\n",
              "    (2): TransformerLayer(\n",
              "      (attn_layer): MultiHeadSelfAttention(\n",
              "        (attention_layer): MultiheadAttention(\n",
              "          (out_proj): NonDynamicallyQuantizableLinear(in_features=8, out_features=8, bias=True)\n",
              "        )\n",
              "      )\n",
              "      (ffn_layer): FeedForward(\n",
              "        (activation): ReLU()\n",
              "        (dropout): Dropout(p=0.0, inplace=False)\n",
              "        (layer_1): Linear(in_features=8, out_features=8, bias=True)\n",
              "        (layer_2): Linear(in_features=8, out_features=8, bias=True)\n",
              "      )\n",
              "      (layer_norm): TransformerLayerNorm(\n",
              "        (layer_norm): LayerNorm((8,), eps=1e-05, elementwise_affine=True)\n",
              "      )\n",
              "    )\n",
              "    (3): TransformerLayer(\n",
              "      (attn_layer): MultiHeadSelfAttention(\n",
              "        (attention_layer): MultiheadAttention(\n",
              "          (out_proj): NonDynamicallyQuantizableLinear(in_features=8, out_features=8, bias=True)\n",
              "        )\n",
              "      )\n",
              "      (ffn_layer): FeedForward(\n",
              "        (activation): ReLU()\n",
              "        (dropout): Dropout(p=0.0, inplace=False)\n",
              "        (layer_1): Linear(in_features=8, out_features=8, bias=True)\n",
              "        (layer_2): Linear(in_features=8, out_features=8, bias=True)\n",
              "      )\n",
              "      (layer_norm): TransformerLayerNorm(\n",
              "        (layer_norm): LayerNorm((8,), eps=1e-05, elementwise_affine=True)\n",
              "      )\n",
              "    )\n",
              "    (4): TransformerLayer(\n",
              "      (attn_layer): MultiHeadSelfAttention(\n",
              "        (attention_layer): MultiheadAttention(\n",
              "          (out_proj): NonDynamicallyQuantizableLinear(in_features=8, out_features=8, bias=True)\n",
              "        )\n",
              "      )\n",
              "      (ffn_layer): FeedForward(\n",
              "        (activation): ReLU()\n",
              "        (dropout): Dropout(p=0.0, inplace=False)\n",
              "        (layer_1): Linear(in_features=8, out_features=8, bias=True)\n",
              "        (layer_2): Linear(in_features=8, out_features=8, bias=True)\n",
              "      )\n",
              "      (layer_norm): TransformerLayerNorm(\n",
              "        (layer_norm): LayerNorm((8,), eps=1e-05, elementwise_affine=True)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bb307d55",
      "metadata": {
        "id": "bb307d55"
      },
      "source": [
        "#### Positional Encoding\n",
        "The transformer model that we have constructed is oblivious to the positions of the tokens in the sequence. The architecture is permutation-equivariant. In sequence modeling tasks, for example language understanding, position of tokens is important to interpret the input sequence. Past sequence modeling approaches like RNNs and CNNs have been crafted keeping the position in consideration. To be able to apply the transformer models to the sequence modeling tasks, it is important to have a mechanism to provide the positional information. In the original paper, Vaswani et al. proposed adding positional encoding to the input token embeddings before feeding them to the transformer model. The specific positional encoding employed are vectors of dimension same as the token embedding and are built using sine and cosine functions as\n",
        "\n",
        "$$PE_{(pos, 2i)} = sin(pos/10000^{2i/d})$$\n",
        "$$PE_{(pos, 2i+1)} = cos(pos/10000^{2i/d})$$\n",
        "\n",
        "where $pos$ is the position and $i$ is the hidden encoding dimension. Thus each dimension of the positional encoding corresponds to a sinusoidal. A motivation for this choice of encoding is that $PE_{(pos + k,)}$ can be represented as a linear function of $PE_{(pos,)}$ which might allow the model to easily attend to relative positions. \n",
        "\n",
        "\n",
        "Let's implement this positional encoding."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 79,
      "id": "b83af732",
      "metadata": {
        "id": "b83af732"
      },
      "outputs": [],
      "source": [
        "class PositionalEncoding(nn.Module):\n",
        "    '''\n",
        "    Implements the sinusoidal positional encoding for the input tokens. \n",
        "\n",
        "    Arguments:\n",
        "        embed_dim: Dimension of the positional encoding, should be the same as input token embedding. \n",
        "        dropout: Dropout probability to be used for positional encoding. \n",
        "        max_len: Maximum length of the input token sequences. \n",
        "    Returns:\n",
        "      A tensor containing positional embeddings for each token.\n",
        "    '''\n",
        "    def __init__(self, embed_dim, dropout=0.1, max_len=5000):\n",
        "        super().__init__()\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        position = torch.arange(max_len).unsqueeze(1)\n",
        "        div_term = torch.exp(torch.arange(0, embed_dim, 2) * (-math.log(10000.0)/embed_dim))\n",
        "        self.pe = torch.zeros(max_len, 1, embed_dim)\n",
        "        self.pe[:, 0, 0::2] = torch.sin(position * div_term)\n",
        "        self.pe[:, 0, 1::2] = torch.cos(position * div_term)\n",
        "    def forward(self, x):\n",
        "        x = x + self.pe[:x.size(0)]\n",
        "        return self.dropout(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 80,
      "id": "db086124",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "db086124",
        "outputId": "d3feaa86-8468-488f-f280-01b6d78620cc"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 0.7247, -0.4644,  1.0725,  1.9649,  1.1270,  0.8108,  0.0023,\n",
              "          -0.0000],\n",
              "         [ 0.6618, -0.4417,  1.0149,  2.0700,  1.1516,  0.8356, -0.0000,\n",
              "          -0.8050],\n",
              "         [ 0.6706, -0.4587,  1.0116,  2.0508,  1.1484,  0.8458, -0.0210,\n",
              "          -0.8031],\n",
              "         [ 0.7984, -0.5981,  0.0000,  1.8073,  1.0850,  0.8858,  0.0000,\n",
              "          -0.7523],\n",
              "         [ 0.7018, -0.5277,  0.9896,  1.9747,  0.0000,  0.8939,  0.0697,\n",
              "          -0.7937]]], grad_fn=<MulBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 80
        }
      ],
      "source": [
        "pos_enc = PositionalEncoding(embed_dim)\n",
        "pos_enc(key)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To see how positional embeddings work, we can visualize the values that get added to each embedding in each dimension of the embedding. Here we generate a heatmap of the values of positional encoding over positions, each row corresponds to the positional encoding in a fixed hidden dimension over all the positions."
      ],
      "metadata": {
        "id": "GTjPzIkRfbkM"
      },
      "id": "GTjPzIkRfbkM"
    },
    {
      "cell_type": "code",
      "execution_count": 81,
      "id": "d2fa6895",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "id": "d2fa6895",
        "outputId": "ac67f210-d99d-4893-8ccc-a319f9c5caab"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 576x216 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcAAAADgCAYAAACU/rbxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydeXyU1fX/33dmMpkl+74nZE9YEiCBAAHZ90UEtCqIglWqtlapWrUurctXrdWqFa1aV1AWRbayhUUggSSQQEhIIAkh+74vs888vz9mSCM/sbTWr/ptPq/XvOZ5nrudu51zz7nnuY+QJIlBDGIQgxjEIP7bIPuhCRjEIAYxiEEM4ofAoAAcxCAGMYhB/FdiUAAOYhCDGMQg/isxKAAHMYhBDGIQ/5UYFICDGMQgBjGI/0oMCsBBDGIQgxjEfyUGBeCPCEKIx4QQ731L+K1CiP3/C3R8KIR49vsu59+FEOJ2IUTmgPteIUTkD0nTfxpCiEohxPSrhE0UQlz4lrTf2n9CCEkIEf2foPPbIIR4Wgix3nEd5ugn+fdd7rVCCPG2EOKJH5qOQfxwUPzQBPyUIYSoBPwBK9AH7AHukySp99/JT5Kk5wfkHQFcApwkSbI4wjcAG74T0f8HIUmSyw9Nw/8mJEk6BsT90HT8K5AkqRr4UfWTJElrfmgaBvHDYlAD/O5Y4GDAo4AU4Hc/MD2D+AlB2DE4DwcxiB8AgxPvPwRJkuqwa4DDAIQQC4UQ54QQnUKIr4QQCZfjCiEeEULUCSF6hBAXhBDTHM/7TUbAUcd/p8N0NO4bTH/jhRAnhRBdjv/xA8K+EkI8I4TIcpSzXwjhMyB8ixCi0ZH2qBBi6LXWVQixSghRIoToEELsE0KEDwiThBBrhBBljrq/KYQQA8J/7kjbI4QoFkKMcjxPcNDc6Wi3hQPSeAshdgghuoUQuUDUFfT0m/Qc5r83hRB/d5SRI4SIGhB3pqPNu4QQ64QQR4QQd16lns5CiD8LIeodvz8LIZwdYSVCiPkD4iqEEC0D6pMmhDjuqE+BEGLyFX3znBAiC9ABVzPfJgshzjpo3SSEUDnSTxZC1A7Ib6QQIt9R302A6op6PCSEaHDUYdU31PFlIUS1EKLJYRZUDyxHCLFWCNHsyOOOq9CKEGKIoz17hBAZwMDxFuHoJ8WANnjW0Ua9Qoidjn7e4Ojnk8JuBbmcPl4IkSGEaHf0340Dwq7a58KOVx30dwshCoUQwwake3ZAPj8XQpQ7ytghhAgaEHbVcS2EiHbUu0sI0erog0H8FCBJ0uDv3/wBlcB0x3UocA54BojFbhKdATgBDwPlgBK76aoGCHKkiwCiHNdPA+sHPJcAxYDybgcyHddeQAewArsp+2bHvbcj/CvgooMWteP+hQF5rQJcAWfgz8CZAWEfAs9epc6LHHVJcJT7O+D4gHAJ2AV4AGFACzDbEbYMqANSAQFEA+GONioHHnO00VSgB4hzpNsIbAa02BcYdZfbYUCZ0QNobwPGOOjbAGx0hPkA3cANjrD7ATNw51Xq+gcgG/ADfIHjwDOOsCeBDQPizgNKHNfBDhrmYl9kznDc+w7om2pgqIMOp6uMrVwgyNHXJcAaR9hkoNZxrQSqgAcc7bjUUadnHeGzgSZHu2mBT69or1eBHY4yXIGdwP8MKMfiaAcnR310gOdV2usE8Ar2MTXJ0YffOJ4dbVCOfTHjDhQDpcB0R5t8DHzgiKvFPmfucISNBFqBxGvo81lAHvbxKLCP28Arxzn2MdeK3ZLjDLwBHL3Gcf0Z8Lijr1VA+g/NmwZ/18jDf2gCfso/B5PqBTodTGgddmHzBLB5QDwZdqY9GTvTb3ZMdKcr8nv6agzD8ex2/iEAVwC5V6Q/AdzuuP4K+N2AsHuAvVeph4ejLHfHfT9j+Ia4e4DVV9RNB4Q77qWBDAC74Pqt43ofcP835DkRaARkA5595mgPOXaGHj8g7Hm+XQC+NyBsLnDecX0bcGJAmMDOWK8mAC8CcwfczwIqHdfR2Bm8xnG/AXjScf0I8MkVee0DVg7omz9cw9haPuD+JeBtx/Vk/iEAJwH1gBgQ9zj/YOzv8/WFT+zl9nLUvw/HAswRPg64NKAcPV8fg81A2jfQG4ZdWGoHPPuUbxeAjw+I+ydgz4D7BTgWZcBNwLEryvsr8NQ19PlU7II1jQHj68pxDvwNeGlAmAv2cRdxDeP6Y+AdIORf4R+Dvx/+N2gC/e64XpIkD0mSwiVJukeSJD32VXvV5QiSJNmwM9pgSZLKgV9jZ+7NQoiNA00t/wK+VoYDVdi1j8toHHCtw+GEIISQCyFeEEJcFEJ0Y2e2MMBk9S0IB15zmIE6gXbsjPSflotdS754lbrUONrpyrr4Yl/V11wR9m24WvlBA/OR7NyrlqvjyjaucjzD0Y8lwAIhhAZYiJ3hg72Nll1uI0c7pQOBA/IaWJ9/tR5X0ljnqMtAOgeGX63tfAENkDeAzr2O55fRJjmcsK6Bjg5JkvquUtY3oWnAtf4b7i+XEw6MvaI9bwUCBsT/xraSJOkQ8BfgTezz7R0hhNtV6B84Z3uxa5XXMq4fxj4HcoXdfP81M/MgfrwYFIDfD+qxT1rAvg+BnfnXAUiS9KkkSemOOBLw4jfk8c8+0/G1MhwIu1zGP8Et2E2Z07GbnyIuk3oNaWuAux1C//JPLUnS8WtMG/UNz+uBUPF1Z5DLdWnBrlmEXhH276ABCLl84+iXkKtH///aOMzx7DI+w256XgQUO4Qi2Ov5yRVtpJUk6YUBaf9Tn2FpAIIv70cNoHNg+NXarhW7oBk6gE536d/zqm0APIUQ2quU9V1QAxy5oj1dJEn6xbUkliTpdUmSRgOJ2DXgh74h2pVzVgt4cw3zSZKkRkmSfi5JUhBwN7BO/C+8ZjKI745BAfj9YDMwTwgxTQjhBKwFjMBxIUScEGKqsDtTGLAzINs35NHieH41B4ndQKwQ4hZhd8C4CfsE33UN9Lk66GnDrgE8/+3Rv4a3gUeFw2lGCOEuhFh2jWnfA34jhBjtcE6IFnYHmhzsK+qHhRBOwu4wsgD7Po4V2Ao8LYTQCCESgZX/Ar0D8XdguBDieoczxr18XYu4Ep8BvxNC+Aq7A9GTwPoB4RuBmcAv+If2hyPOAiHELIe2rXI4lHybsP13cQL7AuFXjra7Afte2GVsBm4XQiQ6NNWnLgc4NO53gVeFEH4AQohgIcSsf5UISZKqgFPA74UQSiFEOvY+/E9gF/axvsJRRychRKoY4Fh2NTjijXXMwz7sc+6b5ttnwB1CiGTH3HweyJEkqfIaylg2oG87sC9uvqmMQfzIMCgAvwdIknQBWI59I70VOyNYIEmSCfsG+wuO543YHSwe/YY8dMBzQJbD7JN2RXgbMB+7cG3DboaZL0lS6zWQ+DF2c08ddueD7H+hbl9i11g3OsynRcCca0y7BXudPsW+f7YN8HK0ywJHPq3Y91JvkyTpvCPpfdjNTY3Y920+uFZ6ryi/FbsjzkvY2ywRO9M2XiXJs47ws0AhkO94djm/BuwCaDywacDzGuxa4WPYFzI12LWO//h8c7TdDdj3h9ux75dtHRC+B7uT0yHsTieHrsjiEcfzbEd/HuDff8fwFmCsg46nsI+z7wxJknqwLzR+hl1Ta8Q+Bp2vIbkbdiHfgX3MtwF//IYyDmDfu/8CuzYb5SjvWpAK5AgherE7FN0vSVLFNaYdxA8I8fWtg0EM4r8HDpNrLXCrJEmHf2h6BjGIQfzvYlADHMR/FRxmSQ+Hmesx7Pue16wBD2IQg/i/gx+VABRCvO94YbXoh6ZlEP9nMQ67J+pl0/T1Ds/dQQxiEN8z/hmPd/gGvO44kOCscBws4Qhb6TiIoEwI8e/6AXy9vB+TCVQIMQn7e3UfS5I07IemZxCDGMQgBvGfwz/j8UKIucAvsb/LORZ4TZKksUIIL+z78SnYnYzygNGSJHV8F3p+VBqgJElHsW+gD2IQgxjEIP6P4Rp4/CLswlGSJCkb8BBCBGI/hCJDkqR2h9DLwH7K0XfCj0oADmIQgxjEIP6rEczXD26odTy72vPvhJ/c55CEEHcBdwGoVKrRAQEB9PT04OrqilqtRo0ZS1cXhk4dSldnnFy1WNQe9PX10dvbi1KpRKPR4KpRYWlrwtStQ8gEzp6uyDz86ezqQq/XY7VacXNzQ6PRIOtqxtSnx6Izo/Z2Re7mSY/JisFgoK+vD3d3d9RqNc7GHsy9fRi79Kg8tShcXTDI1eh0Onp7e9FoNGg0GrQKgbm9FVO3HrlagdLdFcnFmy5H2UIIXF1dcXV1xdJci7lPj80iofZxQ+bqRWevDqPRiMFgwMPDA41Gg7ynFXOfHlOvCbWPC3IXd/qsAr1eT19fH25ubqjValRWA+auLozdBpzdVDi5uWF2dqGnpwedTodSqcTFxQWtsxPmlkbMfQZ7+3i7I3P3o729HYPBgM1mw93dHa1Wi621HovOgMVgQePnjszFg26DGb1ej16vx9PTE5VKhZO+C3NPH6ZeI2ovLXJXN/Q40dfXh06nQ6vVotVqUctsmFpbMPcaUaidUHp6YNO409nZicFgQAjhKFuDuaEGi96IzSqh9vNA5upFR1cPRqMRk8mEp6cnarUa0dmEWWfA3GdG6+eGzMWDHpMVvV6PTqfD3d0djUaD0tSHqasLc68RZzc1Cg8PTHIV3d3dGAwGnJ2d7WPNSY6pqR6L3oSQC1Q+3uDiSVtbGyaTCZvNhqenJxqNBktTDRa9CavRgibAC5mLx9f68DKN8t42TD19mPvMqH1cULh5orMKent7MRgMaLVaXFxccJbMGFtasOhMKDRKlN5eWJVaOjo6MJlMyGQyPDw8UKucMdVXYzWasVklNAHeCBdP2trt8cxmM15eXqjVaqS2Biw6A2a9BW2Ah70P9Sb0ej0Gg6G/fZwM3Rg7u7DozDi7a3Dy9MIonOjs7MRkMuHs7Iy7uzvOMjA21mE1WhByGSp/X1C70draislkQpKk/rLNDdVYjSasJhvaQG+E1oOO7h4MBgMmk6k/nqy7GVO3DovBgtrHDbmbJ31mGz09PZhMJlxcXHBxccHJasTY3IzVaEGhUeLs44tZ7kx7ezsWiwWZTIaXlxfOTnKMddVYTVaQJNSBfgitBy0tLZjNZiwWCz4+PqjVakcfGrEarf192K2zz3+z2fyPeajrxNDWidVkxdlDg5OnDwaboLOzE7PZjEqlwt3dHSVWDPV12Cw2ZAoZqoAAbEoNLS0tWCz2g3e8vb1ROTtjrKvEZrJgM9vQBPshtO60d9jngsViwcvLC41Gg9TeiKmnj9ouHV0W87UcavEvQ6VSSTbb1V9xNJvN57C/a3kZ70iS9M73Qct/Aj85DVCSpHckSUqRJCnFarUSExPDs88+y8XM3WTclM7kUgsPG73ZOGsCWaWXePpPf8FoNBIeHs7ixYuprigj/9lVLC3VcU+PB38OiKGouoI312/Bzd0dLy8v0tLSyMvLo/C93/Nin4wlTRr+RxHM2f0fsn73ASZOt78nPGLECD799FMu7HiPLamRTL4k42mbP/vvX8bfc/L5xW8eQ6fTERMTw3333celk4c4tmYuU88bWGvw5v2UUZyqbuCFN9/DZrMRHBzM7NmzuXTpEqdfWMO9Fb3c0enGK15RnDt3grc+/YIhMXFotVrGjBnDoUOHOPfBc7zl4casWmdeUIaQ99eH+HTPIa7/2XKMRiPJycm89NJLlO35hF2zRzKxzMYzkj9bl0xm/+ki1j71LHq9npiYGJYvX071uXyO//p6Fpfq+KXOi78mDKeouYUX1/0NjUaDj48PM2bMoLCwkNMvrOHZdis3tLjwik8UhYc/5a3PtjIm/TokSWLChAls2rSJor/9gfXxwUyokPGiOoRDD9/CpgNHWbnml/T09JCcnMxDDz1ExVdb2X/jBK4rMfCE1Y8N09I4VlrBo8+9hM1mIyIigmXLllFXV0fOw0tZc7GPVT0erItO5FzZaV5c9zdCw4eg0WiYMWMGR48e5fQLa3hDrWVGnZrXAqLJ//AJ3tr4JTMXXI/BYCA9PZ3XX3+dkk9e5PNxMYwttfGiKoRdq+azLSuXu3/9EDqdjqFDh3LPPfdQeeowB26dzMILOh4w+vDR+FQKmtp49LmX0Gg0+Pn5sXTpUkpKSjix9gaebjCypM2Nt2OHUpS1lRff+htJo1ORJImZM2fy+eefU/CnX/FesA9pFQpeC4gm85mf88G2PSy9dSU9PT2MHz+ep59+mrId77F9+nCuO2/kWRHA1qVTOFhYwr2/eRSr1UpUVBSrV6+mtqKMg7dN5a4KHXfpvPkwLYXiyvM8+twfCQwKxsXFhSVLlpCVlUX2b5bwCkqm1at5O34YZza+wB/f+YiJU2eg1+uZOXMmf/3rXyl64yHWJ4aQWgZ/9o1i/69v5LOMr7hl1V309vaSkpLCI488QmXWLnbMTGJhqZ7fWv3ZPG8i+fVN3PubR3F2diYoKIjVq1dTVlbGweVTeLzWwNIOdz4cm0JR9g4ef+EV4ofat4ZuvPFGduzYwanHfsZbnu6kXVLydvwwsv98P69/sonZCxfT3d3NtGnTePnllzm//iU2jY5g4nkTL2nD2P3zRew5VcDyO9dgs9lISEhg7dq11JYWsWNmEqsr9Nxt8GXLgomcq77IL9b+Fl9fX9zc3LjzzjvJycnh4PIp/E8fTGnQ8vH4VM5se4MnX36dlLTx9PX1sWTJEtavX8/p51bzbpA3qWXwZkQ8R55axYc79rJgyY309fUxbtw4nnvuOS4d2sKm0RHMKzXwlCyIHbfNJbuylltW3YWzszOhoaGsXbuW8vJyds5K5pEaPcu6PNk8fyLnzhzk3ocfJzomFiEEd955J/v27ePYXXN4VeHM2ColH45NISLQ6/vkvwQGBl71Bxgu82fH718VfnV8/eSiEMezqz3/TvhROcFA/4dgd12LE0xiYqL0/PPPc+jQIXx9fQkMDGTevHkEaBUYs7fTeboAm9mCW0QAqqR05GFDqeux0NHRQWOj/Vg/JycnwsPD8fPzQ91Tj/HkPnRVVfQ1tuEZE4pzzHD6EqZRUVHBrl270Ov1yGQyfvaznzFs2DA61j1C0ceZ1FZ0EpXsz/C756BY+GvefPNNjhw5QmlpKVOmTGH+/PnMCBTUrt/AW68fJ0zjRJS3mlmHP6Re7sPzzz9PYWEhBoOBO+64g/nz5xPYUsiZJ16hpbgVpdaJpNUT8b5hBfmtNkpLS8nOziYyMpKwsDBmzpyJRt+K/qtN6GoasZotuCfGoYwdiSxiJLXNbbS1tdHc3IxcLkehUBAZGYmvry/K2jPoTx+h60IlFoMRr/hwNGmzaFCHcuHCBbZt24azszMeHh7cdttthLiruPDAz6nOqqGyqY/rliUS8bPruRQ0ho8//pjDhw9jtVqZPn06K1euJPT8Xs79dQcf773IKA8VIyaEMGrrLgrOneexxx6jt7cXSZJ4/PHHGTt2LPJtr1C84RhNRS2EpgURuzQd7bKHyMzNo6SkhIqKCiIjI4mNjWXSpElQX0Lfwc2YenRIVhuuw5NwCo/HFj6K+vp6WlpaaGlpQS6Xo1KpiI6OxsfHBylvF8YLZ2g9exGlmwaP2DA0s1dR3tpHXl4eBw8exMXFhYSEBG688Ubc2svJvuMB2so6aNBbWPjbGfhe/zP2lHWwa9cuDh8+TFRUFFOnTuXXv/41XW8/Ss7L+/mqrodUTxXpv56Cyy9fICsri+eeew6FQoGXlxe/+93vSE5OpvI3K2g4VUXLhXaG35xE0Mzr0KfdSH5+fv/YCAkJISkpiWHxMUhVZ+g99nesJrvGoB0+GkVwNAbfWFpaWmhsbKStrQ2FQoGrqytRUVH4enth3P83+i6W03K6DPeoYNzjhqBc9GuKioo4cOAAJSUlaLVa5s6dy/Tp07Ec+pAj966jtUOPi0LG9FdvRjN7Fa99tNkuRLOzmTlzJtdddx3LrxtB7buv8+XrmeitEkn+WqZ++AitUZP48ssv+fTTT1GpVIwZM4Z77rmHEJWZ4gd+RWdVF31NOkbfNwX39KnUeCZQWlrK2bNncXFxISAggDFjxhDsqcVafhL9mSysZgsKlRLnhBTkAZH0ugbT2tpKXV0dXV1dyGQy/P39iYiIwMvJhm7f+3SXVdN2rhL/0bG4JsTTN2YpxcXFbNmyhb6+PrRaLatWrSIpdghNb/yO468cosloYfgwX1KeXo0lbRlPPfUUZ8+epaKigptuuomZM2eSZirn4vptfPRZMQmuSmISfRj/5XrOt+h49913ycvLQ6VSsXz5chYsWIBLxXHOPv0qvU19KLVKRqyZjXr8PApardTU1FBcXExQUFB/vd2FAfO5TEwV57CZLSjc3FFGj0AWGEWnwm55qKysRK/X8+CDD1JeXv69aIDOzs5SUNDVjy6urKzMkyQp5dvy+DYeL4SYh/3gi8tOMK9LkjTG4QSTh/1rHWA/lGK0JEnfyWfkR2UCFUJ8hv0Eeh9h/97ZU5Ik/e1q8Z2cnPD09KSnp4fy8nLUajUBAQFMnjwZJ4WS8p2naSvvQOEkZ9QvmvFb7ERlm4xTp06xfft2AgMDiYiI4P7770fdWkb5C89yYft5+sw2hs+JJGDuXIzDZ/HR3/5GYWEhZWVlTJo0iTFjxjDM1xn9pufJfSUDk86Mj6eaEfcuQj3nTg5nZVFSUoLJZGL06NFMnjyZUaNGYTryAYb2bqJdlESFuhKQ7I9w88PabTdFeXl5odfrsVgs6HQ65P4RuAa60lHRSX1jHwF5pWhCM/BPu4X6+nouXbpEVVUVrq6uhIWFMWrUKAxNrZzbcIK6qi7CYvIZdnsdHj+LorCwkOPHj3PkyBFiY2MZPnw448aNQ1byFTm/fo7WC230mW1MvC8d9egpXJS82fj++5w5cwa9Xs+kSZOYOnUqgd3lNG/axpkdpbiqFMTFeBJ592oYMZPjn3xCXV0dKpWKUaNGMWnSJGJjY2nPeA/JKpHkriI6zhvfpDAkuRKZTEZgYGC/Way3t5fe3l78wmNQe57EKkFDXhMugYVEDs/EyysYmUxGbm4uZWVlFBUVER4eToRvAA3Z56g+WkllfS+p088RfeMUJP9EMjMzOXr0KKWlpcTHxzN16lTSR4/AePgjjty7DlOfCZtNYuqLS1FPWsLh0+c5cOAAJ0+exNfXl6SkJObNm4f67B4qPt9OeVErPq5KUtOC8bvl53RqQ8jb+Abd3d34+/szdepU0tLSkDVewNDWhVLrxCgPFSFDfXGLCsEqSahUKvz8/DCZTAB0dXXR2dmJe3QwrSUNtJusVB+tQK5SEjZ+ES4uLvT19XHw4EG8vb2pr68nKCgIT2cNF7Ycp72sg/o2HZPvqCFwzjTqzS4cOnSIffv2odfrSUhI4KabbsKHHro+eI1jT27HZpNw8dMQsXQOzpNuZMPGjeTm5nL69GmSkpIYNWoUM2fOxLDzDUre20lrhx4fTzUJNyWjmXsnl9p0lJeXY7PZiI+PZ8KECSQnJ2OpOY+lz0Co2glnpRy/oT7I3b2Ry+V4eXnh7e2N1Wqlt7eXtrY2QpOTcQn2prOqi4qWPrz2nSHMbCHgnilUVlZSWVlJeXk5Xl5e2Gw25s+fj6yvm/w399Nd24PZKjH+wVZ85l7P2Utt5ObmsmvXLvz8/IiOjmbNmjV4dl2i6u3Xyf8oD4AhE0LwuG461qHT+GjdOs6dO9e/UE1JSWFEkCu9W16h6OMTWCWJmGBXRt47G+fJt5Kfe4r29nZUKhXJycmkpKQQHR2N9VgmMrmMWBclYSGueEZ6gJChVCrx9/fHxcV+vGpzczNtbW24hw1F5amis6qLsrIO3P6eRYTNik/6SqqqqsjJyek3b7q6upI2JhVTxTmy/2cnHS06vAK0pD7Qhcv8WDIzMzl16hSHDx8mNjaWvr6+q7HM/wSPRqH498XGN/F47J/aQpKkt7Ef8TgX++lEOuyfwEKSpHYhxDPASUdWf/iuwg9+hBrgvwKVSiWNGzeO5ORkHn74YfxtnRy/6edUnW+jtNfEkuvjiF42laZRS9i5cyc7d+6ks7OTuLg47rjjDqaMiKJr61/Z8tsvMdkkPJ1kXP/mbWhmrOCDbfs5duwY2dnZJCYmMmHCBO655x5se97izJ+3cvhkA0EqBalzokh89XVq9AqefPJJampqaGlpYeXKlUyYMIHRUh2XPvyUQ5vPARA/xIPxf7qXzqGz2LdvH2+//TaSJBEQEMDatWtJTU2l5qGVVOwv43BFB9PjvYmcEUfICx9w+PBhNmzYwNmzZ/H09GTp0qWsWLEC53MHOHjL76nrMNBhtrL07rEEL17IKVkIBw8e5Msvv8Td3Z24uDjWrl1LlLyLij+/zN6PTmOVJEZEeTL+T/cin3QLL774Ivn5+RQVFTFlyhTS0tJYuXA6XVvf4tiT2ynrMeHvLGfmY7PxWfM02QXFvPTSSzQ12Q/yv/POO5k4cSKBOZ9StvU4O/dcJNZFSfgwX8Z+so5Kk5pPPvmEvXv3YrFYSE9PZ9WqVQwL8+X0yhXUZNdzssPAvPEhxFyfgnHpg5w4cYJ169bR29uLt7c3Dz74INOmTUP36TPsX7uZGr0ZH6WcaQ9Nx+9nq9mWV0FWVha7d+8mOjqaoUOH8thjj6EuO8ap37xEXn4jJpvEzIUxxD98Py1+w3nppZc4d+4cDQ0NzJs3j6lTpzI12InmLzez9Y+HkAsIUimY9cGvkE9fxZYtW/jkk09oamoiPDycFStWMHXqVPTrHqd85xl25zUwMciVsAkhDP3oSwqKinn11VcpLS3FarVy2223MW/ePEK7y8j91TNcKGyhw2xj2uxIYm9fRHlYOllZWbzzzju4uLjg6+vL008/zbAhQVx64pfkrj/DxT4TadGepD4wG9cVT/Du+x/2a2QjR44kLS2N+39xF8ZDn5Cx+g2q+swA3PDQVAJW3M2pJhPvvvsuRUVFKBQK5s2bx7x584hpOkX1F39n44dnCFUrCA10YcrW1+j2S+T1118nMzOTxsZGpkyZwqJFi5g6NpmLj91L6Y4Schp7mTrdkewAACAASURBVBjlScKNo3B78E/k5eXx3HPP9TPlhx9+mAkTJqDet46Tf9rNqbJ2+zxaNpTotb8hq8HMoUOH2Lx5M/7+/sTHx/Pkk0/iZ+0gZ/kays42U6kzs2BOFHEr59I99ibWr1/PoUOHqKurY9SoUSxatIiFaUPp3vUhmx/6wr7okAlufGslmpkr+WzPV+zevZuCggIiIyOZMWMGt956K8q9b3Luw0P8/UgVsS5K4scFM/KD96nVwVNPPUVlZSWtra2sWLHCvqh17uTCn14nd1c53RYbo4b6MvrhpXSMuZFjx47xyiuvoFAocHd354knnmD06NHUP7aa0h3FHKvuYtIQD+IWDyfsmXf4+979bN++nezsbEJCQpgyZQr33nsvytKjHFjyKHUdBlpNVm5aM5bgJddzRhHGrl27yMjIQKFQkJSUxF133cVQ5x5GTV1MQV3z97YHGB5+5Rn8/0Bpaek/1QB/TPhJC8CwsDDpvvvuo6WlhdTUVMLDwxkzKgmpym7Ss/b2ItdocE4cgzx8BEalKy0tLfbvQDkcBTQaDaKn2Z6hwhmjw2nlsglFLpcTHByE1FqN6fQBLM11GDt70cYnooxOotsjkvz8fEpLSykrKyMwMJDY2Fhmz56N/NJJzj76B1qKW6lr7mP0zEiGzElFLHmI/fv3s3PnTnJzc3F3d2fcuHHcd999hLvKybvtDkpz6znWpmdeoAuR0yNIXPcReecv8fnnn7N//37kcjkjRozgjjvuICUlhZ51j1KbWUzx0WrC473xGx5A9OO/p0sbxMmTJzlz5gxGo5HQ0FCSkpKIiopCW3cGS0MlxqoynPyCkHv64pQ0DbOTlra2Nmw2GzKZzO7k46yEXscxo05qjDJnuru76e3txWq1olKp7ObUtktYqs7Rk58DgNJVg3bqEggexldffUVpaSnFxcX4+/szZMgQZs+ejVtDAQ2bNpD/zglMNgmvAC1pTyxFPeVmPj+UTWZmJhkZGXh6ehIZGcnDDz/M8BAvGtY9x64/HaZaZ8bXWc6820cRdstNnBZBnDhxgg0bNqBSqfDw8OCBBx5gzJgxKI5v5NwbG2k9345cKSN6XiKBs6fTPXwOly5d4ujRo8hkMlQqFePHjycsLAwPXQPm8tNY2xoRKi2KoAjkIQmYXAPQ6XTo9XpUKhUqlQqN3AYmA1iM4OqDBTlNTU0YDAZ0Oh2enp64uLjgrmvEcqmQ3sI8TN06lG4a3CbNRopN5/z58+zbt4+mpqZ+7TshIYEEmunI2EXuqwfp1JtxU8qZ8MQ8PJbdS0FNG3/5y18oKSmhra2NSZMmMW3aNG6cmopu/3q23r+RSp0Zk03i1luGEnnHzbRGTWLDhg2cOHGCyspKEhISSE9PZ/Xq1SgqT1H0u2foqOhE32EgYekIAqZMQEy9nYKCAkpLS2lsbMTHx4chQ4YwatQoXIQJU/5+JEMfktGAMnoEwjMQs2cYOp2u30lGrVajtBrA2IdktSBcfTBKMjo6Omhvb8dsNqPVavH398dVbsWY9QWmuiraS6pwjwpGHRaBfPoqKioqyMjIoLS0FL1ez9ixYxk7dizDwgOofflRqg6dJ7egmQRfDaHjghn6l3ep6bHw0ksvUVBQQH19PWPHjmXu3LksW7aMvvef5NzHmXyRW0+YxokRYW5c9+GTkDyHN998k9zcXEpKSkhISCAlJYVbb70V79ZiWrZ/TvHGXBQqBZ6RnsTdswJl0hTyymqpra2lsrKSgIAA/Pz8SE5OxlMpYa06i6WhEiGTI/cPRR4UBx4BdHR0AiCTyXBzc0OY+pC6m0HmhFBp6cOZ3t5eGhoasNlsyOVyIiIicFfJMeXuovdsPpOffZ/CxrbvRQCq1WopMvJq5/NDcXHxT0oA/qhMoP8q/Pz8mD9/PuvXr+fVV1+ltbWVgIAAZs6cyfTp0xnr70RXxhdsm3IflTozXWYbC8cGEXP9aPzue5Yj2ac4cOAAX375JTKZDBcXF+644w7GjBnDUKmB+s2bOPG3XPI7DbgoZFw3KoBRDy3Da/Gv2LRzHzl/t2sySqUSd3d31qxZw8SJEwnqvUTRrQspPVLN0VYdw91VxEZ6kLB2DfLRdnr37t1LYWFh/z5FUlISgYGB2MqyMHQYMNkknAT0mW30NemwVp0lPHwYycnJ5OXl0dzczPHjx3FycqKiooIV9z6HW/Ln9Db8ldyCZhpz6kn9YhlJSxKZcucdhMyfz6lTp3j77bfp6enBYrGwaNEiJk2axJybF9O1/gWKX/qU3dmPoJQJIjROzHp0Jt5zrie/RsvRo0fZvn07ra2tqNVq5s6dy+LFixmVNJzqJ+7i4v7z/LmgiQCVggiNE7P/uAztnNs4XdPJ+lc+IC8vj5qaGoYMGcLIkSNZsWIFoR5qyh9dw4kd58ls6sVLKScpxI3UB2ajWfow+fn5bNu2jcrKSnQ6HfHx8cTExODv74/UUYWxowcZIBegt0p0lLfgU3SK2CW/Qa/Xk52dzYULF2hsbOStt96ioqKCOXPmMPR+ObU79vHl+3kcf+M48r8cZ8nNWxm2cApDVq4kIyOD3bt389prr6FSqYiMjOTuu+8mecJ0XI98QPHzr7F/z0V6LTZ8nRXMvGU4/mvuojd8DH966z1OnTrFiRMn8Pf3Jzg4mIceeojk5GSkna9x8qXtFJS1U6e3kOalJmFuFHEvvkqL5MLTb7xBXt5LlJaW4u/vz4gRI7jtttsYNywGfcZH7Pnlh5T1mKjWmZkf70345Eg8VzxMSXUTu3btori4mM7OTry8vAgLCyM4OBh0XZj7DKjlArkAk02i+VwLXrnZBCVNYcyYMRiNRkpLS/vN/DKZjLFjxzLiD49R88F7FK7PZ92rmXi9eYKxse8z9ulbSF1wE19kZHLixAn++Mc/IpPJ8PT0ZNWqVUyfPpsgUyMFv3mc6uO1HGjuY7i7iqhQVya+8xiKMYvYsn0727Zt49y5c5hMJhITE5kxYwa33XYbzlWnyLnzl2QWtpDfaSDNS03M6EBGPr+WjtBUjhw/zptz59La2kpvby/Tpk1j3Lhx3HHbckxHN5LzwPscy66j3WQlwdWZ6DmxhN+8lAtN3Zw6dYqCggI6OztRKpUEBgbi4+ODUumETibDSeuEWi7otdiorO9l+JGDeKu0XHfddVgsFhobGzl58iRlZWXU1dWxfPlykh/9C2bdai7tL2H9F+fx2vmkff48u4hRN9xNSUyM3ZGoqIiLFy8SFRXFsGHDWLt2LWGeGip//wBFn50hp82+x5oS4U7yXdchfvUyBecv8vLLL1NRUUFdXR0jRowgJSWF1atXE2Ssp+6jt9j8RiaNBvs+8Jwp4chk369v4/ed//8mftIaoKenpzRr1iycnZ2ZM2cOiYmJDPMEfeYOLmw6QuOZZuROckLGBRF18wKc0xaQda6C8+ft+zwAWq2WRYsWkZycTIiTnqYPXqcht5Sa7Hrcg13xivEi4aF7UQybwpETueTm5nL27FmMRiNhYWFMnDiRqVOn4tJTR+XLz9BcWE99YQt+MZ74JvoRdf8vMQ0ZS35+Ptu3b6ehoYG+vj5SUlIYOnQo8+fPR5Qdp+qdt6k+WkFfsw7PSHfCpybgt3AxrQHJXLhwgc2bN9PZ2Yler+e6665jxIgRTJkyBdPhT2jYe4ALX9pPFtL4qBm+ejIu46ZT6RxKUVFRv3nI2dmZtLQ0Jk2axNChQzF9/iL1mWcp3XEeJ60TGh81I+9fgHr8QqpMKjIyMigpKaG8vJzg4GBiY2NZsmQJYcGBdP7tKWoOF1CXW49cKccn3ou4W6ejWfhLKuub2LJlS/8eTkJCAlFRUSxevBh/hZG+3R9Q9OFXdNf2ADBkeiTBk5JR3PCQ3ftt504uXLhAe3s7sbGxpKamMm7cOIJ6L9GTtY8zbx3E2G3CZrUx9JZR+E0Yi3XScvLz88nJyeHEiRM4OTkRHBzMvHnziIuLw7fqBO3HjnB+80n0HQaULk7ELxmJ//VLscZfx549ezh//jxZWVl4eXnh4+PDjTfeSEJCAqqi/VRv/JyGkzV0VnXhE+dNUNoQQu97BJ17KNu2baO4uJiSkhI8PDyIjIxk1qxZpI4ehfnkTkr+9Fc6q7rorO4mbHwIfknhBP3ySRqNco4dO8ahQ4doa2tDo9Ewbtw4kpKSSIsLxZifQfFfPqW9rAN9h4HwSWGETR+J26KfU9LcR2FhIbt27cJisaBQKJg/fz7x8fGM8Fej/2oL5VszqTvZgEKlwH+4L3GrFqNMnUN2aS3FxcUcOnQIm82GVqtl3rx5pKSkEKKBpndepPFUOdXHa3EPtu9nJT58D4oR08k6mc/x48c5d+4cvb29REREkJqayty5c3HRNVH9yjM05lfTVNSC1xAP/Ib7E/mLn2OKnURRURGff/45zc3NdHZ2MmbMGOLi4li4cCHyi9nUfvw+l/afR9emxzXQhcjZQ/GdMZO2IRMoLS3liy++oLW1Fb1ez8SJE0lOTiY9PR0pcyNNh77i3Kf5yOQynN2UJN01Bdfx06lzi+b06dMcO3aMyspK1Go1ycnJTJ8+nYSEBCzbXqEhq4DSHSUoVAr7HLh3NupJS6iXXNm9ezcXLlygtLSU4OBgIiMjWbZsGRFhofSuf4ZLe07SkGd3qPMb7kv0ojRcfvYINU2tbN68mYqKCiorK4mLiyMmJoYFCxYQrLKiP/QZZ9/eS09DLxaDheg5MQRNGI7yxkcpKytj3759FBQU0N3dTWRkJGPHjiU1NZUwUz267P3kv7kfQ4cRm9XWr53Lpt1BTk4Oy5cvp6qq6nvRADUajRQXd/WPhZw5c2ZQA/zfgiRJWK1WFAoFKpXKsTlrAYUTak81nkM8UKgVeMWHIw8IAxcvnJxq8PDwYOjQobi6uuLh4UFCQgIBAQHYLhzDSavCM8ofl0B3vIcOwSVxOCJpFjX19dTX16NUKomJiSEqKorIyEiSk5NxunCE7rxMWovtEyF8TCDxt0xGlTyJOm0EZw4c4NixY5SWlqJUKhk1ahQzZswgMTER45YXqDlwisyt5zFLEm4KGcPGR+E3fxG9EWls37SJnJwcjh49iru7Oz4+Pv0ekKYjG7jwzkbOZFziZIeeMI0TKS5OqILDkIcO4+yBY2RlZZGZmYnZbMbT0xMAtVqNQt9J/dlyWooauNBpIMhgQa6UIWRyUKox9hgwGAwYjUZkMvtmvkqlQi6Xg9Xufafxc8MzUofaU4VnXBBOobFY5UqsVit+fn4olUqCg4NJSEggJCSEAD9fbOU5WM0WfIcH4pPoj9uQQDzTJ+OUMI6i0lLq6uqwWq3ExcWh0WiYMGECUVFRuLWV0rxjC61n7V+ZCUj2wyshhOC716L3CGPPnj2cOnWKkpISFAoFERERTJs2jcmTJ2MrPED+79fRVNjM6Q4Do300eIS743/9UvSR4zh28CAffPAB9fX1tLe3k5ycTEJCArGxsbi0l3Ph3Y85uaOMom4j7k4yxnca0Ph6QmAcl86d48iRI5SXl1NbW0t0dDSBgYH2d+s66jHXlNJS3EpHi45Wk5VAxz4cSjXm3j4MBvsrU3K5HK1Wi0ajQaVSIdlsCIUTWj9XAKwmG34jo1BHxiE8AqD5Ii4uLgwbNgyVSoWbmxvJyckEBwdjLc/EZjLjGuJFjJ8bLsG+eI0fh3LCUuqb26iurkan0xEREdHvRZyeno6qpZSuLzZRf+ICulY9/sN8GTI7Cc+0CbSHjaU0N4+tW7dSWVnZv4hLS0sjPT0d51PbqN57gAPv5mKwSVglidh4b0JmjMUQnc6xI0fYvXs3Bw8eBMDX1xdvb2+ioqIQ5w7SuONLTryXS0GXvT2maZ1w9nBFET6UixcvkpeXR3Z2NmazGYVCgclkQi6X42Qz0ltzkY4L9dR12C01XnKBkMsQCidMJlP/e3XOzs64uLjg5ubWP5atShUaP0984rzR+KjR+Lk7TLdBmJva8fLyYsiQIfj7+xMWFkZYWBgRERFItYWYdQbcwnxQapV4JoTjPmI4zmmLuFhjd1DT6/X9jnbp6emEh4cTqK+he99Oqg+cxtBpwMVfQ/D4KIJvWYF82BR27txFYWEhmZmZ/V7CU6bYNXWfnkouvPAi1Zm1HK/pIkqrJDTMjcDZ01GmLSLjq6/YunUrbW1t3xvP/a5OMD82/KQ1wNGjR0ubNm1i79697Nu3j9raWnp7ewkNDSU+Pp4VK1YQFxeH9uxuarbuomzXebLre5ALQZBKQdrcKEKmJOO26hlKS0vZv38/GRkZ1NXV9Wt48fHxLF++nNjYWJxObKLq891c3H+RnMZe5EIQqlYwbnE8IVNGorzxUc6fP8/evXvJyMigubkZg8FAZGQkiYmJrFy5ktjYWGx73qJ8UwYXDlWS3a5HLZcRpXUi/eYRBE8fh2XaaoqKivpd61tb7Xtv8fHxjBw5kttvv50h4WF0ffAUJRsyKTzTRGGXAXcnOUnuzqStGY/f1Mm0x02joKCAHTt2cOLECbq7u1GpVCQlJTFmzBhuvvlm/JRWGt96jvNbTnH2YgfVOjMBKgUjAl1JfXAGbukzqdYOIT8/nz179nD69Gn0ej2urq6kpKSQkpLC0qVLcelroO6tP1K6rYCSqm4aDRaC1Qrih3iQunYumnFzKDW7kZ+fT0ZGBgUFBRiNRtzd3Rk7diyjR49m8eLFqNsvUrPuFS58WUhJfQ+NBgtDtEriYjxJWbsQddocznXLOXnyJAcOHKCwsLD/Zen09HRSU1NZvHgxsopcKl5/jeIvzlHQZaTLbCXO1ZmEeG9Sfrcc59SZ5FxsJicnh/3791NWVobNZsPf35/p06eTlpbGnGmTMeftpuTV98g/WElJjwmrZPdmTRgbRNLvfoHTqFlkHD1BZmYmx44do7q6GoVCQXh4OHPnziUlJYXxcSEYc/dQ8OomzuY1cqnPhFouY7SPhsjpEcT+9hGkqLHs2rWLrKwscnJyaGpq6n9lY8GCBSQlJZHkq0R/Yhdn1+2m8EwTdXoz7k5yRoS5ETEtmsi1v8XkH8+uXbs4ceIEJ0+e7NcsY2JimD9/PsOHD2eoqxld5k7OvJ1BYVELdXqL3fw8xJ3IGXGEP/g4fW4hbN++nczMTAoKCmhtbcXV1ZX4+HhuuOEGhg8fTpSlnq7DuznzzlecvNhJi9FCgEpBapw3kXMSCX3wD7RYlGzevJnMzMx+rdHT05OkpCRuuukmhg0bhm/FMZr2Z1Dwfi45bXp6LTaiXZSkpAYSvWA03vf+D5eqa9mwYQO5ubmUl5djsVjw9fVl3LhxLF26lISEBJRHPqJufxZ5m4so6jZiskkkuTuTNCuSIQvTUVz/IKWlpWzatIm8vDyqq6v7PZEnTJjA4sWLiY2Nxbr9VWr251C4s5SSHrun7khPFXGL4gmZPRHbNPt7jFu2bOH06dPU1taiUCgICQlhwoQJLFy4kJiYGIybnqdq/0nO7L5Iaa8JuRCM9FSRsHQYIXOmoE+7kdLSUjZu3MiZM2f6F9gRERFMnDiRG264gaghEbT95REu7TtDTlZt/9gZ56dl2PJRBMyZRWvUJM6fP8/HH39McXExBQUFmEym70UD1Gq10rBhV39DLTc39yelAf6kBWB4eLh07733UllZiVarxcPDg9TUVGJiYoiICMd05FP6Sgqp3JePqdeMzWwlYFQoXkOH4DZlETr/RGpra8nMzKSpqYm6ujrc3Nzw8vJizJgxREdHE+zrheHAR/ReKKX6cCFmvQXJKhGUNgSfEdGo0xfS5RJKTU0NmZmZtLS00NDQgJeXF35+fv35+KoE+oyP6Cy5SO2xC1hNVoRcEDoxFs8RCagmLKRFcqGqqorjx4/T2tpKa2srfn5++Pn5kZaWRnR0NG6WbnR7P6D9fBX12XZtSK6UE3pdIh4jhuGctoDaXmt/Pu3t7XR2dhIcHExgYGA/PaqOKrr3rKejpIrG/BqEXOCkVhA+PQnXpJEoU+dxqamD6upqsrKy6OzspKenh4iICIKCghgzZgxRUVHI6wpp376BjtIaWooaUKiccNI6MWRuCtqksSiSZ1JSVkFtbW2/ENbr9URHR/fnMyQ8DFt5Ds1ffEp3ZQMtxc04uznj7OZM5PyxqJLSkQ2dwtmzZ6muriY3N7f/BI7Y2FjCwsIYPXo0Id5uWC4cp/HLL+itaab1fJt9Ve+jIXz+JFTJkzCHJHHmzBlqamo4deoURqMRq9VKYmIiERERJCcn4+9sw1xynPptO+ita6OjohOtvwatnythi2bgPGICOs9ITp8+TVVVFadPn+by6RjDhw/vz8fD0oG5KIuabbvpa+iku7YHtxBXtIEehF4/F6dhE+hUePbnU1hYCNj3WEaOHElERARJSUloOy9hPJtF9fYM+pp76GvS4RnpgUuwN0HXL8RpaDrNBtHP1IuLi5HL5Tg7O5OSkkJYWBgjRoxAWXsGQ0EmVbuOomvVoWvV4xPvjUuoHwGLl6CIG09dew+nTp2iurqa0tJSnJyc+g9euJyP7dxhjEUnuLj9OMZuI8ZuI76JfrhFBOK3+CZkseOprK4hJyeH+vr6/leU3NzcGDduHCEhISTERmM+9Xd0Rae4tPsU5j4zpj4T/klBeMaG4rXgZqyhSVRUVPTnU1lZ2W+1GTduHBEREQwJ8MaUu5OegtNUHSjArLdgM1sJTAnHMyEctznL0XuEcfHiRXJzc2loaKCurg4vLy88PDwYP348ERERBLsqMB7fTufZImqOFNtPhwGC0iLt78XOWkm3kwfl5eVkZ2fT3NxMc3MzPj4++Pj4MH78eMLDw/GV6TFkfknH2RJqjpXa+1MuI2RiHB5DY1FPW06LQaK8vJzc3Nz+eR4YGIivr29/Pu599eiPfUnr2XLqsy8h5AK5Uk7Y1GG4D03E+f+x995Bkt7nfeenc855eronz04Om7CDXSxyECEQQbBE0rJ0Pp1DFeWz7urOJ/vqfK4rqyzrSne2TEulYJZkK5ASSYBEWpBcYAEsNs7u7MxO6gk9PaFzmJ7O+b0/evolIBjknSiczSp11VZtzzvzPh1+v9+Tvt/v8+iXCKeO2N7e5ubNm+I+9/l8/O7v/i57e3ufiQPU6/XC9PT0p16/du3a3zjAH2lUIvkq7WnmiQ4Z8pjo+HWgFwgBP/ujlL5VKpUwNjaGw+HgpZdeYmpqirnRXirXvs32n77O3UtB4tV2+WO2x0T3uW6G/9k/o+mf5cqVK1y+fJmlpSXR8Xm9XjGaHDU2Kb33CoGvf8DC1QNyjRZ6uZTJGRfe8wN0/+N/TlZm5Pvf/z7vvPMOGxsbpNNpsazzhS98geHhYTyZNVKXXiPwzXnurKUpN1u4VHJOPuzDd3EC+5d/jf14itdff53Lly+zt7dHoVDA7XYzOjrKl770JU6cOIFx5W3C336T9VdWuBYt0BQE/FoFDzzip+eJGaxf/tdsbGzy2muvifcplUp0d3czPj7OL/7iL7bLivPfYvcbb7L+nQDvp0oA9OmUnH+8l54nZzD/g3/F+vq6CLGORCJiuWx8fJxf+IVfYHh4GPX1r7P7ytusfyfAjUwZmaQNnJl7bhD/46dQffF/JRAI8NZbb/Huu+8Si8Wo1Wr09/czOTnJF7/4RYaGhmi99Tvsfuddtt4OspStoJFJGTIomfz5k7genqP+0N9mfX2dt956ixs3bpBKpZBIJIyOjjI9Pc3LL7+Mz+Mk/8e/xvZrt9j58ICNQg2rUsaAz8D0L13E9ODDxD0nWV1d5a233uLOnTsUCgV0Op3IeXvxxRcxNY6I/e6/ZvuNJdbW08Qq7YzmxJCFk19+Ct2FZ9moG7l3754IOulIvXXKgS+88ALq5AbB3/x1Nl9fZzFeJFlt4NcqmBmxtbPYY8LznTt3eOONN1hfX6dSqeBwODh37hwPPvggL730EvK9u2z+xm+w8foG85kyR/UWfbrj+/zPL6CZ+2kW4hVu377NW2+9xcbGBrVaDafTyYMPPsjc3BwvvPACku2bbP3mb7Lx+gZL2QqFRosBnZLxGRcz/9MXUJ15ivlQmvn5eS5fvizy+9xuNxcvXuTMmTM8/cRjNO5fZvsrv8v2d4MEspV2mVOvpO+RHkZ+5e+hmHyE9+eXuHPnDh988AEHBwdIJBJ6enp4/PHHmZ2d5eyQl+rd77H2219j58MDQqU6GpmEIa+B/qeG6f3lX6Hpn+WNN95gfn6emzdvkkqlUKvVjIyM8PTTTzM1NcWIukT55iXu/tZbBNbTRCoNHCoZowMWBp6dwPvL/5Sc0sa3vvUtbt26xd27d8nn8+j1eqampnjuueeYmprCn98ie+Vt7nzlXRYiefG7mh6yMvziLJ4v/28k63K+9rWvcePGDe7fv0+hUMBsNjMzM8OLL77I9PQ0XYlF0u98jztfeY87qRLJartycXL4+D7/6H8nXka8z8rKiijBd+rUKX7mZ36G8fFxXPs3SV95l4Xfu8pSskSm1mRQr2RyxsXQC6ex/b1/TjiT5xvf+Aa3bt1ifX1dbG888MADfP7zn+fLX/4yS0tLn4kDNBgMwszMzKdev3r16t84wB9p9D8zEkMikfwGkBEE4dclEsmvAhZBEP6XH3af6elp4atf/SoLCwssLS2RSCREbc5OecTn8zE5OYm2FKe+fovE9y9TDKfIbLXr5FKFDPuYG8uwD/3USZQzT1CW6wgEAkSjUUKhEJFIhGq1ikwmw+12Y7VamZycxOl00mUz0li7Sj24QvidWxSiRxSiBWRKKSqjCse0H/vsGKqxs1T8p4jH48zPzxMIBAiFQiQSCbHW34n6H3zwQSyyGtWbrxP6+uvE78e5H8jQFARkEgmTJ6y4Zzz0fukllOdfZj+aEA/kW7ducXR0RKvVwufzMTExwblz53jyySexqyDzH3+D0NsLrF8PczdboSmAVSnjydVPowAAIABJREFUwX4z3rNdDP2P/xjZ2MPcW7rP5cuXWVlZYX5+XuyjjI2NMTQ0xLPPPsvExASmeobMn/028TsBAm/vkKo1KTdbDJrUOMbs+B8bx/HCF5D2zrK4tkEwGBTRhh1VnQ5isZMddFv1NFY/oBpY4ODdBUrpIqVUGaVOidqixnV6COPoCKqRU9S8U6RSKQKBAPv7+6L6iUwmQ6PR0NPTg9vtZnx8HKfTiSK5ReX2d8lvh4hcW6dyWKGaq6Jz6dC7dLjPjWM49yiy3il20iX29/e5ffs2gUCAVCpFtVrFarXi8Xh49NFH25/xgJ/60jtk3nuHta/dIhXOi07YrpQx9cIIXecn0T398xzUVGL5LBgMsrOzg1QqRavVcvLkSc6fP8/s7CwzvS5q998j8Nv/iZ0ruyweVYkdH/LDeiWnXxql6/wk+i/9U0IHUW7fvs0bb7zB/v4++/v7bc1ZnY7HHnuMsbExLl68iFcLtaUrbH31ayRXUmwFs7QAtVTC0JQT7/kBnOcfQPXEL5I8bCMmNzY22NraIh6PI5PJMJvNjIyM0N3dzZkzZ3C73Shia5RvvU1uc4/Y/Bb1Yp1GpY7eY0DvMeM6N4lq4hyy7lEihSbJZJKNjQ0ODg7IZDKUy2U0Gg1Wq5WRkRGcTicjIyPo6jkaW/Nk3rtMbidKfDFCs9ZEaArYR2yYBjw45s6geuBZSioL8/PzbG9vc+/ePXZ3d8X79vf3MzAwIH5f2v07ZL//OuGrKyy+s8thvclRvcWEUYWtx8j0338Mw/mnqfed5YMPPmBlZYU33nhDBO90vv/HHnuMRx55hOHhYVTzr5K+dp37f3SThePSvVIqYdSgZPCEjZO/8tNoTj1KRN3N8vIyN2/e5MqVK2SzWUqlEl1dXXg8Hj7/+c8zMjLC9PQ0tQ++Rm7+Fit/cp2jcJ5IpYFdKcNk1dD3+ADeJ+ZQzz5MSttNOBzm9u3bBINB/uAP/oBUKvWZOcDTpz/dv125cuUnygH+F+lmCoLw/rEczkcfz9NWCAD4I+AK8EMdYKPR4OjoiFwuJ4rcdkqhLpcLl8uFw+FAJ23S2F+nvLFManmfQrRAPlpAZVSisajR2IxofN0o+qfI1KVEdre5ffs2e3t77O/vUyqV0Gq1+Hw+3G43PT09jI+PI8/uU37jz4hdvUs6ECe1nkYqk6LUK3DN+LAM+bA++zLSwQc4iES5873vsbe3x+LiIuVymXq9zuDgoOikJycncbmc1N77UzLLi4SvrpAOpKkV6owcq0tYBp14X3wOxcAMh2onG/MLbG9vs7m5SbFYxOv1Mj09jc1m4+zZs+Lm18VXqAQWSC1tUy83sFs0PGXRoLaocE266H50FtWJWZrDF9jf2xf7oDqdjuHhYbGEdfLkSfx+PydOnMCsaNEIrlPNttGcRrsGXa2FRCbBMWbHPuHDPDWB1DNModYkm82SzWYpFAoix7BTcnY6nTgcDqxWK0JshWpggeTCJvH78ePyWB1LnxmZUore524DFXpPsrGySiQS4dq1a6LKhkqlwuFwiK+zu7sbr6pG4/a3ib//HpHrAfKRAqV0GY1Fjc6po/eJKQyjI6geeplEsUFoPcT169eJxWKEQiEAETA1ODjI0NAQExPjCAfLZP/Tr5Na2ia5HKVyVMVoUvPwsJWuM34sIz2Ynv156BplfX2dpaV2mbJcLovqP/39/XR1dXHmzBkGBwexqqDy7p+RC2xSOaygd2o5rZajtWkw9ZhwnezH8fRPIeubYT+aEB2JwWCgq6sLm82Gw+HAbrdz4cKF9vv3dtFcfpdGfB+hKaDUK+gyqVDqFKiMKtwnfdimT6AcmiZfrnF4eEgqlRIF2gFRUs1qtYogFmUxSTUwT/L2CodbcRL3kwDIlFIs/RbUtjaoRDZwmmSuzOLiXfb29lhaWhIFojt7dXBwkMnJSdxuN9x7i6PFm8SurxC7F6Oaa/fiLP1mzP1W/J9/EuXoGUr2YVYCAXZ3b3H9+nWxFNj5bGdnZxkZGaGnpwdbKUx9/lX2XrtEajVG7iCHw6TCb1Fj6Tfjf2IW7egUyoe+QDgcJvDhhywsLBCNRjEajTgcDjQazcfK091OG63QAsnbt8ntRFHqFcx2tYFL1iELjgkvjjPjaB79IjW9k/2FBVKpFM1mE6/Xi8vlQi6XMzQ0hNfrZXJyEq/XixDboBHfo5LNI9fI0Tu19ORqGLsNGLuNOGaH2nvAOcDhfkzkUOZyOX6YWPWP+5BIJH9Dg/hrMfyX9OAkEklWEATz8f8lwGHn+V/6O3EahEwmO9XX14fFYuHhhx9mdHSUn/qpn8KtlVC5+gprv/8tYvcSXAnnaR6/z4t2LV0n3Yz9nUfRPv3fcIiWS5cusbS0xLVr14hGo9RqNcxmM4ODg8zMzPD888/T29uLevUy8TffYPedda4uxMnUmhQaLSZNKnqtGmb/wUNYH34M+dnnuXbtGqurq7z66qvEYjGSySR6vV4sVTz++OOMjIzQrypRufsugT96k8AH+2wUamLprVer4OwLJ+i6MInhb/0KB4clFhcXeeWVV8QeTQcBe/r0aaanp4+BIKdprV8l+JWvEL6xx92NDHulNvpwUK9kesyO76EB/P/wv6fln+H69et8+OGH3Lt3r9NAFyWu+vr6ePHFFxkbG6NLI3D49X9H9PoK11/bYr9cJ1lt4tXIGdYrGXumn4EvPody9jHWM3WRbrKwsCBOkXA6nXR3d/P8888zOjrK2ZPT1G9+h+Tl73PvD26wk6uyeoy2dKjkPPhAFz2PjeN47iVy3lmCwSDf+ta3RHpGB9jj9/uZm5tjamqKp59+Gl05wdErf0DgL66zv5TgRqZ9iGtkUh7ym3DPOBn58s+jPPMs++k8b775Jqurq1y9epV8Pi/qbvb19XHx4kUeffRR+vr6aL79e8Q/uEXglWXmwzmS1SYyCUwYVfT1mTn5y0+iPf0IOe8sCwsLLCws8Oabb5JOp0mlUiLF4oknnuDs2bNMTU1hz25Rvv19Fv79d9nZyIgADqtSxmmXjpEXx3FfPIP6+X9MMBjk/fff55133mF3d5dQKIRarUav1zM3N8fExATPPPMMfV0O6vNvsfE7f0TsXpwPtg8pNFo0BThjUdM/6WTohdNYf/bvUza2FYMWFha4ceMGwWCQWq2GWq1meHiY4eFhXn75Zfr7+7FmNki++jX231vlgw/2ydSapGpNRg1K+vVKpn7xNO6nn0B59jnurG6yvr7Ot7/9bXZ3d0kmk8jlciwWC5OTkzz99NOMjIww0WWmdu8yu3/+GivfDrBRaHMdrUoZXo2c888M0DU3ivXn/iEJDKyvr/ONb3yDYDBIIBAA2s55YmKCkydPMjc3x4ULF5BHVzn4/d9i78oGqyspVnNVALwaOaeHrHSd8TLwj76MbPQh7t1vo3nv37/PjRs3xEkLAwMD9PT08Nxzz3Hq1Cn8LivFb/xfRD68z91vrIr7tZOZD5/tYvyXfgr1qcfYFSysra3x/vvv8+GHH4q8Rbvdjtvt5rnnnmN8fJy5uTkk97/H0dV3uPOVd9lNlbh/1G4HWJUy5sbt9D01iuupJ2mc+jyhUIjXXnuNe/fuia0XpVKJ2+3m3Llz/Pmf//lnRoMwmUzCuXPnPvX6d7/73Z+oDPC/Sgd4/PxQEATLD7vH8PCw8C//5b8kFAqJWaBOp8Nms2G32xkYGMBqteL1elGUMwjpA+qhVZr5Q5rZDMJxpCQ3W5HqDMgsTmSOblAbkFi8CAo15XKZarVKo9GgXq/TarUQBAFBED4RaUmlUvFfB2at0+mQt+pQyiJkDmjm0jRjezQLWZqFAs1ae1yMwqBH7vK1X0PvFA2tjUQiwe7uLplMhq2tLTHbValU6HQ6fD4f/f39uN1uBnxdCNF1yjffJnzlDun1JAcrSWotgaYAXq8ec4+J/p8+0y7xDT3A6s4Bu7u7XL58mbW1NWKxGPl8HrVajcViEQFFTzzxBP39/bTW3iPyZ/+J+N1dbn54QKzSIFVrYlfK8GsVTEw5GHr+NMbTZxHOvsju7i63b9/m2rVrhEIhdnd325+3XC6CVx599FHGx8fp7u5GuP1tKoFFQm/fIR1Ik4sWqLUE9HolJp+B7oeGsAz50D3yIhL/FLFYnI2NDZLJpCgEXK/XMRqNGI1Gurq68Hq94hqQlQ4RUrs0IkFa+UOauR9ICUr1ZqQaHTKbG6nJ3l4DRieodNQazU+sgWazKa6Fzv8/ug7kcjlKpRKFQoHBYEAhk8JRDOEwQjObpH6whVDKU0kfITTbf6u2GZE7vcgcXuT9szR1dsLhMOFwmFgsxsbGBrlc7mNrYHBwkJ6enjY832NHiAQ4+t43SdwJELsX5SCYpdZq73GfR4+px8TIz11APXUO+czTLC+vsLOzw6VLl9jZ2eHg4IBKpYJSqaSrq4upqSlGR0d58skn8fl8NK59g9ilS4Q/3ObWnSjJatsBdqnluNVyTl3oxv/oJOaHn6A28ggHBwe89957LC0tsb29TSQSodVqodFoGBwcxO/38+STT9LX10efv5v6re9QCSyy/dotcgc5kSuqMqqwDVlwzvZiHR9Ac+EFJO4h9sMR9vb2SKVShEIhccyTwWBAr9fj8/nEbNXj8SCr5hFimzQzMZpHaVrFHLRaIJUi1eiQqI/XgMYAWhMSvRVUOhrIqNfrNJtNarUazWaTZrNJo9Gg1WpRr9fFtXB8fqFQKFAoFCL9oj12ah+OYjSSYRrhbRq5I0qJLEKzhUQmRes0o7C7kHcPiGtgZ2eHeDzO3t4e29vbZLNZcrkcGo0GvV7fDk67uhgeHsZrN3Hm7APML61+Zg7wwQcf/NTrly5d+olygP81ETriEonEIwhC9HgCcOJH/YFer+fMmTNotVpu3LhBNBplY2NDRPbZbDacTqeYHfX29jJ49lmau0scvv0d9t9fJrWeYS2Sp9xsc5d6tQocdi39Tw3hOT+NeuYira4pkskkN2/eZHV1le3tbXZ3d6nX21mV2+3GZrNx4cIFhoeHOXHiBFZJmcb2ItFXvklycY/YvQSBbFvhRSaBXq0Cu9fA4HOT2GbHUZ/7HFmVg3A4zAd/8i22t7dZW1sjmUxSq9VQKpU4HA68Xi8PPfQQfX19TE5OYqylqa/fYvu3vk1qNUZoMUGk0qDcbIN2erUKbIMWBp+bxTQxhvqRL5As1tm53yZ9h0IhlpeXyefz1Ot18bAYHBzkgQceEA9XogFqW0uUohmquSpqqQSrUoZSKsGtluPRyFFb1EiVcmg1UamUGI1GPB4Pfr+fVqslZlaVSoV4PE6j0cBoNNJoNMjn80ye/Txy7yC9gEy5gEQqYW37kFCiSDlawHc/iVOnpP+JBdznxrGfvoBt7kni8ThSqZSVlRWCwSDb29sfWwM2m425uTlGRkYYHh5m4OTTtPaXyb76NRIL20TvxNhMFCk02o7Ip5Fjt2gY+ukRHDND6B56joZrlEQiwdWrVwkGg6ysrBCJRMT5hDabDZfLxYULFxgYGGBqagqnRkpj+xbRb/4F6dUwe9faoI9Co4VGJqVLLcfVbWDg2XFsk0NoHvsiBZWVUCjEtT9rK+AsLi6STqepVqviRI7+/n5Onz5Nb28vp06dQl8/onbvHXZ+6xLpQJyNWxFStXZfy6qU4VLJ8E058T8yimV6HPVjf5vDqsD69RvMz8+zs7PD2tqaqNfpcDgwm81MTU0xNTXFwMBAe9RNaod6dIfqYQGh1UInk9JUIq4Bu1KGXC2n1WrRyh+iVauwWCz4/X4KhQLVapVisSjOiTw4OBBneabTaYrFIhNzLyH3DdNba5C4E6BVb7EZOuIoV6W2d4Tr6gEe/S0GnrqJfWqArsc+R/eDjxKJRGg0GuJ3s7+/L5baO+XQ8+fP09/fz8mTJ/HNTCHbvUfim39CemWH3ff32c1WOKq314BLJcPr1NH/1BD2qQGMT/8cgmuEcDjM9evX2d3dZX5+nmQySbFYFPujPp+PBx54gP7+fk6dOoVdp6Sx8h6JN19nc3mP7ff3iVcbJKtN9HIpdqWMngEzPY8M4ZgdRvPkL1DR2lkOBLjzF68TDAaZn58nn89TKBTQaDSiru/U1BSDg4PtNSBUqH74Lfbev05p7+AzOaDhb3iAf32GP5kB/p9A+iMgGKsgCP/kh91Dp9MJU1NTIu+qw+cZGBigt7cXXWKN2sYCe9/5Hon7MdJbh+yX22AOtVRCn6/dVxt8/hyaiTPIxx8mFM8QDoe5du0awWCQ3d1dstkszWYTvV5PT08PXq+Xhx9+GK/Xy8jICELgKtXVWwS/9S7pzQwHwSyH9RZNQcClkuP2G7GP2Oh74SKq4VkYf4ydnR1CoRAffPAB4XBYJM52Du3u7m6GhoZ44IEH8Hq9DHhsNLduU7r7Aduv3SK7k2U7dCSSjnu1Csx+I11nfHQ/cxHl8CylrinC4TDr6+vcunWLcDhMMBik0WjQbDbFGV4zMzNMTU3h8/nwagSaO/fIvHeZg/dWSG8esh0r0DxeJv0WNcZuA31PjmGbO4ty9CxptYdoNMrdu3dZWVkhFAoRDocRBAGJRCL2Oh588EEGBwfp6+vDVkvR2LlP5Duvk1g8IBVIs3sMylFKJfTYtZh6jAw+dwrD5DTKmceIVWXid7Ozs0MwGBQHiKrVarq7u/F4PDz00EP09PRw4sQJ1MkNaoF59l65RHo9SXQ1RbzaOBY/l+F2aLGPWBl8/hzqoUkUD3yeg1hSpJF0Ss2FQoFGo4HJZMLlctHf38+FCxfweDxMjI0ihO5Sufc+O9++wmEwS3AtTbHZotZqrwGHR0fXKQ/+p86iHJ5FmHySaDRKIBDg5s2b7O3tsbW1JWaadrsdl8vF5OQkMzMz+Hw+BrvstPbuk3v/LfbfXSSzmWErmP3YGjB69PQ+NoTrwRlUE3OUuqaIxWIsLCywvLzM7u7uJ9aA3W5nbm5OFHjwaqGxe5/kG98hfjdIcjVFKF4UM8k+owpjt4GBZ6ewTI+jOvk4WY2LaDTK9evXCQaDbG5uEovFqFaryOVyPB6PGCD09vYyOjqKqZ6hsbVA5NXvkF6LELkTI1puB286mZQukwrroIXB506hHx1Fde55khWB/f19PvzwQ/b391lZWSGfz4v9arvdTm9vL3Nzc/j9fiYnJ1GkgtSW3mf/9cukAwlCiwlyjRaFRgu7UobLqsE56aD36ZPtHuD5v0U82QZWdbAAnQChM/zWZrMxOTnJ9PR02874GMLBMuVrr7P//XlSgZS4BspNAZ9GjsmqoediD13nJ1FPnEWYbAdvi4uLokzaR9dAJ3g7e/Zsu13S389AlwMhskbmjb8gfidA4n6S0O4RxeMA3qdR8KtHO2xUSp9JBmixWIRHHnnkU6+/+uqrPzQDlEgkzwD/FpABfyAIwq//pev/N/Do8VMt4PxIa6wJ3D++ticIwuf/qu+j8/gv4so/ZSTGrwN/LpFIfgnYBX72R93HYDBw5swZcY6XTCYjFoshl8upVqv09w9ifvo0g2MP0Lu/QT26Q25zj0alSqNYQaZRotBqkGm1tPJZWvvL9HpH8Pl8DAwMUCgUODo6olgsiqUNtVqNUqnEZrOh1Wqp1+so/ZOoLR5GhmdpFXMIpTxCq4lEKkOibZdWJXorEnsPdYmczDGx3W638/jjj4vllM69OyNQzGYzqlIKIZeg+uEVGokwpUQWy6ALc5+DPpkUjdOCyqxHOzaLvKsfSfcE8VSGdDpN4JiM3wFJyOVyxsfHRe7S+Pi4eJhryymae0vE3nqdo+0wkTtRktEiuUbrWDhAhtGjp+eRgTaw44kXkfSf4eDggNvHMlPXr18XZ7FVKhW0Wq14WPj9fvr6+vD7/dh1Sqrz73O0ssr+B9vsbWbYLtaJVRrIJG1UquWogjqnollvHC8aqfg5VSoVCoUC+XxenAbeicLr9bpIRrdarfi9YyiFFo6ZbeRqJc16E9lWux9WawlUc1WOdnPE5wOYDwuYVWq6h89iO3kSvV5PMpkUD/POQavRaMS11mg0UCgUOJ2DWF44yejMRRrxPWaCK5QTWWr54vHcOhUqsx6J1kCrmEOR3MZvd+NyXWRsbIxisShOVW82m8hkMpRKpahcotPpQG1A6h3F9IwR/dmLCMU850o5kMqQSGVILQ5kBgsSazeY3JQqVQ7Taer1Ot3d3djtdhHNq1QqxQHHOp0Op9OBkD6gFd+kfGeJWjJBo1LF1GND5zIwpFWjshgw9rYBSHJ3D03vJJlMhv39fba2vksikWBra4tyuYxKpRLVlvr6+hgcHMTtdjMyMoI8F6N251WiH14nvXbAwY0wh4UaqVoTjUyKSyWn+4QV13QXrtMn0D/xt5B0jbEW2GBjY0OUrEun0ySTbdBNZ+pCh87UCYYUh3vU1m6QvnWHg2u7xA/yLB5VxH5oTatAma1gyrXPBKFWAeEH5exWq0Wj0aBSqVAulymVSmLp02AwYDKZRD1fp38KVeEQd7GM0rhDo9yglC5zWGgDeOrFOpnNNDJ1AGupgklroNvVj+5YdD0ejxMIBDg6OiKTaZfnFQoFrVaLVCqFXC5HJpNhsg5j+7u/ivW5XYYiQU5vr1DLl6ikj5DIZGh+J/ljntA//CGTyf5KfyeRSGTAvweeBA6A2xKJ5DuCIKx2fkcQhP/hI7//j4DZj9yiLAjCp3Mw/iqv6SeZCH/69Gnh7bffZmtri4WFBUKhEHfu3CGdTnN4eCgiQ81mM11dXXR1dfHQQw/h9/vb8mcmrRhNpVd2OLixz8ZeW8UkXm2ikHAMZVbRZVLhP99N14UJdKOTKOde4rBYIRAIcP/+fUKhEPPz8yJ6rmO7g5jz+XycP39eRJC6zDqEgxUOL32T9MoOe++HCEYKhMv1j9k+YVDh1SvpfdSP++wo+olplHMvcVSusbq6yvr6OltbW8zPz5PNZsWM6C/bnpubo6enp400tZkRdu+R/d4rpJd32L0SJBQpiKAWAI2sbdujkdP/RC/OmSGMp8+iPPvT5GsCy8vLbGy0D6SFhQXxMOocsB0n29GK7O3tZWpqCpfdihC6S/Z7r5BZ2yX43U3CiSKh0sdtD+qVdGsV9D/Ri31qAPPZcyhPPUOhJWdxcZFQKMTq6uqn2rZYLAwPDzMzM/MDgrvLSWvjOvkP3iSzssPWW4FP2NbLpfi1CnwaOQOP92Ib78V28WEUM49TEJTid720tMTS0pI4XPmjts1ms1ii6sjlOZ1OhM3rFK9dIrW0zdZbAWLJEtvFGplak6bwSdvWUT/2xx5HMfEwRalGFAK4d+8eKysrpFKpj9m2Wq2YTCYmJiYYGRlhcHCQ2dlZHA4Hks1rlG5dJrW0zfaldeLxIhuFtu26AEa5FK+mDbzqe6QH22g3jiefQjH6IEWFkfv374uUkI2NDRKJxKfaHh4eZmRkRKQKSTauUl36kMT8OtuX1kklSizn2uo8HdtutZwBXdu2ZbgL1+eeRTF8hqLSLJY0b968ydbWFrFY7BO2jUYjk5OTDA4OtjWBJyZwOp1INz+kunyDxK0Vgt8LkIwWWc5VKTRaVFqCaHvIoMR/vhvLiS7cP/0c8oFZSho7q6urHBwciO2CSCRCIpEQZzmazWbMZjOTk5P09/eLPVOn04ls6xq19TvErt5l98o2qXCexaOP23aoZIwYVXSf82Ib8+L+3OeQ909T1LkJBAIcHByICkMHBwefsG0wGJiamqK3t5eTJ09y4sQJXnzxRe7evfuZZIBWq1V46qmnPvX617/+9U/NACUSyRzwLwRBePr4+T8FEAThX33K71+jPRP2e8fPC4Ig6H/Mt/BxGz/JDtDlcgk/8zM/Qy6XEyNmh8OB0+nE7Xa3IeVWKz6fD1k2TCu2Tfnu++T3E2TW9shHClRzVRqVBjKlDJVRiaXfgsZhxnFqFGX/OLLuE9QsPRweHhIKhYhGo8RiMXZ2digWi6JtuVyOy+XCbrfT1dXVRsxZrfT09CDPx2mGA1QWr5LfS5BeCVGIF6kcVj5m29RjQusw4zw7jsI3hLxvkpqlh2w2SzAYJJFIEIm0m/4dMMRHbVutVhG5aLPZ2rYLSZoHa1QWr1IIp0gtBX+0be8A8oFp6tZestks29vbpFIpDg4ORKJ+Npv9hG2/34/P5xMdn7yUphlaorp6i/xenNRSkFKqTClVollvisLFxm4jOrcF59lx5J5eFMOnadp6yWQyBINBcdp1JBIhl8t9wrbZbKa3t5fu7m4cDgc9PT0oKkc0tuepbdwjH4p8wjaA1qZF59Jh8Frbtl0+FGNztKw9pNPpH2m7Q9voqOO4XK627VqexuYtaltLFHYjJBe2KB9WKEQLtI4BLx+zfXqk/b7H5sDeJ4J60uk0W1tbIv8sn28DQjq2O0hll8uF2+3G7/ejbJRobNygHlwht7VLemWHUqr8Cdtauwaj345jdghFVx+KifNIHP0kEglCoRCHh4eio/thtjt96e7ublTNMo31a9T3NshtBD/VttqixjLowjbeh8Lbi3LqIhJHH4lEUgR9bWxsiCopnd5kJ6s3m80MDQ2JiOKuri401GkEjm2vb3G4vkshXvxU29bRHpS+/rZtWw/xZEqslKyvr3N4eEg8HherPyqV6mO2O2vc5XKhlTRoBK7RONjmaDXA4foupVSJ3EFetK2xqFFb1FiHPT+wPXEBib2HeKrddkmn0x+z3QF1dShIVquVgYEBHA4HfX19OJ3ONsUrcI1GJER2cZmj7TAvffMdVrO5z8QB2mw24ZlnnvnU63/6p3+6C6Q+8qPfEwTh9wAkEsnLwDOCIPx3x8//DvCAIAi//JfvI5FIeoAbQLcgCM3jnzWAe0AD+HVBEF79cd/PT3Q3s1gssrCwgFwuFyNvu92O3++nu7ubwcFBTHotzZV3KW8skF3dJnJzm1ISmHWSAAAgAElEQVSqRHYvJ/Y09HolKqOqza+ZGULb60d96gkknhOkMocEFxdJJpOsr68Tj8dFrpFUKkWhUIgH4YkTJ/B4PHi9Xvx+P7J8gvrNVyiG1sis75Ja3qdyWKEQbyuwSGUS9B49hi49lmEflunxNvpr7GHy1QbRaJSdxSsiArBjt16vi+hCj8eD0+mko4jT3d2NWVpFSIYoff1PiG3sk1lr99gqh1Wy5ToamRSFTIJtyILeY8B1egjTzAzK3lEa/pOk02l2dnZYf+tDwuEwKysrHB0dcXR0hCAIyOVyseHv8Xg4deoUHo+HgYEBDI0czf1VDv/db5NZ3yV6J8Je6IhUrUmm1qYMaI4nZ1u69DgmPHjmxlGdmEV+8nNkskcEAgFWX39PzOiz2aw4Nb6T0XcO3o44cRtJ6kU4WKb4Z79GdH2XvSub7G1m2C83RBqITNJWvnGpZAx5DJj6nNhPjoqowo2NTXbufv9jklfRaJRSqUSj0RDLuh2HY7fbGR4exufzYdUpqd96lcL2CnuX75BcSZHczxEq1UWQlUPVBov02LSoLVrMwz7UMxeRugdJ1GREFhfZ3d1lcXGReDwuBlqd71yv14sBh9vtFgEqLpeT1vpVqvsbJN6/Rm4nzmGwLb9WrTVpCm00rcaixtJvwdjnwTw5gnr6ISSOPuL5Kon79zk4OBCdTzgcFg9hvV4vrreOOPTQ0BBWqxVdPUf93luUw9skbi5RCKfJ7h5ROazQagpIZRJ0Th1auwbn7ADGPg+qiTlkfTNU5TrWtrdJrr5PKBQiGAySyWSIx+MisrbjePr6+ujv78fhcDA0NIRBo6S1dYvKG98ktrlH5HqAQrxI7iBP8bgHb1TKMHj0GH0GuuZGMA72oJ59BEnPDKnMIevr60Sjd1hbW2Nzc5NMJiNq78pkMiwWC1arlaGhIfr7+/H5fO3+pUZJY/ldiu/8B3ZWdti9ss1hpMB2scbRsW2TQtZGxzq02E44MA/7sD7+DPKB0xQkahaWlwlfXWR+fp7Nzc2PZdWtVguz2YzRaGRkZASXy8Xw8DDT09M4HA60qQ2q775JdHGF4KX7HAazLGUrZGpNks3qZ3bm/r8AwaT+mlCgXwC+0XF+x48eQRDCEomkH3hHIpHcFwRh+8cx8hOdAZ48eVLoqF90IuVOaahDuG40GkgkEnQ6nQhisdvtIhijE8lpJQ2EyBqV+20UVfxOgNx+ntxBntRhmXJToNJqT5bWyCQ47Fr0Li3WQRv2qQG03Z72xrL3UJJpiUQipFIpcWGHw2EODg4oFosUCgXxPZhMJgwGg5g9OJ1OMbJ2OBxoKxnIRigvvEduc4+j7TDJ1SSlVJl4pkylJVA7fl16uRSTQYltyIqhS4/r7Cja3n4U/RNIB85QKFcJhULEYjHRsXV6OIVCgUqlIjo4jUaDy+XCZrMxNjZ2TKZuj4OxWCxID5aobS1S2lwncnWZ3EGO2HqGSKUh8iM7zs6nkWNXy+k65cY67MY20Yf27GNInX3kFGZisRiRSESUpetkHp2Bu4A4b06v14sgp76+PlHhpdvjQthboha8T+LKB+R24oRvRdlPlYhXG8QqDRHI41DJMClkjNg02E/YsJ5w0vXoAyj8w8hHL3BYaZJIJFheXiYSibC6uiqCoTrl7VarhVarRaPRiM7Q5/MxOzv7AwqOTokQ36Z8623ywQPi85viZIiOUyw324hQjUyCT6PA4dFh7jHhvTCKob8b9dQFpL4Jii0ZwWCQVCrF1taWCADa29ujUql8LEsymUwiFWhsbAyn00lfXx9erxeLxYIiEaAZC4liA/lwhsT9BOXDCqlKg8pxYKiTSbGoZGjtGhxjDgxeK/apAVQnZpB7+mh2jZPNZolGo+zt7ZFIJEReWodE32g0EARBHBjs9XrFILWvr0+ctmCxWBAi69QDt6lHdojdWiO3l+ZoN0cqnKfQaFFsthHUCokEl6Y9usg6aMU+4cPgd6E/8zBSVx8tex/hcFhUCOpUTXZ3d8nn82SzWZHG1DkXnE6nOGuys6Y8Hg/6WrYdTN66TGZ9l/RamNi9BKl8jf1yXewlamQS0eF5+kzH3+E4+uFhlCdO0fRNi1n13t6eKAaQSqXEAKsT4Gm1WgwGAz6fD5fLJZbxvV4vAwMDqGp56kvvUlpfJLOyw977QfLRAuu5Kslqkz9shEkI1c8kA7Tb7cJzzz33qdf/8A//8K+lBCqRSBaALwuCcO1T7vWHtEGU3/j//CY+ep+fZAfY4QF2hGmLxeLHACtGoxGNRiMKSnf4YB2pNGXlEOEwQm3jLo1EmFwoRm4nSvmwTDldollr0Wq2UGjkyNUKdC4teq8drcOCYXQEmbMbmW8E7H3kcjkikQjZbFYsVRYKBeLxONVqVZRSk8vlYjnDbDbT09OD0WjE7XZjt9vRqhQIB8s0Yrs0wtscBXaopI/I7iSp5qrUi3VazRZSmfR4hl87k7CO9KBx29ul055xJGY3R3UJuVxOFOjOZDJiKa9QKIgKHwqFQgRaeDweUZnF7Xa3FTCsZoTMPq14iNrWEtVEgsP1XcrpHKVUmWquKooHK3VKlHoFRr8VrdOMwe9CNTCG3OFF6p+kJtdyeHhINBrl6OiIvb09MpkMR0dHpNNpkWunUCjEcltnDFTnu+vq6sJsNqOoHNHcX6aZDFMKrFAIJynFMuQOjmiUG9SKNaQyKRKZRCxB6dxWzMM+lA4nquFZJHY/ErufeDxBPp8nEomQTCbJZDJEo1ERbNNZU51yVCcLtFqtovqK2WxGL1QQcgnqwSUa4W0K4STZjX2quQrFeJFGpUGr2UKmlKHUKdvl5z4napsJ48ggiq4+ZO5eJN0TFApF4vE4yWRS/KxyuZxIqO4gOTtT1h0OB0ajke7ubmw2m/ja9HodQiRAM7LZXudr65QThxTCaUqpMrVijWatXZKWKaXonDrUFi3GXg/GPg9ylx/F0AwSk4eyXEc6nSabzRIOh8lkMiQSCbFU2QlYZDKZyMXrtCU+uv/sdjvSwwM4ilFdm6caj5ILRcnvJajmqpRSZYSmQKvZQqlTItfIMXgM6Lx2tE4L+rFJZA4vMu8J6non+XxeBGDF43EikQj5fF6Ur6vX62KbxGw2i/vP7/eLyF6Hw4FSqCEcrFCP7NAIb5Pd2KOczpHbS1M9qn3i+9O5tKjMeqyjPajdbhT+YWS+ESQmD6lckXw+TzKZJBwOi8FCh9JQrbYzNZVKJe6/Dm+1U9Y2GAxY1LI2fzQWpLa1RDmW4nBjn3K6IEr5ffRM+G/n735mJVCHwyG8+OKLn3r993//93+YA5QDG8DjQBi4DXxJEISVv/R7I8AloE84dlASicQClARBqEokEjtwHXj+owCav8rjJ9oBKpVKwePxoFQqsdvt4riW3t5eent7GRsbawNB1BKa+8vUtpaIXblJdidJ7F6Cg2N9xeSxYDaAWy1vCym79diGLFiH3bgfOdde2CMXyBTKxONxVldXxexgb2/vY+CXTqlMrVaL4Buv18vJkydxOp0MDAzgtFkQ4ttUbl2iuLNH9FaAxP0EqUSJjUKNwjFMW3mccfq1CvyGNvzc99AApgEv+rMXkffPUlNbRL3GjY0NUcc0GAxSLpf/s9JjVquVqakpkcrR09OD1WpFk9mmsb9BYfE28VtrHAYP2VlJEa82SVYbIldOI5OK3K/+URvmHhOeuTH0UydR9IxSc42I5OR2mSkqRrwdyaYOWEev16PVavH7/Xg8HkZGRjhx4oRY9lHV8jR3Fijcfo+j7TD7H2xxtJtjJ1clfByF1wXQySRiedVhUuGeceE+PYhxyI/m7NNI3MOkckWCwSDxeJyFhQX29vaIHM96LJVKlEolkcjeKat3QFNdXV1MTEy0EX82C63NG9RDa8Tfv056LUxm85DNcJ5MrfmxjNOqlOFQyejVKvBMuzD3W/E+dgZl/wTyvmmyEp3Y/wmFQiIvrwNq6sy066BCO4GTx+NhdnYWn89Hd3d3e05dapfawvcprK2JwK6jeEkEvHS+P728rTIyrFdicmnxne/BOtKDfnwC5fSjSOw97OzsEA6H2d/fFyXBOplwJ2PpEP6dTqeoE9rX10dvby8jIyNYLBZMzTzN3fvUtpYIX7lDNpghuhhnv9zee5nj4Ekm+QGfcKDXhHXQgnW0G8fF8yh8w0iH54jH22XC5eVlYrEYKysrHBwckM1myWQybQ7iR7Jzr9eLz+fD7/czMzMj9s8sahlCbJPyzUscbYeJ3twksZIieVRlu9jef+WmgOZ4TfVqFXiPJdN8Dw1jGvCiOfUo0p5pii0Zm5ubxONx1tfX2djYIBqNiuPZSqV2y+Oj2bnD4WBmZkZExvr9fmw2G7K9Beq762Tv3iF2M0B294jg1iGp4zUltm3kUrwahcjxtPRb8cyNo5k6x9zf/SfML618Jg7Q6XQKL7/88qde/53f+Z0fRYP4HPBvaNMgvioIwq9JJJL/A5gXBOE7x7/zLwC1IAi/+pG/exD4XaAFSIF/IwjCf/hx389PtAOcmJgQvvrVr4qZTaFQIBaLiUTbDlFdKpWi0WhE2LderxdRip3yR4dgqqjlEQoZWvEdmkdpmocJqrEYjWKFcuaIZrlGs96gVW8gkUqRyKTI1UpkaiVqswGFUYvCZERu8yDRGpA7vKCzINGaEfQ2EcLfgVV3IsGO4kytVqNSqYgbufNQKpXiP61Wi1wuR6/Xo9PpUKlUGAyGdiagUkIhhVDO00oftOkdhwma2SS1XJFKOke9VKZZrlEvtYePCs0WUoUcmVKOyqxHrtOgsRlR2F1IDWYU3YOgtyExOikJckqlEul0WlSm6WRKyWRSHHfUyS6hHeF2AEomkwmz2Ux3d7eY+ZrNZnRqJcLBMs10lNrWEkebexSjaVKrcUrpEoVESUQs1loCSqkEtVSCw6Q67muZMQ+626WwyVPIPb1IvKNkyw2y2Sw7OzsiwGN7e1tcM6VSSZS9AsThsnq9XgQ59Pf309/fj81mo7e3F71ShhBdp3r/GuWDfaLXVzgMZjnazbF11CZTJ6ttB9ii7ZhNChlutZweuxZDl46uB/owDXjRDo2iGJtDYvMRT7erBx04fIdWkEqlyOfzYtDQarVQKpWoVCqRK+Z0OhkeHhYHJrvd7nbJuJFvVzkCdyhub1EIJ0ktH1BKlclHC+SKdcrNlsi/VEolWMxqERhl6rGh8zowjo60VWp8o0gcfRTLFRKJhAjZj8fjYrWhWCyKDvKj+68DJNHr9aIknE6na+v16nQYDAY00iZC4RAhvU/zMEHrMEEjHaOWL1E9LFAvlWnV2vuv85CplUgVctRmA3KdGqVRh8zsQKIzInd4kejNoDEjMbtpCBKRjF+pVD5Go+nsv1KpJNIfOoFjZyi0XC5Hp9OhUChEwfFOBqdWq9FqtW21mXIespG22sxRmkYyTD1foJzIUi+VaRQr1IsVhGYLodXefxKZFKVBi9KoQ2XWo3bZkZlsyL0DSE1OJEYnNZWRYrEoIt07mW9n/2WzWf74j/+YZDL5mTnAn/u5n/vU61/5ylf+Rgnm/69HJ6OpVqtotVrRcZTLZVEuqF6v02g0kMvlYgSm0+nEEoPVaqVcLmM2m8VJ5hKA9AFCrUIzm+FoO0w5meUweHhc8quRLdfFCF8jax8cRo8etVmN0WfAPNCF1m1DOzqFoqsPlBrR+cViMRHhFgqFyGazxONxstmsuDGhXUbq9E464sYd1KHZbBYV9A0GA/LMLkIiQTW4THF7i2IszWEgQjFRpBAvUc5WxJ6TUipBJpFg1ClQGZUYPAZMPWbUNiPmsQHkDi8K/wgS9xB1uYaDWIxMLEN6ZV7MAPb398XPOJ/PizJQnc+5A0232+10d3eLaNxO9mk16hCOEjR3F6nf2eIwEiG9vEM5XeBo94hSqkwlXxV7LAAmhQyVUobe057coHPqsY33ofXY2oLLngEkli6O6pL257wcEMtzH+3hdfh2Go1G5PR1pKo6BHSLxUJ/fz8mk0ksT+v1unZ5OrFP/WCL3PoWpViaymGFZq2JXC2nqyrHrmwTnzsOxWhSo7aoMHTpMfc50DgsmCdHkbv97dd87FA6ZenO56jVarFaraLIu16vF0uxnaCiU8qz2+2i41YoFAAIggAKFah0SE021I4MrVoDnTOLRCb5Qdm6/IPMvikIVPLtkppcI0cqk7TLkAYtqloFiVKNVKZAZ26X5wBROq3Dvy0UCmQyGZE32ZniLpFIMJlMYgDq8XhEBK/FYsFut+N0OlHqLW3ZuFqFZi7TLvcls2Q2U5TTZcqHFXLFOk1B+JjjNto1aO0adE4dlhNdaB0W9OMV5J5epCod9RYUCnlRm7eD7O6UcdPptKhU9FHeb2efdfqqnWytQ7w3GAxoKhmE1B710CqFYIBSLENmbY9CvEgx0Z5mUjnOKqENxjJr2vtP59Rh7jWhMZuxTw0gd3pR9Iwi9Qwj6G1Eo9H22t1eFfddZz3n83mx19o55zqBx2fx+BslmB/XoETiA/4j4AIE2jDZf/tXnQfo9/vFA0Kn0+F2u3E4HHg8Hk6cOCHSAex2OyaDHiF0l0Z8j9LqAodru+T2UsTvJykcVogcl0Nrx8CSzsZyq+VYFDJc3QbMfWZMPTZcZ8d+0BvpGqNQKouahOFwmO3tbZLJJDs7OxwdHVEoFCgWi2Jm1yG9dw6uDsDEbrczNDQkbjaTtI5wFKW2fJ1SKMjRVpj44gHFRJFwuCCOcmk7iva4JKtShl4upUenwHCs/+g62Y/ea0czdQ5Z12Abfp3Ois5sf3+fRCJBIBAgk8mIZcparSY25zvTAPR6PW63W1R4GRkZESkYTqcTVatKc3ueRnRHLMXlDo7aAJBCjXi1jQitHSuYdEAgXcflZ2ufGfuIDZ3Xjv3MdLsv5vt/2HvzYFu2sz7st3ruPe+zz3jHNz+eASGCYhcFsRHYmAJMXMQUDkElHEJRDo5s5hACdmQnIDtUoqoYG4yUUlIpVEiOY0RJBYShEuNIxdMAenrjffe+O5z57Hn3PKz8sfpbe3Wf3uece++7aGJV7ere3b3PPrt7rfV93+/7fb/1HNK16zIXNhwOJWOR9km0gDx40mQlB4KWsSEaO93vtbU1ON4R+GgX0Qv/H2Y3bmP2xj4OP7OPxaGH3XGIkwI+DDIhDKAzYMM20DI0XO076F5ti/v8dc+icWUH1ld8HfRLz4J3t3Hv3j0Mh0Ps7u7KJbBu3LiB8XgsnQh1AiMHgu7z1atXsbm5ieeee06yP7e3t4HJPpJXPiFzxcd/eguL/TmOXxlJjc5pImoMM87RMjRJSlrr2Ohd72Ljq3fQuryB3td+jYAZt5/CwmjL+0w1dy+//LKMmknSjuBiQMgSOo4j8+0EX29ubuLatWvY2NgQ5Kn9F5Ed3UX00vMYvXwb8ztHOPjMIfyTAPeKesygUNDR2XL8dQwNl7ab6D/RQ/tKD5tf9yycy9dgPfO1YJefQ5DruHv3LkajkVQ8Ojw8xM2bNzGdTmXeTY2gyYEgyb7q+Nva2hJEpvEekpc+gfju6xi/dhfHfyJKqA7uzOQ9Juanzhi6piZh0+5WA50rHWx+zdUlOnH1GbCNxzEKUnmfqcaPoO/Dw0N5n2n8kcNPBDVS13n22WclK3prawtvf/vb8fzzzz+SCHB7e5t///d//8rzv/RLv/RFFQF+PgzgDoAdzvmnGGNtAJ8E8DcB/ADucz3Axx57jP/UT/0UFouF7NyU7CbYhQgerVZLQi+NRkN6oq7rotfrgSUh+PykgA8XyIYHyENPKLuEPngSI09SWdejmwY0ywSzHDCnCWY70LsDMKcJrSGEdGG6YI0uYLngmiEjVFL7IGNYFdXWdV0uO0KeM02MquA2YxBCvrx4sWKZEqYBmgbOIb+HBj5t6XiWZVLcW3lG0HVd/h+2bUPXdSHuqwFII8CfCpgnCZBPT8DjUMCt3kzsBx7yJEUWL6EqgouZ7UBv9cBsV6iXdAZCeLi7Dbgd+TyDIMDJyQmCIJBQWxiGmEwmEjZWYW7btmGaJvr9voye6HlvbW1JVRVjcQy+OEF6+2Wkh3cQ7B9i/OpdhMO5pO+rUb7qDHWaZrGElou1p9fhDDoYfOUTMHYeg7HzOLQrfwGp1cLR0ZGEpIi9eefOHUmCODk5QZIksk8QxEaOnBodUX3jtWvXZAkIDl9DPtpD9MLHMX9jr2AHH8E79HH7RMDFNDFT76Ic6WVX5NnWtptYf3aA5k4PW1/3rKjJe+w5sCtfiZhZODg4kGU/N27cwHA4lGxPqgVN07Q0QVPurdvt4vLly/I3XLp0SaIArVYLa70u+NHr4JNDRK9+GtHBQUGCOS7g2TmiWYwwWOa9LI3BtnSYTROtLVHT177cR/vaFhqbPdiPPwt94zK0wRWw9euI0hzD4VAqOo1GI/i+L7eUhqCxQMQrSpdQyUuz2ZTzRaPRkHCnrQNYjMDDOfjsBLk/Qz6fgHsz5HGhKpPnQF4w+TVdzBeGCWY70BodsW31wNyWmC/cNmC5gN0E1y15f2nsrhqvNGY1TcPb3/52fPrTn34kBnBnZ4e/853vXHn+Pe95zxeVAfwzj2U55/sA9ov9OWPsJQCX8QDrAQ4GA3zf932fZBXOZjNZvHx4eIiXXnpJDlSCNijXQ9EBCQwTM+3KlSsS/lpbu4K1K8LjbrVa0Ea3kR/fQXZ4F4sXPwvvYIjJjQMJjR5MQszSHKM4Q5Dx0sC1NIYNW0fH0LDVsdHeaaG51cD6V11Fc3uA5pNPwXzsObD+DvL1xzGfzzEejyVrkwrhKcKkPBzlLeh30fJIpmliMBhI0gvBNuSN93o9QfNutYCTW8j3X0d6vIvF516AdzDE8MYBxjcnCMYhdsfid02TrARJEkFnwzbQ1Bm2OjZ61ztwB+J3tS5vwL32GKynvwbobCDrX8NsNpPPazwe486Lr+Pg4I/k7yJJsFW/y3Ecqb5BOTkiqpACjHheTfDD15Ht30B28Arm/+5lzHaPce/OCUY3xgjGIfamUe3vIkh7wxaRx+W+g971LhrrLta/6hqalzfgXHsc1pNvATqbSLuXMJ1OBTPyj/8Ek8lERk8UkdAkHASBjPbIaLRaLViWJWvOSPCafhflTnd2dtBwbPDD15DdewXp0S4Wdw/gHwzhn3hS1AEQEBs9H/V56cW0GOcc8SJBMA7B9Bnmd4/QLsaVZViwOoIF7DgOut0uLMvCZDJBt9uViw7T7yJnhfJnmqbJ0ggaX0Qoqv6uXu8qtv7a29AzNXRPbiO9/SKy411MX3wFi91jzO4MMXpNPK/9IMVoEWNx5CG4MSo9L1cXqip9U0e/J3LCjXUXg698DJevbeKxS4/DfO4tYJ1NRO4axuMxptOpLEDf29uTrNa9vT14nmBwUv2nmgukaJf6IP0uYt1uX/5KySy1kYAP7yG99Vmkw31MP/cq5ncPsdif4uTlEYKJQJ4IxaF8LCCILjRvrFs6Ol0Ha0/30dpqovf0VXSevCLEG558C1h3B5HVwLCQvnuU7c/XA3yzvlwIYv8/AL4KQtz03PUA1UZLc1CtH+XMXNeVyfVOp4PNzU0JNW5vb6PZbKLf76NpG8D8BNn+a8iGB0j338D8jT2Ewymmt0RdVDgO4Q8DZHEGL1FIKYo32lh3YXdsdK/34Qw6cuLX+xswrj0H1tkEbw1kNHN4eIjpdCoNHFGl5/M5giDAbDYD5/zU72o2mxL+UH9Xu92Wv6vX66FpauDzE+T7ryE73kV6vIvZ6/cQDqeY3TmRpQv+MEASpjIvUf1dra0G7I6N9pUe3I0emjsDNB57Alp/E8aVZ8D6l5DZQsnf930Z8YxGIxwcHMjShul0Ct/34Xle6XeRt03Lumxubsq84ebmpvxdFLk1WArujZHvvYr0eFdMlK/viud1W+TiwkmIYByu/F12x5LPi0o1GtsDuI8/KZaiuvQ0WP8SUrMho4fhcIjhcFgqdaHfSSsc1P0uQhiIIUnwJRWzt9ttAWnxCHx+Ioz18S7SI/G7guEM83tjBEMf0SyGfxIgTbLS73J1BtMxYDVNNDebsLsWOtcGaGz20dheg/PYU9AH29C3nwDrX0Ki2fJ3jUYjSVwiR4vEFojIRHV86u9SYbhut1v6XfTc2u023DwQv2vvBrKje0iO9jB5fRfhcIb57lj2w3Acnfm7Wjutoh/20bq8IaK960+LZYs2r4OtX0ecM1keov6ug4ODU89L/V2ErFiWJUsRCK5VnUR6Xu12G3Y8F7/r4CbSvVtIR8cYv3YX4XCGxf4U3qGP2BPPK1EcYfpdVsOE1RLPy+07aF9ZQ/PyBtxBR0Sxgx1oG9dkFHtycoLZbCZLPNTfRc8siiKpU/rxj38cs9mjKYO4dOkS/6Ef+qGV59/97nf/eQR4kcYYawH41wD+Aed8JmyeaJxzzhirtczqgrgEU6jeNNVE2bYtIUMAEmqkpYVoeRmjuQattw1mWGCajo5horE5hdVpIp55QmR27COLM6RhCk5JbEuDZuqwWiasdgNm00FzewC734LRW4Pe34DW7gO6UfzfIqEOCGULldLu+75khlLOjf5n+g0U/ZBHTougkjfa7XYFMULj4NMjIBTSVcxpQO+sofN4Bnezh8bOAFkYI4sT5Ekq16LTLAO6acBqN6G7Fqx2A3p3AK3ZEbBSswu0+mDdHXDTwWw2QzgPERxNZIStUr7VqEaNumnCoTIRmnRsWyydQ4aj1WpBz2Lw0V3Av4vs3gHCgzvIFxMsdgWjNZosEE0WSINUyrq5A0GCYDqD4RhwujaMhiMmzV4b7mYfxtY1aN0B9K3HwNoDoLOJyWQq4bHprc9KshIZQDIKs9lMqvVTo//fdV1JjqCcY6/Xw+bmplwMud/vgwVTEZ3efQnJ3i0M37gD/3iMyY0DeIeeZL0u0j3XjaQAACAASURBVBzjJJe5UoCiOg0dQ0PT1NC50kZjvYH2Thu9Z65KKNDYeRysv4187Tp838fJyQmO77yI6XQqiVf7+/sSORmPxzKvp5IoiNxCRo8gWoIz19fXsbGxIQ3ger8LPt4Dv3sD4aufRnJyiOELtzDfHWGxP8fwtTHGUYbDSEQ9QSYK3OW9VKLv9SRHP87QWHcBAGbTgVOwk43LT4J1N8G2nsR4PJFsyMlkgoODA0kWodww5eDV8QUsxe1brRbiOEYQBHL+aLWWspNUw2tZFvhsBoRzZOMjZNMhvIMR/P0RgnGA+d4C/kmAwEsK1vLSAOpMMJe7OUcjyWE4Ym6wOz7MtgfDsWAFHnjoAZF4Wa6Ivl3XlSte5HmOVqslUyd5niOKIpimKYXUH1X7UiPBfL5WgzAhjN//wTn/P4vDF1oPsNCV+1VAiGF/4AMfkJHGYrHA3t6eJHHs7e3h7t27ODw8lAaGBgHBGgBKXiAZFJq8er3ruPz0ZbnAar/fF8XhBXW7YTDw0V3w2QmSmy8gmxxj9upNzH7njxCcLDApGI2LgvwRZDmmxaSmeoZWoeTi6gzrllAqaRVLD7l9B93HN0Vkub0Os/0V0PsNaBsbQP8SfD/AcDjEfD4vUdJpQqDaO4J1yNBW7wHlcMi49no9tFotWRxPRBfKqVEU+thmD9wbIT/0ke6fIIuOMdm9JSOY2b0ZwnGEk3GAWZpjkuS4l56+BwQ9tgwNfVNH19Swti285FYhW+YMumg/+RiMDbFwrLb9JNBcw3Thy1IB1UumqHT3pV143r6cDJMkkWQZugeUhyNonPI+VJi8s7Mjc1n9fl/2gcFgADMNwGdHyHZfRTY+QvjGDXgvfQrewQiTmyPcO/HxypGPIy+Gl3EcR6lk5dL8T7ClpS2JTJccA511F07fweDpNThrHZHzeuwajMGOIFR0NoHejojePE+UJ9ybYvgnn8S9e/9WGjtiZhLETMaO7oG6mDONBXrt7OxIBIVKVygqKhn2O3+KbLiH9OAOFq+9jmA0xXx3JAz7SQBPGnRIo642Ojct9FrDnEO7KcZQGqYIjiZwN3exNjqGsbYB48qT6G0/gf5gHRsbb5VQ8/HxMRaLBe7cuSPHABn74XAo2dbkfB4dHZX6AL3I2VSj+CtXrhS//wmsfc3b0PmPOri6vo5GwwUf3gM/fgPZ+AhxsSLI/O4hJreOEY5DTG7PMPMS7C5iHL/gI8g4Fn9wWzJadSYMJSnLuDrDlm2g0zTR3Gxg83oX19db+IYnL6N1fQP64AlYT3yVKFNavw4vjPGN3/iNDzQvX7Q9SgP7Z90+HyQYBpHjG3HO/4Fy/L7XA7x27Rp/17veJRUo0jRFFEWS3EFJYcMwYNu29Ooo8uh0OjLRTZEH1dN1u12w2ANfjIHpAfLFBOnxrqjpmU3h7Y+Q+AHC4QzxLEASpIhmEXjGZR6GmhotGo4Jp9+Q9T6N7TWY3Y6A3wbb0Nt9oLsN1uiCu10ZccxmM8lwnM1m8H1fyr2FYSiZhGrOQr0HFBHT76QcBm1VQhB5+q1WC4gW4OM9wJ8imxwjO7qH3J8jPDpGNF4gnhe1hV4klGqCFHmSIYsL77RQzdB0BsMxYbiFAkqvCbPpwN3swx10oLdaIjJr96D3t4DWOliji1h3JMxINYaUSyNxaqrnolrKIAhOkYsMw5Crc6uwKxEb1tbWZCRN96fdbsN1XcAbyag6PRRRaDY+RnBwgsQLERyPkXgholkoVXEILSCFDqYz6KYOwzWgWxrcvguj4cDut9DY6MNoOrC3dkRk2h1AG1wWq9J3NxHnTP7eIAgko1EldPi+L8sO6D6sugcU4TSbTUnoILIQET7o/ojo1gImB+D+FHx2LAhi8zGSoz25DE8wnIla2bEwVKvugeEYhYqKCbvjwGg6cAcdWO0mnEEHxvo2tHYPxtY1sGYPrNkHetuI40T+ft/3MR6P5big8aDWoBJBiiL1LMtgmuYplKjdbsv5gMYDlUrRGLBtG64O0Q/8KfLhLvL5BOlwH+noBPHMh3cwROKFSOY+wqnIxSZejDzjEjVS5wIhQm8XY6EFu9+G1W7A3V6H1uqJ9MnmFdEH+peARhdBEMpnTI4clXzR718sFnjve9+L3d3dRwKBXrlyhb/rXe9aef6nf/qn/xwCPad9A4B3APgsY+wzxbH/Bg+wHuDR0RHe9z4hBmBZFnRdl3kyqtFxXVdGMCRf1Wq1MBgM0O12ReHq7EBg+ievInnxBrLJCLtF7Z93OMfs3gzRLMZiHGKaZAhzLkkTRH0GRNJa1eR0ujZaOy20tppw1jroPnUZ9lpPrLawfU0UyG88jijNJYliMV3g+MbLMmdBuQzKb1AujRivAEr1d2TkqfCc1hUkFiTJK3U6HbmmoZ1H4NMDEcW+/hnk0yH8e/sY7h4jHM4wuzdDMA4QjiN4swhBlmNG66kVK9wDQh2GtFI7XafItzXQvtSC3Wuh++RlOBtr0AfbMK8+Iya49WuImQXP87A/Gokczs0RRqObssh+Pp9LGJKgVnJ2qP6QWKpUqKySZSiXSB78+vp6aS28hmODH98CvDGS3T9F9vIuopMhjihftT+Hd+ghmsUYT0IJ2xFdH4AsjWgZGsyiFKXZsWF3LLSutOD2XTQvr6N9bQvWYF0oC/W3wHrb4L1LIodaSMJNp1Mcvf6KjGLpt6tyY1SrRhEcgFIfoHtARCjKq5JUGkW2BDu7eQA+PUI+2kPy+qeQjY8wfX0Xw4MRvMM5prenCMYhRrMI4wK6rEMyiJCiIhndrUYhht1A76ltuGtdNK9fhnFZ5F217SeBzibm8wWOxmPMZjMc3TvCePwSRqORXDme5PMIySDRCBXJIIe30WjANE25RiCNA8rrbW5uotvtSni61+uhoefg8yHyvVeR7X0O6eEdTF69g+FwhvGNY/gnviS7eRmXyjpBJpSIqNFSZhTFdwwN2z0HTtdGc6uD1k63kOS7Br23AePaMyLn11qHl+uYeZ5Qljo4wWJxF3fu/JFUBSKN3GoUSyxRAHKNxEfRiJ3+pdK+qJVg3vrWt/Lf+q3fKjHsRsUkulgsMB6PpYekeobVyRNAifJPkRJ5ihQVqVTowWAgo4ROpyOT6LZtw9IZMDsSaiyTA1EeMBsiGx4g8QK5UGo88xFNPMRejHiRIA1TpEGKrIigVM+R6QyazqBbeuFFa7A7NsymBdM14Kx1YDQdOL02nEEHeqstSALNDrTOAKyzDmY3gfY6UuiyYDmOY2lYVZUauocEG6uRBRnfaoRBq2OoJSjkjBCkRnlbumcUZRHJR9xzB/DGIhcyO0IezJGPj4UySOghPBRedzwXeUARfUXn3kO6j6ZrgGkMVsuE2bRgOIaIyIpoxO61YXUaQgmn2RGlGv0t4Y13NgG3I6GzOI5l/6KILAxD2Q8pUlf7XxzHp/of9UEyYgTDUhRGRfp079bW1mTfbDabJXo+mx8JNZXFSESs0yGiw30ERxNEkzkWu0OptxmMQyRegkmQSMF3gmWXxn0Jzbq6hqYunLzmwJV6tK0toV7SvrolIrm1DRiXHofeHQC9S2DdTSRck07MZDLBeDyG53ky1zoej6WSzMnJiex7dM+qsH1VoYUidypcJ1IOrapAzi+tXei6Lvr9Poxc1Nry4S6y6RDJ3VcRjybwD4aY3zlCMA4kjO/NIlkTOk2WbO9QcQK04p5R3SU5Ay1DQ3dNFOu7fQeda2twBgLSpujfvPK0zLUHGUp9ichyBNlSrS6R6wgleuGFF+D7j2ZF+KtXr/If//EfX3n+R3/0R/88AvyzanEc486dOyXKMsEhQRDIiZs8I1LIAARxgQYQRQ4qNKJKG6nQCMFnZOwINkIcgE/3gfEc2WKMbHgA7s2QDg8QTRaIZwIqSvwQ4dhH4sVIghSJl0i4KC8ma55xAZlZmoAPTQ2ma5yarO1eC+6gK+SfNjaht/sCQutvAXYTrLcNOG3EaSYm6yjG7Pi2vD+z2UzCp2TsSCWF8kNJkpyC06r3jYwdQUYEKTUajVOwMhk7ilL1aA7uTwH/BNkbnwX35ggLckE89xEcjQtj5yOahUgUZ0G9b2ToSo5Cx4DVtKBbOpy+I0SVFcjN3ewJyLXdF85Cqwe0B2CNHuB24YcRgiJX5nkegomH+Wt/KiXs1PtHxk2NTqtOAjkG1HdUSJr6nFp/RgaQ4Gm6rtFoCDgu9AQ8P99Hdk/I9gX+HMHRGNFkXjhYC8ReLEokCrIQCTqTg2A4Bjaa5oUcLHfQhTPoCHLUQMCVencDaAsHi+T+gsKJiqIIs9vH8P3bCMNQkm3UcUrXkWOQ57lkXA4Gg1J/o/tG/UclgvV6PTkeCdqke+c4DhqODUQe+OwI8I+RH06RvbCL0JshHR0XMoEh/KMxUi8s3TcAsFomdEtDq6a/EbRrOEaR6jCUVEcDzqALq92AsbYBrTsQ921tR4zTzia400YcxzgpUh7erTul9AY5XCROQSpXmqZJJ7LdbiPPc9y4ceORzblfahHgF7UBvHHjBr7ne76nxC5U2ZLtdluSFFTSBnnQasF0o9EQUNh4DwgmyA7eQHayj2xygNmrn0Y0mRclBD6OxyFu7i/gJblUYgkyUccT57wEh2hYes+kENEphIjdpon2JbEeYHOzhebldbiDLhrXrkDvbyzp0I0eUrcnV7sYjUaYFnkQ8gL3PrkHz7sjCUCUK6L6R/KeVfYi3TdK+BNkpC7ASeop9J5YjrRtNBpomhrgT4Ri/XBPkEDuvIbw1gz+wRCL/QnCcYi7BZQ8CZISjEZQYt19I0hNEGM06UW7fQftSy10rq3B7ovIw97chN4dwLz2DOB2wXrbSMymnHgXiwXGBXxELM/p6xNMJm/g+PgYQRBI1ICgJbpnZ903inIJcmy1WlLVX408SAeVDJxECkb3RORx7wbS0evwb00wv3OIaOJhvrfAvaGPcBzhyIsR5rxUMxZkHKqMQvW+rVm60Ey1dTQHovxj7ak+nH4Dje01tK9twex2YBZwJOttga1dRaZbEm5cLBa4NxzCKwg2o5MR5rfmODz8I0mqob6mMi1VeJbuG0VrFOE6jiNKkooSnqtXr8pxSpEb9TVCYCydgZ/cBvwp0qO7SA/vIJscYvbZj4sVSo5mWOzPMZ7FmO0vME1yWe8ZZFyqJlX7m0rCIuhy3RFOVPtSE+6VBhrrTXQe24Hdb8G5dEmQsfqbYlWRZh+J4UpnfDgc4rgwWkdHR1jMFzh47QCTycuSsBaGIabT6SkdYLW/EUFNJWgRMtBut2WJF6kafeITn3jouXVVe1gWKGPs2wC8F0IM+9c4579YOf8DAP4ZxGoRAPC/cM5/rTj3TgD/bXH8n3DOP/DA/wh933kQKGPsGQA/CeA6FIPJOf/mh/3yh23PPvss/+Vf/mXpRaqRC1GaiSKsqp9QI0+GIhnVwyQmHIlPk0qHOumRILHjOKUCdMuywJIQSCOploI4AA898DQG9+fgUSgVU3gaA2kCHofgWSYVZ0gol0oVqDFd/N+aaQhygaZBs0yhNGGYgGGKsg67UJ0o9mEI5RrNdgHNEIoTugEYNphuiH3TBXQTMCxA05FlmXxRNFNVkVEVbej+0jV0nrYApIpF9f1ZTS2ToedG3uiq7aoXEYNI7YaiC4ADcQhkCZCn4pllqVC+SSMgS8GjQDzDOASPQiBNkIcekMTg9AzzTByPE/A8Rxan8jnSs6QtPUumC2F1eqa6Yy+VQyxHKIYUikPMaUJzm+IZOW0wyxXP0mlLxSGK3sMwRJIk0qinaSpzyESeoXFDqiMqgaQaxao0e8MwJGxLKACNGzXKpwhNJaERVGlZllBUiTxwfwLMh8j9ObLhvkgd+PMS2SieUY2djzRMJfSdJzn8QoCcykZUdi0gctQ6AxxDK5ZY0uH2nYKY5siIrbk9gNlpwOr3ixKgDvTBDtBcA2t0kVotmUKgKJYcJ5K3o9ITIm8RskJRHSEE5FxV5yVy6NV0jIpCkXNFBpAc+U6ng3e+85343OcezWoQ169f5z/7sz+78vwP//APn7Uckg6xHNJfA3APYjmk/1Rd0qgwgG+rrhJfSGU+D+BtEBKanwTwdefJZZ7XLmLKPwTgXwL4VwCyc679M22NRgNvectbStEOFV2rCjBUAqDWcdEq29QRsyyTxbHUqAidOiIZSMpdqUw6qgEj2I9gU4pCbbuF9s4TslZPSihNDgtW4RGy8TFSL4B/MJJ1bv6Jh8RLEIwDJF6C2Evg+QninMMr8g8Z55I+To3yNqSnSPVVzYYJqykgmsZ6o8jhiPyNZKW2WyIC7W9Ca/Vg9rZgOW2g0UMMUw5kgv2m06mEalSGKjET6Z4TcaE6+MmQqvedJle65wSxmqZZYqqqEZUqVaVChvQcEC2EhNvkCDxYCFbvfIJkMcHs4ATx3Id/NJZMvmDoI14kiGYxYi9GWCxRRStSVCdaYKm2QqtVUERhOgbsjiWZf411F1bHhd1bskBPwdhuG6y9DjT7iOME82Ly9H0fs+kMURRiNjuS+Vo1v0bPhyAz1aip+Ue679TXKQ/OGCtFaQQl0qRLkzDdb7UeVUVgLMsC88fCuPlTZMMbyA9nyMZHSEcnWMx8HB4MZT5c5CRjBONQQt2zQjc2zJf9XCXeUOkAIS0mW6qoOG0bVkuIThMM3tjsiv7ebqKxM4DW6ol7PtiG5raBzqaQMCxk+ai/e8cegkAIO8RxXMrzEupCBq86x1SRBCrCN01TzjHqPSeHm5R0qL8TakW5X+ImqM/Itu1HNOM+NAT6FwHc4JzfLP7WByEUwC6ypt9fB/C7nPNR8dnfBfBtAH79Qf8Z4GIGMOWc/4uH+ZJH1V555RV867d+q6xhAlDynCgyI31CYoQSeUClwKvkjOrkKZc3WZwgn55ICnQ+nyAczuAf7yE9CeG/OEU0E+UAwTjEPMhwPI9kvZeXlY0V1UERg5AMlqNAWG7TFDmYvoPGulBmaWx2YDYdNDb6S8LLxuWV+ZhQyccQWSgIAhwUFHqCsIJFgMnzQwTBPfj+p2SkQHqJ5Cio0RxFXKqxontO0XO73cba2hqee+45mSeksgMqhKeBLAe+aQDhHHxyAEQesmlBv/dmiA6K5XEm9xC8/AISL5KqIrEX43AcISqICjRpqsaq7r67ugYNYuJsmlqRa3XE4r5X27JsobkzgNVuCKLHYHu55E6jAzR6Ip9jOrJkRb3ni8UCo0r5wmw2w/zuHEEwxGx2SxKNKFojo0VRODW67yrzk6Ivup+0Hp5qpNSSD8rPqgQuU4MkcPHZsVjOZzFBdryLeOYhOJ4gGt1DPPfhHc0LByGCN44wDVPcKNasqyPTqIxpMlY6EyvPuzpDwxZOguGIxW/Npgmna8Pd6MHqNMV2rQe91YO+eQVaow3W2QBr9QGng4hrsr+rDnEQBJgWqQOCw0VON8Dks69LY0UEObr3qkO8ao6hPt9oNKRDvLW1VXIQqv2d5iNyGprNJlg4F/19doTcmy5JX4sJ/P0h4vkJopM34L84XarMeAm8RYJhMccI54zjZH77kc25F4BA1xljzyvvf7Wo3QaE5OVd5dw9AH+p5m/8J4yxvwwRLf4o5/zuis9evt//v9ouYgA/whj7LwH8GwARHSRL/PlsvV4P3/Ed3yE9LBWmU5sKfem6jjzPZTRCiWWCbdSJuwp3uu46rK1LMK8IqMc0DDQtE4NgDqSx8HLTGIg85P5cQJyFODSPQuT+DEgTZGGErFjTLA1jCY0JmCyTgtuahMd0ufagEOE25Pp9mmWC5xny6VCI8E6HBWTmQmu04VgOHMNCz24Crgm2dR2wHMCwkUFDnudIkkRCXeqES9skSWTEoMKfAErGUL3fBFnSfVcFtslYVkW+aUv/i2E2Yew8C8YAI41hxAHAc7gETWaJuN88A5IY4LmAIdNYihBzEiKu/I/FPwem6YCmy31mWgIe1jTAsAUsrBtgtG/Y8jjXDGRZhkiBCZNFgCxblBAF6j+0nBG9qvBwFQpW76EqdqzeRxWGVKPm6j7iQEK5PBKQLZIAPLgLvkiQ3xX9NIpD8MAT9zH0kMcJsiRFFsbI4xSaaYiatY7IIQIAV+4t0zTZbzXTlApD1GfNpgsYpoBz3aaA5Ns9sXWagNMGTEsQkUwXsBxE0RLO9Ytctud5iKMY0b0ZwvBIrulHeTSCdlW4kWBhFdal+uBmsynvr8pmrjp09BxpjlDnC4J1CQ4mo2jqmmA1x4Fwor1D8JM5steOkAceFqOTYq3DUBDliprScBwiDQUCkRQEpjBIkXGOIFs6dDnKTp32SMBPddicGQGePCQL9CMAfr1Y+f2HIWrGH1m67SIGkKS/f1I5xgE88eb/O/fXLl26hJ//+Z8vJd9V+IG8QLVgnHQrqZg4jmPM53MJVZAxVRXYzzKqqjdIRBKVTUoFteVyClGT1+gv2X5q0S0RKhzHgclykR+ZnwD+VHjjwwPkgSfIJsOpLMSlQuRwLBhs4VgU5M7DVEZDC0WBRfXQq+aBqNwqtOQW9HcBqQqigOMaRR5FRKlu34HZtOFu9mRxs9XvQ2u0oa/vCOV7pwPWbgrFe7crJylSaCHZKoLwVAYclbVUIW2KnGiiVCGnKqmAmrq6BpEzqpAreehVoQCpT1qsLEITH0HjFHXZtg09FrJW3BuDz8fI4yLH5c2RzSbwjydysWLBdo2khx9OI8S+WPV+Vjy7QKlBrE6E1WeoQrKurklIlmDxTsFcbKy7oii7a6ExaJ4qCbGvX1+WhHQGkr0Iu4lMt2S06/s+PIURS1EYQYXjwzGCYI7ZbFc+QyruVgXrq7DheWNQfYbkzKrPkJAI13VlfSxBiRSlkREjBimNRcuyiiL4CRDMkU+OkHszZON95OMJkvkC/sEIiSeEMYZjH7GXwD/xS2mLIOOYpXnp2VXRiSqhCShHy/Ts1JrjTpHScPuOQIg+sfcAs+nF2kOSYHYBXFXeX8GS7AIA4JwPlbe/BuCfKp/9pspn//BB/xFq5/4SzvnjD/slamOMORAC2Hbx/R/mnP9DxtjjAD4IYACR4HwH5/zMlR1v376NH/mRHzk3Wa9GgDRYaDJTE/hEl1Y9Oho8qoIGnafJkiZMA5kgTgQCzkAcIJ+PBNHFnyObj8HjEMl0H+k0RLIXIp57yMJYqEh4Aro7UUojsjgDzziyJCtUNZYUbFLYAMr0f80UXnjnShu6pWOnUF/RLR1muwHDsWA2Han7affaYI02NMuB1h2AuU2xVEuzBxgWWGut8MZdCRNRTok8bzo2LwgY5FyEYQh/30eaBvA+9Vk5sVWJFnUR/FnP0DRNSQCoezbViJ7yh+ScUGRPpCXaZ2Ss4hDwJ0ASIpuPxTI3cYh8OkQeeEi8MeIDH1kYIxrPkYYxsjCG78WYx5ksb8mSDGmQguccWZydUgaR46J4fmIrIFizacIduDBdQ1ESEs/OaDowHBt2rwXdsWC2W8K5WLUsl9sWxgqaJMaQw0c5wjiO4YchhsWzUfVp/Vs+4vgIQXBbPkMiztDzIwQBwCljRc+QIiu1YP/y5cvy2TSbzVLEpZJp1GdMz5AMFRFwDEMHivHH40A4jcEcPPQFShIFyBYTZIsh0nAf0Y0F0jASK7T7IbIwxmwRYxSmyOIcaZggi3P5LKtjsO4Z6pYOpoltY91F50oH264O3dRhtSwYDUfofnYa0B0bVkeQb+Szsxxo7b54nqYt6gJNsUQS3I508tSxRM8wDEPkf+fvnD0JP2R7iBzgHwN4upjrdwH8bQDfp15AcpjF2+8C8FKx/9sA/gfGWL94/60AfuZB/xFq5xrAQrfz7wL4y8WhPwTwK5zzB11zIwLwzZzzRfG3/x1j7GMAfgzA/8Q5/yBj7F8C+EEAZ+YewzDEyy+/XP1/JeRGMJDq3dMxVRJL9RIJlycDqHqBKgzSaDTA8lSQKhYjIA6Rz4diDcH5REyYoYd4LGqKEm9p7IjJloZJwWDLCgmxYqDFOfIsR54s5cSokaQU08RESbWCVLtlOCbMpllILTliciyMndl0xKBrtaA5YpVwzW2W1yNr9YWxc9uI40QyCOPpGHF8WBpsxLal6JuK6cnw0WeJhUtb1VlRJ0qq06xOlKZpSiNHDkuVoUtwU3VyVD9Dxs6yTMH29CfgyRyYe+D7E+S0pqE/E5D1YoLEC+TzS8MIydxHFgtIMPbikuQXTZIApAQYICZGkgGjSZLksOiZia0F3bGKybGpPLvGcg1Ftwmt2REOi9NcGjinCbhdcKbJZ0MTZezFCPbvyQjZ9325lQauwp5WyRtVWBw4beDoWbmuWzJwuq6XxOnrnBP1GD2vagrCMAzoPAWCwrAFEyCeIj8SYy0PPSTTIaLAExDiZIEsjIRYeuGchNMIeZLV1pFmcQae89JzUxsZNTnmXAOGY8j6P7tTOJhNWwjJF44lPT+t3RNjrSME5k+tGeq2kSSpfDaEasRBjMXxiRxThGqRc0LkM7qe1tB8VO1hIkDOecoY+3sQxkwH8H7O+ecYY+8G8Dzn/DcBvIsx9l0AUgAjiHViwTkfMcb+MYQRBYB3vxlpuIv8kn8BwATwy8X7dxTH/osH+UIuEh2L4q1ZvDgEzkvewAcA/COcYwCfeeYZfOxjHyuRBtTIgyaCqtKJymIk9QQ50Rc5BpKaoglAzX8BKG3JIyovVstK8ChNCMsaqBbsgY3mtaaEalQhapoYWq0WnCKfoE4cai2VmHgcMamHM1l6wRdiYuDeHPl8jDz0kc0miE5GSLw9RJNPIQtixHMfiRdJPVOhY5ggCcQkEResU4JtcuAUqaTGIYbFAAtlNqpap2ZpYskbtfBaLyIdKsC2ey3olgWr3YTVyaA7Fqy+Debo0FwHWrcvPWbYzdIixClnchJXiUA0wdOkX1XFCWUUGyOOM/h+gjjOEEUZgiBHyer1nAAAIABJREFUkmSI4wxpypGmefEqG4iqygsAIF0qDhmJAT0i1iWKLYNhcNg2YNsMlpXCdTlsO4dtJ2i1OCzLQ6OxkP2BYPNyrtqVBLBWq7WEdXVNlOT4U2FEIg/cnwmnbToEDzzkfop0lorc09hH4gfIghhRsBAOmyJEQH0kDVMkBfmFIFliyhLpCwBSDsxqxrEKtesMMBXY3dIYHNeQ/YKMjt21YDgmrCZFUkuDYzYddJ66LqLgdk84DE4DWqsPWEX/KPpKnGbS4KhlIuQIEOJBogd1cwilXNI0hT/2ER/OC+fjjRKMS0tm1ZUNUauW66j5XdWJJ6dCTbeQzJvrumdNmw/VHrYOkHP+UQAfrRz7eWX/Z7AisuOcvx/A+x/4y2vaRX7Jf8g5/xrl/e8zxv7kYb60qAf5JICnAPxzAK8DmHDOKVFzIYbPdDrFRz/60ZImIBkwNeGtQqQE0aierCqUTa2ubow6n7r6OOUcVNYpGSq1wJz21RpDikSpY9N5qiFEHAjCAhFrwhl4fAQ+ni9rCEMB1029AFkYIw1jpF6ALEmRemGFbMORBGlRm5ZLeJXneUmFBoCcaAARwQAQEJ1WCDsXAtflSMaS5Byj6UA3DeiODbPpQDMNWJ2mrGvTmh1BhqDJyRZ1bDAsUddmNwHDBjcd+fziOEaSpvCKZ6yWsUSHEZLkuMTiU5eXUvOCVYUbtX+oMGx1cqJ+UnV4KFqtIgxVNjL1C7VmVM05qn1K/QxR5R3HAeO5MGKRB56IfoEkAI8C5P4QSBLkk5kwZnEooL8wxMIPkQWifyR+gDwWfSJP0mJ/CdmSQoyoRV32jyrsR32EoloA0Ey9JHy9VJcxwXRN9gUR6dowHKuAcy1ojiP6guWANTuinxQRLwxr6eCYFuCIYxkXilBZlgnUIcvgF882yzJpwJJZguh4giwbIghek/OEWv9IeX+qHz4Poq+iF4RSnDV3qFs1Z0n7qnOrzhF0vbpfdaxN0wTiAP/mX3/4vKnzodqXmxJMxhh7knP+OgAwxp7AQ9YDcs4zAG9ljPUg2KVfcdHPqusB6rqOn/zJJTdH7XjV6EudWIgyXvWiibBCUVhdREafVwt/bdsWxbbeWBisyAP3BASajffAF4IVSnAoyVTFYYyJogVK0Vc0i+UCvCS0S+oVVeKDGn2pTDCxLZNYdAZJhFBrAxu2yDeZLtHQzaJmTUCozqADs+kW67F1wZymyFU02mKC6q4vIy9bkFuiJF0y9wo4lKTpKDqP/AiLvTl8/7AUoZH6CO2rKhlVggQZripRolpYrFLYqX9UtTdVeFUVFV+uimCXIHJVmkxdQkjtZ5ZlidKCyCvlhrPpMXggSE35cII8oFU1QiR+gGC8QOqHCKcREi9BGorV27M4R+Il8KO0VAtaJTVVSz6qReHV0ptSxKXUL9oFvG53LBiuUdQxmjBcQ6zi4Yjo3O63YDYc6L01AfE5TWj9DeHYNPtCh9YWkG2aZiXCy7QSccVxjMWEah73sFi8KslqFHGpq5+oDk9Vhk4lQa3qG7St6xuqcaHnSc+YUBsiP6l6t+q8oc41FJ0vIX1dROQkmOGNkQeeQGwWol9kkxESPyxqJedIvBCLyUJG4oItmiJeiAh9cvz6RafT+25fjusB/iSAP2CM3QTAIBRh3pQsK+d8whj7AwBfD6DHGDOKKPAUO0j5jFwP8Ku/+qv5hz/84ZJQrlo/VR0Y6nWq7iBtSXGdBkw1GiBIa1WCH0CJSg1ARgdlL3AdRtOA3tFhX11GCqpHSNtBMTGTh0nHKbepHqPvoO+k63RdFxNfKqjvSCJRRhB5QJ4IFZMkEuUaoYgseCyUanhSqJsUajVZGCGbech395DHKbJERJjqArs8y5HRvqKAYuQ5tCyHC6AniTynsVOKJgCAGQyaXZQo6G1x3jSEakqhgsN0TdLtma6LCEOh3jNdRKfMtIQyjuUIlRVVKcdtilIIVSnHtADNFLR8tQxCbk0JjVcnWupjatmDgNIZssxGim3kdo5UT5G1RD9LryyXsaK+V2WyVuEzI8+h5TmsPEejBp6nVjfpq+9VQwBNQwxgZhi1jkPVyayWZshXoEOPdRhBDF3PoOseTHN8qvyl3++XDI/6EqUuQoUHaSSi3qLPlvpuoc6DNBEqS2mhzBOFJaUlKuugyJf6KfVhnufI4/RUH87jAtrOE+TZEDzkQAjkR+X+y7Ncqvuo/ZnrDBGARNcwL/osK/qvrvRTUgLSTEF4Yrq2zAu3G2g/cUX010IdiBkmmNMUfbtAV/67v/VA2akLtS87LVDO+e8xxp4G8Gxx6BXOeXTWZ85qjLENAElh/FwIWZz3APgDAH8Lggn6TgD/9gJ/qzRYOOdyqy7wqT6w6gRShUnPImyoa4tRzVY1z0OsRarhUokdqmdZhURU+ItgkCpkRlCJCo+ppRe0InS1aBcAdMsSE3da3Auei1eWgke+yBVGYSHTFgj4zJ8jjxPEc19CZpQTImJBlqQVIo9gOtKKDIIcsiT15FmOPD8dtVbtYDVS0QoCAq2rp1vqvi7JCAKKNSTspjsWdMuA2W5At0xJLDEcC3qjsYTc8kxMJpoOblhgmrE0fgX0BtMG1y3RZ8IYSeKVYPcSJKuo3aiOWB08SxGt+hl1X4Xvq9HMRfsiOWbUJ6rOEkG7al8kyFeNkNV+SE5XFd6nvkl/U3XU6O8hjQr4NgKPZsI5iwPkwVzIyHkzZKFfQLtz8DRBMl/UwvxpUaNIpCS1LyZBKlnUBO0u2ZzLvniRfkjvqS8S25qIaExjYr1H6pMF8xO6LtMDZtMp+qQpyDGWAbPhQGs0ZGqAjJpWwMDMEekAWK7oiwUcnHHxzKNK2idmJh5l+7KIABlj38w5/33G2HdXTj3FGIOykvv9th0AHyjygBqA3+Cc/xZj7EUAH2SM/RMAnwbwvvP+0CuvvIJv+ZZvAVCGMqov1bOkgVglpRAkoa5sQBNAtdhVLY1QtUBVpqHjOBK/1xnEYA/n4GkkmGxF7i6fDmXBfO7PkYchosmxMDYnAZK5jzSM5aK7pH+YxRlCL8G8YI8SJBZkHAlfwmKqdNRZsCk1Mjjqfh1MRpJTBJUZ5lIJXzM1OH2ntPCpZglCi8j5uEIZ37Ggd3rQnIZgN7YLQkuTSApFQXQx4ON4aVDIcVHZpnEcI1Ap/QrLkYgLkRdhcbhAmi6wWBxIY6Muk1WVDDsrCgPqGZF1fVGNllaRo+g99SVSJlLZyFX4vq4fqgaL8oeWZQnmcgHDSrg+CpAvJiJvWLCYeRwimS+QegKWjYflch2VMJUGKRZF2UeVMJUoRdtqfzyvL1ZhfGBZ00grplOfdCo1ckSsEiuomHLf7tiyFIicILvXFjBup7ksBWr3RZG+0xB90bAlsQqmC246Miet5pOrhfeLoh+qhfmlPnlMpKwhomivpMm6qh/W1Ser/TDPcxwcHJw3dT5wY4yVdHm/2NtZpvyvAPh9AH+j5hwH8EAGkHP+pwC+tub4TQituAu3y5cv493vfndpgqIOonYSNYpbxeSsNvqbnudhMpkAKNei1e1XYSIV5lFhSvKWxfF1YZz7OszNpS5gyWDrOpwVUJNhCA1BVVkFWSKiuyQSos5ZsoSOsrR4kYpKLiDPNBHKKYUgN/JseazYIiv2geUxQKitFMorAMTnsRR8Pqtlswmy2UQRhRZRCzTaVtRaADQMC4yietMqzmsC1tR1oGeKrdYGtK44X0R3zLQApomXXii+MF3sM00IgTMBTYljxd+WnzGX73UxfDiHVHJRX9X+RcfpGO1Xt2p/XCUUXne8bmJS4VdqwkC7RUSxIQ5ulP8GbQ0AlqahWfMdF9kvf+fDtbpxqt6H8/br7ivnHByiNuvM+58CPEkBzItX+TvIWW632xd+NufdP1VRiQwP3Ud6rx6n/a//+q8/9V1vZiNk4UuhrTSAnPN/WGwfbVXlQ7R+v4/v/u7vLjG1VPanKoFEMJXqqakMQdpXPTC10DTLMvlZgkjrKM4ESRFcqrJNL5KTqeZjJIRZoURXyyyqyXo6RnBWXQ2dqtxPMJXj9IWxti1Ybav09yi6rcJaUmYr8gS9vmCt8jgU0GpQ7HtzmVtM5gvkSYp45ou8YrGfJ6mouSsgVBVeTWVxsijPSLJypJvwcrRbjTaA86Pf5X45CqEouEosomV06JitrN9IogQkQmC4ApY1XQNGw4FuGZJEolkCHjMbTpkN6TZFjsd2hDiBKaAw2E0wyYwU8myUc6QxQPsq5Er9mMbFqvFA/b4qXEDXV+sEVUH56nio5tLPc0BXjQc1/6jW+dJ4UJm36rhQCSyqaIK6eoWaWqDomj6nRtBqVF2Xlzc0JvLsaSzWa0yjgrXrI48C8NBXxsMMvIB7KaUgxkBS1J3GRdQdyZrFeCHylPEikbAujYewUAoaTd644Ax6/43g9C+VdpFC+L8P4H+FcHv+FYD/AMB/zTn/nUf8v53bXnvtNXznd34ngNMesaoiAiy9KlWXUs2NqEXXVUp7t9s9VbJA+6vyH+qLBqlaEqHm/9R9Irfoui7gqjgA8nQ5mIpcCU8iCaPyKJRL8uShXxgeoWwhcncesnCELEiRnixLIxIvEvk6Re0iDVOkGUekKF8khZIJzzjSotA7zk8blapBqWtVaEvdqrkVKrMgIyLgVEuu6E7lF+LYkl4vciouNMuQqimaaUArdCdpOSFmmEstStsF7IbI+VmuIMnoZlGKYSHLlmQUdRWLqpEho6I6YlmWYayQsOrKdaIoQu7liEYRksRHms5qc9HVfDQ5W+q2bizUtbPGAzlddeOBDAiNAcofquOhblzQPhkUGgcyTaCQulSUpKRnWpQGiby1J1CMJADiUBiSQsBgSeZK5BjJwkgYlCBGGo6RhhFyP0WyF54qBSFHi8pC4jhDWOSvZfkQKftcENI9azxUUw60NUxd5rnVhXeZziQjV801Ut67+ZFHB4ECX35lEP855/y9jLG/DiFT9g4A/zuAz7sBtCwLV69erYU1gdOkAGAZvqtRVpUgoBrFKllA9Uir5JgwDGEYRqneTDV2cRyfEiumSUCtMVRVUMQxB2anXXi6AOIQjCDMtGB0xkGJFYc8E0YxTZaszhpWHM8yWQ/Gs1yKc9ex4mhtOwDyegDIkoIhV2F2ngWBqkw5Yn2SDBgxO4Ey45P2SRScGJ+S5VlcIxigyrqIhinWQyzWS2SGKVhzBczKswwsiQFdEIN4lgC6CZZEgG5C1w3ohg3bsYBms8TQVLeqyLV6TBUTrwqLVwXG6VwVxj/rpfb7OphwVTvF/sTpvOWqPKZKQKMxVCWl0blq7pOiNzWSU8eh+nnGmDB0cbCE9nm2hKN5DjAdzMqgm/ZSED0OJXzPkxgmQflpIiD7JBYMZVU4nSD/LFuyPxWheurT1TUdz7rvqrFQ136krWQxK/0dQCHKrpXge2g6mGkq/bi4xrRKfd35xMt4VO3LLgKEKH0AgG8H8L8V0jVfEFnQxx57DO973/tK3rgKxVQhoVXHVPinWk5RhULrWHrkpdPfVqFRgoNWQUFnTWbVfWpViIi2Z72qOUMyriqUQ958HZQqFSjWVUmxegGAs0o11EhCZcTStWLC1JcL00aeMPDFosLEWiUoNQ+WUTAdy3wf8cxbMgQLSIkEAbJQsAWXsGpW2Qq2YBqmSJOsRN4IFTLRKqi1bkutTjGn2uoiAjpeR0xSYdk6SJaK0nVLg27qtUxFwzVKTEURQduSgk+RtFyY1zDBzCaYKVYeYU5DTMCOKxiLpiX0K3VTrD5iuoCml8YIjVXf90ulI2okTQXuNLarMK06ZmXRuwLPrmLW1r1U56XqeFRTGffreDyIo1Edp1V4Vx2naiR+72Ryfid7iPYFMv2/Ke0iBvCTjLHfAfA4gJ9hjLVxWnj+89L29/fxC7/wCyVyC3VkYJl/q0aIKoWc2lkdWDU45P0QY1T1otWSB9X7VdUgziPMrCLOkJGoY7USCaZKoKnzwlVvmz5brbVCli5XQlePpcuaQFoNXdZfZaJ0AlECHmQClk1SCbeWo8tiiZ0sRxDG0sum60T9FVc0GnOoIsSkUAKgVsVG1XKsEy2mRgo3gNBY1XQhxyYjUk1bRqXFMd3SiiV/WOG5k/ixJusO1WhVKOOY8nxpOasiUpV1iZYDZpqyzguaJoyLIaJZWZ9YqUdkuiEMjWHJLWfaqQhzVcSqOo5xlmGhXKee55wvha+THLFHjMQp0nR4ii1b932r6mtXkdTqEJ2LGhsar+TgNZvNU+Ozuq+OU1U8fxWxbdVLHZPVv6OOXfXceWO2RHArnEFkxfJfhAhlCf7f//u3z7w/D9O+HCPAHwTwVgA3Oec+Y2yAN6kQ/mHbaDTCb/zGb6w8r3pb5+U8qvtVMWbysKhzViMl9ZgaHVX31UhIhUCrUdNZOUnD0EXnJ8JJloAnxYCIFyIBHy/zHzyJRfI9jZEHAhZN4xBRsS4h1VaJCCk6VyaLaq2qKvlZnMv9NMnOjIrUNczuN2cituX31eWbgCVtXtPKCv26pS1rCq1l/mQZKZ2WdtOLfcOxpSEjuTfNcWSBPXOby/3CeFHOEYYlCCy6sYyOlGNpujRU1fUZaT/LMiRhgjRNkGUh4ngkDRQZl6rgg7pVqfSqYoq6pqaa66waLjUyquYggdV1iecxI6v1s6ty9ZSfrI6L6hhRxySdU+XDqmOyeowMUTVFoR5TjZeaz5fwbZ4uHUhaw5LGaRqCR2PwOJYIBtJE5vI57ec5It8vSRqKMRkp47MYq4lAMYLdWg2RN609TA6QMfZtAN4LIYb9a5zzX6yc/zEInekUwDFECu52cS4D8Nni0juc8+964H+kaBcphM8ZY4cA/gJj7AuqAvLZZ5/F7/3e75WgRlW4mgZt9Zjq3VYnh/Ne6gRBkxKxROvKLlZpTFYnEAC1RIb7ITZQu8jEou5XPeClV2yJ/KNlQnNO617WTUDV/KXqMQtZMA2ucbokpOoxqxNMXbRbF0lXPWfxO7RSFCsVRfJ8eUwqi6SyJORUlJuKa/I4BIrcEY9C8CxHPJnJXGlWTEoU5fKsyKEW+dNqlCuXSMrzkgYnCQqUNTmX+2rUSxEvL0SnWcZhZjlMAC6APK/vOxlfOhLVpmnl6BiAXIWEjpU0YmX+lpWiZtKP1a2lZuiS2PFwEbSma9Ade1niQpG0WY2qLanyQ3kyWugYhi3yidVIWtOW0TRfokjVPC8d9zxPlr5U55i61/I6jjTVkecu0tREmlqnIuiYx8j1HDnLkWopcidH2khXzjcj+1MXnivutz1MBFjUfv9zCPGTewD+mDH2m5zzF5XLPg3gbUWw9Xch1gP83uJcwDl/64P/96fbRVig7yn+gRex1ADlEGv6fV5bGIZ46aWXSp1SVcuoQjB1x+r2q5CM6gkT/KKue6ZCNlmWncL56Vy1nUXSobYq30fXVr9LPVYl7lRhnyrEchbUU3esSlYgA6bWJarX0LFqjuMsMgRTFGuQRrK+kWcpkIdyJXh4UUFmSJFHgTRSSRTK+kZpuJJkSQQqiA+qRBYZraVM1mljliuQ7ZIgREYrl0atDrqtwrbSgElYd2mwqkvz0DXaMgSGUTMhSaOlLQlGpeOKcVq+10pGSoV6ZdRcGCJp1LSlFJ04VjZWSyMmzmtOAe2SISJJOiJ7WA6Yrhgzkq5j2lKaTlMNVmG0NEOUgxT1mVmWn8q9V7c03jnnyIIMaRqV5oPq2Fdf9LdV5/iseee8uUYV6b+fuYb21a1a8/ko2kNAoH8RwI2i5huMsQ8C+I8hbAsAgHP+B8r1Hwfw/Q/6ZRdpF4no/iaAZx9G/qyuFd7A8wB2OeffyR5gQdybN2/ie7/3e08drxqNOoN0keQzTebVJLQKT1YhUoI36yTOqlGPSiCpQqAqBEOfUUkjal5QjZRUwyQinCI3QEZDQjCipKKs+7mESnm0kMzRPAhFzo6g0jMgGCE9lSOpUMkFbLpc71CFSqtqNepSOipsukrkeVWdH3B/yWrqNXXyV1WKet1WJaSQSo563DCXJR1ESNEtHaYrjIPhlokqmqIDyTQNRtORUY/hWLLUg8gpMsKR+0WpB2lHmkXekGBYw17mEi2RX+RMOwW5VtENeh8q+2pJCH2GyGGllTjmKiRLtbknp1ZeqNtW4WAyXioyoxomisjOI5w9LJO2uq/OJ9X3fxZzze/+7u/eR6+///YQJJjLAO4q7+8B+EtnXP+DAD6mvHcYY89DwKO/yDn/vx70H6F2EQN4E2LNvjfVAAL4+xCr/XaK9+/BfS6I++STT+L973//Ka8sz/MSFX2VB3bWsVUv1euqe0/7ACSDVB1Q1cFVFwWe1S4SIdaVetSdoyhLfV9+OQAcGIYQK4YFGI1ljdiqQV0PpxZRnqbBLga7jPLO2Fc/q0awpHqhfoaOq9991nXIM0UTNZERJPKCap+JaE9S7/MiEuXL8+IhpkX5RKGgUyjj8LTw39JEnkOeF+cr+6IziO9T/oboNDXvCRovIlEkC+TZTB4DUKLuq9vqPrBCZaVyDYNY3/GirSoKDdSUBegA7PK1TLcAWEp0qpbMUGSrlBLQmKiqB9ExulY5JpWE9KK8QLlOvi/+llQYWqFOJP82XceKv8O0pWqQVigO0bkVx1n1uHotHRc/vP44GD784Ue3HNIFIND1wkhR+9ViAYP7/Z7vB/A2CEUyatc557tMrEj0+4yxz9IqRQ/aLmIAfQCfYYz9HhQjyDl/14N+KWPsCoDvAPDfA/ixoqzim3GfC+Lato2nnnqqZIxUmGMVC60O0lDhDBXKuAi8URUrBnDqWHW/jvl2kfzgWXlC9Vj1+P00tYOr+cLq+1WEBTpWNYTEkq1Cs+r7uii3et0qJl5d7rGOYXfRl/heHbouonDd1KE7ZeEEFeZleVbIy2WSnCQNYZYs841FLpJyjQTF8iSuMGuLOrYUhcE8G6pNw+jUagbyuoIgUWXWLoXKlznHLBb9ZikarUC4yqrp1SJw4HTpBxGeZJ+sEZ2+CPmptp9WlHvUY2o0XxfJ07VVoXWmlaFgIk8RREwsYHVNTFWUXezXQL8W1avaKyFizTTkiiWyrk8TTGCZ57SV3CZByBTZ60VuM3mzY5VyU52YmnbCOX/binO7AK4q72tX/WGM/VUAPwvgr6jII+d8t9jeZIz9IYSk5iM3gL9ZvN7M9j8D+CkA7eL9ABdcEJcp6wGapolv+qZvOvfL6qIkdf+sSR1QVlSoYaZRvouuUyfxKsWZyibqmZ1G7QSvMkLrSCUqgaQuD7eKhq1GZqvycDL64rnUFIUymfPs9DFJJIlD8DyrJZJwmvCLyf/0EjUx8tgHz4tJvTLRZ2Esc240qdMETgxVdTIn1Y6YjhVqNmcpeVTZqmdJqZ23ssVFJ3h1Qhfvy8erE3fdvsqGXaW2Q5O6nND18uoGVsss8nlVBZJlLlAzl0v4lEo7LPOUOIGEadXcXlHyoU7mpYJumuDV3B+VfWjaErpViCxU+qHm9qoO7yqRAoJwq0S6VU5y1TlWRaqrx1QVn3ShEumiU85xnZOs/r9VR7hu+/Lrty7U3x6kPWQZxB8DeLpId+0C+NtYBj30978WwK8A+DbO+ZFyvA/A55xHjLF1AN8AQZB5qHYRFugHmFi26Brn/JWH/ULG2HcCOOKcf5Ix9k33+3l1PcArV67wd7zjHQBQiqLU98BpbL+unuhhoUlqF4Eo6Tp1wFDxrvo3qtAl7ZNBrkZYdIzOq9epx1blJKpwZRV6rO6Xj5nQdUccNzTo9mnIse49GWvGGMwVcOZyiwKerMCWRJIpYEquwpQ158Hz0t/hWVZWBFHEvR8IugRO/y3gFHQJUB1jDUSZ1xyruY7a/eStzmtVD78MT9Yom1RhTUASZqqf4XEIaLqEhyWUKL74NEypQI8qTMkMS36mClFqmgZbFVCvwpMEGWoAHBWGJHH0MjRZgiUBEWmJf6h8rQpJFr8BgMxFqkLpQJnhXT1H16vn647RvvqZb//2b699rm9We9AcIOc8ZYz9PQC/DQF+v78QVnk3gOc5578J4J8BaAH4UPE9VO7wHIBfYYzlEE/uFyvs0QdqF2GB/g0A/yME/P84Y+ytAN79EDUY3wDguxhj3w7AgcgBvhcXXBBXbdvb2/iJn/iJldChmgivQo51BcJ1Hp/KJDsLDj0PKq16eGdBpGcxUdXfSNcBy6J/+s0ATnmGQFnJYpWDcB4poG6yPWsCroNMqsdUA75qu+pYHfxaF6mrRAPg9GLFQFkKry6Kr4vYq2UZ4mXLY6Uo3jnNqK0yZtWo/CyItq6EpRrdV5m/ABcRe54rK4JkAn7l5WNnQrU1zFq5mkieycie5zkSPzyXWasuTFstE6muK3lqTT9Fp7MK1V60HrVOuec8IfU6GLeupnWVFsNZLksdyKiWrJwF/x5nj64O8CEjQHDOPwrgo5VjP6/s/9UVn/v3AL76gb94RbsIBPqPIOirf1j8I58pkpAP1DjnPwPgZwCgiAB/gnP+nzHGPoT7XBD31q1b+IEf+IGVER61syK9B22rJnCgnoRy1nF1oiOx4Wrkt5zAypFd9bw6GVYjwGq0d94xdRK+X6LKWefqSCnqZ+qILOIcykSVLCnv80KzkfJstK+QV6TeYzFJy/dJvMzXUYR3jl5knqTFhOyXSiLycFkuwbOsmJDzU2UUXDle1p6kvJsol0jzHElN7R+VVACQkK/YV5VyTh+rKufQeWq8Ujd4lprORZuqugMsyzOAJbFFvU7Vh6Vr6ko6iEUr/qZWvn5FSQcdAyDzcFVdTrW8Y3ndsl6x+lmqSxTnl3qdMgqlJbx0fanhqWp5KtfJ6wkCXrFsF1OX8NI0UQqiafj9b1B5I29+OycH+EXVLmIAE875tBL2PgoptJ/GfS6ImyQJjo6OaleCuJ9W59Gov/eUt6vGAAAgAElEQVQsUkhdhKI2TdOQ53ntdXVG6yyDVneuDto8y7Ctqvm7qFE773ydgaszdlUDt/Qs+bJgnSdAki4jE2Jl5jXQp/xMsb4hGTiCLClqoXUMJTmlIJqoxwpjRxEM1fwBkASTvCCbAECeJCUjd9qgnW/oAJyqHQRW1QleTP6tasjkpK8zcY1ilPTamOPsVjVqQMWYVQyWep6Mk7hOPVZzvmLAxN9Ri++1khEDAL0wUpSPpH2tYI2qn9EUY1c1gmTMSsapKkxdEqxeGrRTBlDXpZFS155UDdwpo0bnVeOnaaI4v4LmcM5FqdEK4YM3o9E4/lJpFzGAn2OMfR8AnTH2NIB3Afj3b8aXc87/EMvI8ibuc0Hcp59+Gh/5yEdKMF9dslg9Rh2lTtmh+tmLnK8yQi9yTH1VGaoAamFR9Vgdc7QKmQKn1yK8X2bpWfJWKqSqbi+6r7YHicpXDUL1+Hn7tL0o05XOqU7AqmNVI6/rqjqNviSdnOPoVBmydcfPit7PcpDKijnl1RzuJ6JXHSM6V43eL+IYlXK8RLCqODlcdXhqnKHqIs5qxC4dohUrQshtEktHJZ55peg9VxwZ1fFRV0ypatsSYQtASfmnTiSBCFoqG7eqAFTHyAUAInTNRzfvczTdX/tyM4D/FQQlNQLw6xAJzH/8KP+pi7bxeIwPfehDp4yKOinXvT9rv66ub5XiQh3kWmcQzoJk696fdxyAnHTOaxcxFNX36t+tMxxnQblnnVeJPNVIuO599dxZx1YRg6qRJm2r/w/9z3UT+Xnb6mfUV93/XXdefa962UsjzFcTgCSp5wwSEBkLYGk8FBLQkrCTgqe+uI7qF9MzSEB1tYs1RCJpCCoRMYBlJF2JkgGcgpDFsfI1Zy5ZVLqmfrmuXImsxfHydeVjpyNt9bNqy2ug4/uBk3Vr9fiuj74L1vqn7lz4Ox6kPSgJ5guxXYQF6kMYwJ999P/O/bXd3V383M/93IWurTMEF40QVm3PK5+4SKSwauI+CwpdFRVUCRt116+CSs+DSOsihbMigfNg07Oig/MiBlnAftEogedC+abYV6MD1etHnoFHKyDSmoiCcoBq/q9M5siQrjivQqLq6heU96vTAlXLN+q0QWlyJYIIgNp6vVUkD5UQApy+jvbVc3XHzioDOWv+P69cpFomUj63+toqSaTu2KqyEjpXJ7qunqseO6u28CzdVM0sHDGlxnCVJB3lHqvrZVqvvnTmfXyY9rAkmC+0dhEW6Nvw/7d35vGWFEW+/8bthabZV+kGWtZuZEAWGQfUAUFAVARxGIFhGBB5CuMA6pMHTI8I+uADDKOjz1ZkRHBpZIdBREA2ZVSWpm2aRS5rY2/QNNAsIr3cjvdHZt1bp24tmXWqzpq/z+d8TlVmRGRknToVlZGRkfCvwFZxelV9d31quWHatGn8/Oc/H/F/J76jkVd8SUTSFerzHR8VJt2LLqPQMmVFI9Ikbdryj7SyZH1WXdZItmgUmzUPW1UQUh7SXnbS/rRZL0BJ+ryXozhtY51ZGjKqfhwMrDGQwZM9Wi4aPcePXUfSYwcGGF8wuk4bvSZHqa4j5fi18BllRy+USX0iOWmj76zReFbZ8AgbTGAVjARXQepIezjoKm20DSMj7nh5YtRtRGePnA1/ytKbxCjcyDF8Y6+5izrRVyNAYCZwGmYbivqfXh5QVd5+++3SBs3V4LXC6EXnSWOVZ+yi47yy+HdRfRpNVn2y3OW8qLwZpBm9eF3026XRphmzVatWpRq+NMMY9SdJk1Y/ZsyYht8yXjYw0BgwNTQ0NCwrKk+e5/U5qVcRX5rRi5f7uobLfMevnashyzOcqYYu8hAosCLblWwMVZTubmXCiGWvKY1HGQONxzbaGBheCzkcmAUjy0li60eHU+qttp4KyA2yWr0iN4Vy08i777oNLgbwJbtAseMwODjI/vunLhtpQN4fPq+s6DjrO36cdIFCfnBF/DitLJId/2SNFpL1cfcojLhfIxdp/DhNTvyTlws0kp1XljbXlhUgkeVyTR4n59Ti32nHacsuGlys0cMtevgl5t5SXa3xB+XqVaMeiMmoVFYnll5E82oZkak6tDy2/CLhSl0x4mqNlmAMxdyvaQ/NaP4tzQ1btJtFcqsmYJRLNgrWgJGI1sglC+5u2aguzkNGXd55vMy1PFmXRLOrRLK2pMrKDJSsS5a/uPLV5hTKQd+5QIGvisgPgGQu0Otr08oRU6ZM4aKLLho1ooqO88qa+XatSx5njZ5c3Idpo6a8kZTPKCttlFsGRSMSl/LknyvrJSRJ6/qyUlTv86LTzPfAwBiMm3QAmNhII8ZNGnf9QWMIeryuiM71WICxDm2kHcdHW2l1cZos3rzypL5ZvCPtRRbUvpjAyHc0ynOtt3W6OkGnQw31Sfo0Gann9rvB5YkdOcbpC8ojGb/89GnUiX5zgX4a2AGzI0T0VFWg7QZwvfXW48ADD2xIIxQP04+7/uI0kZszrSztOz6XmFwOkDe/mFYP+S7ULLdqGfdrmjs1y+Ua1zEpy8UNm9XH6DjJk7asIi2pdx59hLyypLy4Pslj3zKfelcUuZeKvBZZZfHz+AMsb41rWlkab5pnI1mft242+k7zZMRps+iS5Vm8ybI0OWmfovosnrQANzd38Nj8+rHpbuOoPyvHTaQuxF9EegEuBvCvVXValY2KyDzgDWAIWKWqe4jIhsBVmGCbecCnVDV3LP/kk0/y4Q9/2Gnhe3KxvIeuTnRZboEkf5Iur97nQeUygnFdnpCsi9KE5T2ckvLzaH2XQxQ9iHxo00Ywed/R9XZ/eKUvt4i3Fy+LtxPVZ811Oc1v5cxneQVxuM5zRa7ZaBQSzXUV5VCNlk/QOMeVtYQCyE4sMDTygqVDI1G2kR7DyyhWRmWN7uDhstWN8hrr48kJRlzEo+vcExfokKLAqpj7uJEmPdlB3KUcLzfXQHnr5WepE/1mAH8nIjtWkXg0gX1VdWns/AzgTlU9X0TOsOen5wlYc8012XnnkfRwRa7EMkEZdQRs+CDvZsuqS5bnuRWT5/FjF3dkkRGGkWCO+EgwrZ0sI5z3XWWdz7GXa9HRlegjL6s+zW042k054n6VgXzXZdzIxsvix/G6hmjKuNGF0S7DtDodMT6jXI0NLsoMmQn5cbdimuswmbzciGyMtjTiCsoSkZjD9Yn2ssrTkqRHSCZLz9rnMfpe+5tXUBf6cQS4J2Y/wOcwc4ACaA3LIA4FPmiPf4TJEJNrAKdMmcKMGTMK3ZpFGdjL8Oa5TOPu1zyauMwi/iJ3Z1552bIsF2eWKzS+H2IeXZp7NC/6NY8nukZZ58m6Ivr4eVZZUo/kcV5dHp3LeVG5a32dqOKlLa0s60Utr67oOM+lm1ZWxXney1cznpXoeHG9QaBNzQGKyEGYzQ/GAD9Q1fMT9WsAPwbeA7wMHKGq82zdmcBnMJ7DU1T1ttKKWLgYwIOabSQFCtwuIgp8X80WR+9Q1cW2/gXgHUVCFi5cyPTpjevz43/8skEnLg+htLI8V2wnP7CKUPTGl1cfHyFGx5FLNU9GM6NYn+CYLPq8EWoaXRpfmRFtnLcsf9EINy1oJK0u67gMTxG/64g2Ka8oUKYZntTRLYwOnIncyll1MGqkmjrCjXhGyUkZ5cbaSI5yf3tn03YhF2WjQMW4HmYAB2D2fH1QRG5KeBc/A7yqqtuJyJHABcARIrIjZv/AvwImA3eIyFRVzX7oOiDTAIrIuqr6Omaurmp8QM3W9psCvxKRJ+KVqqrWOKbp1bAh7q233hrxVKpgs3N/WTLS6H3mCX3mCF3KXR7ceQYi/kbrKi+NN+1tN/ouos9zWaaV5R0X1RXVF7k4W1mWPk+Y79pMzjPG64rKDTLmIaGwfMQo6Oj5RxiZl1yVVl6wuDxqJ77Hoy1rmKMEVmcsME/d6zF2DhBP+5acXxw+jpWbecs0uqHYnF86bzIVXES7/KXhfWQrR5Mu0PcCT9u8z4jIlRjPX9wAHorZgQjgWuA7Ym6uQ4Er1ewQ/5yIPG3l/b6sMpA/ArwCOBh4CPPqE39KK7BN2UZ1ZGv7JSJyA6YjL4rIJFVdLCKTgNRfUWMb4u622256zz33NLgU4y7LuBszqy5enlZXdJx0VdZ5HLWb5xasoyzru4gma+G9y4L9rIw28fM03qroXOpdvQqu0alp9a40aXR5tFn0PvydCJfRSd7LrWswWxatz8tsUX2WJyN5njxetKreZQoFg4ONRWRW7PwS+8wG2ByYH6tbAPxNgn+YRs0Guq8BG9ny+xK8m/tr34hMA6iqB9vvrZttJA4RWQsYUNU37PGBwNeAmzD7AJ6P436Ab731FrNnzx5lmGJ9aCgr+q6TpqjMp9712MUF7LN3oq/r2IUni69MeVJ+1kMi4o/C1F3QLhd1mbdtF548Q5HHn1aXVlaUei7tvGywVpLXx9Wdx5NnZMrWu3hG8upcaOfMmUOdEM39LyxV1T1qVaBC5LlAd89jVNXZJdt8B3CDfYsYC1yhqreKyIPA1SLyGeB54FNFgp577jmOPvrokmo0wuWh4ftgyCovehAU8eQ9COLnRa7QIveoSzCAa1nRHzlviUZ07sITZZuJ6l3crgMDfss60srKuFVd6dNos1ygWccu9Um5ybK8epc5tzyXa7wsfuwip8HtCo0u1oSbFGisS0ad5kaVZkSj5kWbJtKaNbhgI+/OcDTp6kaXLDTstjFCuzrmnh0te52hP1MbNBbl64+FwJax8y1sWRrNAhEZC6yHCYZx4fVG3ivwf9jvCcAewMMYN+i7gVnAXmUatP7fXVLKXwY+5CNru+22Y+bMmbmjvqzzPJpm+Fo18qtqlJg3yitD53Kel0+0iN6HL6ts5cqVo8ry6PPKI7i4Cn1Gke0acbYaviPcZke3RTKy6rJkNvPimya3DJ/JKgQmsBKWrCg9R+cAZTjJtz8eBLYXka0xxutI4B8SNJEn8PfA4cBdNibkJuAKEfkGJghme+CBsopEyHOB7gsgItcDu6vqI/Z8J0YmKduKcePGMXny5Ia5PBhtoJLzgEmaImOZdu5Dm9VOM7Li51V9V81Tlh8a59+iB0BUPzQ0NKqs7Dyei5u4Gfo8njzZRbQu52kywX2uMGuOsJl5Rdc5xWaD2toxd9lMjswqgu4iOa+//nppPZxQ8uXMzun9C2ZP2THAD1X1MRH5GjBLTc7pS4Gf2CCXVzBGEkt3NSZgZhXw+WYjQAGk6EYTkcdU9a+KytqBCRMm6Dvf+c5cmuRDoYkIpkK4yi6iq8rVCm6T92VostynRbzNRri6uIKTLkyfel86FxqXZRQuLuC872SZ6xxS3ndStgtt1neW+9Wlrhm6ZmUk3bJJ3jS6rPoi2Vmu4HT+2JIMaHDx/vWe72PWQ7NriYTZY/fd9MF7786sH1h7g4d6Yg4whrlikmH/1J4fDcytTyV3bLrpppx88snD5y5vxGVossp897zLKq97/aCrO62Zt+Zec9k186KU9ZaeNUocGBgoTELezEtTmYAXH1dgs27AZFldNM2c18XrWudKJyK8/sab1IemXKAdB9dk2CcBp9rz3wDfq00jD2y66aaceOKJAKNcnMllDVk08e+4OzI+Mk5zX+bJyNKjWVeni0u2qD5rntN1HjSvvtdcsWmuS1f3qitdkUvVx53r66r1dedm8eYt0yhb58vry58nx/UFuMryojqX+ohm0aJFhXSl0VwQTMeh0ACq6tvAN+2no/DMM89wxBFHNJRVOSIqS59Et62lSkMVe4D5jKpa4U6ue2SURV82QCKZdMBVls/ygjL08fMqtrPK46vjOKuurMy6+OLTAnlln/vc56gNitn/skfgvhCqA6GqrPDY/bgqQ9TsBH3dcJ1Qd4Gv4avKcBXRtMKwtcKolXHlFbkhi9pthUHLo22nsaviuGoZroYtXp5XVuX/fzRslp4eQVcbwG233Zbrrx/ZlrDILZlHk0bvQ1Nn+2llRa7KMnydIj9+7FrWDp46+epwYabtYVlWriuPq/5FbVUdIevrLi3jck2jqYMuSTt//vxUmmqgYQTYLERkfeAHwE6YQfXxwCCe+wEuXbqUyy67rKHM1X/fijJfWt+gmjKyyspzrS8aZbu6k13pyozqy7q0m3WFt0pmO5CW9NwHzQQdVS2zDJ9rn6vykOS1t2DBAqc2SkEZSQTQAyg0gCIyFTgNeGecXlX3a6LdbwG3qurhIjIemAj8K577Ab7wwgtccMEFTajhhjr+nFXI77Q5taL6LNeMb0LxLJ5macvkdnSV5SKnyI1Z1v2YPK875Vhee83K9pVbdglN1nGVtHXVX3fdddQHhaH+igK9BrgY+C/MPkxNQUTWA/YGjgNQ1RXAChHx3g9whx124JZbbhk+T3Mn5h1Duisur9yVpwp5devRirq0ele3mu95lbLSzovaq0JmWZ5m+MrkdE3j8+XP8140k/3HV7eiujyeTvS0vPLKK7m8TUEV7TMX6CpVrXLZw9bAS8BlIrILZreJUymxH+CqVatYtmyZs5FI1lf5UC9ruFpR3+rzdvJ0Ap9PWVZ5s0nDq1qjWubh3yuGxIXfh87H3e3j2m+5G72fXKDAz0Xkn4EbMDvCA6CqZV8zxgK7Ayer6v0i8i2Mu3MYqm77AY4ZM4YDDzywpBrl0aku0bK83eZKLaovy+vros2LtmuXi7ZqujK7OrjQ+C61SPJU5SIucicW6erqxi06d23Dhe7ee++lNmj/uUCPtd+nxcqU8vsBLgAWqOr99vxajAH03g9w6tSpOmPGjMxlCXW8mfuWl3kb9pFTZRtleZqpq5O3lTLqkAX1jgTKjBzKLiWqYpRSx0inztFTnkehzrXBecnem0f/LYSvdD9AVX1BROaLyDRVHcTsAPG4/RyLx36Aa6+9NnvttVckN95GA12WOzJJ5+oCzePzcYW6yiyS0866ZH0z7ttWuZXrdCn7zht2g3u6FTLL0ri4iptxMdcpq1leH5mVocYRoIhsSMFKABHZFZOJbF1MTMq5qnqVrbsc2Ad4zZIfp6q5myO6RIFOBL4ETFHVz4rI9sA0Vb3ZuWejcTIw00aAPotJtzaA536Ag4OD7Lfffh2/ML1fUO8C3Gqy0UBzepbRoUx7ru34yG52t/Qq5TSzZVFRfdm6KjcJzpPnK8tXft3Q+kaAZ1C8EuAt4J9U9SkRmQw8JCK3qeoyW3+aql7r2qCLC/QyTKDK++z5QkxkaGkDaK1yWsZwr/0A11lnHfbZZx+gt90/Vbpp6p4w79S0by0PFGgz6p6nbhZ1P7yr7H+zsprh75Q1lcNQhVXu2bc8UbgSQFWfjB0vEpElwCbAMkrAxQBuq6pHiMhRttG3pO5XfUdsscUWXHjhhQ1leS61ZH2R29HHjeYru8jt1gx/lbLqkNeMu7Lq+mb1a0d9HW1U4dZsJ083yKlSVp7rdOzY+vKbKGp2tK8HXisBROS9wHjgmVjxuSJyFnAncIaqLk9ltnC5UitEZE1M4Asisi2xaNB2Yv78+XzpS1/KpemGTCHNjE6qGnFVPUJqZXBBK9GpI9yA9qBdbkjIHuktX17j41kVVuXOAW4sIrNi55fYwEUAROQOYLMUvumNzWSvBLByJgE/AY5VHV6XcSbGcI7HBEqeDnwtT1kXA/hV4FZgSxGZCbwfu4i93fjzn//MfffdFx5KXYR2PjDqdFx0wvxkhLK6lG3btz3fdnzkVz0nCtUt/amSJk/3Wl8QVdF8A7hUczbEVdX9s+pExGklgIisC/wCmK6q98VkR6PH5SJyGfDlPEXBLQr0VyIyG9gTEOBUVV1axNcKTJ06lTvuuKPQHZlXVuQmbbe8qtuoo52q6VrVdrv1rIu2WV2r0CGr3Ie2m2XXLSev/Pe//32mrOah6OraBhw3UbASwAZO3gD8OBnsEjOeAnwCeLSowUwDKCK7J4oi6zpFRKao6uwi4XXjzTffTF30WfQG5PKG1KyMuutbJaOqdlqpT7tk1UHXjbRQfyaTVgWxddN0CJg+ttkF2gzOJ2UlgIjsAZyoqifYsr2BjUTkOMsXLXeYKSKbYAZqc4ATixqUrDdAEbnbHk7ARGw+bAW/G5ilqnuV6WGVGD9+vG6yySbtVmMYnR5554J29KHONquWXYW8dkUVtpLPl8eXvg5Xp4/cqpepVLlEZdasWbzxxhu1+PvfM20b/d33v55ZP2Hff3wozwXaacgcAarqvgAicj2wu6o+Ys93As5uiXYFmDp1KldffXVDWdGawLw3rHbxdnPbreBvhQ5VyeikUXcnjqbbMZLuhtF7lTKfffZZZ1ne0NV1LoNoOVyCYKZFxg9AVR8VkXfVqJMzBgYGWHfddUeV5z3IytSVNT7dzle2rTraK2qz1dezrjab4a1L32bru1V2K+rraKPuVWo1LoRvOVwM4FwR+QHwU3t+NDC3bIMiMg2T7ibCNsBZwI/x3BB3cHBweCF8QOeiG1zD7dKxVe120oazrZBZhZxmDUkzkcHNtL10aY0xivUuhG85XAzgp4GTMFsWAfwGk4utFNTk/9wVQETGYDLL3IBbGpwGTJo0ienTp6fWddPkdbt4q+DvNRl1yKpDXp1Lf6rWtVWyk2jH8qhW9i8L8+fPr022Fi+D6Cq4LIN4G/im/VSNDwHPqOrzZTbE3WijjTjmmGNGlVftAoXybimfpQjt0KPT2ut1XbqFL4+3X1zVzfC2U99rrrkml6Y5KNS3DKLlyFsG8Qg2+0saVPXdFbR/JPAze+yUBie+H+CECRM49NBDK1AjHS7BGZ2GkBQgH934mwb4oUMyNbYNixcvLiYqC1W01u2WWou8EeDB9vvz9vsn9vsfyTGMrrALGg/BpK9pQF4aHI3tB7jhhhvqWmut1awqXugEF4cvulHnViBclwBfdMN8du3ZlvphBKiqzwOIyAGqulus6nSbGeaMdE5nfASYraov2nOnNDhxbLPNNlx11VUNZc2Gy9ddX0U4fj/oWMXSiXbr2AodWrGEpAoZrVqGEmTAE088USi7NFRZ3U9zgICIyPtV9bf25H2YvfuaxVGMuD/BIQ1OEkuWLOE73/nO8HknBkF0ejBFHTLrcsPWNWKrcyRYt0u6FaPYVo6UWz0qb9eUQZ39fPXV3OD55qCKDvWO58TFAH4G+KGIrIfJBPMqcHwzjYrIWsABwOdixalpcPKwZMkSZsyY0YwqAR2ObnA5dSp65dp1Qz86RceBgQFef/312uSrKqtXrqpNfqvhEgX6ELCLNYCo6msFLIVQ1T8DGyXKXsZzQ9wddtiB22+/3bVNH/1qofV96+sEnX3p6+xjkN16+d2sey/IL9PGCSec4N2GMxRWr+gDAygi/6iqPxWRLyXKAVDVb9SsWyFWrFjRsOalbIRfWXdEr7cX2uzddruRt938nbBW1YV/xYr6FqqrKqt7KNI8bwQYhVeu0wpFyuDpp5/mkEMOabcaAT2CTnFjdQJ66Vp0U1+q0HXhwoUVaJKBGl2gIrIhDtnARGQIiNJz/klVD7HlWwNXYryLDwHHqGru20BeFOj37fc5vh1pFbbeemsuueSSYsIMVLUmrMoJ7SrXqXWqXtCZAUv9Kq/fZfaa3LPOOquWNiPUGATjmg3sL6q6a0r5BcA3VfVKEbkYE7+Sm7UszwX67TxGVT0lr74VmDhxIrvssoszfZmHeKt4yvxROrk/4Rp09jVoVTutbKsX+1SmrQkTJpRqxwWqylB9QTDe2cAi2E1w9wP+IcZ/NmUNIGYIGeEc4KsuirQSg4OD7Lvvvu1WAwiLqgMC6kY3uTLbiXnz5tUnXAtHgBuLyKzY+SU2eYkLnLKBARNsG6uA81X1Rozbc5mqRtZ5AbB5UYN5LtAfRcci8oX4eadg/fXX5+Mf/3hL2up0A9fp+sXRTbrG0Stp5rr1+rui1/uXh6GhIRYtWlSbfIdlEEs1Z0NcEbkD2CylqmFXg7xsYMA7VXWhiGwD3GXTdpZaneCyDhAqSH1WByZPnsw557hPUXaDu6MZ3lbztTqatVv4uuW6NMPbLX1sxzVt97W55557SslxbKypZRCqun9WnYg4ZQNT1YX2+1kRuQfYDbgOWF9ExtpR4BaYnYZy4WoAK4WIfBE4AWNYH8FsuTQJzwie559/npNOOqlpfXrlzR56++23l/uWhX7scxF66f9aB+qNAq31nizMBiYiGwBvqepyEdkYeD9woR0x3g0cjrEjTtnE8oJg3mBk5DdRRKL0AoIZoY7eit0BIrI5cAqwo6r+RUSuxuwK8VE8I3jefvttHn/88TJqOKPq6MeAgICAOlH7OsAVteUCTc0GJiJ7ACeq6gnAu4Dvi8hqTErO81U1MgKnA1eKyP8F/gBcWtRg3hxgnev/xgJrishKYCKwmBIRPFOnTnXOBJOGqt5kqjCSVRnaTpNTxTXutD6Fa9M9cjrpP94qOQcffHBmXQUtozWNALOyganqLIzHEFX9HbBzBv+zwHt92my5C9ROXl4E/An4C3A7xuXpHcHz2muvccstt+TStNOF1G73VT/3vRN0aHf7naIDBD3SUJcur73WdLbKbDQ5B9hpaLkBtD7cQ4GtgWXANcBBHvzDG+KOGTOGU089tQ41A2pCCGUPCHBD2f/KCy+8ULEmI1CF1X22G0TV2B94TlVfAhCR6zETmU4RPBrbEHfnnXfWG264oZQSrZrba9UbZyvnKlv5Ft1rvxOE36oKtHok1019O/bYYyvSJAX9thtEDfgTsKeITMS4QD8EzAK8I3hEhHHjxg2ft+JP4ftHKLs7cyf2JYJvn7rhJcC1T93QlziK+tVt/YmQ169u7VOEZN98+xNtWFAHVJWhFb0ThduOOcD7ReRaYDZmJf8fMCO6X+AZwTM4OMjee+9dp7oBAQEeCC7u9qPOhfAo6OreiYxvyzpAVf0qo1OreUfwbL755px33kTexhEAABLtSURBVHmpdZ002e2LbtYdul//CL3Qj17oQxy91h+ovk/nnntupfLiUCWMADsFG2ywAZ/85CeHz1vhp29FG93uwml1G61qJ/z2nddGq9rppt/lu9/9bgWaZEAVHQojwI7AU089xcc+9rHh85AhIqAbEZItBFSJZ555pj7hYQTYORg/fjyTJ09utxp9j/DiEZCGXnRXdgPGjx9fm2wFVoc5wM7AVlttxeWXX+7F04637XY9CNrR13aNZvqpr+24n/rp+nZ7X/fZZ5/KZI1CiALtHCxevDgzCKYXEN6g/RGuWXmEkXx1aOd9WOtC+NWwekXv/Me62gC+8sorXHnlle1WIyAgIKBjsGzZshqla22ZYERkQ+AqYCtgHvApVX01QbMv8M1Y0Q7Akap6o4hcDuzDyN6Ax6nqnLw2u9oATps2jbvuuqvdaqSi0wMbOnmk1OnXLujXHDpdv07+b0Dx9fvoRz9aY9swtLK263MGcKeqni8iZ9jz0xvb17uBXWHYYD6NyScd4TRVvda1wXbtB3gq8L8wWyv9l6r+p4v1T+Ltt9/miSeeqFnb6tDpf6yy6PQHWlmE36v70Ku/Gbj/bsuXL69TCbS+XKCHAh+0xz8C7iFhABM4HPilqr5VtkFp9Z9BRHbCpDt7L7ACuBU4EZPg+pWY9d9AVfM6z/jx43WTTTapW+WAgICArsFLL73EihUrasmHNnXNtfS72+yYWX/A47OeB5bGii6x+ZsLISLLVHV9eyzAq9F5Bv1dwDdU9WZ7fjmwF7AcuBM4Q1Vz3wbaMQJ8F3B/ZLVF5NfAJ/G3/my77bbeUaB56OW3xyL08qigCP38u0fo598feuse+OxnP1ur/IL9AJeq6h5ZlSJyB7BZStX0hjbMDu+ZN6WITMLsC3hbrPhM4AVgPCa95unA1/KUbYcBfBQ4V0Q2wiTD/igmGfY7VHWxpXkBeEeRoDXWWIPtttuuNkXrRLf/4XrhgdnNfehm3SN0ex869T88YcKE+oSvVoaaiAJV1f2z6kTkRRGZpKqLrYFbkiPqU8ANqjq8PX3MfiwXkcuALxfp045k2H8UkQswE5d/BuYAQwmaTOsf3w9w3LhxIRl2QEBAQAzPPvtsbbJrzgV6E2YnoPMp3hHoKMyIbxgx4ynAJzCDrVy0Kxn2pdjdHkTkPMwO8E7WP74f4BZbbKG17n1VAp36VhjQGoS1dAFZaNWz4eKLL65Req25QM8HrhaRzwDPY0Z5iMgewImqeoI93wrYEvh1gn+miGyCCa6cg4ktyUW7okA3VdUlIjIFM/+3J2aHeFfrD8Bmm23GaaedVthet7takuglI9tLv03oS2eil/ri8t8vu0m4C1RhZU3PH1V9GbM/bLJ8FnBC7HwesHkK3X6+bbZrHeB1dg5wJfB5VV0mIqnWPw/PPfccxxxzTM2qBsTRS8Y3IKBb4PO/mzdvXm16KNBDm0G0zQX6tyllqdY/DytXrmTJkrx50v5GL735BgQEuGHVqlW1yVZgRUiG3RnYfvvtufnmm9uthjf62TD1c9/7ffTcz799K/teZ2CgKgz10O/Y1Qbw1Vdf5eqrr263Gh2Ffn/I9gPCb9xf8P29X3nllZo0AUXDCLBTsHDhQr7yla+0W42AgICAjsHixYuLiUrCuEBrE99ydLUBnDZtWse4QPvZvdNqhBFQaxHu7dai2fv7sMMOq0iT0Qgu0A6Cqtaa+DU8aNuL8OBtL8L9336U+Q/U+buFIJgOwuDgIPvt5730IyAgIKBn8eKLL9YmOyyD6CBMmTKFiy66qN1qBHQQwqgloN1o9z14+um5ewg0BdUwAuwYrLfeehx00EGl+dt9o/Yrgmuz9QjXvPVo1zU/77zzapUf5gAdICI/BA4GlqjqTrYsddNbm7z0W5idId7CbGU/u6iNJ598kgMOOKCeDvQYQo7KgID+wJNPPlmbbEVZGQygEy4HvgP8OFaWteX9R4Dt7edvgO/Z71ysueaa7LzzzhWrHRAQEJCObvAaPfXUU7XJXk1wgTpBVX9js3bHkbXp7aHAj9X4DO4TkfWjnSHy2pgyZQozZsyoUu0ARwSXWnegGx7YAdX+nx588MHKZI2ChiCYZpC16e3mwPwY3QJbNsoAxvcDXHfddXtqIXx4WAUEBDSLRYsW1SY7LIOoCEVb3ufwDe8HOH78eL3ssssq1y0gICCgW/Hyyy/XJtssgwgGsCyyNr1diNngMMIWtiwXK1eufHPRokWDJXXZGFhakrcT+DtBh9CHztCh3fydoEPow4iMtZqUkYmXWHHbd/X5jXNImtW/tVDV2j6YaM9HY+f/Dpxhj88ALrTHHwN+idnJd0/gAUf5s5rQrTRvJ/B3gg6hD52hQ7v5O0GH0IfqZPTTp85lED/DBLxsLCILgK+SseU9cAtmCcTTmGUQn65Lr4CAgICAAKg3CvSojKq0Le8V+HxdugQEBAQEBCQx0G4FmsQlbeLtBP5O0CH0oTN0aDd/J+gQ+lCdjL6BWL9xQEBAQEBAX6HbR4ABAQEBAQGl0JUGUER+KCJLROTRsvQisqGI/EpEnrLfG2Twbikid4vI4yLymIic6sNvaSeIyAMi8rCVcY4t31pE7heRp0XkKhEZX9CPMSLyBxG52ZdfROaJyCMiMkdEZvn2wdKvLyLXisgTIvJHEdnL4zpOs21Hn9dF5Aue1/GL9vo9KiI/s9fV9xqeavkfE5EvFF0Hn3tHDL5tdZkrIrtn8P+9bX+1iOyR0O9Myz8oIh/O0eHf7e8wV0RuEJH1s2Rk8H/d8s4RkdtFZLJPH2Jy/reIqIhsnMWf04ezRWRh7J74qE8fLN3J9jo8JiIXel6Dq2JtzxOROSV+h11F5D4rY5aIvNfzXthFRH4v5r/5cxFZN6cPXs+irN8iIIZ2h6GWDPXdG9id2BILX3rgQhqXZFyQwTsJ2N0erwM8Cezoym/rBVjbHo8D7scs97gaONKWXwycVNCPLwFXADfbc2d+TPLxjRNlzn2wND8CTrDH44H1fWVYujGYTEDv9PgdNgeeA9aM9f04z2uwE/AoMBETAHYHsF2eDj73DiaSOb6c5/4M/ncB0zCpAPeIle8IPAysAWwNPGOvVZqMA4Gx9viCmA5pMj6Ywr9u7PgU4GKfPljaLYHbMBHdG2fx51zHs4Evp/xOrn3Y1/6Ga9jzTX34E23+B3BWid/hduAjsb7f43kvPAjsY4+PB76eo8PmeDyLsn6L8In97u1WoLTiiTWGvvTAIDDJHk8CBh3l/DdwQBP8E4HZmGTfSxl5iO0F3JbDtwVwJ7AfcLO9qX345zHaADr3AVgPY4CkrIwYz4HAb334GUmXtyHGeN0MfNjzGvw9cGns/CvA/ynSwfXeAb4PHJWky7pXGW0AzwTOjJ3fBuxVdL8DhwEz82QU8J8JfM+3D8C1wC7xeyuLP+M6nk26AXTqA+blZ/+y/LF6sffW9r6/g607wh4fBVzhcx2B1xiJxdgSeLxIh1hZ7rMo77cIH/PpShdoRcjKS5oJMcm9d8O8zXnxi3FfzsFkv/kV5o1umaqusiRR/tMs/CfmYR0lDN3Ik1+B20XkITH5VPHsw9bAS8BlYtywPxCRtTxlRDgS+JmPDqq6ELgI+BMmR+xrwEP4XYNHgb8VkY1EZCLmDXnLEn3wzWnrirL8x2Pe9L1kiMi5IjIfOBo4y4dfRA4FFqrqw0324V+se+6HMuJ6dpUxFfN73i8ivxaRvy6pw98CL6pqtI2CD/8XgH+31/EijOHykfEYZjMAMC9oUUasXH7HZ1Gz92PPo58N4DDUvB5pHo2IrA1cB3xBVV/35VfVIVXdFTOSey+wg6t+IhLtq/iQK08KPqCqu2O2nvq8iOyd0K+oD2Mx7pvvqepuwJ8x7hYfGYiZozsEuCZZl8dvH46HYgzxZEy6J6/dkFX1jxh34e3ArcAcYChBU9iHZuirhohMB1YBM315VXW6qm5pef/Fo82JwL8yYjTL4nvAtsCumJea//DkH4vxCOwJnIZJsiEl9DiKkRcyX5wEfNFexy8Cl3ryHw/8s4g8hHFrrihiaPZZFDCCfjaAL4rJR4o05iUdBREZh7nhZqrq9b78cajqMuBujEtmfRGJkhHk5T99P3CIiMwDrsS4Qb/lwR+NoFDVJcANGCPs04cFwAJVvd+eX4sxiL7X4SPAbFV90Z678u8PPKeqL6nqSuB6zHVxvgYAqnqpqr5HVfcGXsXMo/j2IYu+VE7bGLz4ReQ4zKbTR9sHX1kdZgJ/58G/LeZF5GF7T24BzBaRzXzaV9UX7YvhauC/MPekTx8WANerwQMY78jGPjrYe+eTmI26I/hcw2Mx9yKYlzqvPqjqE6p6oKq+B2OEn8nj93wWNXs/9jz62QDehLl5sd//nUZk3ygvBf6oqt/w5bcyNhEbpScia2L89n/EGMLDi2So6pmquoWqboVxH96lqke78ovIWiKyTnSMmYN71KcPqvoCMF9EptmiDwGP+8iwSL5tu/L/CdhTRCba3yRq3+kaRBCRTe33FMyD74oSfciivwn4Jxt9tyfwmhbsaZki90gRWUNEtsZsEP1ARj8OwrjED1HVt3xliMj2sdNDgSdc+6Cqj6jqpqq6lb0nF2CCM17wuQbRQ9viMMw96XMdbsQEwiAiUzGBWUs9+MG8WD2hqgtiZT78i4B97PF+QORGdboOsftxAPg3TCBXng4+z6Jm78feR7snIct8MA/QxcBKzJ/vM770mDm0OzE37B3Ahhm8H8C4FOZiXGZzMHNHTvxWxruBP1gZjzISbbYN5qZ+GvP2uIZD3z/ISBSoE7+le9h+HgOm23LnPlj6XYFZth83Aht4Xoe1gJeB9WJlPvznYB7UjwI/wUTIeV1D4F6M4XwY+FCRDj73DiaYYgbmLf4RYI8M/sPs8XLgRWKBO8B0yz/ISHRhmoynMfM70T15cZaMDP7r7HWcC/wc2NynD4lrOo+RIJhR/Dl9+ImlmYt5WE/y7MN44Ke2H7OB/Xz4Ld3lwIkp94nr7/ABzFz0w5j5uPd43gunYrwQT2JyJUtOH7yeRVm/RfiMfEImmICAgICAvkQ/u0ADAgICAvoYwQAGBAQEBPQlggEMCAgICOhLBAMYEBAQENCXCAYwICAgIKAvEQxgQFshIkNiMuk/KiLX2CwjPvyTReRae7yrNO4ocIiInJHN7dXO76qQExAQ0DkIyyAC2goReVNV17bHM4GHtHGRr4+s4zBrnZzTegUEBPQvwggwoJNwL7CdmP3NbrRJku8TkXcDiMg+MrJ/2x9EZB0R2cqOHscDXwOOsPVHiMhxIvIdy7uViNxlZd5pM8EgIpeL2TPtdyLyrIgcnqaYiLxpvz8oIvfIyL6IM21mmiT9KWL2bZsrIlfasrXEJH1+wOp/qC1fU0SuFLPH4g1ikjvvEW/XHh8uIpfb401E5DoRedB+3m/Lz7Zt3GP7c0qM/5+sPg+LyE/y5AQE9APGFpMEBNQPMTkZP4JJUn0O8AdV/YSI7Af8GJOF5svA51X1t2ISAr8d8avqChE5i9gI0I4II/w/4Eeq+iMROR74NvAJWzcJk2VjB0xGkmsL1N0N+CtMGqzfYnKS/k+C5gxga1VdLiOb1U7HpLE73pY9ICJ3AJ8D3lLVd1ljP7voemFywX5TVf/HGvPbMPsMYvuxLya58qCIfA+zc8K/Ae9T1aUisqGDnICAnkYwgAHtxpoyshP3vZhch/djkzOr6l1iti9aF2NsvmFdpder6oKUwVcW9sLk/gSTguvCWN2NahIyPy4iLts5PaA2d6TVfStGG8C5wEwRuRGTNg5MDtZDROTL9nwCMAWzUeq3bX/nishcBx32B3aM9X9d+1IA8AtVXQ4sF5ElmO1x9gOuUdWltp1X8uSo6vDIMyCgVxEMYEC78Rc120QNI8uoqer5IvILTP7D34rIh4mNApvA8njznvRDpP+PPoYxbB8HpovIzlb236nqYJywwIjHJ+knxI4HgD1VtaH/VpaLfrlyAgL6AWEOMKATcS9mk1ZE5IPAUlV9XUS2VbMTwQXAg4zeU/ENjNsvDb/D7KSBlX1v5VpbiMnsv6Wq3g2cDqwHrI1xL54czRmKyG6W5TfAP9iynTDJ0yO8KCLvsjIPi5XfDpwca7PhJSIFdwF/LyIbWfrIBeorJyCgZxAMYEAn4mzgPdYVeD4jW718wQa8zMVk1P9lgu9ujDtvjogckag7Gfi05T0Gk4W/LowBfioij2B2Afm2mn0gvw6MA+aKyGP2HMzGsGuLyB8xgTzxjY/PAG7GGPD4VjanAHvYoJbHgRPzFFLVx4BzgV+LyMNAFGnrJScgoJcQlkEEBHQYROQe4MuqOqvdugQE9DLCCDAgICAgoC8RRoABAQEBAX2JMAIMCAgICOhLBAMYEBAQENCXCAYwICAgIKAvEQxgQEBAQEBfIhjAgICAgIC+RDCAAQEBAQF9if8PygAQLEHg3noAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "pos_enc = PositionalEncoding(100, max_len=200)\n",
        "pe = pos_enc.pe.squeeze().T.numpy()\n",
        "\n",
        "\n",
        "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(8, 3))\n",
        "pos = ax.imshow(pe, cmap=\"RdGy\", extent=(1, pe.shape[1] + 1, pe.shape[0] + 1, 1))\n",
        "fig.colorbar(pos, ax=ax)\n",
        "ax.set_xlabel(\"Position in sequence\")\n",
        "ax.set_ylabel(\"Hidden dimension\")\n",
        "ax.set_title(\"Positional encoding over hidden dimensions\")\n",
        "ax.set_xticks([1] + [i * 10 for i in range(1, 1 + pe.shape[1] // 10)])\n",
        "ax.set_yticks([1] + [i * 10 for i in range(1, 1 + pe.shape[0] // 10)])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can clearly see sine and cosine waves with different wavelenths that encode the positions in the hidden dimensions. "
      ],
      "metadata": {
        "id": "adZ4edCv5BbS"
      },
      "id": "adZ4edCv5BbS"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Further, we visualize the positional encoding for the hidden dimension 1,2,3 and 4. "
      ],
      "metadata": {
        "id": "t54UG21m5Lzr"
      },
      "id": "t54UG21m5Lzr"
    },
    {
      "cell_type": "code",
      "execution_count": 82,
      "id": "ded8aa52",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "ded8aa52",
        "outputId": "9ac7ffec-7e23-4e39-c6a6-96a5bb284fcd"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x288 with 4 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAAEWCAYAAACZq3rNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3gUVReH35NCEnpHKQm9i4hRQQGpSgdR1BgVFbsgYtcoTaJ+CqKABRUVJUapShUsgCIK0gk1tIRO6DWBJPf7YwZMMGWzO7uzu7nv88yT2Z2dc3+7O3ty5pZzRCmFRqPRaDQajUajyZ8AuwVoNBqNRqPRaDS+gg6eNRqNRqPRaDQaB9HBs0aj0Wg0Go1G4yA6eNZoNBqNRqPRaBxEB88ajUaj0Wg0Go2D6OBZo9FoNBqNRqNxEB08a/6DiCwSkYfN/WgRWeCmdjaISBsnz1UiUjuXY3lqzvr+cjhW3bQd5IyugiAiX4nICHO/lYhscXebBUFE5olIX7t1aDSavNE+W/ts0D7bk+jg2csQkV0ick5ETmfZxtmlRykVp5S6xU22GymlFrnBrts0uwul1B9KqXp268iKUqqzUmqi1XZFpK2ILBSREyKyy2r7Go0n0T7bErvaZ1uAG332CyKSICKnRGSniLxgdRu+htvv1DRO0V0p9YvdIjQaN3EG+AKIB161WYtGYwXaZ2v8GQHuB9YBtYAFIrJbKfWdvbLsQ/c8+xAi8oCILBGRkSJyzLwD7JzleFkR+VJE9pnHf8hy7BER2SYiR0VkpohUznKso4hsNnsCx2H8ULK1meWxEpHHRSRRRI6LyIciIuaxQBEZJSKHTW398xpOM3tsOpj7Q0Vksoh8bd7dbhCRyHw+kg656Lhcc17vL9D8PA+LyA6g62UaS4nIBBHZLyJ7RWSEiAQ68n3k8H6vEZFV5vv7HgjNcqyNiOy57LN5QUTWicgZU0Mlc1julIj8IiJlsry+uYgsNT+LtZJlaNUc8nxDRP40z10gIuXNY6EiMklEjpjn/iMilbKcd3EoOEBEXhORJBE5ZH5PpcxjF4dN+4pIsvlZxuT2OSilliulvgF25PYajcYf0D77P2if/e/rfclnv6OUWqWUSldKbQF+BG7K7fWFAR08+x43AFuA8sA7wISLDgj4BigKNAIqAqMBRKQd8BZwJ3AlkAR8Zx4rD0wHXjNtbif/H0U34DqgiWnzVvP5R4DOQFOgGdCrgO+th6mrNDATyG/oMzcdl3Dg/T1i2rkGiATuuMzEV0A6UNt8zS1A1rl3eX0fWXUUAX7A+I7KAlOA2/N5f7cDHYG6QHdgHkZPbQWM3+7Tpu0qwBxghGn7eWCaiFTIYuse4EGM66KI+RqAvkApoBpQDngcOJeDlgfMrS1QEyjOf7+flkA9oD0wWEQa5PP+NJrCgPbZ+eu4hPbZl/BKn21+Vq2ADfm91q9RSunNizZgF3AaOJ5le8Q89gCwLctriwIKuALDwWYCZXKwOQF4J8vj4sAFoDrGUMzfWY4JsAd4OEubS7IcV0DLLI8nAy+b+78Bj2U51sF8fVAe77WDuT8U+CXLsYbAuTw+p7x0XNLswPv7DXg8y/FbLmoGKgFpQFiW41HAwvy+jxz0tgb2AZLluaXACHO/DbDnss8mOsvjacDHWR4PAH4w918CvrmsvflAX3N/EfBalmNPAj+Z+w+ZOprkoHlRls/pV+DJLMfqmddQkHkdKaBqluPLgbvzudY7ALvs/s3pTW+ubGifffGY9tl+7rPN1w0D1gIhdv/27Nz0nGfvpJfKff7cgYs7Sqmz5g1zcYy716NKqWM5nFMZWJXlvNMicgSoYh7bneWYEpHd/zWRswbgrNn+xXaynpufnfzshopIkFIqvYA6spLf+7tcc1KW/QggGNifpWMi4LLX5/Z95KRjrzK9Tw5t5cTBLPvncnh8sZ0IoI+IdM9yPBhYmJNOsn9W32D0YHwnIqWBSUCMUupCDvqz6k3i339W+bWh0fg72mdrnw1+7rNFpD/GzU0rpVRaXq/1d3Tw7D/sBsqKSGml1PHLju3D+LECICLFMIZ79gL7MX6IF49J1scFZD9QNctjZ+1YSX7vb/9lj8Oz7O/G6MUon8c/g4LoqCIiksUZh2MMSbrKboxejEcKeqLpcIcBw0SkOjAXY0hzwmUvzXYNYWhPx/jnUBWNRlNQtM/OGe2z88Auny0iDwEvA62VUnvye72/k++cZzEWNFy+BXtCnMZxlFL7MeZXfSQiZUQkWERam4fjgQdFpKmIhABvAsuUUrsw5l01EpHeYiwSeRpjSNEZJgMDRaSKeUf8kivvySLye3+TgadFpKq5mOPliwfMz3QBMEpESpoLMGqJyM1O6PgLw3E9bX43vYHrnX1TlzEJ6C4it4qxmCZUjMUs+TpIMdLGXSXGgpqTGMN6mTm8NB4YJCI1RKQ4xjX0vTP/oMzPMRSjp0VMvUUKakeTM9pn+wbaZ+eK9tl5YJPPjjbP76iU0gu9cWzB4CogBdgKJJr7u8RYgXqtO8UVYmZJ9pyhMxw87z6MH9Jm4BDwDIA5nPg6xhys/RipZu42jx0G+gBvA0eAOsCfTur+DMNxrQNWY9wRpwMZTtpzGQfe32cYc83WYlzr0y8zcT/GYo2NwDFgKsZcxYLqOA/0xphzdxS4K4e2nEIptRvoibEwJQWjV+MFHPt9X4Hxnk4Cm4DFGMOCl/OF+fzvwE4gFWMOnzO0xhjCnIvRG3IO47rRWIP22Z5H+2yL0D47X+zw2SMwRj7+yXKNf+KkLb9Ask/nyeEFIp8BU5VS883Ht2CsKP0S+EApdYPbVWp8EjFSAH2ilIrI98UajcYStM/WOIv22RqNYzhyl9P8ohMGUEotAFoopf4GQtymTONziEiYiHQRkSAxUvEMARztgdFoNNagfbbGIbTP1micw5Hgeb+IvCQiEeb2InDQnG+T0zwbTeFFMBYyHMMYAtwEDLZVkUZT+NA+W+Mo2mdrNE7gyLSN8hh3oy3Np/7E+LGdAMKVUtvcqlCj0Wg0DqN9tkaj0biXfINnjUaj0Wg0Go1GY5BvnmcRqYtRFrJ61tcrpdq5T1bOlC9fXlWvXt3TzWo0Go3LrFy58rBSqkL+r3QN7bM1Go3GGnLz244USZkCfAJ8jo3pawCqV6/OihUr7JSg0Wg0TiEi+VUnswrtszUajcYCcvPbjgTP6Uqpjy3Wo9FoNBr3oH22RqPRuBFHsm3MEpEnReTKrBWr3K5Mo9FoNM6gfbZGo9G4EUeC574YlW+WAivNzZJxOBH5QkQOiUiCFfY0GncRF/ctNerUJyAwkBp16hMX963dkjQOUgi/O5/12fFxcTSuW53AwAAa161OfFycO5rRaDQal8g3eFZK1chhq2lR+18BnSyypdG4hbi4b3li0Auci+xLtWency6yL08MeqEwBGE+T2H87nzVZ8fHxREz6DHGtkgh9dXijG2RQsygx3QA7SPoGx9NYSLXVHUi0k4p9ZuI9M7puFLKkhrvIlIdmK2UapzfayMjI5VefKLxNDXq1OdcZF9CI5pcei41aR1hKyayM3Gzjco0+eFN352IrFRKRbrRvk/77MZ1qzO2RQpta/y7FGfhznQG/FWBhK27nBOr8QgXb3wmdFa0DA9kSXIG/eYJsaPHExUdbbc8jcZpcvPbefU832z+7Z7D1s1yhbkgIo+KyAoRWZGSkuKpZt1GIRxC9nmSdiQSUrVhtudCqjYkaUeiTYo0jlLIvjuf9tmbtifTMjww23MtwwPZtD3Zaokai4kdFsOEzoq2NYIIDhTa1ghiQmdF7LAYu6VpPEhhGn3INduGUmqI+fdBz8nJUcenwKdg9GLYqcVVLg4hF23fn2q9GnJuz0aeGPQCANHR99isTnM559Mzmbh0F8HlqpG2Z2O23su0PRsJr1nHRnUaRyhftWaO312JShHsPX6OKqXDbFRnLb7usxvUCmdJcvae5yXJGTQIL2+9SI2lbNqeTMu7imd7rmV4IJsmJUP6eQgqYpMyjafINvpwV3GWJKfQb9BjAH45+pBrz7OIPJvX5kmRdlPQ3uK09AySjpxh6bbDTFmxmw9+SeTFqWt5/PnXKNq+P6ERTZDAIEIjmlC0fX9eGzrcQ+9E4ygLtxyi0we/Ezt3Ezfc1o/Tv4wjNWkdKiOd1KR1HJn3ATU73MuFjEy7pWpy4dtlyairb+PY/DHZvrsTC8ZS9IY76DBqMR8u3EZauq2pkC3D1312zJBY+s0TFu5M50KGYuHOdPrNSiOm2Wn4dThk+sf35I80qFqGJcnZv58lyRk0KCcwqh7MeQ52/wO6orHXUeDe4ox0OLEHkpdBwnRYOhZ+eoXYFx8rVKMPeeV5LmH+rQdcB8w0H3cHlrtTlDeRU2/x48+8wJ7jZ2nWtjt7j51l34lU9h47x97j59h3/Bwpp9P+4yMqlgjh9MEkyuY0hDzZL4eQfZKdh88wYvZGft18iBrli/HFA5G0q9+VuNa1eG3ocJImJxJRsw4PPfMK81NrM+Db1Yy95xqCAx1JXKPxFN8uS+bVGevp3rsPN/dpwtDhIy59d6PGjOTmLr0YMXsT787fwtSVexjSvSFt6lW0W7ar+LTPvtg7NWBYDJsmJdOgVjixHwwjqtQK+GMUHEiA3p9CWGmblWqysWUeMZHn6Dc7nQndyD7n+aXnIfwQrJ4E/3wOZWtCk7ugyZ3GvsZWcu4tfhSObieqbWM4uQ9O7jW2E3uNx6cPgLqs0yi4GJv2n6FleIlsT18affBDcl0weOkFIr8DXZVSp8zHJYA5SqnWLjcuEg+0AcoDB4EhSqkJub3ejgWDuS04OvrLJ1Tu9xEAIUEBVCkdRuXSYVQuHUqV0kWNv2XCqFI6jCtKhRISFOhVi5c02Tmdls7Y3xL5YslOigQG8HT7Ojx4Uw2KBOUeFH+xZCfDZ2+k61VX8sHdTQnSAbRXcDFwble/Ih/f24yQoMBcX7t4awrDZm5gx+EzdGxYicHdGlKtbFG36HL3gsEs7fiXz1YKVkyAeS9Bmepw97dQoZ5rNjXWcGA9TLgVKtQlPrQvsSOGsWm7ceMTMyT23+H61BOwaRas/Q52LQEUVLvBCKIb9YaiOg25HeS6SHdeKglPmtNwgotCySpQqorxt2QVKFkZSlU1/pasAqGlaFyvhl8u+M3NbzsSPG8Bmiil0szHIcA6pZTHvZcdwXNAYCDVnp2OBP57QaiMdHaP6s3qpCNUKRNGuWJFEJF8bWXtxQ6p2pC0PRs5Pn8ME8aOJNoP5wT5ApmZiumr9/K/nzaTciqN25tV5aVO9ahYMtSh8z//Ywcj5myi+9WVGX3n1TqAtpmCBM4XSUvPYMKSnYz9dRuZSvFkm9o8dnNNQoPzP7cgeDB49k+fnbQUJt8PF1Lh9s+gXmdr7Gqc49RB+KwdoOCR36DEFY6dd2IPrJ8Ca7+HlE0QEAx1bzUC6bqdiP9+KrHDYnIOwjWWEhgYQOqrxQkO/Dd+uZChCH3zNBn71hvBcWhpcCC+yTHjyuzzxI75kqjoe935NtxKbn7bkfLcXwPLRWQGIEBPjFyfhYKImnU4l8OCo4hadbi6WsGGDy8uCrw4/F+2cg1KtLyP4HoudwhpnGDN7uMMnbmBNbuPc3W10nx637VcE16mQDYeblWTjEzFW/M2EyDw3p1NCQzI39ForOdi4Ny2XgWHA2eAkKBAnmxTm15NqxA7dxOjf9nKtFV7GNytIR0aVnKzarfgnz474kZ4dBF8Fw3xd0PbGGj1PAToG1aPc+EcfBcF547CQz85HjiD0WPZchDc9IzRc73ueyOY3jyb+M1BxPx8igndggvFojO7yXWRbq1wqNSoQLb+M+2qalli2yiiwg9aqtlbyLfnGUBEmgGtAAX8oZRa7W5hOWFHz/NTw8fw6XtvUq7zwEu9xWd/HcfHo991OUNGRqbivgnLWJV8jFn9W1KnUon8T9K4zKFTqbzzkzHXtUKJEF7qVJ/e11QhwIWg96NF23jnpy30vqYK7/a5WgfQHiZr4PzJfdc6HDjnxJ/bDjNk5ga2HTpNu/oVGdK9IRHlirms0VM9z2Zb/uuzL5yDWQONoKtBd+j1MYRo3+kxMjNh2kOw4Qe4axI0sCALYkY67FxM4za9GNs+0++G/r2V+HefJ+bN95jQI8z6/NxKwdSHYOOP8OBcCG9ujWgP40ye56xkAJlZtkLB7qNnWZhel+v6PEHYionsfq83YSsmWhI4AwQGCO/f1ZTiIUE89e0qzp3Xq8mtJmumlOp16vPw4NG0G7mYH9fs5bGba/Lbczdzx7VVXQqcAZ5sU5vnb6nL9NV7eWnaOjIz9apyT2Fl4AxwU+3yzH26Fa92qc+yHUfoOPp33luwhS8nTvKlHO3+67ODw+C28XDrm7B5DnzeEY5st1tV4WHx27BhBnQcZk3gDBAYBLXbm4vOdK5vj3DmMFH8QOxttRmwtDyhb55mwF8VrCtsIwLdP4DS1WBqPzh71HWbXkS+wbOIDATiMBaIVAQmicgAdwuzm8xMxfNT1hIgwpR3X2Rn4mYyMzLYmbjZ0pzMFUuGMvqupiQeOs2wWRsss6v5b2nm1Mi+TBzzP8odWM78Z1rzSucGlAgNtqy9/u3q8EyHOkxduYeXp+sA2hPEL7c2cL5IkaAAHm1di9+eb0OnRlfw1rgJPPaMb5T5LhQ+WwRaPAX3TjdW/3/WFrb9Yrcq/2fdFFj8P2h6L9z4tOXmjWkEOaS8qxVueVuFGqVg9iA4d5yot2aQkJhERkYmCVt3WTs9JrQk3PElnD4IPz7lV6kKHel57gfcoJQaopQaDDQHHnGvLPv5aukulu08yuBuDd1eSKFVnQo82aYW3/2zmx/X7HVrW4WJ14YO/09e7XKdB5L8Wxw1KxTP34ATPNOhLk+3q83kFXuI+WG9DqDdSPzyZF6Zbn3gnJVKJUMZE3UNwet/oGynp30lR3vh8dm12sIjC6FkVYjrA39+4Ff/oL2K3cuNACjiJug22qFFZAUlx1zfs88TMyTW8rYKNQnTYNNMaPtqgec2F5gqzaDjcNgyF5aNd29bHsSR4FkwhgAvkmE+57fsSDnNO/M307ZeBfpEVvVIm4M61CUyogyvTl/PzsNnPNKmv2NXaeZBHevyVNtaxC/fzes/JuDIugJNwcgaOH98r3sC56wcSN7uS2W+C5fPLlsDHv4ZGvSAnwfDtIfh/Fm7VfkXx5Lgu3uM7At3TXJbxcCo6GhiR49nwF8VjGkEi8KIbRNMVEPtQy3j1AGjaE3V69wyepAjzZ+Aup1hwWuwz5blF5bjSPD8JbBMRIaKyFDgbyDXvJ6+Tkam4rkpawkJCuTt25s4lILOCoICA4werqAAnopbReoFPf/ZVcJr1CFtz8Zsz6Xt2UiEm8tqiwjP31KPx2+uRdyyZIbM3KADaAu5PHC2OqVcTkTUtOdacpJC5bMBKFIM+nwF7QdDwjTiH29K49pVHa+apsmd1JNGdpP083DPZLfnZI6KjiZh6y5jGsHOQ0R1vRnmx8DpFLe2WyhQCmY+Delp0OsTY665JxCBXh9B8Yow5UHjmvJx8g2elVLvAQ8CR83tQaXU++4WZhef/r6D1cnHGd6zEZUczPVrFZVLhzHyjqvZuP8kb83d5NG2/ZEbevfjyLwPspVmPvvrOEYMHez2tkWElzrV49HWNfn6rySGzdqoA2gLsCNwBhgxdDBnfx1ny7VUUAqbz76ECLR6jvgSjxHzw3bG3nSM1FeLM7ZFCjGDHtMBtDNkZsC0fpCyBe6cCBXqerb9gADo/j5cOAvzX/Fs2/7I6kmQOB86DIXytT3bdtGycPsEOJ4Ms5/x+elVjiwYbA4kKqXGKKXGANtF5Ab3S/M8Ww6cYvTPW+nc+Ap6XF3ZFg0dGlbi4ZY1mPhXEj8l7LdFgz+w9eApVgY14ta+A92SKcURRIRXOtfnoZtq8NXSXYyYs0kH0C5wMXBu4+HAGYwc7R+Pfte2a6kgFCafnROx479nQvdQ2tYIIjhQaFsjiAmdFbHDYuyW5nsseA0SF0CXd4355XZQoR60es7IBZ34sz0a/IHjyfDTK1C9FVz/qD0aIlpA21eMOdervrZHg0U4UmFwNdBMmS8UkQBghVKqmQf0ZcOdeZ4vZGTS68M/OXAilQWDWlOueIhb2nGE8+mZ9PlkKTsOn2Hu063cVi7YX1FKcdenf7P14Cl+e64NZYu5Z35eQfQMm7WRr5bu4tHWNXmlc32PTQfyF7IGzp94OHC2Cg9WGCwUPjs38qyaluFfWfvcyoovjIwMzZ+ETm/ZqyU9DT5pZeT4fupvY5qOxnEyM+GbnrB3FTyxFMpE2KglAyb1huRl8OhCqNjAPi0O4EqeZ1FZImylVCaOVSb0KT5cuI0N+04Se1tjWwNnMNJkjbvH+D/XP34159O1wy8IM1bvZfnOo7zUqb7tgTMYPdBDujfk/hYRfPr7DqJeGuVL+YJt41KO7oBA+nZtRfixVT4bOHuYQuGzc0OnO7OA7QthzvNQ5xa4ZYTdaiAoxMgZfCIZFr5ptxrf45/PYefvcGusvYEzQEAg3PYphBSHKQ/47OJeR4LnHSLytIgEm9tAYIe7hXmShL0nGPfbNno1rUynxlfaLQeAamWL8r/bm7B293FGLthitxyf4cS5C7w5dxNNq5Xmrshqdsu5hIgwrEcjrk5LYNqnI30iX7CdZMvR/dx0ynZ4nNUzPmHa5O/tluYL+L3Pzosc053NSiXmFT1n1iFStsLkvsZ0idsnGMGONxDRAq59EP7+yG8yNniEI9uNLDS1O0CzvnarMShRCXp/asyln/ei3WqcwpHg+XHgRmAvsAe4AbBpwoz1pKVn8OzkNZQtVoRhPRrbLScbXa66kvuaG72Vv232z/rwVjNqwRaOnjnPiF6NXa4aaDUiwrrZX1Ku80BfyRdsGznl6C6mPydH8WufnR//SXe2pDSxbUOJKqeLUOXL2aPw7Z1GKrqo74wiF95Eh6FQrIKRMSIj3W413k9mBsx43Pg+e4x1S25up6nVDlo9C6u/gfVT7VZTYBzJtnFIKXW3UqqiUqqSUuoepdQhT4jzBO//ksjWg6f53+1NKFXUumpzVhHTtQENrizJc5PXsv/EObvleDXr95zgm7+TuL9FdRpXKWW3nByxK/e0r6E/J+fxd5/tCNnSnW3fS9QjA40FSntX2i3Ne0k/D9/fCyf3wd3f2j+8nxNhpaHzO3BgHSz72G413s9f42DPcugy0sjR7W20eRWqNYdZA40ech/CkWwbFUTkVRH5VES+uLh5Qpy7WZV8jPGLt3NXZDXa1q9ot5wcCQ0O5MN7riEtPZOB8WtI1wteciQjU/HaD+spXzyEZ2/xcDqlAuBj+YJto1yVGvpzchJ/9tlO0+ZlI8fsnOeNxVMaAOLj4mhct7qRD7t6eeJnL4KeH0K16+2WljsNe0K9Lsbc52O77FbjvRzaBL+NgAbd4ao+dqvJmcAguP1zCAiCqQ8aC0N9BEembfwIlAJ+AeZk2Xyac+czeH7yWq4sFcZr3bx7tWfNCsV587arWL7rKB/8qnveciJ+eTJr95zgta4NKBnqfSMIF8kpX/Cpn8d6Zb5gu0g5lUbodXdwbP4Yn8ir7IX4pc92idCS0PEN2LfKGCbWEB8XR8ygxxjbIsXIh90+g5jFGcSvP2+3tLwRMVLnSQDMftbn8wW7hYwLMOMxCCkJXd1TSt0ySleDXh/D/rXG3GwfwZEV2EWVUi+5XYmHeWf+ZnYcPsO3D99ACS8Oti7S65oqLN1+mHELt3FDjXK0rFPebklew+HTabzz02Za1CxnW35uR7mYF/i1ocNJmpxISPlq1OrUjzvvvttmZd7DOz9tJrR+a97qWJexo94haXIiETXrMMpL8yp7IX7ps12myZ2w8kv4ZajRG+fmSnneTuywGCZ0VrStYYQBbWsEMaErDBgWQ1R0tM3q8qFUVaOa5LwXjfmyTby0Z9Uu/hhlBKN3fgPFK9itJn/qd4EbnjCm4lRvBQ262a0oXxzpeZ4tIl3crsSD/L3jCF/+uYu+LSK4sbbvBKFDezSidoXiPPP9Gg6dSrVbjtfw1tzNnLuQwRu9GvlE/uTo6HvYmbiZzIwMZi5cxumqzZm4dJfdsryClUnHmLJyDw+1rMGzjz906XPambhZB86O43c+2xIu9limHoeFsXarsZ1N25NpGZ49k0bL8EA2bU+2SVEBue5hqBIJP71sLHTUGOxbDb+/C03ugoY97FbjOB2HwZVN4cen4Phuu9XkiyPB80AMZ5wqIidF5JSI+Gxh8tNp6bwwdS3VyxXlpc717ZZTIIoWCeLD6GacTrvAoO/XkJGph6uW7zzKtFV7eKRVTWpXLGG3nALToUFF2tSrwPu/JHLoZOG+IcrIVAyZmUClkiEMaKfnNruAX/lsS7niKiPoWvGF0TNXiPH5fNgBgUbu59TjRiVEDVxIhRlPGBlJOv/PbjUFIygE+nz5b0n4jAt2K8oTR7JtlFBKBSilQpVSJc3HXpa/xnHenLuJPcfOMbLP1RQt4nt1A+pWKsGwHo1YMHM6lSLqFOpCGxcyMnn9hwSqlA7z2WBLRBjavRHn0zN5a95mu+XYSvzyZBL2niSma0OKh/jeb9Nb8DefbTltYyCsbKFfPBjz7FP0m3kuez7seULMEB/qlb+iMdz4NKyJgx2L7VZjP4vehJRN0GMchJWxW03BKVsTur9P/NwlNK5e0VjIWrc68XFxdiv7D45k2xARuVdEXjcfVxMRL16KmzuLt6bw7bJkHmlVk8jqvjvfLX3rH5xa8g1BLfsV6kIbX/25iy0HTzG0RyPCinhJIn8nqF6+GI+2rnmpMmJh5NiZ84xcsIUbapSlexPvKFTkq/iTz3YLYaWNIeI9y2Hdd3arsQeliApZROytZRjwZzkjH/ZfFYgdPd775ztfzs0vGkHX7GeM8t2FleRlsHSsUQilTge71ThN/Lo0YhalM7bdeWMha4sUYgY95nUBtKh8VqqKyMdAJtBOKdVARMoAC5RS13lCYFYiIyPVihUrnDr3xLkL3Dr6d4qHBjF7QEufLvFbo059zjTBTu8AACAASURBVEX2JTSiyaXnUpPWEbZiIjsTC0fv5f4T52g/ajE31irH5309filaztnz6XQYtZiSYcHMHtCSoEBHZlT5D69MX8/kFbuZ+3Qr6l3he9NvHEFEViqlIj3Qjl/4bLeSmQkTOsLxJOi/wgioCxPrJsP0R6Db+xD5oN1qXGfHYvi6B7R6zlhIWNg4fwY+aQmZ6fDEUgjxXR/auG51xrZIubSQFWDhznQG/FWBhK27PK4nN7/tyH/oG5RSTwGpAEqpY0ARi/W5neGzNpJyOo1Rfa726cAZdAEJgDdmbzTmyHZvZLcUSyhaJIjXuzVk84FTTPo7yW45HmXdnuN8908yD9xY3W8DZw/jFz7brQQEQNeRcOYwLHrbbjWe5dxxmB8DVa71nnLNrlLzZmgaDX9+AAcS7FbjeX4ZBkd3QM+PfDpwBt9ZyOpI8HxBRAIBBUYCfoxeDZcRkU4iskVEtonIy1bYzEpc3LfUqFOfgIBAxj7Vg+aZm7i6mu/3MBT2QhuLt6Ywd/0BBrSrTbWyRe2WYxmdGl9By9rlGfXzVg6f9p1k8a6Qmal4/ccNlCsWwsAOheP69QBu89mmPbf6bY9R+Rq49gFY/ikcLESluxfGwtnD0PU94ybCX7hlBISWNqrVZWbk/3ofJ1uBmwdGEX+2NdRoZbcsl/GVhayO/HLGADOAiiISCywB3nS1YdO5fwh0BhoCUSLSMO+zHCcu7lueGPQC5yL7Uu256ZTt8Dgzv3jPL+YF51Ro4+hPYxg2+HW7pbmd1AsZDPkxgZrli/FI65p2y7EUEWFoj0akXsjgf4Vk8eCUlbtZu/s4r3ap79XFbXwMt/hscL/f9jjtBxsFVOa+UDiKbexbA/98bmQcqdzUbjXWUrQsdHob9q6AfybYrcat/KfATedQYr5a6HXzgp0hZkgs/eZJ9oWss9KIeW2I3dKy4Ui2jTjgReAtYD/QSyk1xYK2rwe2KaV2KKXOA98BPS2wCxhFKIq2709oRBMkMIjQiCYUa9+f14YOt6oJ24iOvoePR79L2IqJ7H6vN/LXl5RqdR+nqt5gtzS3M37xDnYdOcvwno0JCfLt6Tc5UbticR5qWYMpK/ewMumY3XLcyomzF/jfT1uIjCjDbddUsVuO3+BGnw1u9tsep2hZaD8Ekv40im34M5mZMOdZKFreyDjij1x1B9RqD78OgxN77FbjNrIWuAkOFKPATWdF7DDf/16joqOJHT2eAX9VMBay/lGS2LZFiLpyl93SsuHQmI1SarNS6kOl1Dil1CaL2q4CZM2Evcd8Lhsi8qiIrBCRFSkpKQ4b9/d5wVkLbezftZU77ryLD35NZOfhM3ZLcxtJR87w4aJtdL+6sl9XWBzQrg6VSoYwZGaCX+fyHvXzFo6fPc+wnr5R3MaXcJPPBgf8trM+2zaa3W8UZ1jwGqSdsluN+1g1EfauhFtj/XeBpAh0e8+YtjHneb8dTfCVecHOEhUdTcLWXWRkZJKwYz9R9z9oZBLZv85uaZfw+glPSqlPlVKRSqnIChUcLzNZ2OYFD+vRiJCgAGJmrCe/DCq+iFKKwT9uoEhgAK91bWC3HLdSPCSImK4NSdh7kvjl/uEML2fDvhNM+juJe5tH0KhyKbvlaCzEWZ9tGwGB0HUUnD4Ai32ssISjnDlslCWv3gqu8vNS1mWqQ9tXYes82DTTbjVuoUHNqj4xL9gyOr5hjBLNetpr5rPbGTzvBapleVzVfM4ScpoXfPbXcYwY6p9pbCqWDOXlzvVZuv0IU1f633DV/A0HWLw1hWc71qVSyVC75bid7k2upHnNsrw7fwtHz5y3W46lKKUY8uMGShctwnMd69ktR1Mw3Oq3baNqJFxzL/z9MRzyw/UGPw+B86ehy0ijd9bfaf4kXNGE+BGP0bhOuFcX23CGmG61fb/ATUEoWtaomLhvNSz7xG41gL3B8z9AHRGpISJFgLsBy24TL58XHLZiIh+Pfpfo6HusasLriLounOuqlyF27ia/ytZwJi2dYbM20uDKktzfIsJuOR5BRBjWozGn09J5d/4Wu+VYyozVe1mRdIyXOtWjVFG9SNDHcKvftpUOw6BIMZj3on8N9yf/DWsmQYv+ULG+3Wo8Q2AQ8RmdiJl3iLE3HvHqYhsFZtNsokqtJPapPv/OC/bVAjcFoVFvqHMr/DYCjtmfzjXXIikicgoz1dHlhwBlRblXEekCvA8EAl8opfK8bfLahPtexLZDp+jywRI6X3UFH9x9jd1yLOGtuZsY//sOpj3RgmsjfLcypDO8MXsjX/y5kx+fuokmVX1/nuLJ1Au0G7mYKmXCmPHEjQQEFIJeMBN3F0nxhM8223HYb/ucz17+Gcx9Hvp8BY1us1uN62Skw/jWkHoC+i83bg4KCd5WbMMSzhyBj26AElfCI79BYCHrfDi+Gz68ASJuhOgpHhlFKXCRFKVUCaVUyRy2ElY5YaXUXKVUXaVUrfwCZ41j1K5Ygifb1uLHNftYuOWQ3XJcZuvBU0xYspO7IqsVusAZ4JkOdShXLITXf9xAph8sHnz/50SOnEnjjZ6NClXg7Ak84bPNdvzXb1/7IFS6yigict4PFl8vHw+HNkDntwtV4Ax+uqhuzrNGkZvbPil8gTNA6WrQ/nXY9jMkTLNVisPTNkSkooiEX9zcKUrjGk+0qUXtisV5bUYCZ9LS7ZZTYC4VtwkMpFnTJpzf+jsvdS4kw42XUSI0mFe71Gft7uNMWbk7/xO8mC0HTjHxr13cfV24X/SiezvaZztBYJBRefDkXvh9pN1qXOPkPlj4JtS5Bep3s1uNx/GVYhsOkzANNv4AbV+BSv5RWdcprn/UqI457yU4e9Q2GfkGzyLSQ0QSgZ3AYmAXMM/NujQuEBIUyFu9r2Lv8XOM/nmr3XIKRLbiNs9Op3ibRzm86Gvm/eDnOVjz4LZrqnBd9TL87ycjtZsvYmRLSaBEaBAv3qoXCboT7bNdJLw5NLnbSI11eJvdapxnfgxkpkPndwrHIsHLyLHYxpwM31xUd+ogzHnOCBpvHGi3GnsJCITuYyD1uJFe0i4ZDrzmDaA5sFUpVQNoD/ztVlUal7muelnuuSGcL/7cybo9x+2W4zA5Fbcp2XGAXxS3cZaLiwePnz3Pez52M3SRWev2s2znUZ6/pR5lihWxW46/o322q3QcDsFh8NNLvrl4cPtvsGE6tHoOytawW40t/KfYxi9C7M2BRLWqZbe0gqEUzH4GLpyDXp8YoyOFnSsaw41Pw5o42LHIFgmOBM8XlFJHgAARCVBKLQTctuhFYx0vd65P+eIhvDxtPekZmXbLcQh/L27jLA0rl+S+5hFM+juJDftO2C2nQJxOSyd2zkYaVylJ1PU+OmTqW2if7SolKkGbV2DbL7B5jt1qCkZ6mlEgpGxNI8AoxGQrtrF9L1Gta8PUfrYO9xeYtd/BlrnQ7nWoUNduNd7DzS8a1/gs88bCwzgSPB8XkeLA70CciHwA+MFKCv+nZGgww3s2YuP+k0xYstNuOQ5R2IrbFIRnb6lHmaJFGOxjiwfH/pbIwZNpDOvRmEC9SNATaJ9tBdc/AhUawE+v2PLP2WmWjoGj242czsH+nxPfYUJLGVlUTh+EH570jRGFE3uNub3hLaD5E3ar8S6Cw6Db+3Bspy3FjRwJnnsC54BBwE/AdqC7O0VprOPWRlfQsWElRv+yleQjZ+2Wky+3RD/BkXkfFJriNgWhVFgwL3Wqz8qkY8xY7Rt1KbYdOs2EP3bS59qqXBtRxm45hQXts60gMBi6vEv8ku00rlXZNwptHNtlLHRs2Atqt7dbjfdR+Rq4ZYRRffDvj+xWkzdKwcwBkHkBen1kzPXVZKfmzdD0XvhzDBxY79Gm8w2elVJnlFIZSql0pdREpdQYc0hQ4wOICMN7NiIoIICYH7y7dPfirSn8nFabllH9C1Vxm4Jwx7VVaVqtNG/N28zJ1At2y8kTpRRDZ24grEhgoc2WYgfaZ1tH/NJkYhalM7btee8vtKEUzH0RAoKg01t2q/FebnjMyD7y8xDYs9JuNbmzaiJs/9WYf1+2pt1qvJdbzNLdMz1butuRbBu9RSRRRE6IyEkROSUiJz0hTmMNV5YK48VO9fgj8TA/rPHOHsvEg6foH7eKeleUZPb7L7MzcTOZGRnsTNysA+csBAQIb/RszJEzaV6fSeWnhAMs2XaY5zrWpXzxELvlFBq0z7aO2GExTOgWTNsaQQQHCm1rBDGhsyJ2WIzd0v7LlrmQOB/avAwlK9utxnsRgZ7jjEIjUx8w8iZ7G8eSjGwpNVpDZD+71Xg3RctCp7dh3ypYNt5jzToybeMdoIdSqpQ7Eu5rPMO9N0TQLLw0b8zexNEz3pXu7OiZ8/SbuIKQ4EAm9I2kWIheTZwXV1UtRdT14Xz9VxKbD3hnTHT2fDpvzN5I/StKcG/zwlFS3YvQPtsifKbQxvkzMO9lqNgQbnjcbjXeT1gZ6POlkQt7Zn/vmv+cmQk/PgUI9PwQAhwux1F4aXy7kc/8txFw3DO/TUe+lYNKqU1uV6JxKwEBwlu9m3Aq9QIj5mzM/wQPcT49k8e/WcmBk6l8dv+1VC4dZrckn+CFW+pxYevvRF5zNQGBgdSoU5+4uG/tlnWpwE3x0BBWvPcQN7KZoEDt/D2M9tkWkWuhjZrVbFKUC7+PhBPJ0PW9wll5zhmqRkKHobBpllGW3Vv453PY9QfcGguldXYihxCBrqOM/TnPeeRmyJH/aitE5HsRiTKHA3uLSG+3K9NYTr0rSvD4zbWYvmovfySm2C0HpRQxM9azfNdRRva5mmvC9YIyR5n7w1QOL/ya4m0epdqz0zkX2ZcnBr1gawCdrcDNc9Mp2+FxRr45zCuC+kKG9tkWkWOhjZnniGlbGtJO2y3PIGWrUdDl6nsgooXdanyL5k9BnVthQQzsW2O3GjiyHX4ZArU7QrP77VbjW5QOh3avQeICj5TulvwWkInIlzk8rZRSD7lHUu5ERkaqFStWeLpZvyL1QgZdPviD9EzF/GdaE1bEvhW84xdv5615mxnYvg6DOur8lQWhRp36nIvsS2hEk0vPpSatI2zFRHYmbtaavBARWamUcnu+Ze2zrSU+Lo7YYTFs2p5Mg1rhxPTrSVTq1xB+I0RPhiLF7BOnFHzdA/avhf4roXgF+7T4KmePwictIbAIPPY7hNo0wykzA77sAimb4Mm/9bx1Z8jMgM87wInd8NRyYz60i+Tmtx3JtvFgDpvHnbDGGkKDA3mz91UkHz3L+7/at+Ds540HefunzXRrciXPdNA5nAuKNxaTSdrufZoKI9pnW0u2QhtbdxH10gfQ+zNIXgrf3mXMN/Yw8XFxNK5bncDAQBq/MJf4C7fqwNlZipaF2ycYc2VnDbRv/vPfH8Huv41y6jpwdo6AQOgxxrghWvC6e5vK7wUiUlVEZojIIXObJiJV3apK41aa1yzH3ddV4/M/dtpSrW7DvhMM/G41TaqUYmSfqxHRhTMKSm7FZEpdWZ0LHq4mqZTik8XbCSpXVRe48QK0z/YAV90Bt30KSX+aAbTncujHx8URM+gxxrZIITWmOGM7hxIzNt470+f5ChEtoF2MUdJ85Veebz9lC/z6BtTrCk3u8nz7/sQVV8FNT8OaSbBjsduacWTO85fATKCyuc0yn9P4MK90bkCZokV4Zfp6MjxYre7QqVQembiCUmHBfHZ/JKHBOvG7M4wYOpizv47LVkzmxIKxBDW7nXs/X8bh02ke0XHufAYDv1vD2/M2c/Odj3LmMk26wI0taJ/tCZr0gdvGGwF0vOcC6NhhMUzorHwjfZ4vcdMgqNUefnoZDiR4rt2MdJjxuDH9p/v7xuI3jWvc/BKUqQGz3Ve625HguYJS6ksz4X66UuorQI8P+TiligYztEdD1u05wZd/eqZ0d+qFDB79eiXHzl7gs/sjqVhSl451lujoe/h49LvZisl8PmYknw57mjW7j9Nj7BLW7XFv/tK9x8/RZ/xSZq3bxwu31mPBuFf55DJNusCNLWif7Sma3Am9Poadf8B3UR4p4+0z6fN8jYAA42YotDRMecBzC0L/fN/IUdx1FBSv6Jk2/Z3gMONG5OgOWPyOW5pwJHg+IiL3ikigud0L6GpVfkDXq66kXf2KDH5vPNVq1XNryjOlFC9OXcea3cd5/+6mNK5SyvI2ChvR0ff8p5jMbddUZdoTNyIi3PHJX0xbucctbf+z6yg9xy1h1+GzfH5/JE+1rY2I5KhJ43G0z/YkV99tBNA7FkO8GwPos0dh9rM0KCc5p8+rpdOauUzxCnD753B0O8x51v3znw8kwKK3odFt0FgnxLGUmm2gaTTxn7xL41pVCAwMoHHd6pZNb3IkeH4IuBM4AOwH7gAetKR1ja2ICM3SN5CycCIXrn/ArSnPxvy6jZlr9/Fip3rc2ugKS21rstO4Silm9r+Ja8PL8NyUtQyducHSedDfLkvmns/+pkRoMD88dSPtG1SyzLbGErTP9jRNo6DXR7BjEXx3D1xItc52Zgb8MwHGNoOVXxHzQGf6zSN7+rx5QsyQWOvaLMzUaAU3vwzrvoc1bpxHnn4efngcwkpDl1Hua6cQE3+sKTG/nWNsy+OkvlqcsS1SiBn0mCUBdL6p6rwJf0h75G14Ir3Y7HX76P/tano3q8IovUDQY6RnZPLWvM1MWLKT62uU5aPoZi6Vyb6QkcnwWRv55u8kWtetwNi7r6FUUV2QwVE8larOmyh0Pnv1JPixP9RuD3fFQbCLU9OS/4a5L8CBdRDRErq8A5Ua/Td93pBYoqKjrXkPGuOG5ZtesPsfeHQhVGxgidls31uVUsRcl0ZU7BSo39US+5rsNK5bnbEtUmhb49+qxQt3pjPgrwokbN3lkI3c/HauwbOIvKiUekdExgL/eZFS6mkH9VtGoXPEHiAgMJBqz05HAv+9uFRGOsmjejN5+S46Nb6CEqHOB0hrdx/nzvF/0aRqKSY9fAMhQXqBoKeZsXoPL09bT9liRRh/37U0qVq6wDaOnE7jibhVLN95lMda1+TFTvUJDNA3QQXB3cGz9tlewqpvjJLPtTvCXZOcC6BPHYCfBxu9nyWrwC1vQKPeejGZJzl1ED65CYqWg0cWQpGiLpm7mCVlQmdFy/BAliRn0G/2BWLHfKlvfNxEYGAAqa8WJzjw39/NhQxF6JunyXBwNNaZPM8Xy7uuAFbmsGn8gNxSnoVWCOeFqeuIHPELT8WtYv6GA6SlZ+RiJWf2nzjHI1+voEKJED6591odONvExXnQAeY86KkFnAe9Yd8Jeoz7k7W7j/P+XU15pUsDHTh7J9pnewPN7oPuY2DbzzD5PkgvQOab9PPw5wcw9lrYMANaPQf9/4HGt+vA2dOUqGTk807ZAvNecNlc7NBX/5slpVuwzpLiRhrUCnfb+oCg3A4opWaZu2eVUlOyHhORPi63rPEKRgwdzBODXoD2/Qmp2pC0PRs5++s4Ph/9Dg1a3ciPq/cye91+5qzfT6mwYLpcdQU9m1bh+uplCcgjgDp7Pp2HJ67g7PkMJj18A+VcmC6gcZ2L86D7f7ua56esJWHvCWK6NiA4MO9lD7PX7eP5KWspHVaEKY+3cKrXWuMZtM/2Iq7tCyrTSJX1/X1w1zcQlI8P3PYLzHsZjiRC3U5w65tQrpZn9GpyplZbaPUc8R+9RezAKWzafTj3aTIXzhmFVo4nw/EkOJaU7fGm7cm0vLtEtlNahgeyaZLOkuIuYobE0m/QY0zonP5vb/88IXa06+sDcg2es/AKMMWB5zQ+yMVsCK8NHU7S5EQiatZhVJb0Ys3Cy/Bat4b8ue0wP67Zx49r9hG/fDdXlgqlx9WV6dm0Cg2uLIGIEBf3rWFnRyLFK0UQEnkHk999nrqVSuQlQeMhyhUP4Zt+11+aB71x/8lc50FnZipG/byFDxdu59qIMnx8bzMqltCpBX0E7bO9gcgHAQWzB8HkvnDn1xBU5L+vO7YLfnoVtsyBsjXhnilQ9xZPq9XkQvz+6sQsusCEbmdoGV6cJckp9BvYDzZMJ6pZKSM4PpYEZw5lPzGwCJQON7Yrm9Cg6jcsSU7LNv9WZ0lxLxdvcAYMi2HTJGN9QOxoa9YH5DXnuTPQBWPV9vdZDpUEGiqlrne6UaMXZCjQALheKeXQpLhCOX/Oyzh7Pp2fNx5k5pp9LN6aQnqmom6l4lQ+sopp40dSrMO/PdgnF4zlszEjdboyL+SH1Xt5adq6HOdBn0q9wDPfreHXzYe4K7Iaw3s10lNuLMADc57d5rNN+wX229pnY2TKmPMs8YcbEjt/H5t27DZ6L18bQlTVfcY0jYAgaP08tHgq/x5qjUfJddHZvFQSYq76N0AuEwGlL27hULySkTvaJMc5z/OE2NHj9ZxnL8aZBYNXA02B4UDWEmGngIVKqWMuiGkAZALjged18OybHD1znjnr9/Pj6r3MHBxF2Q6PuzVrh8ZaEvae4LFvVpJyOo3OYduZ+dU4knYkElo+nOLN+zDy5Se5r3mEzo5iER4Int3ms037Bfbb2mcbxL/xMDHvfcGEHmH/Bk6zUoltG0JUVBR0HA6lqtgtU5MDViw6u4jOkuJ7FDh4znJikFIq3U2iFqGDZ78gt6wdu9/rTWZGwRYaajzH0TPn6TbwLf6Z+jHlOg+8NGpw6udxfPqBrg5oJZ5KVedOn23aX4QOngtErr2Xf5QkYcd+G5Vp8sOKdGca36XA2TZEZLK5u1pE1mXZ1ovIOrcp1fgkuWXtiKhZxyZFGkcoW6wI+xfHU67zQEIjmiCBQYRGNKFEx/68NnS43fI0BUD7bO8l15LaSQdtUqRxlJghsfSbJ7oojSYbeS0YHGj+7eaMYRH5BciplFyMUurHAth5FHgUIDxcT6z3VnLL2jFq9Lt2S9PkQ9KORKr1apjtuZCqDUmanGiTIo2TuOSzwRq/rX32fzFSZqXoxWI+iDsXnWl8l7xS1V0cSzoMnFNKZYpIXaA+MC8/w0qpDlYIVEp9CnwKxhCgFTY11pNf1g6N9xJRsw7n9mzMNl9djxr4Hq76bNOGy35b++z/4s6UWRr3ExUdrYNlTTbyTvJq8DsQKiJVgAXAfcBX7hSl8U2io+9hZ+JmMjMy2Jm4WQfOPsKIoYM5++s4UpPWoTLSSU1ax9lfxzFi6OD8T9Z4I9pnexlR0dHEjh7PgL8qEPrmaQb8VUFnWdBofBhH8jyLUuqsiPQDPjLLv65xpVERuQ0YC1QA5ojIGqXUra7Y1Gg0zqFHDfwOy302aL/tKrr3UqPxHxwKnkWkBRAN9DOfcynpq1JqBjDDFRsajcY6oqPv0cGy/2C5zwbttzUajeYijkzbeAajOtUMpdQGEakJLHSvLI1Go9E4ifbZGo1G40byzfN86YUixQGUUqfdqihvDSlAkhOnlsdYROMqVtmx0pY/a/Ln92alLa3JN+xEKKUqWNC+Q2ifbakdK21pTZ61Y6Utb7NjpS2tKWdy9NuOFEm5CvgaKAsIkALcr5Ta4IQIWxCRFVYUJ7DKjtbkWTtak9bkjXbchfbZ1tvRmrQmb7SjNdljCxybtjEeeFYpFaGUCgeeAz6zSoBGo9FoLEX7bI1Go3EjjgTPxZRSl+bLKaUWAcXcpkij0Wg0rqB9tkaj0bgRR7Jt7BCR14FvzMf3AjvcJ8ktfOpldqy05c+a/Pm9WWlLa/JNO+5C+2zr7VhpS2vyrB0rbXmbHSttaU0FwJE5z2WAYUBLQAF/AMOUUsesFKLRaDQa19E+W6PRaNxLrsGziIQCjwO1gfXAF0qpCx7UptFoNBoH0T5bo9FoPENewfP3wAWMXovOwC6l1DMe1KbRaDQaB9E+W6PRaDxDXgsGGyql7lVKjQfuAFp7SJNliMgXInJIRBJctFNNRBaKyEYR2SAiA520Eyoiy0VkrWlnmIu6AkVktYjMdtHOLhFZLyJrRGSFi7ZKi8hUEdksIpvMSmcFtVHP1HJxOykiTgUBIjLI/KwTRCTe7J1zChEZaNrZUFA9OV2LIlJWRH4WkUTzbxkn7fQxNWWKiEOpeHKx8675va0TkRkiUtoFW2+YdtaIyAIRqeyMnSzHnhMRJSLlXdA0VET2ZrmuujirSUQGmJ/VBhF5xxFNHkD77H/tWOKzTVte57f92Web9izx2/7ks/OwVWC/bZXPzs1WlmMO+22f89lKqRw3YFVej31hw/jn0QxIcNHOlUAzc78EsBXjH1VB7QhQ3NwPBpYBzV3Q9SzwLTDbxfe3Cyhv0Wc+EXjY3C8ClHbRXiBwACNReUHPrQLsBMLMx5OBB5zU0RhIAIpiLLT9BajtyrUIvAO8bO6/DPzPSTsNgHrAIiDSBT23AEHm/v8c0ZOHrZJZ9p8GPnHGjvl8NWA+RrENh67TXDQNBZ4v4Peek5225vcfYj6u6Mo1btWmfXY2O5b4bPN8r/Pb/uqzzfMt8dv+5rPzsFVgv22Vz87Nlvl8gfy2r/nsvHqerzbvHk+KyCmgycV9ETmZx3leg1Lqd+CoBXb2K6VWmfungE0YP/CC2lHq32pfwebmWInHyxCRqkBX4HNnzncHIlIK48KdAKCUOq+UOu6i2fbAdqWUM1XKwHCaYSIShOFE9zlppwGwTCl1VimVDiwGejt6ci7XYk+Mf1yYf3s5Y0cptUkptcVRLXnYWWC+N4C/gaou2MrqI4rhwHWex+91NPCiIzYcsFUgcrHzBPC2UirNfM0hV9uxCO2z/7Vjic82z/dbv+2lPhus8dt+5bPzsFVgv22Vz87NlkmB/Lav+excg2elVKBSqqS5lVBKBWXZL+lqw76KiFQHrsHofXDm/EARWQMcAn5WSjllB3gf48LMdPL8rChggYisFJFHXbBTA6Oa2ZfmsOTnIuJqftm7gXhnTlRK7QVGAsnAfuCEUmqBkzoSgFYiUk5EigJdMO6s6wtICAAAIABJREFUXaGSUmq/uX8AqOSiPSt5CJjnigERiRWR3UA0MNhJGz2BvUqpta5oyUJ/c2jyC0eGXHOhLsa1sExEFovIdRZpcwnts3PGVZ9t2vA2v+2XPhss9duFzWeDi37bCp9t2rHSb3ulz3akSIrGRESKA9OAZy67S3MYpVSGUqopxt3h9SLS2Akd3YBDSqmVzmjIgZZKqWYYi4yeEhFn50oGYQyXfKyUugY4gzG05RQiUgToAUxx8vwyGD0FNYDKQDERudcZW0qpTRhDYguAn4A1QIYztnKxr3CyN8tqRCQGSAfiXLGjlIpRSlUz7fR3QkdR4FVccOKX8TFQC2iK8U95lJN2gjBKXzcHXgAmi4hYolBjKVb4bPBKv+2XPtu0YYnfLkw+G6zx2676bFOHlX7ba322Dp4dRESCMZxwnFJquqv2zKGxhUAnJ06/CeghIruA74B2IjLJBS17zb+HgBnA9U6a2gPsydIrMxXDMTtLZ4x5mwedPL8DsFMplaKMlF3TgRudFaOUmqCUulYp1Ro4hjGP0hUOisiVAOZf24f/ReQBoBsQbf5zsII44HYnzquF8Q90rXmtVwVWicgVzohQSh00g6BMjHLVrlzn083h/OUYvYgOLWTUeA6rfTZ4j9/2Y58NFvrtwuCzwS1+21mfDRb6bW/22Tp4dgDzDmUCsEkp9Z4LdiqIuRJWRMKAjsDmgtpRSr2ilKqqlKqOMUT2m1LKqR5VESkmIiUu7mMsPnBqpbtS6gCwW0TqmU+1BzY6Y8skCheG/zCG/ZqLSFHzO2yPMffRKUSkovk3HGPu3LcuaAOYCfQ19/sCP7pozyVEpBPGkHIPpdRZF23VyfKwJ85d5+uVUhWVUtXNa30PxiKwA05qujLLw9tw8joHfsBYgIKI1MVYZHXYSVsaN2CVzzZteZXf9nOfDRb6bX/32WCd37bCZ4O1fturfbZyccWhN28YP+L9GLlP9wD9nLRzsVLXOoyhnzVAFyfsNAFWm3YSgMEWvMc2uLZquyaw1tw2ADEu6mkKrDDf4w9AGSftFAOOAKVc1DMMwwkkYJQrDnHB1h8Y/1jWAu1dvRaBcsCvQCLGSuCyTtq5zdxPAw4C8520sw3YneUad3S1dU62ppmf+TpgFlDFGTuXHd+F49k2ctL0DUbxkHUY/wSvdNJOEWCS+f5WAe1cuUb15vg1UAA7lvhs05ZX+W1/99mmLUv8tj/57DxsFdhvW+Wzc7N12XGH/Lav+ex8y3NrNBqNRqPRaDQaAz1tQ6PRaDQajUajcRAdPGs0Go1Go9FoNA6ig2eNRqPRaDQajcZBdPCs0Wg0Go1Go9E4iA6eNRqNRqPRaDQaB9HBs8ZlRCRDRNaISIKITDErDBXk/MoiMtXcbyoiXbIc6yEiTle8uqydpVbY0Wg0Gl9G+2yNxjV0qjqNy4jIaaVUcXM/DlipnCxMYFZKilRKOVUaVKPRaDR5o322RuMauudZYzV/ALVFpKyI/CAi60TkbxFpAiAiN5s9HmtEZLWIlBCR6mYPSBFgOHCXefwuEXlARMaZ51YXkd9Mm7+aVaMQka9EZIyILBWRHSJyR07CROS0+beNiCwSkakisllE4nKqcy8iT4vIRrO978znionIFyKy3NTf03w+TES+E5FNIjJDRJaJSGTWds39O0TkK3O/gohME5F/zO0m8/mhZhuLzPfzdJbz7zf1rBWRb/Kyo9FoNA6gfbb22ZqC4mqVFb3pDTht/g3CKFf6BDAWGGI+3w5YY+7PAm4y94ub51QHEsznHgDGZbF96bF5bl9z/yHgB3P/K2AKxs1gw/+zd+dxUZX7A8c/XwaQTUXcQQV3xcoyy0qz5Zb3tljZcistvS1WmlumpgZuoOKWWTdNs11ssbL1Z2mlZcs1Nc0SRU1BRcAFAWVRhOf3xxxsNJYBZubMDM/79TovZjvP+Z5h+PKdc57zPMCeSuK8FsgBWhjr/Az0KuP1hzBmtgJCjZ8zgAdKHwN2YZ1ZazTwmvH4RcAZrEdjzm7XuH038IZxe3npdoFWWKcSBpgC/ATUARphnbXLD+hibK+R8bqwitrRi170opeyFp2zdc7WS80WXzSt5gJFZKtxez3wKrABuAtAKfWtiDQUkXrAj8BzYj1V+JFS6mAZBxDKcyVwp3H7bWC2zXMfK6VKgCQRaWpHW78opQ4CGLFHAT+c95ptQKKIfIx12lqAPsBtIjLGuB+ANfn1Bl4w9nebiGyzI4YbgGib/a8nIiHG7S+UUqeAUyJyGGiK9R/aCqXUUWM7WRW1o5Q6e/RE0zTNhs7ZOmdrNaCLZ80RCpRSF9s+UF5yVUoliMgXwM3AjyLyT6DQATGcst18FV9fTNl/C7dgTbB9gWdF5EKj7buUUsm2L6zkn4nthQUBNrd9gCuUUufsv9GWPfFV2I6maVo5dM7WOVurAd3nWXOW9cAAsPZXA44qpXJFpK1S6nel1CxgI9DpvPVOAHXLafMn4D7j9gBjG04hIj5AS6XUWuAZoD7WU5ZfAcNL+9uJyCXGKt8D/Y3HLsB6GrBUpoh0NtrsZ/P4amC4zTbP+WdWhm+Be0SkofH6sGq2o2madj6ds/+ic7ZWIV08a84yBbjUOBWWAAwyHh8l1gtNtgFFwKrz1luL9XTWVhG597znhgMPGes+CIx0WvRgAZaJyO/AFuAFpVQ2EIe1L9s2Edlu3AdYBISIyA6sF9BstmlrPPA51n8k6TaPjwC6GxeTJAFPVBSQUmo7MB34TkR+A0qvjq9SO5qmaWWYgs7ZpXTO1iqkh6rTNCcQkXXAGKXUJrNj0TRN0yqmc7ZWFfrIs6ZpmqZpmqbZSR951jRN0zRN0zQ76SPPmqZpmqZpmmYnXTxrmqZpmqZpmp108axpmqZpmqZpdtLFs6ZpmqZpmqbZSRfPmqZpmqZpmmYnXTxrmqZpmqZpmp108axpmqZpmqZpdtLFs6ZpmqZpmqbZSRfPmqZpmqZpmmYnXTxrmqZpmqZpmp108axpmqZpmqZpdtLFs6ZpmqZpmqbZSRfPmqZpmqZpmmYnXTxrmqZpmqZpmp108az9jYisE5FHjdsDRGS1k7azXUSurea6SkTalfNchTHb7l8Zz0UZbftWJ66qEJE3RCTeuH21iCQ7e5tVISKrRGSQ2XFomlYxnbN1zgads11JF89uRkRSRKRARE7aLP81Kx6lVKJSqo+T2u6ilFrnhHadFrOzKKXWK6U6mh2HLaXUTUqpNx3drog8JSJ7RSRXRA6JyHxX/OPTNGfQOdsh7eqc7QDOytmlRMRfRHaIyEFnbcNT6OLZPfVVSoXYLMPMDkjTHOhToJtSqh5wAdAVGGFuSJpWIzpna7XBWOCI2UG4A108exAR+Y+I/CAic0XkuIjsE5GbbJ4PE5HXjaN5x0XkY5vnBovIHhHJEpFPRSTc5rkbRWSniOQYR0zk/G3a3Fci8oSI7BaRbBF5SUTEeM4iIvNE5KgR27CKTqcZR2xuMG5PEZH3ReQtETlhnB7sXslbckM5cZwfc0X7ZzHez6Mishe45bwY64vIqyKSLiJpIhIvIhZ7fh9l7O8lIvKrsX/vAQE2z11r+23eeG/Gisg2EckzYmhqnJY7ISJfi0gDm9dfISI/Ge/Fb7anVo1TnnEi8qOx7moRaWQ8FyAiy0TkmLHuRhFparNe6algHxGJEZFUETls/J7qG8+VnjYdJCL7jffy2fLeB6XUn0qp7NLwgBKgzNO5mubJdM7+G52z/3q9x+RsY53WwAPAzIpeV1vo4tnz9ACSgUbAbODV0gQEvA0EAV2AJsB8ABG5HusH/t9AcyAVeNd4rhHwERBjtPkn0LOSGG4FLgMuMtr8p/H4YOAm4GKgG3BHFfftNiOuUKxHJys79VleHGfZsX+DjXYuAboDd5/XxBvAGazF3SVAH8C2711Fvw/bOPyBj7H+jsKAFcBdlezfXcCNQAegL7AKmAg0xvq3O8JoOwL4Aog32h4DfCgijW3a6g88hPVz4W+8BmAQUB9oCTQEngAKyojlP8ZyHdAGCOHvv59eQEfgH8AkEelc3o6JSH8RyQWOYj3yvLiC90HTPJnO2ZXHcZbO2We5Vc4GXjT2paxt1T5KKb240QKkACeBbJtlsPHcf4A9Nq8NAhTQDGuCLQEalNHmq8Bsm/shQBEQBQwE/mfznAAHgUdttvmDzfMK6GVz/31gvHH7W+Bxm+duMF7vW8G+3mDcngJ8bfNcNFBQwftUURxnY7Zj/74FnrB5vk9pzEBT4BQQaPP8/cDayn4fZcTbGzgEiM1jPwHxxu1rgYPnvTcDbO5/CCyyuT8c+Ni4/Qzw9nnb+woYZNxeB8TYPDcU+NK4/bARx0VlxLzO5n36Bhhq81xH4zPka3yOFNDC5vlfgPvs+Ly3B+LKes/0ohdPWNA5u/Q5nbO9NGcD/YBVZe13bV30RTru6Q6l1NflPJdRekMplW98YQ7B+u01Syl1vIx1woFfbdY7KSLHgAjjuQM2zykROfD3JsqOAcg3tl+6Hdt1K2unsnYDRMRXKXWminHYqmz/zo851eZ2JOAHpNscmPA57/Xl/T7KiiNNGdmnjG2VJdPmdkEZ90u3EwncIyJ9bZ73A9aWFSfnvldvYz2C8a6IhALLgGeVUkVlxG8bbyp//bOqbBvlUkrtFpHtwELgzsper2luSudsnbPBC3O2iARjPUJ/8/nP1WaVFs8iElbGwyfK+EVp5joAhIlIqPqrP2mpQ1j/WIGzfwwNgTQgHesfYulzYnu/itKBFjb3q9uOI1W2f+nn3W9lc/sA1qMYjSr4Z1CVOCJERGyScSuspyRr6gDWoxiDq7qi8Xc8FZgqIlHA/2E9pfnqeS895zOENfYzWP85tKBmfIG2NWxDM+ic7TF0zi6bztkVMCFnt8d6pHq98UXDH6gvIhnAFUqplKrugzewp8/zr1ivrtwF7DZupxid6C91ZnCa/ZRS6Vj7Vy0UkQYi4icivY2n3wEeEpGLRaQOMAPYYHzovwC6iMidYr1IZATWU4rV8T4wUkQijG/Ez9Rknxyksv17HxghIi2MiznGlz5hvKergXkiUs+4AKOtiFxTjTh+xpq4Rhi/mzuBy6u7U+dZBvQVkX+K9WKaALFezFJpghSR60TkQrFeUJOL9bReSRkvfQd4SkRai0gI1s/Qe9X5ByUij4pIE+N2NDAB6ylGzTF0zvYAOmeXS+fsCpiQs//A+mXlYmN5FGsBfjFVP1PhNewpntcANyulGimlGmK9uOBzrH1wFjozuFrsMzl3zNCVdq73INY/pJ3AYWAUgHE6MRZrH6x0rEf57jOeOwrcAyQAx7B+y/yxmnG/gjVxbQO2YP1GfAYormZ7NWbH/r2Cta/Zb1iLjo/Oa2Ig1m/aScBx4AOsfRWrGsdprN0S/gNkAfeWsa1qUUodAG7HejHHEawJbSz2/X03w7pPucAO4DuspwXP95rx+PfAPqAQax++6ugJ/C4ieVg/I/9nxK45hs7ZrqdztoPonF0pl+ZspdQZpVRG6YL1vSgx7pv2OTGbnNudp4wXiPyulLrwvMe2KaUuEpGtSqmLnRqh5rHEOgTQy0qpyEpfrGmaQ+icrVWXztmaZh97vuWki8gzIhJpLOOATOOUQVmnCrRaSkQCReRmEfEV61A8kwF7j8BomuYYOmdrdtE5W9Oqx54jz42w/kH1Mh76EWtn9RyglVJqj1Mj1DyGiARhPYXUCeuVxV8AI5VSuaYGpmm1iM7Zmr10zta06qm0eNY0TdM0TdM0zcqeoeo6YJ3ZJsr29Uqp650XlqZpmlYdOmdrmqY5lz3dNn4DXgY2Y3MFrlJqs3ND+7tGjRqpqKgoV29W0zStxjZv3nxUKdW48lfWjM7ZmqZpjlFe3rZnhsEzSqlFToipyqKioti0aZPZYWiaplWZiFQ2O5mj6JytaZrmAOXlbXtG2/hMRIaKSHMRCStdHByfpmma5hg6Z2uapjmRPUeeBxk/x9o8poA2Nd24iLwG3AocVkpdUNP2NE3TNJ2zNU3TnKnSI89KqdZlLDVOwoY3gH85qC2PkLg8kTYd2+Bj8aFNxzYkLk80OyRN07yIJ+fs5YmJRLdujcXHh+jWrVmeqPOjpmnV56ycUu6RZxG5Xin1rTGf+98opWo8TaVS6nsRiappO54icXkiQ8cMJWxgGNHto8nbncfQMUMBGNB/gMnRaZrmyTw9Zy9PTGT8kKFMC61Pt/Yd+LUgn/FDrPmx/wCdHzVNqxpn5pSKjjxfY/zsW8Zya422WktNnDKRsIFhhHQOQXyFkM4hhA0MI3ZqrNmhaS6kzz5oTuLROTs+JoZpofXpERSMnwg9goKZFlqf+JgYs0PTPJQ+k1G7OTOnlFs8K6UmGz8fKmN5uMZbtpOIPCYim0Rk05EjR1y1WYfacWwH49ePZ/+e/QS3Dz7nueD2waTsSTEnMM3lSs8+qH6K6CXRqH6KoWOGVquA1kW4ZsvTc3ZyairdAoPOeaxbYBDJqa4apERzB44qeEuPOo4rLmZL+w6MKy5m/JChuoCuJYpzckhOSXFaTqmo28boilZUSj1X463bQSm1BFgC0L17d4+ZDrFElfBD2g+8uf1Nfsn4hWC/YMIiw8jbnUdI55Czr8vbnUdYqzCKiovws/iZGLHmCrFTY8+efQCsPwfC0AlD+S3iN+rVqUc9f+tSv0596+06594P9A1k+TvLdRcg7RyenrM7Rkbya0E+PYL+OsDwa0E+bfz8OPDkMJqOG4t/ZKTjA9bcRnmn2RVwf79+FJ84QUluLsUnTlCck0PJiRMU556gODeHktwTFJ/ItT6fe4IpH37AtLCws5+nHkHBTMN6NFJ3A/JuJ9f/QPqzz9KmTp0yc0pHB+SRikbbqGv87AhcBnxq3O8L/FLjLXupU8Wn+PzPz3kr6S325uylaVBTnr70ae7qcBef8qm1wBloPeKctzuPzNczaXhnQ/r/X39m9JpB+wbtzd4FzYlS9qQQ3T76nMeC2weTciCFpKwkck/lkns6l2JVXE4L4OvjS/LEZJoObPq3Ijx2aqwunmsvj87ZMfHx1sIJ69GhXwvymZSdzbMPPkj+zz/z5619CXvwQRoNeQJL3bqVtqd5HtvT7PBXwTvlkUfoFhdf4boSFISlbl0s9eriU68+f+bl0S2ixTmv6RYYRPLuXc4KXzNZSX4+mXPmkP3Ou/i3a8uzU6cyafqM83JKDgmLFtZ4W+UWz0qpqQAi8j3QTSl1wrg/Bfiixlu2tvUOcC3QSEQOApOVUq86om1XO154nHeT3+Xdne+SVZhF57DOJFydQJ+oPvj5WI8olxY1sVNjSdqTRFS7KF6Z/wrNezVn6s9TuffzexnZbSQPdH4Ai4/FzN3RnCD9ZDqB4YFlnn1o3b41n/f7HAClFHlFeeSeziXnVA65p3PPvX0ql9GHRpfZBShpT5JL90lzH56es0uPBsbHxJC8excdIyNJWLSI/gMGUHT4MEcWLCDr9dfJ+fhjGo8cSejddyEWnSe9SXJqKt3adzjnsW6BQew9fZomY57Gp249a3Fctx6W+vWw1K2LTz3rT/E798xtx59+dNpRR8395P+6hUPjx1N04ABhDz1E41EjaVunDv4tWpyXUxY65MyDPdNzJwMXKaVOGffrANuUUh1rvPUq6t69u3K32ar25ezj7aS3+fTPTzlVfIreLXozKHoQlzW7DBGxu51jBceY9vM0vj3wLZc2vZTpvaYTERLhxMg1V0o/mc5DXz3En9/+ScZHGTQa1Ojs2Yest7JYOHdhlY4Yt+nYBtVPnVOEn9xxElkp7E3e64xd0GpIRDYrpbq7YDtem7ML/thO5syZFGzeTJ1OnWg6YQLBPS53WPuauTqHh/OMr+85Be+G/DxmWywk7dtXpbbO6QJiHHWMzchgysiRPDJnjqND10xScvo0R1/8L8defRW/5s0JT5hJ0GWXOaz98vK2PTMMvgX8IiJTRGQqsAHrWJ+1xt8uzEpMZFPGJoZ/O5zbP76dT/Z8wq1tbuWT2z/hpX+8xOXNL69S4QzQMLAhz1/3PHE949iZtZO7Pr2LlbtXUtmXG839lRbOuady+WjKR7w872VkpZD0WBKyUqpcOAPETY4j660sTu44iTqjOLnjJGmvpdFrYC8n7YXmQbw2Zwde0IXIZW8T8fx8SnJz2T9oEAeHD+f0gQNmh6bVUP6WLTzi60dsZiYb8vMoUooN+XlMys4hJr7iLhtl6T9gAAmLFjLbYuGS3buY7ePDmMsu46r/W0XuqlVO2APN1QqTk0m5598ce+UVQu+6k9affOLQwrkilR55BhCRbsDVWGepWq+U2uLswMpixpFn27GZS48Upr+WTuO7GhN1TRT3dryX+zrdR6PARg7b5qGTh4j5MYaNGRu5ruV1TL5yMg0DGzqsfc110k+m8/BXD5NzKofFNy7mwsYXOqztxOWJxE6NJWVPClHtoujWvxvJUcmM7T6WgV0GOmw7mmO46sizsS2vz9klhYVkvfEmR5csgaIiwgYNpOETT2AJCal8Zc2tnNq9m5QHHsQ3NJSf7uzHzIQEklNT6RgZSUx8vMMu8CvJy2P/449TsGUrEXPnUO+mmxzSruZaqriYY6+9xpEXXsRSvz7N46ZR97rrnLKt8vK2vcVzV6A3fyXi3xwfYuXMKJ7LOz2evyKflOQUAn0DnbLdElXC20lv88KvLxDiH8LkKydzfavrnbItzTmcWTiXpbikmLHfj2VN6hpm9JpB37Z9nbo9rWpcXDzXmpxdlHmYI/Pnk/Pxx1gaNaLJqJF8UVDA9EmTnFKAaY51+mAaqf37AxC5fDn+LZzbXbEkL4/9jz1OwdatRMybS71/1apJjj3e6dRUDo2fQMGWLdT95z9pNmUyvg0aOG171e62ISIjgUSgEdAEWCYiwx0fontK2ZNS5oVZR1KOOK1wBvARHwZ1GcR7t75H06CmjFw7kpgfYjh5+qTTtqk5TmnhnH0q2yWFM4DFx0LC1Qn0aNaD2B9j+f7g907fpuZ+alvO9mvahPCEmUStWIF/q1YsHTGS8Y8+qsf39QBnjh3jwCOPUFJYSMulrzi9cAbwCQ6m5eLFBF58MWlPjyH3yy+dvk2t5pRSHH/3Xfbe0Y9Tf/5J+Jw5RDw/36mFc0Xs6fP8CNBDKTVZKTUJuAIY7Nyw3Edku0jydued81je7jyi2kW5ZPvtGrQj8eZEBl84mM/2fsZdn97FxoyNeoIMN2ZbOC+5cYlLCudS/hZ/Fly/gI5hHXl63dNsOWzK2XrNXLUyZwdeeAGRictYWlJMXNOmeqZCN1d88iQHBj9GUWYmLV9+mYAOHSpfyUEsIUYB3bWrUUB/5bJta5U7f6Kctxcu5MDgx8iYMpWgSy6hzaefUL/vrVW+tsyR7CmeBbAddLbYeKxW6HZ/N9JeSzvnwqyst7KImxznshj8LH6M6DaCt256C18fX+6ceieDnxrskFnqNMfKyMswrXAuFewXzKIbFtEsuBlPfvMku4/vdnkMmqlqbc4WEXYfPqxnKnRzJadOcfDJYRTu2kWLBc8T1O0Sl8dgCQmm5ZIlRgH9NLlfrXZ5DNrflTUz5MRRo3h/9WqaToql5atL8WvWzOww7SqeXwc2GFduTwH+B3jkWMxVtXL3Sna13sV9I++r8egIjtC1cVdW9F1B3pd5NH3IOkGG+AohnUMIGxhG7NRYl8ek/SUjL4OHvnzI1MK5VFhAGItvXEygJZAn1jxB2sk002LRXK7W5mz4a6ZCW3p8X/ehios5NGYs+Rs2ED5zBiHXXGNaLGcL6IsushbQq3UBbTbbiXJKzxzFNWvGq6qEsP79TT3abKvS4tmY0vUhIMtYHlJKPe/swMy2M2sn0zdMp0ezHrw24TX2Ju+lpLiEvcl7TZ3BLcgviOz92WX2w07Zk2JOUJpbFc6lwkPCWXzjYgqLC3l8zeMcKzhmdkiaC9TWnF0qJj6eSdk55wx3FpuRwTPDhpkdWq2nlCJjyhROrFlD04kTqN/X/IuaLSHBtHxlCYEXXEDa6KfJXbPG7JBqteTU1DLPHO1Kc68DQPZcMHgFsFsp9YJS6gXgTxHp4fzQzJN7OpfR60ZT378+s3rPwtenolnMXS+qXZSp/bC1c7lj4VyqXYN2vPSPl8jMy2TI10P0Bae1QG3M2bb+Nr6vCKOiWtPr27WU5OdX3oDmNEfmP0/2ig9o+MTjhA10n+E0LSEhtFz6irWAfmq0LqBN5ClnjuzptrEIsP2Pe9J4zCsppYj5IYb0k+nMvXauW46vXNYEGZlvZLq0H7Zm5c6Fc6mLm1zMc9c+x+7juxm5diSnik+ZHZLmXLUqZ5el/4ABJO3bR3FJCUmpqTzx5huc3ruXjPjpZodWax174w2OLVlC6L330njkSLPD+ZuzBXSXLrqANtG4J54gNiPDIRPlOJNdFwwqm8GglVIlgHsdinWg17e/ztoDaxndfTSXNHH9RQz2GNB/AAvnLjzbDzvn3Rwa9mtIh+tdd7Wydm7h7Krh6Krr6hZXM63nNH7J+IUJ6ydQXFJc+Uqap6pVOdsewVdeSaMhT5Dz0UfkfPKJ2eHUOtkff8zhhFnWcXknxbpNv9XzlRbQAV2iSXtqNCe+/trskGqV4pN59Pr2W55q247ZPj7WM0cWCwmLFrrdOO32FM97RWSEiPgZy0hgr7MDM8PGjI0s+HUBfSL78EDnB8wOp0ID+g842w/74O6DdO3TlfHrx5NdmG12aLWC7agai29czEWNLzI7pEr1bduXcZeNY03qGuI3xOup371XrcnZVdFo6FCCuncnfeo0Tu2t9W+Hy5xYu5b0Z2MIuvIKwufMRiwWs0OqkKVuXVotXUpAl2gOjnqKE998Y3ZItUZmfDxFBw7y+JtvkJSSYj1ztG+f2xXOYF/x/ARwFZAGHAR6AI85MyhDB3dyAAAgAElEQVQzHMk/wtjvxtKqbium9Zzmtt+MyxLkF8Ts3rPJKsxi8k+TdVHkBLbjakd2iOSfMf/keOFxjymcSz0Y/SCDLxzMB7s+4L9b/2t2OJpz1IqcXVXi60v4vLn41KlD2qinKCksNDskr5e/eTNpo54ioHNnWrz4X3z8/c0OyS5nC+hoawH92sSJ54w7rCfccbyczz4n5+OPafTEEwRffrnZ4VTKntE2Diul7lNKNVFKNVVK9VdKHXZFcK5SVFLEmO/GkH8mn/nXzifYL7jyldxMdMNoRnUbxbcHvuX95PfNDserJC5PZOiYoWfH1fa504ediTu5Lfc2jyqcSw2/ZDh3tb+LJduWkLhD/xPwNrUhZ1eXX9OmhM+exaldu8icmWB2OF6tMDmZA08MwS88nJZLFmMJ8az/q9YC+hW+Cgpk8ty5esZKJzp94AAZU6YQ2K0bjYYOMTscu0hlRylFpDHW2amisOk3p5R62KmRlaF79+5q06ZNDm937sa5vJn0JjOvnsmtbW51ePuuUqJKGPr1UDZlbuKdW96hfYP2ZofkFdp0bIPqpwjpHHL2sZM7TiIrhb3Jnnn6t7ikmDHfjeHr/V97/OfeU4jIZqVUdxdsx+tzdk0dnjePY68sJeK5edS7+Wazw/E6pw8cIKV/f8THQtQ7y/ELDzc7pGqLjopiXEkJPYL+Kv435Ocx22Ihad8+EyPzDqqoiJQBD3B63z7afLwSvwjnT9FeFeXlbXu6bXwC1Ae+Br6wWbzC16lf82bSm9zb8V6PLyB8xIf4XvEE+wUz7vtxFJ7RpyUdIWVPiteNq23xsZDQO4HLm11O7A+xTPrvJD3du/fw6pztCI1HjCDwkktIj53EaT3zoEPYTql8QadOfHYonVavLvXowhkgef9+PWOlEx154UUKt22jedw0tyucK2JP8RyklHpGKfW+UurD0sXpkblASk4KMT/GcGGjCxl32Tizw3GIRoGNmNFrBnuy9zB301yzw/EK3jqudh1LHRZctwD/bf7MmjZLT/fuPbw2ZzuK+PkRMW8u+PqS9tRoSk6fNjskj3b+lMoTGzZkQWYmH27YYHZoNeYp4w57oryff+bY0qWE3nM39f71L7PDqRJ7iufPRcTrzmvlF+Xz1Lqn8PPxY9418/C3eMaFDPboGdGTQdGDeC/5Pb7Zr68UrqkhTw8h7bW0c8bVznoryyvG1Q7xDyHj0wzCHw7X0717D6/M2Y7mFx5O+MwZFCYlcXj2HLPD8WhlTak8LawB8TExZodWY2XNWDnp6FG3G3fY05zJyuLQuGfwb92aphMmmB1OldlTPI/EmowLRSRXRE6ISK6zA3MmpRTx/4vnz+w/Sbg6geYhzc0OyeFGdhtJ57DOTP5pMhl5GWaH47EKzxTyQ5MfaHdfO9RHiqTHkpCVwsK5C02dpt2RDvx5wOu6pdRyXpeznaXu9dcTNmgQx5ctI3f1arPD8VjlTansDV0bzp+xcmZuLiMahNHvggvMDs1jKaVInzCR4uxsIp6bh09QUOUruRl7Rtuoq5TyUUoFKKXqGffruSI4Z1mxawWf7f2MIV2H0DOip9nhOIWfxY/ZvWdzuvi0nhSjBp7b/Bx7c/aydPxSUnalUFJcwt7kvV5TOIP3dkuprbwxZztTk6dHE3DhhaQ/G8PpgwfNDscjdWje3Ku7NpwzY+X+/dzesQNp48ZRkpdX+cra3xxflsjJ776jydixBHTqZHY41VJp8SxWD4hIrHG/pYi4/yB85dh+dDsJvyTQM6Inj3d93OxwnCqqfhQTe0xkU+YmXv3jVbPD8TjrD67nnZ3v8EDnB7gq/Cqzw3GasqZ7P/LGEa/ollIbeVvOdjbx9ydi/nMApI1+GqX7P1dJcU4Og+vXJzYz0+2nVHYES716RMyaRdH+A2TMnGl2OB6ncOdODs+eTcg119DgQfeejK4i9nTbWAhcCfQ37p8EXnJaRE6UXZjN6HWjaRTYiIReCfiIPbvv2W5vezs3tb6JhVsXsvXwVrPD8RjHCo4R+2Ms7Ru0Z9Slo8wOx6nOn+79yPIjtLynJTfdeZPZoWnV4zU521X8W7SgeXw8hdu2cXj+82aH4zGUUqRPmcJNPhbipk4927XBXadUdpSgyy6j4eDB5HzwIblr1pgdjscoyc8nbfTTWEJDaT5zhkdNRnc+e6rHHkqpJ4FCAKXUccDjrq4rUSVM+GECRwqO8Ny1zxEaEGp2SC4hIsReEUuz4GY88/0z5J7WXR8ro5Riyk9TOHH6BAlXJ1DHUsfskJzOdrr3zVs3E3R5EJN/1LNVeiivyNmuVu+ffWjQvz9Zr7/OibVrzQ7HI+R8/AknVn1J4+HD+c/48X91bXDTKZUdqfGwJwmIjiYjdhJFmXoOIntkzkzg9L59hM+ehW9YmNnh1Ig9xXORiFgABWcH4C9xalROsGTbEn5I+4FnLnuGCxrVro7+df3rMqv3LDLzM4n7OU4XRJX4YPcHrDu4jlGXjqJDgw5mh+NyHcM68tSlT7Hu4Do9W6Vn8oqcbYYmz4yjTufOpI+fQFF6utnhuLXT+/eTGRdnPQr76CNmh+Ny4u9P+Nw5lBQWkj5xIqpE/4lVJPfLr8hesYKGjz5K8JVXmh1OjdlTPL8ArASaiMh04AdghiM2LiL/EpFkEdkjIuMd0aatxOWJZyd+GHXLKKL+jOLfHf/t6M14hK6NuzLskmF8mfIlH+/52Oxw3FZKTgpzNs7hiuZXMKCzdx85qUj/zv3pGd6TOZvmsDfbM2dRrMWclrPB+XnbTD516tBi/nOooiLSnh6DKioyOyS3pIqKSBs7Fnx9CZ89C7FYzA7JFHXatKHp+GfI+/FHji9bZnY4bqsoLY30SZMIuOgiGo8YbnY4DmHPaBuJwDhgJpAO3KGUWlHTDRtHRl4CbgKigftFJLqm7ZZKXJ7I0DFDz0780PyB5nz3yncsf2e5ozbhcR7q8hCXN7ucmb/MZF+Onlb0fEUlRYxfPx5/iz/Te02vFX3iy1M6W2WQbxDjvh/H6WJ9EZWncFbOBufnbXfgHxVFs2nTKPj1V468+F+zw3FLRxctovC3bTSfOgW/5t431GtVhN57LyHXXcfhufMoTN5ldjhuR505Q9rYcVBcTMTcOYifn9khOYRd1YFSaqdS6iWl1H+VUjsctO3LgT1Kqb1KqdPAu8DtDmqb2KmxhA0MO2fih4aDGtbqiR8sPhZm9JpBHUsdXRCVYdHWRWw/tp0pV06hSVATs8MxXaPARkzrOY3k48m88OsLZoejVYGTcjY4OW+7i/q33kLoPfdwbMkSTq7/wexw3Er+5s0cfXkx9e+4g3o36YuKRYTm8XH41K3LobFjKTl1yuyQ3MrRhYso+PVXmk2Zgn+rVmaH4zBmHlqLAA7Y3D9oPHYOEXlMRDaJyKYjR47Y3XjKnhQ98UMZmgY3Ja5nHDuzdvL8r/qq8lKbMzez9Pel9GvXjxsibzA7HLdxbctrubfjvbyZ9CY/HfrJ7HBMZdsNrE3HNrV1+vJK83Z1c7a7aTpxAnXat2fxww/TOTISi48P0a1bszyxVv7eASg+cYJDY8fhFxFBUy+YPdBRfBs2JHzGdE7t2sURPVrLWfkbN3L05Zepf/vt1O97q9nhOJTbn5dWSi1RSnVXSnVv3Lix3evpiR/Kd23La7m/0/28nfQ26w+uNzsc0504fYKJ6yfSom4Lxl/uVV04HeLp7k/Tpn4bYn6I4XjhcbPDMcX53cBUP8XQMUNrawFdoermbHfjExjI+uuv4/nUFJ5Rii3tOzCuuJjxQ4bW2gI6Y+o0ijIziZgzG0tIcOUr1CIh11xDg/73k/XGG+T9VLsPNAAUZ2eTNnYcfi1b0DTW+874m1k8pwEtbe63MB5ziLImfsh6K0tP/GB4uvvTtG/QnpgfYzhacNTscEw1Y8MMMvMzmXn1TIL8PG+aUGcL9A1kVu9ZZJ/KZvJPtXP4urK6gYUNDKuN3cCcmrfdzawXXiCuWTN6BAXjJ0KPoGCmhdYnvhYedc357DNyP/+cRk8OJfDii80Oxy01GTsW/zZtODR+AmeO184DDWCM/x0by5ljx4iY95xXftEqt3gWkRMiklvGckJEHDFY8EagvYi0FhF/4D7gUwe0C/x94gdZKSycu9CrplWuiTqWOszpPYf8onz+PfXftO7Yulaejl61bxWf7/2cxy96nK6Nu5odjtvqFNaJkd1GsvbAWj7Y/YHZ4bicJ3QDc0HOBifnbXeTnJpKt8Bzv1B3CwwiOTXVpIjMcfrgQTKmTiPw0ktp9Lh3z8xbEz6BgUTMncOZ48fJmDyl1h1oWJ6YSHTr1vhaLFy3ZAnrr7iCwAu6mB2WU5RbPCul6iql6pWx1FVK1avphpVSZ4BhwFfADuB9pdT2mrZry3bih73Je3XhfJ62oW25MvNKfnjtB+hHrTsdnX4ynbif4+jauCuDLxpsdjhu78HoB7mi+RXM/mU2e3Nq1/B1ke0i3b4bmLNztrENp+dtd9IxMpJfC/LPeezXgnw6RkaaFJHrqTNnODR2HADhs2rvsHT2CoiOpsnIEZxYvZqcj1aaHY7LLE9MZPyQoYwrLmZL+w4827Qp8e+967VdnOzutiEiTUSkVeniiI0rpf5PKdVBKdVWKTXdEW1qVfP5ks+JeDii1p2OLi4pZuIPEylWxcy8eia+Pr5mh+T2fMSH6b2mE+AbwPjvx1NUXDvGwFVK0e3+bqS9luZR3cCckbOhduXtmPh4JmXnsCE/jyKl2JCfR2xmJhMnTjQ7NJc5ungxBVu20GzyZPxb/O2afq0MYQ89RNDll5M5fTqn9+83OxyXiI+JYVpo/fO6OIV6bRenSotnEblNRHYD+4DvgBRglZPj0lzEE05HO8ObSW+yKXMTE3pMoGXdlpWvoAHQJKgJU66awo6sHby49UWzw3GJFbtWsKv1Lu4beZ9HdAPTOdtx+g8YQMKihcy2WLhk9y5mFRczskkTev9vA+rMGbPDc7r8LVs4unAR9W7r63WjJTiTWCyEz0oAi4VD456pFZ+V2tbFyZ4jz3HAFcAupVRr4B/A/5waleYytXFUkqRjSby45UVujLyR29t63RC1TvePVv/g7g5388Yfb7AhfYPZ4TjV5szNzNwwk6sjrub1Ca97SjcwnbMdqP+AASTt20dxSQk70tJ45LnnyPvxRw7Pn292aE5VfPKkdVi6Zs1o5oWjJTibX/PmNJsymYKtWzn68mKzw3EqpRTtQkNrVRcne4rnIqXUMcBHRHyUUmuB7k6OS3ORskYlSX89nSmTppgdmlMUnClg/PrxhNUJY9IVkxARs0PySGO7jyWyXiQTf5hIdmG22eE4RUZeBqPXjaZF3RYk9E7A4uMxfT11znaiBvfcQ+j995H16mvkfPa52eE4TWZcPEWHDhE+ZzaWunXNDscj1b/lFurd1pejixZRsHWr2eE4zfG3lzE4IJDYo0fP6eI0KTuHmPh4s8NzCnuK52wRCQG+BxJFZAGQV8k6moc4f1SSghUFNL6zMXtb7/XKK4XnbZrHvpx9xPeKJzQg1OxwPFaQXxCzes8iqzCLqT9P9brPSuGZQkauHcmp4lMsuG4B9fwdcr2dq+ic7WTNJkwgsPulpMfEULDd+66XzPniC3I++YRGQ4YQ1K2b2eF4tGaxsfg1bUra2HEUn/S+P8MTa9eSmZDAvXf2I2Hx4rNdnGZbLCQsWkj/AW57hq5G7CmebwcKgKeAL4E/gb7ODEpzLdtRSTL+zGDs4LF8uPtDXt/+utmhOdT3B7/nveT3GBg9kCvDrzQ7HI8X3TCa4ZcM5+v9X7Nyj/dcVa6UYurPU0k6lsTMXjNpE9rG7JCqSudsJxN/f1osWIAlLIyDw4Zz5tgxs0NymKK0NDKmTCWwa1caDXnC7HA8nqVuXcLnzKYoLY3MGTPMDsehCpOSSHt6DAGdOxMxezYDHnzwbBenpH37vLZwBjuKZ6VUnlKqWCl1Rin1plLqBeOUoOalhl0yjH9F/Yv5m+ezJnWN2eHUiO2Uyv+68l8EbAtgZLeRZoflNf7T5T/0aNaDhF8SSMlJMTsch3g76W0+3/s5T178JNe1us7scKpM52zX8G3YkBYvvkhxVhZpI0ehijx/9BlVXEzaM89ASQnhc+cgvnoUIkcIuvRSGj42mOVvvEGn5s29Yqr3osxMDgwZiqVePVosWohPUO2aYMye0TbuFJHdIpLjhAH3NTfkIz7E94qna+OuTFg/gd+P/G52SNVy/pTKTQc0Zefynax4b4XZoXmN0s+Kn48f49d7/vB1Px/6mXmb5/GPVv/gsYseMzucatE523UCL+hC8/g48jdtInNmgtnh1NixV5ZSsGkzzSbF4t9Sj0LkSGvCwlhw7Bjj/fw8fqr3krw8DgwZQsmJE7Rc/DJ+TZqYHZLLSWV9FUVkD9BXKbXDNSGVr3v37mrTpk1mh1FrZBVm0f+L/hSeKWT5LcsJDwk3O6QqadOxDaqfIqRzyNnHTu44iawU9ibXrkk+nG1N6hpGrxvNoxc+6rFH9g+eOMh9X9xH48DGLLt5GcF+jp1SVkQ2K6WcfuGeztmulzlrNlmvv07z+DhC777b7HCqpWDbNlL6D6Benz6Ez5urL6Z2sOjWrRlXXEyPoL/yyob8PGZbLCTt22diZFWjios5OHwEJ9eto+WihYRcc43ZITlVeXnbnj7Pme6QhDXXCwsIY+E/FnK6+DRPfvMkJ06fMDukKqmtY1ib4cbIG7mz/Z3MfWUuEe0iPG6q9/yifEauHUmJKmHBdQscXji7mM7ZLtbk6dEEX3UV6VOnkb9li9nh2K10OmWLjw9dr7ySVSXFNJsyWRfOTuAt4yAfnj2Hk99+S9NnJ3p94VwRe4rnTSLynojcb5wOvFNE7nR6ZJpbaBPahvnXzSclJ4Ux342hqMQzTssrpWgY2bDWjWFtpqg/ozjy4RH87/b3qKnelVLE/hjLnuw9zOk9h1b1HDYZn1l0znYx8fUl4rl5+DVrxsERIyjKzDQ7pEqdP53yxIYNeX7/ft777DOzQ/NK5U713spz8s3xd94h6803afDgg4R58cWA9rCneK4H5AN9sF6x3RfQUw3VIj2a92DSlZP46dBPzNww0+2HJStRJczaOIuAfwaQ8XqGR02p7Mni4uNo/nBzj5vq/dU/XmV16mpGdRtFz4ieZofjCDpnm8ASGkqLl/5LSV4+B4ePoOTUKbNDqlCZ0ymHhXntdMpmK3Oq94wMHmvcxCNGazm5fj0Z8dMJufZamo5/xuxwTFfppbRKqYdcEYjm3vq170dqbiqv/vEqkfUiGdRlkNkhlamouIhnf3iWVSmrGPbQMCJ6RTBp2iSS9iQR1S7KbadU9gYpe1KIbh99zmPB7YNJ2pNkUkSV+/7g97zw6wvc1Pom/tPlP2aH4xA6Z5snoEMHwmclkDZ8BBlTptJ8xnS37QKRnJpKt/YdznmsW2AQybt3mRSRdysdti0+Jobk3bvoGBnJ1LFjueqL/yOlf39aLV3qthdpFibvIm3UU9Tp0IGIeXMRi8dMGOU09oy20UJEVorIYWP5UERauCI4zb2M6DaCPpF9mLdpHt/s/8bscP4mryiPod8MZVXKKkZ1G8W4y8bxwIAHPGVKZY9X3lTvzds0NymiiqXkpDD++/F0CuvE1Kumum2RU1U6Z5ur3o030mjoUHJWruT4MvfssnTm6FHaBofUqumU3YHtVO9J+/bx8PTptHr9NUqyc0i5vz+FSe53oOHMkSMcGPIEPsHBtFy0EJ9gj74exGHs6bbxOvApEG4snxmPabWMj/gwvdd0Lmx0IeO/H8/2o+4zs9axgmM8/NXDbMzYSFzPOB658BGvKYY8RZlTvb+Wju8Nvryz8x236u5z8vRJRqwdga+PL89f9zyBvoFmh+RIOmebrNGwJwm5/noyExLI+98Gs8M5R8Hvv7Pv7nt4rEEDJh07VmumU3ZXQZdcQuTyRMTPj9QHB5L3v/+ZHdJZJQUFHBj6JMXHs2mxaCF+zZqZHZLbsKd4bqyUet0YcP+MUuoNoLGT49LcVIBvAAuuX0DDwIYM+3YY6SfTzQ6JgycOMnDVQPZm72XBdQu4o90dZodUK50/1busFF6a+xJ97+7LjA0ziP0xllPF5vcDLVElTFg/gf25+5l37TyPG4LRDjpnm0x8fAifPQv/qCjSRo3i9ME0s0MCIHvlx6QOeADx8WHYl6tIWLKk1kyn7M7qtG1L1DvL8QtvzoHBj5G7apXZIaFKSjj0zHgK//iDiHlzCezSxeyQ3Io9xfMxEXlARCzG8gDg/r3bNadpFNiIl/7xEoVnCnny2yc5efqkabHszNrJg6seJPtUNq/0eYVrWtbeoXPcge1U73uT9/LIwEd44foXGNJ1CJ/8+QmDVg0iIy/D1BgX/baIdQfXMe6ycVzW7DJTY3ESnbPdgCUkhBb/fdE6Lu6wYZTk51e+kpOooiIy4qeTPmECgd26EfXhBwRER/+tG4EunM3j16wZkcuWEXDRRaSNfposk7v8HJk/nxOrV9PkmXHUvf56U2NxR/YUzw8D/wYygHTgbkBfkFLLtQ1ty7xr57E3ey9jvh/DmZIzLo9hY8ZGHvryISxi4a2b3uLiJhe7PAatcj7iw9CLh7LgugWk5KZw7+f3sjFjoymxfJ36NS//9jJ3tLuD+zvdb0oMLqBztpuo07o1Ec/N41RyMukxMaZ0XTpz7Bj7H36E48uWETZoEK2WvoJvgwYuj0OrnKV+fVq9utTa5Sc+nsPznzflM3N8xQqOvbKU0PvvI2yQew4OYLZKi2elVKpS6jalVGOlVBOl1B1Kqf2uCE5zb1eFX0XMFTH8mPYjCb8kuPSPfE3qGh5f8zhNg5qy7OZltA1t67Jta9VzfavrWX7Lcur51+Ox1Y+RuCPRJZ+ZxOWJtOnYBh+LD3179qXuH3WJuSLGa/vE65ztXkKuvprGo5/inXffo3OTJlh8fIhu3dol0zIX/LGdfXffQ8G2bYTPnkXTCeMR30oH2dJM5BMQQIsFzxN6zz0cW7zY+qXrjOsOTuX9/DMZU6cRfPXVNHv2Wa/NkzVVbvEsIuOMny+KyAvnL64LUXNnd3e4m4e6PMR7ye8xdO7Qs0WKM2eXez/5fZ5e9zRdGnbhzZvepFmwvojBU7Sp34bltyynV4teJPySQMyPMRSeKXTa9hKXJzJ0zFBUP0X0kmiaPdCMP5b9wQfvfeC0bZpF52z39VVQEAuOZzE+MJAt7TswrriY8UOGOrWAzvnkE1IHDACByOWJ1L/tNqdtS3Ms8fWl2bSp1lFbPvzIOm54QYFTtmU7y2Tnli15ecAA6xmT+c/pL1oVkPKO/IhIX6XUZyJS5jF7pdSbTo2sDN27d1ebNm1y9Wa1SpSoEm6JvYVvlnxDxMMRBLcPJm93HllvZTl0XGWlFIt+W8Si3xbRu0Vv5l4z19tGSag1SlQJi7ctZuHWhUQ3jOb5a5+neYhjh7RTStGyQ0v87vIjpHPI2cdP7jiJrBT2Ju916PYqIyKblVLdndi+ztluKrp1a8YVF9Mj6K9hvjbk5zHbYiFp3z6HbkudOcPhOXPIevMtgi6/nIjn5+MbFubQbWiuc/ydd8iYFkdg1660fHkRltBQh7VdOsvktND6dAsM4teCfGIzMpnx3DweHDbMYdvxZOXl7XKLZ5sV71FKrajsMVfQidh9te7YGvrhtCKluKSYGRtm8P6u97m97e1Mvmoyfj5+NW5XM9e6A+uYsH4C/hZ/5l4zt8YX8JWoEn478hurU1bz9f6v+frfX9PllS6I71+nHtUZRdJjSZQUl9Q0/CpxdvFssx2ds92MxceHLe074GdzCrxIKS7Zlcz+0U9Tt08fQq7uhU9QUI22cyYri7SnRpO/YQMNBj5I07FjET+dJz1d7lerOTRmDH6tWtFq6Sv4NXfMgQZXfqnzVOXlbXsuGJxg52NaLZa6J5Xg9ucOnh7cPpiUPSms2LWC7Ue3V3uYslPFpxjz3Rje3/U+j1zwCHE943Th7CWubXkty29ZTv069Rm8ejDLkpZVuR90cUkxGzM2MmPDDG5YcQMDVw3k/eT36RTWiaatm5Y5cUtUuygH7oXb0TnbzXSMjCxzQpJ2oaHk/fgjaSNHsuuqnhwcPpyczz6j+MSJKm+jMCmJfXffTcGWLTRPmEmziRN14ewl6v2zDy1fXcqZzExS7u/Pqd27q9WOKinh1N695Hz2OZkJs0hOSaFb4Llf2LoFBpGcmuqIsL1auR1aROQm4GYg4rz+cvUA1w+toLm10tnlbI885+3OIyA8gGk/TwPAV3xpG9qWTmGd6NywM9ENo+nYoCNBfuf+8SYuTyR2aiwpe1Jo1a4Vre5oxfHo44y7bBwPRj/o0v3SnK91/dYsv3k5z/7wLLM2ziLpWBKTrpxEgG9AueucKTnDxoyNrEldwzf7vyGrMIsASwC9InpxY+SN9G7RmxD/EK6Iu4KhY4bCQP7Wncjb6JztvmLi462nx+Hs6fFJ2TkkLFpI+3vvJX/TZk6sXs2JNWs4seZrxM+PoKuupF6fPoRcf32lo2PkfPY56bGxWEJDiUxMJPDCC1yzY5rLBF9+OZHL3ubA4MdIGfAAP91+G3OWLiU5NZWOkZHExMefM9SgKi7mdEoKhdu3U7h9OwXbt3MqacfZIROlTh3ahlhnmbQ98qxnmbRPRX2euwIXA9OASTZPnQDWKqWOV3ujIvcAU4DOwOVKKbvO6+lTgO6r9MKssIFh5xYpcxZyzW3XsCNrBzuO7SApK4kdx3aQVZgFgCBE1Y+iU1gnosOiSf0+lXlx82g4qOHZdtJeS2PkxJHMGTXH5L3UnKlElbBk2xIWbl1Ip7BO9D7Sm+dmPUfKnhSi2kUxJXYKba5rw5rUNXy7/1uyT2UT6BvINS2u4RsTnj4AABCYSURBVMbIG+kV0etvX8Tg3C9jUe2iiJscZ8o07S7o8+y0nG20X+W8rXP2X5YnJhIfE1NusQPWI4MFW3+zFtKrV1N06BBYLARdfhn1+vSh7g034Nu48TlttWvQgMF1Arjn+uuJWPA8vg0bmrSHmiucPpjGSzffzHM7dxDXrNlfX8aOH2fSo49ya6PG1oJ5505UaaEcEEBAp04EREcT0KULARd0oU6bNrzz/vt/6/Nc+qVOj/ltVZM+z75KKYcetRCRzkAJsBgYo4tn72BvkaKU4kjBkXOK6R1ZO8jIy2D3s7tp/kBzt7jASzPHdwe+45GERziw4gDhD4ef/RJ16LVDNLmrCRFXR3Bty2u5MfJGeob3rPAItTtxYZ9nh+dso90q522ds6tPKUXh9qSzhfTplBQQYXWDBszbupVpjRr+dZHXkaMkLH6ZAQMHmh225gLRkZGMU+pvfZWnZ2byWXQXAjp3/qtQ7hJNnTZtyh05w54vdbVZlYtnEXlfKfVvEfkdsH2RAEopdZEDglqHLp41w/HC4zQMbkj0kmi3uMBLM0+rDq2w3Gn525eoUx+cIiU5BX+Lv4nRVY8Ljjw7PWcb21mHLp5dSinF6T17yF29mp7PPMPEsDB9kVctVu4FqLt3caaoCLFYTIzOu5SXtysaxG+k8fNW54RkHxF5DHgMoFWrVmaGojlZg4AG5fad9vILvLTzHPzzINHto895LLh9MKn7Uj2ycHYRnbO9lIhQp317Grdvz5/Dh9MtosU5z3cLDCJ59y6TotNcrfQC1LL6KuvC2TXKHW1DKZVu3DwKHFBKpQJ1gK7AocoaFpGvReSPMpbbqxKgUmqJUqq7Uqp748aNq7Kq5oHiJseR9VYWJ3ecRJ1RnNxxkqy3soibHGd2aJoLlX6JsqW/RFWspjkbHJO3dc52rvJG7tAXedUeMfHxTMrOYUN+HkVKsSE/j0nZOcTEx5sdWq1hz/Qx3wNXi0gDYDWwEbgXqLBTjFLqhpqHp9U2pX2kY6fGkrQniah2UQ6daEXzDHGT42rNKBlOUK2cDTpve4KKRu7QaofSPsnxMTEk795Fx8hIfZGfi9lTPItSKl9EHgEWKqVmi8hWZwem1V4D+g/QxXItp79E1YjO2V5MF04aWD8H+nduHruKZxG5EutRi0eMx2rUqUZE+gEvAo2BL0Rkq1LqnzVpU9M076K/RFWbw3O20ajO225CF06aZi57iudRWGenWqmU2i4ibYC1NdmoUmolsLImbWiapmllcnjOBp23NU3TSlVaPCulvgO+E5EQEQlRSu0FRjg/NE3TNK2qdM7WNE1zLnsmSbkQeAsIwzpe6BFgoFJqu/PD+1ssR4DqTLreCOsV6DXlqHYc2ZY3x+TN++bItnRMntFOpFLK6cNP6JztlHYc2ZaOybXtOLItd2vHkW3pmMpWdt5WSlW4AD8B19ncvxb4qbL13GkBNrlTOzomvW86Js+NyZH75oxF52zn/K50TDomd2tHx2ROW0qp8sd5thGslDrbX04ptQ4ILv/lmqZpmol0ztY0TXMiey4Y3CsiscDbxv0HgL3OC0nTNE2rAZ2zNU3TnMieI88PYx2a6CPgQ6z9Rh52ZlBOsMTN2nFkW94ckzfvmyPb0jF5ZjvOonO249txZFs6Jte248i23K0dR7alY6qCci8YFJEA4AmgHfA78JpSqsiRG9c0TdMcQ+dsTdM016ioeH4PKALWAzcBKUqpUS6MTdM0TbOTztmapmmuUVHx/LtS6kLjti/wi1KqmyuD0zRN0+yjc7amaZprVNTn+ezpPqXUGRfE4nAi8pqIHBaRP2rYTksRWSsiSSKyXURGVrOdABH5RUR+M9qZWsO4LCKyRUQ+r2E7KSLyu4hsFZFNNWwrVEQ+EJGdIrLDmCa4qm10NGIpXXJFpFpH0ETkKeO9/kNE3jFObVeLiIw02tle1XjK+iyKSJiIrBGR3cbPBtVs5x4jphIR6V6DeOYYv7dtIrJSREJr0Fac0c5WEVktIuHVacfmuadFRIlIoxrENEVE0mw+VzdXNyYRGW68V9tFZLY9MbmAztl/teOQnG205XZ525tzttGeQ/K2N+XsCtqqct52VM4ury2b5+zO2x6XsysYE68YyDWWE8AZm9u5jhwvz1kL0BvoBvxRw3aaA92M23WBXUB0NdoRIMS47QdsAK6oQVyjgeXA5zXcvxSgkYPe8zeBR43b/kBoDduzABlYByqv6roRwD4g0Lj/PvCfasZxAfAHEIR1lJqvgXY1+SwCs4Hxxu3xwKxqttMZ6AisA7rXIJ4+gK9xe5Y98VTQVj2b2yOAl6vTjvF4S+ArrJNt2PU5LSemKcCYKv7ey2rnOuP3X8e436Qmn3FHLTpnn9OOQ3K2sb7b5W1vzdnG+g7J296Wsytoq8p521E5u7y2jMerlLc9LWeXe+RZKWVRStUzlrpKKV+b2/XKW8+dKKW+B7Ic0E66UupX4/YJYAfWP/CqtqOUUieNu37GUvEUj+UQkRbALcDS6qzvDCJSH+sH91UApdRppVR2DZv9B/CnUqo6s5SBNWkGivU0dhBwqJrtdAY2KKXylfWo3nfAnfauXM5n8Xas/7gwft5RnXaUUjuUUsn2xlJBO6vVX0cs/we0qEFbuTZ3g7Hjc17B3+t8YJw9bdjRVpWU084QIEEpdcp4zeGabscRdM4+px2H5Gxjfa/N226as8ExedurcnYFbVU5bzsqZ5fXlqFKedvTcrY9Q9VpNkQkCrgE69GH6qxvEZGtwGFgjVKqWu0Az2P9YJZUc31bClgtIptF5LEatNMa61TArxunJZeKSE0nZ7gPeKc6Kyql0oC5wH4gHchRSq2uZhx/AFeLSEMRCQJuxvrNuiaaKqXSjdsZQNMatudIDwOratKAiEwXkQPAAGBSNdu4HUhTSv1Wk1hsDDNOTb5mzynXcnTA+lnYICLfichlDopNc4Ka5myjDXfL216Zs8Ghebu25WyoYd52RM422nFk3nbLnK2L5yoQkRCs46aOOu9bmt2UUsVKqYuxfju8XEQuqEYctwKHlVKbqxNDGXop64VFNwFPikjvarbji/V0ySKl1CVAHtZTW9UiIv7AbcCKaq7fAOuRgtZAOBAsIg9Upy2l1A6sp8RWA18CW7GeJncIZT2XVK2jWY4mIs9iPeWfWJN2lFLPKqVaGu0Mq0YcQcBEapDEz7MIaAtcjPWf8rxqtuMLhAFXAGOB90VEHBKh5lCOyNnglnnbK3O20YZD8nZtytngmLxd05xtxOHIvO22OVsXz3YSET+sSThRKfVRTdszTo2tBf5VjdV7AreJSArwLnC9iCyrQSxpxs/DwErg8mo2dRA4aHNU5gOsibm6bgJ+VUplVnP9G4B9Sqkjyjre7UfAVdUNRin1qlLqUqVUb+A41n6UNZEpIs0BjJ+mn/4Xkf8AtwIDjH8OjpAI3FWN9dpi/Qf6m/FZbwH8KiLNqhOEUirTKIJKgFeo2ef8I+N0/i9YjyLadSGj5jqOztngPnnbi3M2/9/evYXYVd1xHP/+1KppEm0TL9iqDGq13kKKebAobTC1LRETopFovcTLiz4kighahDZtabF98CEV+qQEYkSI2mlVJIIxGk3rrZmc3LQVEQyaSFq8BDVU++/D+k+6M5w57tn7DB3j7wOLs846Z//32jN7/mfP3nudRR/z9pchZ8O45O2mORv6mLcncs72wXMN+R/KvcD2iLi7RZyjh0fCSpoEXAi8OtY4EfHTiDg+IgYol8jWRkSjM6qSJkuaOlynDD5oNNI9InYCb0k6LZvmANuaxEpX0OLyH+Wy37mSvpq/wzmUex8bkXRMPp5IuXfugRZ9A/gzsDjri4E/tYzXiqQfUy4pz4uIj1rG+lbl6Xya7eebI+KYiBjIfX0HZRDYzoZ9Oq7ydAEN93NgkDIABUmnUgZZ7W4Yy8ZBv3J2xppQefsAz9nQx7x9oOds6F/e7kfOhv7m7Qmds6PliMOJXCh/xO9QvsJpB3BDwzjnUy7PdCiXfoaAuQ3izAA2ZpwtwM/6sI2zaTdq+yRgU5atwJ0t+zMTeDm3cRD4esM4k4F/Ake27M8vKElgC7CSHG3bMNZ6ygfLJmBO230RmA48BfyDMhJ4WsM4C7K+F9gFrGkY53Xgrco+Xne0dbdYD+fPvAM8CnyzSZwRr79J/W/b6NanlZSZ9zqUD8HjGsY5FLg/t+9vwAVt9lGX+vvAGOL0JWdnrAmVtw/0nJ2x+pK3D6Sc3SPWmPN2v3L2aLFGvF4rb3/Rcvaok6SYmZmZmdn+fNuGmZmZmVlNPng2MzMzM6vJB89mZmZmZjX54NnMzMzMrCYfPJuZmZmZ1eSDZ2tN0meShiRtkbQ6Zxgay/LfkPRQ1mdKmlt5bZ6kxjNejVjPhn7EMTP7InPONmvHX1VnrUnaExFTsr4KeCUaTkyQMyXNiohGU4OamVlvztlm7fjMs/XbeuAUSdMkDUrqSPqrpBkAkr6fZzyGJG2UNFXSQJ4BORT4JbAoX18k6VpJ9+SyA5LWZsynctYoJK2QtFzSBklvSFrYrWOS9uTjbEnrJD0k6VVJq7rNcy9pqaRtub4Hs22ypPskvZj9n5/tkyQ9KGm7pD9KekHSrOp6s75Q0oqsHy3pYUkvZTkv25flOtbl9iytLH9N9meTpJW94piZ1eCc7ZxtY9V2lhUXF2BPPh5Cma70JuD3wM+z/QJgKOuPAudlfUouMwBsybZrgXsqsfc9z2UXZ/16YDDrK4DVlH8GzwBe/5x+zgbeB47PZf4CnN/l/W+TM1sBX8vH3wBXDbcBf6fMrHUrcF+2zwA+pZyN2bferC8EVmT9geH1AidSphIGWAZsAA4DjqLM2vUV4Mxc31H5vmm94ri4uLh0K87Zztku7cohmLU3SdJQ1tcD9wIvAJcCRMRaSdMlHQE8D9ytcqnwkYjY0eUEwmi+C1yS9ZXA7yqvDUbEf4Btko6tEevFiNgBkH0fAJ4b8Z4OsErSIGXaWoAfAvMk3ZbPD6ckv+8By3N7O5I6NfrwA+CMyvYfIWlK1h+PiL3AXknvAsdSPtBWR8TuXM+/esWJiH1nT8zMKpyznbOtBR88Wz98HBEzqw2jJdeIuEvS48Bc4HlJPwI+6UMf9lZXP8b3f0b3v4WLKAn2YuBOSWdn7Esj4rXqGz/nw6Q6sODwSv0g4NyI2G/7M1ad/vWMY2Y2Cuds52xrwfc823hZD1wJ5X41YHdEfCDp5IjYHBG/BV4Cvj1iuQ+BqaPE3ABcnvUrcx3jQtJBwAkR8TRwO3Ak5ZLlGmDJ8P12kr6TizwL/CTbzqJcBhy2S9LpGXNBpf1JYEllnft9mHWxFrhM0vR8/7SGcczMRnLO/h/nbOvJB882XpYB5+SlsLuAxdl+i8pAkw7wb+CJEcs9TbmcNSRp0YjXlgDX5bJXAzePW+/hYOB+SZuBjcDyiHgP+BXlXraOpK35HOAPwBRJ2ykDaF6pxLoDeIzyQfJOpX0pMCsHk2wDbuzVoYjYCvwaeEbSJmB4dPyY4piZdbEM5+xhztnWk7+qzmwcSFoH3BYRL/+/+2JmZr05Z9tY+MyzmZmZmVlNPvNsZmZmZlaTzzybmZmZmdXkg2czMzMzs5p88GxmZmZmVpMPns3MzMzMavLBs5mZmZlZTf8F+rDDnuJtvZsAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "fig, ax = plt.subplots(2, 2, figsize=(12, 4))\n",
        "ax = [a for a_list in ax for a in a_list]\n",
        "for i in range(len(ax)):\n",
        "    ax[i].plot(np.arange(1, 17), pe[i, :16], color=\"C%i\" % i, marker=\"o\", markersize=6, markeredgecolor=\"black\")\n",
        "    ax[i].set_title(\"Encoding in hidden dimension %i\" % (i + 1))\n",
        "    ax[i].set_xlabel(\"Position in sequence\", fontsize=10)\n",
        "    ax[i].set_ylabel(\"Positional encoding\", fontsize=10)\n",
        "    ax[i].set_xticks(np.arange(1, 17))\n",
        "    ax[i].tick_params(axis=\"both\", which=\"major\", labelsize=10)\n",
        "    ax[i].tick_params(axis=\"both\", which=\"minor\", labelsize=8)\n",
        "    ax[i].set_ylim(-1.2, 1.2)\n",
        "fig.subplots_adjust(hspace=0.8)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 83,
      "id": "795c4b37",
      "metadata": {
        "id": "795c4b37"
      },
      "outputs": [],
      "source": [
        "class MyGPT(nn.Module):\n",
        "    '''\n",
        "    GPT-like language model.\n",
        "\n",
        "    Arguments:\n",
        "        vocab_size: Size of vocabulary.\n",
        "        embed_dim: Dimension of the input token embedding. \n",
        "        num_layers: Number of Transformer layers in the encoder. \n",
        "        hidden_dim: Hidden dimension of the transformer layers.  \n",
        "        ffn_hidden_dim: Hidden dimension of the Feed-forward layers. \n",
        "        num_heads: Number of heads. \n",
        "        attn_dropout: Dropout for MHSA layers. \n",
        "        ffn_dropout: Dropout for feed-forward layers.\n",
        "    '''\n",
        "    def __init__(self, vocab_size, embed_dim, num_layers, num_heads, attn_dropout, ffn_dropout):\n",
        "        super(MyGPT, self).__init__()\n",
        "        self.embedding_layer = nn.Embedding(vocab_size, embed_dim)\n",
        "        self.pos_encoder = PositionalEncoding(embed_dim)\n",
        "        self.transformer_encoder = TransformerEncoder(num_layers, embed_dim, num_heads, attn_dropout, ffn_dropout)\n",
        "        self.decoder = nn.Linear(embed_dim, vocab_size)\n",
        "        self.embed_layer_norm = nn.LayerNorm(embed_dim)\n",
        "        self.init_weights()\n",
        "\n",
        "    def init_weights(self):\n",
        "        init_range = 0.5\n",
        "        self.decoder.weight.data.uniform_(-init_range, init_range)\n",
        "        self.decoder.bias.data.zero_()\n",
        "        \n",
        "    \n",
        "    def forward(self, seqs, attn_mask=None, key_padding_mask=None):\n",
        "        embedded_seq = self.embedding_layer(seqs)\n",
        "        embedded_seq = self.pos_encoder(embedded_seq)\n",
        "        embedded_seq = self.embed_layer_norm(embedded_seq)\n",
        "        out = self.transformer_encoder(x=embedded_seq, key_padding_mask=key_padding_mask, attention_mask=attn_mask)\n",
        "        results = self.decoder(out)\n",
        "        return results\n",
        "      \n",
        "    def get_attn_weights(self):\n",
        "      return self.transformer_encoder.get_attention_weights()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "vocabulary_size = 10\n",
        "seq_length = 8\n",
        "num_seqs = 5\n",
        "x = torch.randint(vocabulary_size, (num_seqs, seq_length))\n",
        "x.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KotAU-LIGrXG",
        "outputId": "19675a4a-014d-4bc9-abc6-babc437644c1"
      },
      "id": "KotAU-LIGrXG",
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([5, 8])"
            ]
          },
          "metadata": {},
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ykLMTlcGrKD",
        "outputId": "cc0d8b6d-5c5a-40e0-fe6b-45373b615607"
      },
      "id": "5ykLMTlcGrKD",
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[2, 8, 0, 0, 9, 9, 9, 2],\n",
              "        [8, 0, 1, 8, 2, 9, 6, 3],\n",
              "        [7, 7, 0, 5, 1, 7, 2, 7],\n",
              "        [2, 3, 0, 6, 5, 3, 0, 3],\n",
              "        [3, 4, 0, 9, 8, 6, 2, 6]])"
            ]
          },
          "metadata": {},
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model = MyGPT(vocab_size=vocabulary_size, embed_dim=64, num_layers=2, num_heads=2, attn_dropout=0.3, ffn_dropout=0.1)"
      ],
      "metadata": {
        "id": "ccHYtKXjGqcv"
      },
      "id": "ccHYtKXjGqcv",
      "execution_count": 86,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y = model(x)"
      ],
      "metadata": {
        "id": "yzV_PARoGzU_"
      },
      "id": "yzV_PARoGzU_",
      "execution_count": 87,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kMHwo7PDJGUM",
        "outputId": "eec2553d-552f-4570-aab7-38a677824f6d"
      },
      "id": "kMHwo7PDJGUM",
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 0.5538, -0.9655, -0.2114,  0.3255,  3.7826, -2.0932,  2.4401,\n",
              "           1.9437,  1.6627, -1.1368],\n",
              "         [-0.4130,  1.4705,  0.4886,  2.3956,  1.0408,  4.6443, -0.0127,\n",
              "          -2.8765,  1.8967, -4.8898],\n",
              "         [-2.8881,  3.3545, -0.1635,  1.0542, -0.2818,  3.4571, -3.4251,\n",
              "           0.3699, -0.5231,  0.3577],\n",
              "         [-0.3912,  4.5304, -1.0740,  1.1999,  0.3905,  3.5423, -3.7838,\n",
              "          -0.3086,  0.6252,  1.1353],\n",
              "         [-2.4198,  2.9980,  1.7149, -1.6793,  1.8955,  4.2764, -4.9772,\n",
              "           1.2452, -4.0753, -1.3832],\n",
              "         [-1.6980,  0.4211,  1.1076, -0.7423,  2.7173,  4.0816, -4.2425,\n",
              "           3.4285, -3.9942, -0.8829],\n",
              "         [-3.7988,  1.1590,  1.2091, -1.2984,  1.7106,  3.4663, -3.6836,\n",
              "           2.7338, -4.1580,  0.1947],\n",
              "         [-0.4154, -1.3298, -0.0483,  0.3889,  4.0009, -0.8270,  2.8111,\n",
              "           2.1943,  2.2104, -0.6873]],\n",
              "\n",
              "        [[ 2.2071,  2.0732,  1.6402,  3.3526,  0.6989,  6.5105, -1.9483,\n",
              "          -4.4777, -0.4824, -4.3081],\n",
              "         [ 0.2699,  5.2672, -0.4237,  2.2563, -1.2875,  2.8786, -2.9976,\n",
              "          -0.3958,  0.9130, -0.3525],\n",
              "         [ 0.5133,  1.2900,  1.1764,  2.1020, -1.2602,  1.6165, -1.6208,\n",
              "          -2.2411,  2.5724, -0.6136],\n",
              "         [ 1.5686,  1.2818,  0.6864,  1.3597,  0.9574,  6.2771, -1.6993,\n",
              "          -2.5309,  1.0494, -4.6579],\n",
              "         [ 0.2782, -0.2744,  1.1670,  1.9694,  1.5752, -1.2876,  0.8324,\n",
              "          -0.5817,  2.7685, -1.1574],\n",
              "         [-3.2350,  1.2320,  3.4578, -0.9414, -0.1969,  1.7508, -6.1217,\n",
              "           2.1849, -3.3339,  1.3729],\n",
              "         [ 2.3168,  1.7557,  1.1014,  1.9820,  3.6183,  4.2904, -2.6521,\n",
              "           0.5969,  1.1746, -1.8326],\n",
              "         [ 2.4633, -0.8878, -0.8451, -1.1348,  1.3006,  0.0643, -3.3349,\n",
              "           0.8159,  0.1909, -0.1427]],\n",
              "\n",
              "        [[-1.4681, -1.4137, -1.7140,  1.2715, -2.0618, -1.2251,  1.2698,\n",
              "          -0.7530,  0.7711, -1.6539],\n",
              "         [-1.6459, -3.2051,  0.3891,  2.5847, -2.5356, -1.0783,  1.8088,\n",
              "          -0.2915,  2.4510, -0.3466],\n",
              "         [-0.0890,  3.3685,  0.8654,  2.4700, -4.0259,  1.9069, -3.1216,\n",
              "          -0.6018,  1.3947, -0.2882],\n",
              "         [ 0.6054, -1.4236, -2.4944,  2.1496,  0.4910,  2.1576, -4.7364,\n",
              "           3.0269, -1.8304, -1.8934],\n",
              "         [ 2.1089, -0.0594,  1.8763,  1.2158, -3.8350,  1.9505,  0.9856,\n",
              "          -1.0320,  4.0375, -1.3598],\n",
              "         [ 0.2434, -3.6705, -0.0099,  2.5119, -1.7988, -0.6003,  2.8470,\n",
              "           0.5341,  1.3363,  0.4964],\n",
              "         [ 2.3827, -1.3027, -0.2046,  0.2425,  1.4993, -1.2156,  1.3204,\n",
              "           1.7010,  3.9839, -2.2493],\n",
              "         [ 0.1210, -3.2145,  0.1939,  1.6334, -2.2636, -1.6308,  2.2783,\n",
              "           0.9681,  2.4870,  0.0176]],\n",
              "\n",
              "        [[ 2.0830, -1.5694,  0.9942,  0.9664,  2.8151,  1.3679,  1.5733,\n",
              "           0.4098,  4.3465, -4.5278],\n",
              "         [ 1.7100, -2.4208, -1.4045, -0.7590,  1.6494,  0.8879, -1.6423,\n",
              "           1.7318,  0.4329,  1.3643],\n",
              "         [ 0.5281,  4.2195, -1.5908,  1.8243, -2.2861,  3.3216, -3.7733,\n",
              "          -0.0743, -0.2276, -1.0618],\n",
              "         [ 3.3256,  1.4683, -0.7473,  2.2509,  1.5349,  0.8551,  0.2787,\n",
              "           0.7615,  4.4540, -3.0795],\n",
              "         [-0.0536, -1.5782, -1.9093,  2.8829,  1.1607,  2.8469, -6.0691,\n",
              "           1.1881, -1.7913, -3.3965],\n",
              "         [ 2.6587, -2.7874, -0.5143, -0.6277,  1.3039,  1.4380, -3.6707,\n",
              "           1.5632, -0.4785, -0.4136],\n",
              "         [ 0.6088,  4.2381, -0.1763,  1.8450, -0.6855,  3.8234, -3.7610,\n",
              "          -0.3107,  1.0479, -0.3213],\n",
              "         [ 2.0366, -2.6264, -0.2661, -0.2616,  1.1054,  0.2439, -1.6354,\n",
              "           0.4881, -0.1295, -1.5062]],\n",
              "\n",
              "        [[ 1.7014, -1.8605, -2.8326,  0.1682,  1.7750,  0.7944, -2.5822,\n",
              "           0.8861,  1.2336, -0.6760],\n",
              "         [ 0.6216, -0.2074, -2.8646,  3.7708,  0.2593,  3.1525, -1.6527,\n",
              "           0.9344,  2.0516, -0.0475],\n",
              "         [ 0.8070,  5.1272, -2.1955,  1.6184, -1.2714,  0.2192, -4.9431,\n",
              "           0.3086, -0.4605, -1.7532],\n",
              "         [-0.4143,  2.4398,  0.5253, -0.1655,  0.6852,  4.1020, -5.1620,\n",
              "           1.8527, -1.6058, -2.9006],\n",
              "         [ 1.6146,  2.5278,  0.3296,  1.9101,  1.5464,  5.2364, -3.6615,\n",
              "          -3.1202,  3.1145, -6.0291],\n",
              "         [ 2.2226,  3.2341,  0.1009,  1.5053,  2.1826,  0.2627, -1.1570,\n",
              "          -0.0585,  4.2143, -4.0395],\n",
              "         [ 1.4240, -1.1439, -1.4154,  0.7406,  2.8403, -2.7695,  1.7500,\n",
              "           1.5278,  3.5059, -2.7375],\n",
              "         [ 3.8535,  3.4508, -0.7909,  1.9856,  1.6611, -0.5643, -1.1824,\n",
              "           0.2980,  4.1873, -3.4158]]], grad_fn=<ViewBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AO6Jn7GPJGIc",
        "outputId": "f53742d7-9196-40c6-d461-f37bce6e78e1"
      },
      "id": "AO6Jn7GPJGIc",
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([5, 8, 10])"
            ]
          },
          "metadata": {},
          "execution_count": 89
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "attn_weights = model.get_attn_weights()\n",
        "len(attn_weights)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OMEjNiTbfPLk",
        "outputId": "5a3a80c2-82d2-4a07-9a9d-e77465c4f800"
      },
      "id": "OMEjNiTbfPLk",
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "metadata": {},
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "attn_weights[1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BiB3JIdFfRI3",
        "outputId": "33dade9f-08c2-44f0-a0e4-1d7d7100aaed"
      },
      "id": "BiB3JIdFfRI3",
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[0.1372, 0.1781, 0.2368, 0.2057, 0.0000, 0.1790, 0.0697, 0.1454],\n",
              "         [0.2269, 0.0656, 0.1025, 0.0310, 0.1245, 0.1589, 0.0683, 0.2653],\n",
              "         [0.0000, 0.0852, 0.2156, 0.2704, 0.1579, 0.0879, 0.1205, 0.1530],\n",
              "         [0.0000, 0.0797, 0.3178, 0.2861, 0.1612, 0.1047, 0.1262, 0.1423],\n",
              "         [0.1785, 0.1505, 0.2558, 0.0000, 0.0000, 0.0569, 0.1406, 0.1966],\n",
              "         [0.2842, 0.0000, 0.1101, 0.1658, 0.0000, 0.0000, 0.0702, 0.2990],\n",
              "         [0.0960, 0.0000, 0.2226, 0.0000, 0.0340, 0.1563, 0.0558, 0.2279],\n",
              "         [0.0000, 0.0829, 0.0000, 0.0946, 0.1715, 0.0801, 0.1793, 0.0706]],\n",
              "\n",
              "        [[0.1075, 0.0394, 0.0467, 0.1653, 0.0646, 0.0805, 0.0618, 0.1547],\n",
              "         [0.1561, 0.4013, 0.0924, 0.0533, 0.0000, 0.0478, 0.0841, 0.1081],\n",
              "         [0.0681, 0.1508, 0.0716, 0.0524, 0.1282, 0.2920, 0.1886, 0.1863],\n",
              "         [0.2029, 0.1787, 0.2049, 0.1599, 0.1856, 0.0875, 0.1841, 0.0000],\n",
              "         [0.1666, 0.0997, 0.1611, 0.1481, 0.0000, 0.1931, 0.2731, 0.1449],\n",
              "         [0.1227, 0.1992, 0.0728, 0.1676, 0.1166, 0.1482, 0.1132, 0.0000],\n",
              "         [0.1458, 0.1172, 0.0686, 0.0705, 0.1469, 0.0765, 0.1581, 0.1426],\n",
              "         [0.0927, 0.0966, 0.1774, 0.1398, 0.1606, 0.0794, 0.0000, 0.0754]],\n",
              "\n",
              "        [[0.1667, 0.1940, 0.1510, 0.2547, 0.1463, 0.1397, 0.0606, 0.0463],\n",
              "         [0.1883, 0.1240, 0.0675, 0.1227, 0.1786, 0.0666, 0.1156, 0.0882],\n",
              "         [0.0000, 0.0000, 0.0813, 0.1159, 0.1460, 0.0488, 0.1065, 0.2049],\n",
              "         [0.0635, 0.1125, 0.2146, 0.2346, 0.1337, 0.1448, 0.1304, 0.0404],\n",
              "         [0.1581, 0.1698, 0.1946, 0.2340, 0.0000, 0.1815, 0.1809, 0.0835],\n",
              "         [0.1053, 0.2165, 0.0000, 0.0909, 0.1501, 0.1479, 0.0830, 0.0843],\n",
              "         [0.1676, 0.0000, 0.2330, 0.1898, 0.0814, 0.1715, 0.1651, 0.1733],\n",
              "         [0.2105, 0.2066, 0.1533, 0.1185, 0.1511, 0.0000, 0.1113, 0.1533]],\n",
              "\n",
              "        [[0.0000, 0.1357, 0.0513, 0.0826, 0.2493, 0.1789, 0.1078, 0.0852],\n",
              "         [0.1314, 0.0516, 0.0995, 0.2008, 0.2014, 0.1783, 0.1805, 0.1233],\n",
              "         [0.1607, 0.0924, 0.1172, 0.2330, 0.1517, 0.0866, 0.3432, 0.1061],\n",
              "         [0.1841, 0.0482, 0.1389, 0.0000, 0.1678, 0.0837, 0.2016, 0.1024],\n",
              "         [0.0957, 0.0948, 0.0362, 0.1412, 0.0000, 0.1077, 0.1329, 0.0000],\n",
              "         [0.0671, 0.0638, 0.0868, 0.0000, 0.1487, 0.0460, 0.0677, 0.1453],\n",
              "         [0.1356, 0.0526, 0.1334, 0.2354, 0.1456, 0.1013, 0.3523, 0.0000],\n",
              "         [0.0710, 0.1601, 0.0456, 0.2014, 0.1885, 0.2058, 0.0727, 0.1346]],\n",
              "\n",
              "        [[0.2002, 0.2128, 0.0514, 0.0418, 0.1238, 0.2147, 0.1024, 0.2739],\n",
              "         [0.1652, 0.2334, 0.0578, 0.0883, 0.0820, 0.1388, 0.1623, 0.1435],\n",
              "         [0.1183, 0.0625, 0.1721, 0.0702, 0.0894, 0.1749, 0.1460, 0.0361],\n",
              "         [0.1281, 0.1608, 0.1742, 0.1434, 0.1208, 0.1273, 0.1454, 0.1662],\n",
              "         [0.1837, 0.2986, 0.0000, 0.0910, 0.1786, 0.1737, 0.0684, 0.1906],\n",
              "         [0.1128, 0.0525, 0.0732, 0.0548, 0.1447, 0.2154, 0.0543, 0.1397],\n",
              "         [0.0390, 0.1914, 0.1637, 0.0615, 0.1651, 0.1191, 0.0360, 0.2666],\n",
              "         [0.1154, 0.1485, 0.2404, 0.1262, 0.1627, 0.1942, 0.0924, 0.1227]]],\n",
              "       grad_fn=<DivBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 91
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "biJEXmoRJx_i",
        "outputId": "553b2896-e1cd-4984-b378-ed49dbf6407d"
      },
      "id": "biJEXmoRJx_i",
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "MyGPT(\n",
              "  (embedding_layer): Embedding(10, 64)\n",
              "  (pos_encoder): PositionalEncoding(\n",
              "    (dropout): Dropout(p=0.1, inplace=False)\n",
              "  )\n",
              "  (transformer_encoder): TransformerEncoder(\n",
              "    (layers): ModuleList(\n",
              "      (0): TransformerLayer(\n",
              "        (attn_layer): MultiHeadSelfAttention(\n",
              "          (attention_layer): MultiheadAttention(\n",
              "            (out_proj): NonDynamicallyQuantizableLinear(in_features=64, out_features=64, bias=True)\n",
              "          )\n",
              "        )\n",
              "        (ffn_layer): FeedForward(\n",
              "          (activation): ReLU()\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "          (layer_1): Linear(in_features=64, out_features=64, bias=True)\n",
              "          (layer_2): Linear(in_features=64, out_features=64, bias=True)\n",
              "        )\n",
              "        (layer_norm): TransformerLayerNorm(\n",
              "          (layer_norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
              "        )\n",
              "      )\n",
              "      (1): TransformerLayer(\n",
              "        (attn_layer): MultiHeadSelfAttention(\n",
              "          (attention_layer): MultiheadAttention(\n",
              "            (out_proj): NonDynamicallyQuantizableLinear(in_features=64, out_features=64, bias=True)\n",
              "          )\n",
              "        )\n",
              "        (ffn_layer): FeedForward(\n",
              "          (activation): ReLU()\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "          (layer_1): Linear(in_features=64, out_features=64, bias=True)\n",
              "          (layer_2): Linear(in_features=64, out_features=64, bias=True)\n",
              "        )\n",
              "        (layer_norm): TransformerLayerNorm(\n",
              "          (layer_norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (decoder): Linear(in_features=64, out_features=10, bias=True)\n",
              "  (embed_layer_norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Auto-regressive Language Model"
      ],
      "metadata": {
        "id": "c7pUDNvNeulP"
      },
      "id": "c7pUDNvNeulP"
    },
    {
      "cell_type": "markdown",
      "source": [
        "For a autoregressive language model, the self-attention layers are only allowed to attend to earlier positions in the sequence, looking into the future is not allowed. We will require an attention mask which prevents peeking into future using upper-triangular matrix of ones. \n"
      ],
      "metadata": {
        "id": "5jIArPmmcXt_"
      },
      "id": "5jIArPmmcXt_"
    },
    {
      "cell_type": "code",
      "source": [
        "def generate_square_subsequent_mask(sz):\n",
        "    \"\"\"\n",
        "    Generates an upper-triangular matrix of ones, with zeros on diag.\n",
        "    Shape max_length * max_length\n",
        "    \"\"\"\n",
        "    return torch.triu(torch.ones(sz, sz, dtype=torch.bool), diagonal=1)"
      ],
      "metadata": {
        "id": "XkVScG_mOdUd"
      },
      "id": "XkVScG_mOdUd",
      "execution_count": 93,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "generate_square_subsequent_mask(5)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jBa35kgcOIBB",
        "outputId": "208de61a-78da-4cd8-a277-eaff88144129"
      },
      "id": "jBa35kgcOIBB",
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[False,  True,  True,  True,  True],\n",
              "        [False, False,  True,  True,  True],\n",
              "        [False, False, False,  True,  True],\n",
              "        [False, False, False, False,  True],\n",
              "        [False, False, False, False, False]])"
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To follow the requirements of the PyTorch `MultiHeadAttention` layer, the `True` value at `(i,j)`-th place indicates that the self-attention for `i`-th position can not look at `j`-th position. "
      ],
      "metadata": {
        "id": "S8ihlr9vfvbs"
      },
      "id": "S8ihlr9vfvbs"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Back to our GPT model, we would require an attention mask `attn_mask` "
      ],
      "metadata": {
        "id": "gds3YsC3gWNQ"
      },
      "id": "gds3YsC3gWNQ"
    },
    {
      "cell_type": "code",
      "source": [
        "attn_mask = generate_square_subsequent_mask(seq_length)\n",
        "attn_mask"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TwXpu7yDfWVx",
        "outputId": "c4e5659d-5fc9-496e-c3e9-45740cdbca9f"
      },
      "id": "TwXpu7yDfWVx",
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[False,  True,  True,  True,  True,  True,  True,  True],\n",
              "        [False, False,  True,  True,  True,  True,  True,  True],\n",
              "        [False, False, False,  True,  True,  True,  True,  True],\n",
              "        [False, False, False, False,  True,  True,  True,  True],\n",
              "        [False, False, False, False, False,  True,  True,  True],\n",
              "        [False, False, False, False, False, False,  True,  True],\n",
              "        [False, False, False, False, False, False, False,  True],\n",
              "        [False, False, False, False, False, False, False, False]])"
            ]
          },
          "metadata": {},
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y = model(x, attn_mask=attn_mask)"
      ],
      "metadata": {
        "id": "F_bXSH8pg583"
      },
      "id": "F_bXSH8pg583",
      "execution_count": 96,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S0wAF2OGg-Yd",
        "outputId": "89024d8e-506a-4a4a-92c0-84703455a566"
      },
      "id": "S0wAF2OGg-Yd",
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 1.7974e+00,  7.2112e-01, -7.8530e-02,  4.8879e-01,  3.5751e+00,\n",
              "          -2.4817e+00,  2.0186e+00,  1.4142e+00,  1.8569e+00, -1.3372e+00],\n",
              "         [ 3.0068e-01,  2.5702e+00,  6.7665e-01,  1.9813e+00,  1.6985e+00,\n",
              "           4.0547e+00, -4.8453e-01, -2.6835e+00,  1.9968e+00, -4.3766e+00],\n",
              "         [ 1.1401e+00,  5.4791e+00, -2.5561e+00, -1.1816e+00,  1.5839e+00,\n",
              "           1.5675e+00, -2.4823e+00, -6.4347e-02,  1.4122e+00,  1.0774e+00],\n",
              "         [-2.7872e-01,  4.5330e+00, -2.1399e+00,  1.6308e+00, -2.3378e+00,\n",
              "           2.9814e+00, -3.9714e+00,  1.6348e+00,  7.9497e-02,  1.0263e+00],\n",
              "         [-2.6267e+00,  3.2035e+00,  1.6754e+00, -1.9250e+00,  6.3431e-01,\n",
              "           3.4647e+00, -5.2544e+00,  2.2085e+00, -3.9343e+00, -9.9813e-01],\n",
              "         [-1.5112e+00,  2.5731e-01,  2.6547e+00, -3.5589e-01,  3.0056e-01,\n",
              "           5.2017e+00, -3.1953e+00,  2.0065e+00, -3.8659e+00, -1.0345e+00],\n",
              "         [-1.6251e+00,  2.6326e+00,  2.4651e+00, -1.4558e+00,  5.9543e-01,\n",
              "           3.0159e+00, -4.4720e+00,  1.9356e+00, -3.7138e+00, -1.5284e+00],\n",
              "         [ 9.9393e-01, -1.0201e+00, -8.7653e-01,  5.0938e-01,  3.4485e+00,\n",
              "          -1.7941e+00,  3.2256e+00,  2.5001e+00,  2.9418e+00, -1.7065e+00]],\n",
              "\n",
              "        [[ 1.2972e+00,  1.7809e+00,  2.2259e-01,  4.0713e+00, -1.3923e+00,\n",
              "           5.0530e+00, -5.9173e-01, -5.7630e-01,  2.5651e+00, -5.0136e+00],\n",
              "         [ 1.7957e+00,  4.7338e+00, -1.4630e+00,  2.7324e-01, -2.6355e+00,\n",
              "           1.8102e+00, -3.1467e+00,  9.2883e-01,  2.5225e+00, -1.0687e+00],\n",
              "         [ 1.1659e+00,  1.0025e+00,  2.5189e+00,  8.2368e-01,  3.2010e-01,\n",
              "           2.8865e+00,  1.1674e+00, -6.7550e-01,  4.1324e+00, -1.9171e-01],\n",
              "         [ 1.3453e+00,  3.2422e-01,  9.7951e-01,  2.0517e+00,  1.1150e-01,\n",
              "           7.2115e+00, -1.7067e+00, -3.0284e+00,  1.5352e+00, -4.2482e+00],\n",
              "         [ 2.5046e+00, -8.3965e-01,  7.3391e-01,  5.4281e-01,  3.0549e-01,\n",
              "          -2.7994e+00,  1.8647e+00,  1.1798e+00,  2.9781e+00, -2.4195e+00],\n",
              "         [-1.5474e+00,  2.2055e+00,  9.5698e-01, -1.9278e+00,  7.5805e-01,\n",
              "           2.7642e+00, -5.3858e+00,  1.6517e+00, -4.1315e+00, -8.7094e-01],\n",
              "         [ 2.2251e+00,  2.7917e+00, -8.2648e-01,  3.2241e-01,  1.1064e+00,\n",
              "           1.7454e+00, -6.1135e-01,  1.7613e+00,  2.8780e+00,  1.3759e+00],\n",
              "         [ 1.6068e+00, -1.3869e+00, -7.0506e-01,  3.5996e-01, -7.4280e-01,\n",
              "          -8.3293e-01, -1.9545e+00,  1.3852e+00, -1.0181e+00,  3.4577e-01]],\n",
              "\n",
              "        [[ 2.9072e-01, -2.6447e+00, -2.1548e+00, -6.7715e-01, -2.4994e+00,\n",
              "          -2.8646e+00,  3.5367e+00,  1.4947e+00,  2.7352e+00,  9.1368e-01],\n",
              "         [ 5.8099e-01, -3.8112e+00, -8.9899e-01,  1.1251e+00, -3.5072e+00,\n",
              "          -1.8817e+00,  2.3281e+00,  1.6787e+00,  2.6080e+00,  6.7121e-01],\n",
              "         [ 9.8971e-01,  3.0295e+00, -2.2957e+00,  1.0725e+00, -3.6283e+00,\n",
              "           2.4732e+00, -1.2905e+00,  5.2584e-01,  4.6304e-01,  4.6188e-01],\n",
              "         [ 1.2991e+00, -2.1469e+00, -3.5215e+00,  1.7984e+00, -2.1559e-01,\n",
              "           3.2255e+00, -3.7442e+00,  2.5582e+00, -1.8444e+00, -2.8118e+00],\n",
              "         [ 3.1826e+00,  1.5289e+00,  7.5584e-01, -1.8134e+00, -1.7065e+00,\n",
              "           2.7547e+00,  1.2757e+00, -5.7826e-01,  3.7539e+00, -6.9645e-01],\n",
              "         [ 5.1457e-01, -3.7497e+00, -1.0980e+00, -1.0582e-01, -1.2594e+00,\n",
              "          -2.0634e+00,  1.7537e+00,  8.0227e-01,  3.1019e+00, -8.2089e-01],\n",
              "         [ 2.5388e-03, -1.8444e+00,  9.7286e-01,  2.2546e+00,  2.0803e+00,\n",
              "           9.8846e-01,  3.6411e+00, -3.3194e-01,  2.9085e+00, -2.5998e+00],\n",
              "         [ 2.1220e+00, -2.3061e+00, -2.5418e-01,  1.7230e+00, -1.1573e+00,\n",
              "           2.5074e-01,  2.2669e+00, -4.8671e-01,  1.3498e+00, -5.3795e-01]],\n",
              "\n",
              "        [[ 1.0879e+00, -2.7159e-01, -1.2001e+00, -2.3982e-01,  2.3415e+00,\n",
              "          -1.3542e+00,  2.1319e+00, -1.5279e+00,  5.9089e-01, -6.9176e-01],\n",
              "         [ 9.0105e-01, -1.8679e+00, -9.7054e-01, -2.2430e+00,  2.8334e+00,\n",
              "           1.8911e+00, -6.2929e-01, -1.6273e+00,  5.1937e-01, -7.4379e-01],\n",
              "         [ 6.9679e-01,  4.4394e+00, -2.1656e+00,  4.8239e-01, -8.4581e-01,\n",
              "           2.1880e+00, -3.1517e+00, -4.3942e-01,  5.3157e-01, -1.0903e-01],\n",
              "         [ 2.0518e+00,  2.6689e+00, -3.7028e-01,  7.8054e-01,  1.5232e+00,\n",
              "           9.2498e-01, -1.1643e+00,  8.6966e-02,  3.8974e+00, -1.1714e+00],\n",
              "         [-5.2441e-01, -1.6114e+00, -4.3060e+00,  4.8746e-01,  3.2694e+00,\n",
              "           2.1724e+00, -3.7127e+00,  1.8971e+00, -4.7865e-01, -2.3590e+00],\n",
              "         [ 1.8061e+00, -2.7008e+00, -4.4506e-01,  3.2627e-01,  3.1759e-01,\n",
              "           1.4981e+00, -1.4191e+00,  5.4354e-01,  4.6201e-01, -1.4333e+00],\n",
              "         [ 1.6563e+00,  3.8308e+00, -1.3134e+00,  2.2575e+00, -1.7778e+00,\n",
              "           3.1901e+00, -3.0694e+00, -2.7204e-01,  6.8205e-01, -1.2625e+00],\n",
              "         [ 1.6860e+00, -1.2908e+00, -6.7974e-01, -5.6378e-01,  1.4824e+00,\n",
              "           1.2519e+00, -1.6680e+00,  1.1911e-01,  2.2795e+00, -2.0493e+00]],\n",
              "\n",
              "        [[ 2.2435e+00, -1.6707e+00, -1.1164e+00,  7.0716e-02,  4.8675e+00,\n",
              "           2.5648e+00, -3.7405e+00, -2.7535e+00, -2.2021e+00, -3.2192e+00],\n",
              "         [-1.1887e+00,  5.9713e-02, -1.8622e+00,  2.0445e+00,  1.3945e-01,\n",
              "           2.2778e+00, -1.4313e+00,  4.1265e-01,  1.4055e+00, -9.3456e-03],\n",
              "         [-1.3451e+00,  2.0949e+00, -2.6849e+00,  1.1009e+00, -7.9402e-02,\n",
              "           1.5857e+00, -2.2467e+00,  4.6415e-01,  8.0545e-02, -5.4830e-01],\n",
              "         [-6.8455e-01,  1.6310e+00,  2.8600e+00, -7.8365e-01,  1.9226e+00,\n",
              "           4.1351e+00, -4.2532e+00,  1.3880e+00, -2.2331e+00, -2.7638e-01],\n",
              "         [ 1.9721e+00,  1.0561e+00, -2.0927e-02,  3.5584e+00,  7.3948e-01,\n",
              "           6.0180e+00, -1.4645e+00, -2.6732e+00,  2.7344e+00, -5.8341e+00],\n",
              "         [ 3.9739e+00,  2.8463e+00, -5.5192e-01,  2.9045e-01,  1.8051e+00,\n",
              "           1.5901e+00, -1.2993e+00,  1.9012e-01,  3.4874e+00, -1.7532e+00],\n",
              "         [ 1.5217e+00, -2.2636e+00,  9.0962e-01,  1.4886e+00,  3.2783e+00,\n",
              "           3.5141e-01,  6.2888e-01,  9.8392e-01,  2.1973e+00, -2.7770e+00],\n",
              "         [ 3.0633e+00,  2.3021e+00, -1.8530e-01,  5.4145e-01,  3.2486e+00,\n",
              "           4.8144e-01, -1.3251e+00,  4.3953e-01,  4.3179e+00, -1.6688e+00]]],\n",
              "       grad_fn=<ViewBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This network produces a 10-dimensional output for each token, where 10 is the size of the vocabulary. It can be viewed as capturing the probabilities of the next token over the whole vocabulary. Such a network can be trained using well-crafted training data of sequences e.g, \n",
        "$$X = [A, B, C]$$\n",
        "$$Y = [B, C, D]$$"
      ],
      "metadata": {
        "id": "U_SiHtRXqKaF"
      },
      "id": "U_SiHtRXqKaF"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's look at the attention weights. "
      ],
      "metadata": {
        "id": "8DyUITOwqMsO"
      },
      "id": "8DyUITOwqMsO"
    },
    {
      "cell_type": "code",
      "source": [
        "attn_weights = model.get_attn_weights()\n",
        "len(attn_weights)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "26hSj7EAdhF4",
        "outputId": "df717519-7682-4ffe-deb1-1b43a2ecdef3"
      },
      "id": "26hSj7EAdhF4",
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {},
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "attn_weights[3]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MKzBOwYqdjVg",
        "outputId": "0bd38c02-3e15-4b52-d305-c0b5cfd57e60"
      },
      "id": "MKzBOwYqdjVg",
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[1.4286, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.7654, 0.3419, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.4977, 0.2518, 0.5794, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.1874, 0.4689, 0.2164, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.1868, 0.0512, 0.2236, 0.1321, 0.2359, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.0000, 0.1915, 0.1258, 0.1056, 0.2487, 0.0000, 0.0000],\n",
              "         [0.2112, 0.0305, 0.3023, 0.1016, 0.1891, 0.2077, 0.1193, 0.0000],\n",
              "         [0.1667, 0.1038, 0.2294, 0.0949, 0.1713, 0.0000, 0.0835, 0.0000]],\n",
              "\n",
              "        [[1.4286, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.2348, 0.6909, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.4058, 0.2164, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.1172, 0.0590, 0.1789, 0.1317, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.0000, 0.1490, 0.1841, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.2091, 0.1518, 0.2202, 0.1757, 0.0000, 0.1581, 0.0000, 0.0000],\n",
              "         [0.1000, 0.1088, 0.1581, 0.1256, 0.2577, 0.0515, 0.3604, 0.0000],\n",
              "         [0.0446, 0.1618, 0.1638, 0.1098, 0.1470, 0.1211, 0.1770, 0.0658]],\n",
              "\n",
              "        [[1.4286, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.7212, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.3666, 0.1183, 0.6122, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.2285, 0.2820, 0.4649, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.2136, 0.2191, 0.4069, 0.4127, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.1778, 0.0000, 0.1030, 0.2571, 0.1127, 0.2319, 0.0000, 0.0000],\n",
              "         [0.0867, 0.1748, 0.2851, 0.2362, 0.0842, 0.1998, 0.1126, 0.0000],\n",
              "         [0.1878, 0.0000, 0.1287, 0.0000, 0.0594, 0.1911, 0.1408, 0.1125]],\n",
              "\n",
              "        [[1.4286, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.6704, 0.7582, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.2985, 0.3320, 0.3473, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.4081, 0.2323, 0.1597, 0.4707, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.3839, 0.2944, 0.2262, 0.2676, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0670, 0.0462, 0.1775, 0.2018, 0.2766, 0.2349, 0.0000, 0.0000],\n",
              "         [0.1251, 0.0839, 0.2348, 0.0318, 0.1621, 0.1177, 0.3297, 0.0000],\n",
              "         [0.0000, 0.1731, 0.0820, 0.2748, 0.2419, 0.1068, 0.1561, 0.1305]],\n",
              "\n",
              "        [[1.4286, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.8339, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.2888, 0.3218, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0000, 0.4347, 0.1244, 0.3073, 0.0000, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0621, 0.4682, 0.0692, 0.1614, 0.2117, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0294, 0.3112, 0.0000, 0.0000, 0.0749, 0.0000, 0.0000, 0.0000],\n",
              "         [0.0567, 0.1599, 0.1897, 0.1508, 0.0882, 0.1103, 0.0566, 0.0000],\n",
              "         [0.0830, 0.0755, 0.0730, 0.1119, 0.1338, 0.2375, 0.0489, 0.1074]]],\n",
              "       grad_fn=<DivBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Worth noting that the attention weight matrices are lower triangular which highlights the fact that the attention mask prohibits peeking into the future. "
      ],
      "metadata": {
        "id": "yO8ZIgn6pmEe"
      },
      "id": "yO8ZIgn6pmEe"
    },
    {
      "cell_type": "markdown",
      "source": [
        "This has been the first notebook in my series on transformer models. Next, we will implement data processing and training pipelines. "
      ],
      "metadata": {
        "id": "CoupRwcsIncQ"
      },
      "id": "CoupRwcsIncQ"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.13"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}