16_nlp_with_rnns_and_attention.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 16 – Natural Language Processing with RNNs and Attention**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code in chapter 16._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/ageron/handson-ml2/blob/master/16_nlp_with_rnns_and_attention.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No GPU was detected. LSTMs and CNNs can be very slow without a GPU.\n"
     ]
    }
   ],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "    !pip install -q -U tensorflow-addons\n",
    "    IS_COLAB = True\n",
    "except Exception:\n",
    "    IS_COLAB = False\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "if not tf.config.list_physical_devices('GPU'):\n",
    "    print(\"No GPU was detected. LSTMs and CNNs can be very slow without a GPU.\")\n",
    "    if IS_COLAB:\n",
    "        print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"nlp\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Char-RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting a sequence into batches of shuffled windows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, let's split the sequence 0 to 14 into windows of length 5, each shifted by 2 (e.g.,`[0, 1, 2, 3, 4]`, `[2, 3, 4, 5, 6]`, etc.), then shuffle them, and split them into inputs (the first 4 steps) and targets (the last 4 steps) (e.g., `[2, 3, 4, 5, 6]` would be split into `[[2, 3, 4, 5], [3, 4, 5, 6]]`), then create batches of 3 such input/target pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________ Batch 0 \n",
      "X_batch\n",
      "[[6 7 8 9]\n",
      " [2 3 4 5]\n",
      " [4 5 6 7]]\n",
      "===== \n",
      "Y_batch\n",
      "[[ 7  8  9 10]\n",
      " [ 3  4  5  6]\n",
      " [ 5  6  7  8]]\n",
      "____________________ Batch 1 \n",
      "X_batch\n",
      "[[ 0  1  2  3]\n",
      " [ 8  9 10 11]\n",
      " [10 11 12 13]]\n",
      "===== \n",
      "Y_batch\n",
      "[[ 1  2  3  4]\n",
      " [ 9 10 11 12]\n",
      " [11 12 13 14]]\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "n_steps = 5\n",
    "dataset = tf.data.Dataset.from_tensor_slices(tf.range(15))\n",
    "dataset = dataset.window(n_steps, shift=2, drop_remainder=True)\n",
    "dataset = dataset.flat_map(lambda window: window.batch(n_steps))\n",
    "dataset = dataset.shuffle(10).map(lambda window: (window[:-1], window[1:]))\n",
    "dataset = dataset.batch(3).prefetch(1)\n",
    "for index, (X_batch, Y_batch) in enumerate(dataset):\n",
    "    print(\"_\" * 20, \"Batch\", index, \"\\nX_batch\")\n",
    "    print(X_batch.numpy())\n",
    "    print(\"=\" * 5, \"\\nY_batch\")\n",
    "    print(Y_batch.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Data and Preparing the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
      "1122304/1115394 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "shakespeare_url = \"https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\"\n",
    "filepath = keras.utils.get_file(\"shakespeare.txt\", shakespeare_url)\n",
    "with open(filepath) as f:\n",
    "    shakespeare_text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(shakespeare_text[:148])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\n !$&',-.3:;?abcdefghijklmnopqrstuvwxyz\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\".join(sorted(set(shakespeare_text.lower())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = keras.preprocessing.text.Tokenizer(char_level=True)\n",
    "tokenizer.fit_on_texts(shakespeare_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[20, 6, 9, 8, 3]]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.texts_to_sequences([\"First\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['f i r s t']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.sequences_to_texts([[20, 6, 9, 8, 3]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_id = len(tokenizer.word_index) # number of distinct characters\n",
    "dataset_size = tokenizer.document_count # total number of characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "[encoded] = np.array(tokenizer.texts_to_sequences([shakespeare_text])) - 1\n",
    "train_size = dataset_size * 90 // 100\n",
    "dataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_steps = 100\n",
    "window_length = n_steps + 1 # target = input shifted 1 character ahead\n",
    "dataset = dataset.repeat().window(window_length, shift=1, drop_remainder=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.flat_map(lambda window: window.batch(window_length))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "dataset = dataset.shuffle(10000).batch(batch_size)\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32, 100, 39) (32, 100)\n"
     ]
    }
   ],
   "source": [
    "for X_batch, Y_batch in dataset.take(1):\n",
    "    print(X_batch.shape, Y_batch.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating and Training the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 31370 steps\n",
      "Epoch 1/10\n",
      "31370/31370 [==============================] - 7150s 228ms/step - loss: 1.4671\n",
      "Epoch 2/10\n",
      "31370/31370 [==============================] - 7094s 226ms/step - loss: 1.3614\n",
      "Epoch 3/10\n",
      "31370/31370 [==============================] - 7063s 225ms/step - loss: 1.3404\n",
      "Epoch 4/10\n",
      "31370/31370 [==============================] - 7039s 224ms/step - loss: 1.3311\n",
      "Epoch 5/10\n",
      "31370/31370 [==============================] - 7056s 225ms/step - loss: 1.3256\n",
      "Epoch 6/10\n",
      "31370/31370 [==============================] - 7049s 225ms/step - loss: 1.3209\n",
      "Epoch 7/10\n",
      "31370/31370 [==============================] - 7068s 225ms/step - loss: 1.3166\n",
      "Epoch 8/10\n",
      "31370/31370 [==============================] - 7030s 224ms/step - loss: 1.3138\n",
      "Epoch 9/10\n",
      "31370/31370 [==============================] - 7061s 225ms/step - loss: 1.3120\n",
      "Epoch 10/10\n",
      "31370/31370 [==============================] - 7177s 229ms/step - loss: 1.3105\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id],\n",
    "                     dropout=0.2, recurrent_dropout=0.2),\n",
    "    keras.layers.GRU(128, return_sequences=True,\n",
    "                     dropout=0.2, recurrent_dropout=0.2),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(max_id,\n",
    "                                                    activation=\"softmax\"))\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")\n",
    "history = model.fit(dataset, steps_per_epoch=train_size // batch_size,\n",
    "                    epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Model to Generate Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(texts):\n",
    "    X = np.array(tokenizer.texts_to_sequences(texts)) - 1\n",
    "    return tf.one_hot(X, max_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'u'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new = preprocess([\"How are yo\"])\n",
    "Y_pred = model.predict_classes(X_new)\n",
    "tokenizer.sequences_to_texts(Y_pred + 1)[0][-1] # 1st sentence, last char"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 0, 2, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 2, 1, 0, 2, 1,\n",
       "        0, 1, 2, 1, 1, 1, 2, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 2]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "tf.random.categorical([[np.log(0.5), np.log(0.4), np.log(0.1)]], num_samples=40).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def next_char(text, temperature=1):\n",
    "    X_new = preprocess([text])\n",
    "    y_proba = model.predict(X_new)[0, -1:, :]\n",
    "    rescaled_logits = tf.math.log(y_proba) / temperature\n",
    "    char_id = tf.random.categorical(rescaled_logits, num_samples=1) + 1\n",
    "    return tokenizer.sequences_to_texts(char_id.numpy())[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'u'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "next_char(\"How are yo\", temperature=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def complete_text(text, n_chars=50, temperature=1):\n",
    "    for _ in range(n_chars):\n",
    "        text += next_char(text, temperature)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:5 out of the last 6 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:6 out of the last 7 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:7 out of the last 8 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:8 out of the last 9 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:9 out of the last 10 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 12 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44616830> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "the belly the charges of the other words\n",
      "and belly \n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "print(complete_text(\"t\", temperature=0.2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "thing! they know't.\n",
      "\n",
      "biondello:\n",
      "for you are the own\n"
     ]
    }
   ],
   "source": [
    "print(complete_text(\"t\", temperature=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "th no cyty\n",
      "use ffor was firive this toighingaber; b\n"
     ]
    }
   ],
   "source": [
    "print(complete_text(\"t\", temperature=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stateful RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])\n",
    "dataset = dataset.window(window_length, shift=n_steps, drop_remainder=True)\n",
    "dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "dataset = dataset.repeat().batch(1)\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))\n",
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "encoded_parts = np.array_split(encoded[:train_size], batch_size)\n",
    "datasets = []\n",
    "for encoded_part in encoded_parts:\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(encoded_part)\n",
    "    dataset = dataset.window(window_length, shift=n_steps, drop_remainder=True)\n",
    "    dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "    datasets.append(dataset)\n",
    "dataset = tf.data.Dataset.zip(tuple(datasets)).map(lambda *windows: tf.stack(windows))\n",
    "dataset = dataset.repeat().map(lambda windows: (windows[:, :-1], windows[:, 1:]))\n",
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, stateful=True,\n",
    "                     dropout=0.2, recurrent_dropout=0.2,\n",
    "                     batch_input_shape=[batch_size, None, max_id]),\n",
    "    keras.layers.GRU(128, return_sequences=True, stateful=True,\n",
    "                     dropout=0.2, recurrent_dropout=0.2),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(max_id,\n",
    "                                                    activation=\"softmax\"))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResetStatesCallback(keras.callbacks.Callback):\n",
    "    def on_epoch_begin(self, epoch, logs):\n",
    "        self.model.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 313 steps\n",
      "Epoch 1/50\n",
      "313/313 [==============================] - 62s 198ms/step - loss: 2.6189\n",
      "Epoch 2/50\n",
      "313/313 [==============================] - 58s 187ms/step - loss: 2.2091\n",
      "Epoch 3/50\n",
      "313/313 [==============================] - 56s 178ms/step - loss: 2.0775\n",
      "Epoch 4/50\n",
      "313/313 [==============================] - 56s 179ms/step - loss: 2.4689\n",
      "Epoch 5/50\n",
      "313/313 [==============================] - 56s 179ms/step - loss: 2.3274\n",
      "Epoch 6/50\n",
      "313/313 [==============================] - 57s 183ms/step - loss: 2.1412\n",
      "Epoch 7/50\n",
      "313/313 [==============================] - 57s 183ms/step - loss: 2.0748\n",
      "Epoch 8/50\n",
      "313/313 [==============================] - 56s 179ms/step - loss: 1.9850\n",
      "Epoch 9/50\n",
      "313/313 [==============================] - 56s 179ms/step - loss: 1.9465\n",
      "Epoch 10/50\n",
      "313/313 [==============================] - 56s 179ms/step - loss: 1.8995\n",
      "Epoch 11/50\n",
      "313/313 [==============================] - 57s 182ms/step - loss: 1.8576\n",
      "Epoch 12/50\n",
      "313/313 [==============================] - 56s 179ms/step - loss: 1.8510\n",
      "Epoch 13/50\n",
      "313/313 [==============================] - 57s 184ms/step - loss: 1.8038\n",
      "Epoch 14/50\n",
      "313/313 [==============================] - 56s 178ms/step - loss: 1.7867\n",
      "Epoch 15/50\n",
      "313/313 [==============================] - 56s 180ms/step - loss: 1.7635\n",
      "Epoch 16/50\n",
      "313/313 [==============================] - 56s 179ms/step - loss: 1.7270\n",
      "Epoch 17/50\n",
      "313/313 [==============================] - 58s 184ms/step - loss: 1.7097\n",
      "<<31 more lines>>\n",
      "313/313 [==============================] - 58s 185ms/step - loss: 1.5998\n",
      "Epoch 34/50\n",
      "313/313 [==============================] - 58s 184ms/step - loss: 1.5954\n",
      "Epoch 35/50\n",
      "313/313 [==============================] - 58s 185ms/step - loss: 1.5944\n",
      "Epoch 36/50\n",
      "313/313 [==============================] - 57s 183ms/step - loss: 1.5902\n",
      "Epoch 37/50\n",
      "313/313 [==============================] - 57s 183ms/step - loss: 1.5893\n",
      "Epoch 38/50\n",
      "313/313 [==============================] - 59s 187ms/step - loss: 1.5845\n",
      "Epoch 39/50\n",
      "313/313 [==============================] - 57s 183ms/step - loss: 1.5821\n",
      "Epoch 40/50\n",
      "313/313 [==============================] - 59s 187ms/step - loss: 1.5798\n",
      "Epoch 41/50\n",
      "313/313 [==============================] - 57s 181ms/step - loss: 1.5794\n",
      "Epoch 42/50\n",
      "313/313 [==============================] - 57s 182ms/step - loss: 1.5774\n",
      "Epoch 43/50\n",
      "313/313 [==============================] - 57s 182ms/step - loss: 1.5755\n",
      "Epoch 44/50\n",
      "313/313 [==============================] - 58s 186ms/step - loss: 1.5735\n",
      "Epoch 45/50\n",
      "313/313 [==============================] - 58s 186ms/step - loss: 1.5714\n",
      "Epoch 46/50\n",
      "313/313 [==============================] - 57s 181ms/step - loss: 1.5686\n",
      "Epoch 47/50\n",
      "313/313 [==============================] - 57s 181ms/step - loss: 1.5675\n",
      "Epoch 48/50\n",
      "313/313 [==============================] - 56s 180ms/step - loss: 1.5657\n",
      "Epoch 49/50\n",
      "313/313 [==============================] - 58s 185ms/step - loss: 1.5654\n",
      "Epoch 50/50\n",
      "313/313 [==============================] - 57s 182ms/step - loss: 1.5620\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f8d45d95d10>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")\n",
    "steps_per_epoch = train_size // batch_size // n_steps\n",
    "history = model.fit(dataset, steps_per_epoch=steps_per_epoch, epochs=50,\n",
    "                    callbacks=[ResetStatesCallback()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the model with different batch sizes, we need to create a stateless copy. We can get rid of dropout since it is only used during training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "stateless_model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id]),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(max_id,\n",
    "                                                    activation=\"softmax\"))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To set the weights, we first need to build the model (so the weights get created):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "stateless_model.build(tf.TensorShape([None, None, max_id]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "stateless_model.set_weights(model.get_weights())\n",
    "model = stateless_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:5 out of the last 5 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:6 out of the last 6 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:7 out of the last 7 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:8 out of the last 8 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:9 out of the last 9 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:10 out of the last 10 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "WARNING:tensorflow:11 out of the last 11 calls to <function _make_execution_function.<locals>.distributed_function at 0x7f8d44bc53b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "tor:\n",
      "in the negver up how it thou like him;\n",
      "when it\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "print(complete_text(\"t\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can load the IMDB dataset easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "(X_train, y_test), (X_valid, y_test) = keras.datasets.imdb.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
      "1646592/1641221 [==============================] - 0s 0us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<sos> this film was just brilliant casting location scenery story'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_index = keras.datasets.imdb.get_word_index()\n",
    "id_to_word = {id_ + 3: word for word, id_ in word_index.items()}\n",
    "for id_, token in enumerate((\"<pad>\", \"<sos>\", \"<unk>\")):\n",
    "    id_to_word[id_] = token\n",
    "\" \".join([id_to_word[id_] for id_ in X_train[0][:10]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDownloading and preparing dataset imdb_reviews/plain_text/1.0.0 (download: 80.23 MiB, generated: Unknown size, total: 80.23 MiB) to /home/aurelien_geron_kiwisoft_io/tensorflow_datasets/imdb_reviews/plain_text/1.0.0...\u001b[0m\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d19e9f5f622440b9feb5ed1a98db7b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Dl Completed...', max=1.0, style=Progre…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "001634a67f0e42f69450a95ed84492b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Dl Size...', max=1.0, style=ProgressSty…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /home/aurelien_geron_kiwisoft_io/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteK5RNB1/imdb_reviews-train.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c661939803ee4442be234cc9e1485599",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=25000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /home/aurelien_geron_kiwisoft_io/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteK5RNB1/imdb_reviews-test.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bdd0d660b82e4f93b4824e6e3573f197",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=25000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /home/aurelien_geron_kiwisoft_io/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteK5RNB1/imdb_reviews-unsupervised.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "91133fb7161d440383d4d50bd06f16ba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=50000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDataset imdb_reviews downloaded and prepared to /home/aurelien_geron_kiwisoft_io/tensorflow_datasets/imdb_reviews/plain_text/1.0.0. Subsequent calls will reuse this data.\u001b[0m\n",
      "\r"
     ]
    }
   ],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "datasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, with_info=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['test', 'train', 'unsupervised'])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_size = info.splits[\"train\"].num_examples\n",
    "test_size = info.splits[\"test\"].num_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 25000)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_size, test_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review: This was an absolutely terrible movie. Don't be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting  ...\n",
      "Label: 0 = Negative\n",
      "\n",
      "Review: I have been known to fall asleep during films, but this is usually due to a combination of things including, really tired, being warm and comfortable on the sette and having just eaten a lot. However  ...\n",
      "Label: 0 = Negative\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for X_batch, y_batch in datasets[\"train\"].batch(2).take(1):\n",
    "    for review, label in zip(X_batch.numpy(), y_batch.numpy()):\n",
    "        print(\"Review:\", review.decode(\"utf-8\")[:200], \"...\")\n",
    "        print(\"Label:\", label, \"= Positive\" if label else \"= Negative\")\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(X_batch, y_batch):\n",
    "    X_batch = tf.strings.substr(X_batch, 0, 300)\n",
    "    X_batch = tf.strings.regex_replace(X_batch, rb\"<br\\s*/?>\", b\" \")\n",
    "    X_batch = tf.strings.regex_replace(X_batch, b\"[^a-zA-Z']\", b\" \")\n",
    "    X_batch = tf.strings.split(X_batch)\n",
    "    return X_batch.to_tensor(default_value=b\"<pad>\"), y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(2, 53), dtype=string, numpy=\n",
       " array([[b'This', b'was', b'an', b'absolutely', b'terrible', b'movie',\n",
       "         b\"Don't\", b'be', b'lured', b'in', b'by', b'Christopher',\n",
       "         b'Walken', b'or', b'Michael', b'Ironside', b'Both', b'are',\n",
       "         b'great', b'actors', b'but', b'this', b'must', b'simply', b'be',\n",
       "         b'their', b'worst', b'role', b'in', b'history', b'Even',\n",
       "         b'their', b'great', b'acting', b'could', b'not', b'redeem',\n",
       "         b'this', b\"movie's\", b'ridiculous', b'storyline', b'This',\n",
       "         b'movie', b'is', b'an', b'early', b'nineties', b'US',\n",
       "         b'propaganda', b'pi', b'<pad>', b'<pad>', b'<pad>'],\n",
       "        [b'I', b'have', b'been', b'known', b'to', b'fall', b'asleep',\n",
       "         b'during', b'films', b'but', b'this', b'is', b'usually', b'due',\n",
       "         b'to', b'a', b'combination', b'of', b'things', b'including',\n",
       "         b'really', b'tired', b'being', b'warm', b'and', b'comfortable',\n",
       "         b'on', b'the', b'sette', b'and', b'having', b'just', b'eaten',\n",
       "         b'a', b'lot', b'However', b'on', b'this', b'occasion', b'I',\n",
       "         b'fell', b'asleep', b'because', b'the', b'film', b'was',\n",
       "         b'rubbish', b'The', b'plot', b'development', b'was', b'constant',\n",
       "         b'Cons']], dtype=object)>,\n",
       " <tf.Tensor: shape=(2,), dtype=int64, numpy=array([0, 0])>)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocess(X_batch, y_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "vocabulary = Counter()\n",
    "for X_batch, y_batch in datasets[\"train\"].batch(32).map(preprocess):\n",
    "    for review in X_batch:\n",
    "        vocabulary.update(list(review.numpy()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(b'<pad>', 214309), (b'the', 61137), (b'a', 38564)]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabulary.most_common()[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53893"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocabulary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = 10000\n",
    "truncated_vocabulary = [\n",
    "    word for word, count in vocabulary.most_common()[:vocab_size]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22\n",
      "12\n",
      "11\n",
      "10000\n"
     ]
    }
   ],
   "source": [
    "word_to_id = {word: index for index, word in enumerate(truncated_vocabulary)}\n",
    "for word in b\"This movie was faaaaaantastic\".split():\n",
    "    print(word_to_id.get(word) or vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = tf.constant(truncated_vocabulary)\n",
    "word_ids = tf.range(len(truncated_vocabulary), dtype=tf.int64)\n",
    "vocab_init = tf.lookup.KeyValueTensorInitializer(words, word_ids)\n",
    "num_oov_buckets = 1000\n",
    "table = tf.lookup.StaticVocabularyTable(vocab_init, num_oov_buckets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 4), dtype=int64, numpy=array([[   22,    12,    11, 10053]])>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.lookup(tf.constant([b\"This movie was faaaaaantastic\".split()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_words(X_batch, y_batch):\n",
    "    return table.lookup(X_batch), y_batch\n",
    "\n",
    "train_set = datasets[\"train\"].repeat().batch(32).map(preprocess)\n",
    "train_set = train_set.map(encode_words).prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[  22   11   28 ...    0    0    0]\n",
      " [   6   21   70 ...    0    0    0]\n",
      " [4099 6881    1 ...    0    0    0]\n",
      " ...\n",
      " [  22   12  118 ...  331 1047    0]\n",
      " [1757 4101  451 ...    0    0    0]\n",
      " [3365 4392    6 ...    0    0    0]], shape=(32, 60), dtype=int64)\n",
      "tf.Tensor([0 0 0 1 1 1 0 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 1 0 0 0 1 0 0 0], shape=(32,), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "for X_batch, y_batch in train_set.take(1):\n",
    "    print(X_batch)\n",
    "    print(y_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 781 steps\n",
      "Epoch 1/5\n",
      "781/781 [==============================] - 118s 152ms/step - loss: 0.5305 - accuracy: 0.7282\n",
      "Epoch 2/5\n",
      "781/781 [==============================] - 113s 145ms/step - loss: 0.3459 - accuracy: 0.8554\n",
      "Epoch 3/5\n",
      "781/781 [==============================] - 113s 145ms/step - loss: 0.1913 - accuracy: 0.9319\n",
      "Epoch 4/5\n",
      "781/781 [==============================] - 114s 146ms/step - loss: 0.1341 - accuracy: 0.9535\n",
      "Epoch 5/5\n",
      "781/781 [==============================] - 116s 148ms/step - loss: 0.1011 - accuracy: 0.9624\n"
     ]
    }
   ],
   "source": [
    "embed_size = 128\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size,\n",
    "                           mask_zero=True, # not shown in the book\n",
    "                           input_shape=[None]),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.GRU(128),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set, steps_per_epoch=train_size // 32, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or using manual masking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 781 steps\n",
      "Epoch 1/5\n",
      "781/781 [==============================] - 118s 152ms/step - loss: 0.5425 - accuracy: 0.7155\n",
      "Epoch 2/5\n",
      "781/781 [==============================] - 112s 143ms/step - loss: 0.3479 - accuracy: 0.8558\n",
      "Epoch 3/5\n",
      "781/781 [==============================] - 112s 144ms/step - loss: 0.1761 - accuracy: 0.9388\n",
      "Epoch 4/5\n",
      "781/781 [==============================] - 115s 147ms/step - loss: 0.1281 - accuracy: 0.9531\n",
      "Epoch 5/5\n",
      "781/781 [==============================] - 116s 148ms/step - loss: 0.1088 - accuracy: 0.9603\n"
     ]
    }
   ],
   "source": [
    "K = keras.backend\n",
    "embed_size = 128\n",
    "inputs = keras.layers.Input(shape=[None])\n",
    "mask = keras.layers.Lambda(lambda inputs: K.not_equal(inputs, 0))(inputs)\n",
    "z = keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size)(inputs)\n",
    "z = keras.layers.GRU(128, return_sequences=True)(z, mask=mask)\n",
    "z = keras.layers.GRU(128)(z, mask=mask)\n",
    "outputs = keras.layers.Dense(1, activation=\"sigmoid\")(z)\n",
    "model = keras.models.Model(inputs=[inputs], outputs=[outputs])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set, steps_per_epoch=train_size // 32, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reusing Pretrained Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "TFHUB_CACHE_DIR = os.path.join(os.curdir, \"my_tfhub_cache\")\n",
    "os.environ[\"TFHUB_CACHE_DIR\"] = TFHUB_CACHE_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_hub as hub\n",
    "\n",
    "model = keras.Sequential([\n",
    "    hub.KerasLayer(\"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\",\n",
    "                   dtype=tf.string, input_shape=[], output_shape=[50]),\n",
    "    keras.layers.Dense(128, activation=\"relu\"),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe.descriptor.txt\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/saved_model.pb\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/variables/variables.data-00000-of-00001\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/variables/variables.index\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/assets/tokens.txt\n"
     ]
    }
   ],
   "source": [
    "for dirpath, dirnames, filenames in os.walk(TFHUB_CACHE_DIR):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirpath, filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 781 steps\n",
      "Epoch 1/5\n",
      "781/781 [==============================] - 128s 164ms/step - loss: 0.5460 - accuracy: 0.7267\n",
      "Epoch 2/5\n",
      "781/781 [==============================] - 128s 164ms/step - loss: 0.5129 - accuracy: 0.7495\n",
      "Epoch 3/5\n",
      "781/781 [==============================] - 129s 165ms/step - loss: 0.5082 - accuracy: 0.7530\n",
      "Epoch 4/5\n",
      "781/781 [==============================] - 128s 164ms/step - loss: 0.5047 - accuracy: 0.7533\n",
      "Epoch 5/5\n",
      "781/781 [==============================] - 128s 164ms/step - loss: 0.5015 - accuracy: 0.7560\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "datasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, with_info=True)\n",
    "train_size = info.splits[\"train\"].num_examples\n",
    "batch_size = 32\n",
    "train_set = datasets[\"train\"].repeat().batch(batch_size).prefetch(1)\n",
    "history = model.fit(train_set, steps_per_epoch=train_size // batch_size, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automatic Translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = 100\n",
    "embed_size = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_addons as tfa\n",
    "\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "sequence_lengths = keras.layers.Input(shape=[], dtype=np.int32)\n",
    "\n",
    "embeddings = keras.layers.Embedding(vocab_size, embed_size)\n",
    "encoder_embeddings = embeddings(encoder_inputs)\n",
    "decoder_embeddings = embeddings(decoder_inputs)\n",
    "\n",
    "encoder = keras.layers.LSTM(512, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_embeddings)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "sampler = tfa.seq2seq.sampler.TrainingSampler()\n",
    "\n",
    "decoder_cell = keras.layers.LSTMCell(512)\n",
    "output_layer = keras.layers.Dense(vocab_size)\n",
    "decoder = tfa.seq2seq.basic_decoder.BasicDecoder(decoder_cell, sampler,\n",
    "                                                 output_layer=output_layer)\n",
    "final_outputs, final_state, final_sequence_lengths = decoder(\n",
    "    decoder_embeddings, initial_state=encoder_state,\n",
    "    sequence_length=sequence_lengths)\n",
    "Y_proba = tf.nn.softmax(final_outputs.rnn_output)\n",
    "\n",
    "model = keras.models.Model(\n",
    "    inputs=[encoder_inputs, decoder_inputs, sequence_lengths],\n",
    "    outputs=[Y_proba])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples\n",
      "Epoch 1/2\n",
      "1000/1000 [==============================] - 6s 6ms/sample - loss: 4.6053\n",
      "Epoch 2/2\n",
      "1000/1000 [==============================] - 3s 3ms/sample - loss: 4.6031\n"
     ]
    }
   ],
   "source": [
    "X = np.random.randint(100, size=10*1000).reshape(1000, 10)\n",
    "Y = np.random.randint(100, size=15*1000).reshape(1000, 15)\n",
    "X_decoder = np.c_[np.zeros((1000, 1)), Y[:, :-1]]\n",
    "seq_lengths = np.full([1000], 15)\n",
    "\n",
    "history = model.fit([X, X_decoder, seq_lengths], Y, epochs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bidirectional Recurrent Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "gru_10 (GRU)                 (None, None, 10)          660       \n",
      "_________________________________________________________________\n",
      "bidirectional (Bidirectional (None, None, 20)          1320      \n",
      "=================================================================\n",
      "Total params: 1,980\n",
      "Trainable params: 1,980\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(10, return_sequences=True, input_shape=[None, 10]),\n",
    "    keras.layers.Bidirectional(keras.layers.GRU(10, return_sequences=True))\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Positional Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(keras.layers.Layer):\n",
    "    def __init__(self, max_steps, max_dims, dtype=tf.float32, **kwargs):\n",
    "        super().__init__(dtype=dtype, **kwargs)\n",
    "        if max_dims % 2 == 1: max_dims += 1 # max_dims must be even\n",
    "        p, i = np.meshgrid(np.arange(max_steps), np.arange(max_dims // 2))\n",
    "        pos_emb = np.empty((1, max_steps, max_dims))\n",
    "        pos_emb[0, :, ::2] = np.sin(p / 10000**(2 * i / max_dims)).T\n",
    "        pos_emb[0, :, 1::2] = np.cos(p / 10000**(2 * i / max_dims)).T\n",
    "        self.positional_embedding = tf.constant(pos_emb.astype(self.dtype))\n",
    "    def call(self, inputs):\n",
    "        shape = tf.shape(inputs)\n",
    "        return inputs + self.positional_embedding[:, :shape[-2], :shape[-1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_steps = 201\n",
    "max_dims = 512\n",
    "pos_emb = PositionalEncoding(max_steps, max_dims)\n",
    "PE = pos_emb(np.zeros((1, max_steps, max_dims), np.float32))[0].numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkkAAAFLCAYAAADCoBiiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOx9eXhURfb2W91JCCFsYd8F2fd9CULSJOybIiCIDDhuuKHjNo7KbwYVFGfGXXTEEVQQRBEcdkPSCWEVgQCCgMgORglLyEK27vP9cXKTTtLL7e6q7g5fv89zH8jtuufWrVu36tQ5b50jiAhBBBFEEEEEEUQQQZSFwd8VCCKIIIIIIogggghEBJWkIIIIIogggggiCDsIKklBBBFEEEEEEUQQdhBUkoIIIogggggiiCDsIKgkBRFEEEEEEUQQQdhBUEkKIogggggiiCCCsIOgkhREEEEEEUQQQQRhBwGvJAkhHhNC/CiEyBdCLHFR9i9CiHQhRKYQ4lMhRBUfVTOIIIIIIogggrjJEPBKEoCLAF4F8KmzQkKI4QCeBxAH4BYArQDMVV25IIIIIoggggji5kTAK0lE9C0RrQFw2UXRGQD+S0SHiegqgFcAzFRdvyCCCCKIIIII4uZEwCtJbqATgAM2fx8A0EAIUcdP9QkiiCCCCCKIICoxQvxdAYmIBJBp87f2/+qwY4USQjwI4EEAqAP0ugUAQkJgadkSFBHhVUWuXjXgwgUDLBZACD5HBNSoQWja1ILQUK/E68aJEycAAK1bt3ZYxmKxwGg0+qZCAFBYCOPJkxB5efx3SAhQVAQIAWuDBrA2aOCV+Lw8gXPnDMjNFTAYuN2JgNBQoHlzCyIjOVehL577ypUrAICoqCil93EHhvPnYbhc/DmEhgKFhQAAioyE5ZZbAC/axGIBfvvNgMuXDWX6PQA0bGhF/frWkvP+gs/7uw1EVhaMp08DVitgMPDgYLEAYWGwNGsGioz0Sv6VKwZcvMjjjtHIogGgenVC8+YWCOG/Z/c3rAUFCD19GuLGDT6h9X2DAdZ69WBt2NAr+TduCJw/z+OObduHhAAtWpSOO76GP/u7LQy//QbDH3/wHyEh3EBEoBo1YGnaFN5MikVFQHo6jzuGYrOP1QoAezOIqJ7XlSeiSnGAeUlLnPx+AMBkm7/rACAAdVzJ7talC9E33xC1bElUsybRnj3kCaxWopkzeVqOjiY6dozPFxUR/fvfRFWqENWpQ7R/v0fi3UZMTAzFxMQ4LXPp0iXfVIaI6PRpolatiGrUIPr0U6IrV7jRdu0imjCBG+7VVz0W/913RCEhRHXrEn35JYsmItq5k6h1ayIhiN54g8/54rkXL15MixcvVn4fXSgsJJo+nQig3PvuK+2c588TzZtHFBpK1L8/0fXrHonPyOBXazAQzZ5NlJnJ569cIZo8mV+tyUSUnS3peTyET/u7LVau5Dbu3p0oJYWooIAoL487art2RNWqcUf1EA8/XDrupKXxuaIiog8/JAoPJ2rYkGjTpquSHqaS4fRpKrz1VqKqVYkWLya6fJnP795NNHEiN9wLL3gsPimJx50GDXhYs1j42LePX63BUDru+Bp+6+8aCgqI7r2X2/j++4lOnOCBOT2d6B//IIqIIOrWjejaNY/EZ2YSdepEZDQSPf44kfa4164RAfiRJOgegsg/Gq67EEK8CqApEc108PuXAE4R0YvFfw8B8CURuVwidO/endLS0oAzZwCTCbhyBdiyBejd2606vv8+8PjjwPPPA6++WnFRfvQoEB8PhIcDe/cCNWu6Jd5txMbGAgCSk5MdlsnIyEDdunXVVgQAzp4FBg8GMjOBzZuBvn3L/m6xAPfeC3zxBbBgAfDcc26JP3EC6NULaNMG2LQJKP9I2dksftUqvn2PHuqfe8mSJQCAmTNnKr2PS1itwNSpwMqVwKuvIuOhhyo+++rVwKRJwMCBwMaNgBvWVIsFGDUKSE4GEhL4NduCCPj0U+CBB4A//xn45BPvH8lT+Ky/22LFCmDaNGDAAGDdOqBWrbK///YbMGgQcPkyN2K3bm6J/+QTbtunngL++U+UrKY1HDoEjB8P5OdbcOiQEQFk2FSPixeBvn1hzc6GYf167t+2IAIeeghYtAj497+5EV3AarUiPT0dV65cQUFBIQoK2CgYFma/fGEhf4KhoRXfzU0P7eFDQuxbqa3WEoueO9ak0NBQ1K4dhccea4i1aw3YtInnVlsIIfYSkXuTuD3I0LRUHmCXYDiA1wB8Ufz/EDvlRgBIB9ARQG0ASQBe13OPbt26laqmZ84QtWjBVqXcXN0a7Q8/8EJxzBheRThCaiprvZMmlVo6VCFgLElWK1FcHFuQ9u51XK6oiGjqVF51JCbqFp+TQ9S1K1Ht2kSnTjkul51N1LEjUf36RIcOZeivv4cIGEvSJ59wm86fT0RO3vmXX7K57YEH3BL/4oss/uOPnZf729+43PLlbomXCp+vrC9c4H4/cCB3VEc4dYqoaVOiRo3cWlXv2kUUFkY0dCh/Po6wdy9RaKiV7rhD/bgTMLBaicaPJwoPpytms+NyRUWlFqW1a12KPXHiBP36669040YeXb9upcxM521vtRJlZbGR1tncoAIWX9/QFgUFbOrJy9NXTud8a7VaKS8vjw4f/pU+/PAEvfee/XKQZEnyuxLksoLAP8BuM9vjHwCaA8gG0Nym7FMAfgdwHcBiAFX03KOMkkREZDZz07z0ktOXpeHqVdarWrQoteQ6w+uvs/j339cl3mM88cQT9MQTTzgt45NJY9kyfuAPPnBdNjeXFdT27Yny83WJv/9+nts3bHBd9qef2Oo+aFC+04FNBjZu3EgbN25UexNX+P131h4HDSoZoZ2+86ee4sbcvVuX+M2b+dXed5/rsgUFRAMGEFWvTvTrr7rES4fPlaSJE9nf9csvrsv+8AO3/VNP6RKdlcV61S23sLvTFebOzSaA6KOPdImv/PjmG+6cb7zh+r3n5fEKqmVLohs3nBbdv38/WSwWys3lub2w0HVVioq4bHa2b5VUvylJFgtrhVlZ+h74xg39jUlc7OpVC+3evd+h+P9vlCRfHBWUJCKie+7hJdrRo/bfgA2ef57Htl27XBYlIu4/w4fzZKFncFMJ5ZPGlStsuunb1/lyyxbr1nHXfP11l0X37+eiTz+tv0qLFvE1n32m/5pKi3vuYRPnkSMlp5y+88xMJrD07u3yfRUVEXXuTNSmjct5pQSnTjHtb/hwfeVlw6dK0nffka0FTxfuv58JLjbvyxFefZXFb9umT/Tvv1+iYcN4kXDhgv4qVUpcucL9uGdPosJCfe89IYEbdN48p8X27dtXovS44Wyg/Hy+RufaTwr8piRpGqTeMd9q1a1UWa2sbF6/zu/CEYJKkmolKT2dR/O4OKcvLT2duWd33+2wiF389BMrVn/9q3vXyYbySWPWLGYuOunMdjF+PDfsmTNOi40ZQ1SrFlvz9MJqJerSpZBatWLrxk2LLVv4E58zp8xpl+986VK+7j//cVrsiy+42IoV7lVrwQK+bscO966TAZ8pSTk5bObp3Nm9TvbHH9yh4+OdjjsZGezFGz9ev+hLly7Rr7+yDjZ7tv7rKiVmzWJeQ/G4o/u9T5jA4865cw6L7Nu3j3JyWAdwRwexndx9ZU3yi5JUWMiNo3flpEFzu7nQIm2LBZUkfypJROwPA4icuEyeeIK/xePHHRZxiGnTeFX322/uX6tP/jSaNm2a0zJKJ40TJ0q3O7mLU6e4cf70J4dFtm8ntxfqGpYuzSSA6TqqsGrVKlq1apW6G7hCdDT7gMsNVi7fudVKNHgwUVSUw91u+fnsmeje3X2eRXY2Ub16zKPxNXymJL33HnfOlBT3r333Xb52zRqHRZ5+mj+tn37SL1Z79vvu4522N6016fx5tp4+/HDJKd3v/dQpdo86WfXu3btPF9XGHjT9wVfWJL8oSZ5qgjq0SI3fpRmcgkqSv5Wk/Hyixo15VWcHZ8+yR04PH8Mejh9nBcsFbchj+J24/dhjPFh5OhrPns3L3vPnK/xktRLFxrInz5Nt5X/8cYn69GEdQtWA5VfitqZBvvtuhZ90vfOdOx1eT8T0MkAfD8we/vUvvj411bPrPYVPlKTCQiYKRUd7fv2ttzq8/swZVnJmznRPrPbsmjXp8cc9q17A49lnWYO0Ib659d7/+lc289shzlmtRLt37/PYGuRra5LPlSTND+mJBqnjes1lqRlnfaEk/f+2IdE9hIUBs2dzOIADByr8PG8e7yCdM8cz8W3aADNnAh99BJw/711VAw5XrvC+72nTgMaNPZPxxBO8RfSDDyr8lJzMx0svAdWquS9aCODllznqw3//61n1Ahr//CcQFcV77j1B//58vPNOaWS8YhQWct8fNAgYMcIz8Q8/DDRoAPz9755dH9D45hvg9Gm3w1iUICSEx50dO4Affqjw8z//yePOP/7hmfhWrYAZM4CPPwYuXPBMRsDi2jUeUCdP5gf1BI8/ztvV33+/wk87d/KQVKUKPAqMKgRfS1QSx/XmQn4+AOC1f/8bffr0QY0aNVCvXj2MHTsWP/30U5mir732WsUyP//MbV9QUBqJthhELN5o5E/EVwgqSa7w4IM8C7/5ZpnTV68Cn33GSk6LFp6Lf+kl/lj+8x/vqhlw+OgjIDcXePppz2W0agXccQfLyskp89P773MspAce8Fz88OGsB/z731qE1psEx48D330HPPKIZxqkhqeeAn79lWP72OC77zj8zF//6tlEAXAYpr/+FUhKAvbs8byKAQci4I03gHbtgLFjPZdz771AjRrA22+XOZ2dzePOpEnejTsvvsi6b7lhrfLjo4+ArCzPFVQAaNKEG/i//2VZNli4kP/1JmuC0chHfn4FPaByw2Lh8NdhYUhOScEjjzyCHTt2ICkpCSEhIYiPjy/JQgBw/D67ZbKzuWGKisqILyri02Fhno87HkGGOaqyHw7dbRpmz67gNtJoA+7yke1h9GgOj6Jz96Nu+M3dlpfH4WdHjPBeluY2sgkfcOECuymffdZzsdpza+RjN8Iy6Ybf3G0PPsj+mPR0uz/rfueFheyPLNeH4uL4tLchFK5dY46sm2GZvIJyd5tGlpdBdnvqKfaL2ZCIP/6Y3NrRZovyzz5pEmcA8NQzEnC4cYPHnWHDKvzk9nvftauCu/mPP5he8cMP3g/6GvlY9caR8u62S5cuEQB68803qXfv3lSlShVq06YNbd682fubaTva7Lj4srKyyGAw0P/+9z+Hl5eU+e479keW41Hk5FR0UwbdbYGCJ59kLfm99wCwNvvxxxyQu0cP78U/+CAH3S23YPcaAwYMwIABA+QK1YPly4HffweeecZ7WQMGAP36AW+9VeL2+fRT/u+DD3ov/s47OQDyokXeyyqPpk2bomnTpvIFO8OVK2xqmDGD/VneICSEXQ8pKcD+/QDYSJWYyG3vbUqomjXZK7J8OVtIbgq88w63+z33eC/r8cfLuJuJgA8/BLp0AaKjvRd///0c5Pu777yXFRD49lt5406/fmxmfvfdEjPzp5+yF0hGKrSQELaG+Nrltr/4O37//ffx+uuv4+DBg+jatSvuvvtu3NDy2hVj/vz5iIyMdHqkpqZyYc1/6CCseFZWFqxWK2rXru2wbiVloqLYXGSxlIz5VmuJkcr3+R9laFqV/XBpSSLivbYNGxIVFpYYNxYtcn2ZHhQWEjVpQjRypBx57kDJynrwYE5aJIuZ+OWX3OBbtlBREVHz5mzN8Aa2zz17Nq8Q/Z3mSAo+/NClidOtd37tGlFkJMfvId5VFRIib0em7G/JFZRakn7/nRvnuefkybzzTt5lmJdHu3dXMKq6hfLPbrGwRdDBvpTKh2HD+IHsWDI8eu/Ll3ODr1tHFgvv5oyJcWy90Cz3tscHxS8rJyenwm+DBsXQwoWLyWLh+tm7fkVxfI2zZ89W+E0PyluS3njjDTIajXTUJv7fiRMnCECF57p8+TL98ssvTo9cLVCUxqh24A6ZNGkSde/enYqcmJ/LlLFaywSi0mJNln+1QUtSIOFPfwLS04HERHz8MVC9OjBlihzRISHAffdxzrEzZ+TI9BtOnwa2bgWmT5en8t9+O/Mzli7F5s2cBm7WLDmiAeY1FRQAn38uT6bf8PnnQOfOQPfucuTVrMnmtq+/Rt61PCxezDQxL5Oml2DAAKBjRzWWPJ9j+XJe7v7pT/Jk3n8/Wwc3bsSHHzLFTIaRCuAF/3338b6UU6fkyPQbLlzgB5k+XV6CtDvvZOLjF19g82Zuo4cfliMaKB0efWlNSktLw9ixY9GuXbuSc2EOks5FRUWhdevWTo+qVatyYS3/mh0z21NPPYVt27Zh1apVMDoww1UoIwRbpQoLQVZCYSHPk37JfSdD06rshy5LUl4eUe3alDdpGoWHc6wymThzhned6syEogsTJkygCRMmOC0jfWX9yiu8+jp9Wq7cP/+ZqHp1mjgqhxo08N6XX/65o6PlGr+IiFasWFGyEvQJjh8nLQ2DM7j9zr//ngiglNnfKOFvvfUWV/vAAbly7UGpJalnT6JeveTKLCwkql+fCsZPpPBwooce8lyUvWc/e5Z3y8scd/wCLdeTg4B1Hr/3Rx4hqlqV7h57nerXdx3A0F1kZ+vP3OEJyluSOnbsSHPnzi1zbtWqVRQeHk455XILzps3j6pVq+b02Lp1K5t3HGzbf/LJJ6lhw4b0888/O6yjwzLFQaWKbhQ4NFIFLUmBhCpVgMmTYfhuNYx52V7tqrKH5s15O/Vnn8nbaXX58mVcvnxZjjA9IAK++AKIifFu6409TJ8OZGUhbNP/8Kc/ebe7xB4eeAA4dox3XcvCjRs3Kvj5leKLL3ipNW2aXLlDhgANGyLkq6Vo3hwwmeSKnz6duQaffCJXrk/x00/Avn1yrUgAL5+nTIFhw1qE5WVKF9+sGY87Gs+vUoKILajR0RxXRSamTQNu3ED4ptW46y7upzIRGsrjvS/aPi8vD8eOHYO13ATz9ttvY8qUKYiIiChzftasWUhLS3N69O7du9QUVm5QfuKJJ/Dll18iKSkJ7du3t1snp2U0i1JhIYSQwwXzBEElyR1Mn47Qglw80uBbKYTt8pg6FTh3zm5olMqBH35gZu/06fJlDx6M7NpNMdW6VJqb0xZ33sl68MqV8mX7BFYrK0nx8Z7HpXIEoxF5E+5G79/XY8bYK9KJk3Xq8G75r7+uxKEYPv+cFZqpU+XLnjYNxsJ8PFB7Ffr3ly/+3ns5pIPGwa102LsXOHKENyvIxoAByKrXEpMLl+Guu+SL1/QKX7jcDh06BABYvnw5UlNTcezYMUyfPh0nTpzAa6+9VqG8LndbeHgpm93GF/boo49i8eLFWL58OWrXro309HSkp6cj22aHhssyQoBCQ2GgIoSGWH1P2C5GUElyA5faROMkWuLBiC+UvLBx43ilUmkn6i++AMLDgYkT5cs2GLC+5jSMwCb0aPKHdPHVq/OKetWqSjpRb9vGfDDZpoZibKp7D8JQiD/X+FqJ/IkTmfK3fbsS8WpRVAQsXQqMGgXUqydd/NXWffALWuOh6suUcDJGjeK4VV+rebXq8dlnJZZ+6RACG2rejXhswYCW6SrEIySkNAaQSqSlpaFNmzaYO3cupk6dih49eiArKwt79uxBQ09JhhYLV7ycFWnhwoXIyspCXFwcGjVqVHL861//cqtMkQiFABAmysZM8iWCSpIbWL1GYCnuwa2nE5WEqq1ZkwMcfvNNJZyoi4qAFSuA8eP5QSQjIwOYd3Y6QmCB+GqFdPkAx4+7cAHYtUuJeLVYtoxZvbffrkT8R7u643hoR7RIXapE/pgxrF9Xyok6NZVjeKiwoAJY8x2PO63PmZWMOxERrCh9+20ldLlZrTxgjhnDsTwk4+pVYN7paTDCCsM3alavoaGsZ6hu+7S0NHTp0gVTpkzB+fPnkZubizVr1qBJkyaeC9UCPpZTkhzxe/5hEyZeT5lCixEWGCAs/gtPHlSS3MDKlcDOFlMhiJQFF5k0SZ7LLS4uDnFxcd4L0oPUVA66osImDWD1auCQtRNutOmqbCYdO5YXpLLEt2zZEi1btpQjzBmsVu6Po0Z5F2HbATIygC2JAqcGTIPYtk1JDp3ISGDkyEpqyVuzhjW8kSOViP/qKyC12TQed776Ssk9NEueTE6eT7BrF1d8wgQl4levBg4VdUBOux68EFEALcWGapdbWloaunbtKk8gFcdG0oI+SQYVB922GkMhLBa/DQxBJUknLl0CzGag9/QOnHJgzRol95HpcpszZw7meJpYzl1oE8WwYUrEr1zJnMzwqXfwSH7pkvR71Kgh15IXExODmJgY7wW5wq5dHETvjjuUiF+9mle5zWYXy//f/5TcZ+JE5sbs3KlEvBoQcd8fNkydgroF6DetNdCtm7JxZ/Ro/ny/+UaJeHVYvZqtGKNHKxH/1VecHSlixmReuSqw5PnC5UZEOHTokFwlyWrlCitKpKYpjYYwHxK37CCoJOnEt99yn5g0CexSMps5maJkVEqXm+KJ4tIlzvE1eTIgbh/PDSM7PHkxJk1iQ8nu3UrEq8GaNTxRjBqlRLymoHa4oz3Qtq0yK+qYMWzJq1QT9b59HLhLkYKqucDuugs87mzfzpqTZFRKSx4RK0lDhihz8ScmctuL28fzybVrpd8HUO9yE0Lg+vXrGDdunDyhmtKiUEkSAjCEGJgUXuQfXlJQSdKJr79mA1KXLmDeR1ERsHGjknvJcrmNHDkSIxW5AMogLY0nCkV8mDVrbBTU7t05XoKiFfXYsWzJk+FyW7p0KZYuVcPhKYE2UZhMSiaKK1d4PTBpEiAMonSBkJkp/V6yLXk+wZo1PICPGaNE/HffAbfeykYkjBundIEwcWIl4+T99BMnYFakoG7cyErLhAkA2rcHWrdWtkDQ9Aw/6QGeoaiowq42WdAUxtDQYk9eSAif8MPAEFSSdCAzk9NX3X578Qvr14/zMymcqI1G770aPovTo3iiWLeOwy517Qp+AePHAwkJQG6u9HvVrMkGsTVrvDd9FxUVoUj1qHfkCHDihLKJYvNmHptKEtqPH89LPEULhIkT2ZL3449KxMvH6tXA4MEcmVkycnLYkjF2bPG407Mn0LSpMnfnmDG8QFi1Sol4+Vi9unQ8UIC1a4FGjbjZS+6TlKTkXprLrbBQ/S43KdAUFtkB64qhDZslRirtPn7QIoNKkg4kJPC7KdEBDAb+YDZsAPLzpd+vVi3gttuA9euli1aDNWu4wgq2P+flMSdjzBgbbuDttwM3bgDffy/9fgDTG06dAo4eVSJeLjRFXaYZ3Qbr1vFr7dOn+ET//nxC0Yp61Cj+vDZsUCJeLn75BTh8WJmCmpTEw0vJuCMEv+fNm7n/S0aNGmyQrDTjzurVnNdGVo4cGxQUcDOPHm1jKBk3jn9QZM0ICWEFqVJYUStoMWrElwSQNBi4/weVpMDEunVA7dooG8ht/HhOXa5oZTF6NHDwILvdAhonT3JFFbnakpPZYFSGlzloEGuSCidqoJJM1KtXc8eUHUASpR7lUaNsBiujkU0bGzbwhCEZderw41SKiXr1av5XkSVj3TqO3zVokM3JceP4g0hMVHLP0aM58vyvvyoRLw+nT7ObX5GCum0bcP16OeN4dDQQFaVUSQIqicutqIgVF0WutqKicpvmfBlQqhwqhZIkhIgSQqwWQuQIIc4IIe52UG6mEMIihMi2OWK9ubfVyhPFiBHllOYhQ5jtqHC3CVAJJmpNUVE4UURElEuFoe1mWbtWyYjSvDlzzwJ+or5wgaMNK2r7nTs5TkwFL+r48TyDpKQoue+oUexuS5cfu08u1q1jjpzsFDzgeWDdOuZolUmFERvLmpOiBYJGYQz4cUeroEILapUqHMC+BCEh/DFoARQlw+BffrJ+aHlUFLnaHMSnLJ2AfRzMq1IoSQA+AFAAoAGAaQA+FEJ0clB2JxFF2hzJ3tz4xx+BP/6ws8M0PJxHsA0blHwwHToAt9zi3UQ9ZswYjFHEEyrBxo2cxr1VK+miifj54+K4uctg/HiOy6RoG9ro0Rz6yRt+ctu2bdG2bVt5lSqPzZv5X0Xbn9ev53Fp6NByP8THA1WrKuPGaI+zaZMS8XKQmcmhKBTtKExL43AIFT7fKlVYk1m7VolFo3Vr3sAY8ErSpk085sjO1QYed9au5XVwhc26mlKmaKL2Iz9ZP7RnV+xqqyDeVwGlyiHglSQhRDUAdwKYQ0TZRLQNwP8AqAlvWw7r17N2P2KEnR+HD2eW6c8/S7+vEDxZJCZ6Tj945pln8Mwzz8itmC1yc4GtWx00jvc4coSt6nb1vPh4fjGaoiAZo0fzx5qQ4LmM6OhoREdHy6tUeWzaxG62zp2ViF+3jjnJFTbNRUSwRUNR23frxo8V0Ja8xESeLBT1/XXreAywuzl19GiOi5WWpuTeo0fzBsacHCXivUd+PtMcRo5UEsTw+HHeC2F33NHiwCky92jWk4COfF5UVLw3X436oG2aq/Bq/eRyC3glCUBbABYiOm5z7gAAR5akHkKIDCHEcSHEHCGEV+ruunXMDaxTx86Pw4fzv4qWvGPGsB6SnKxEvPdISeEBS9FEoU2SdhfrtWvzLkNFE3X//nyLgJ2oNQ1uxAglE8WpU8xJdmiIHD6cicunTkm/txD8zr//3m/x41xj0yZmOqvIOAsed/r1A+rXt/OjNlEr6vujRvFnbTYrEe89tm1jDU6hggo4MNBWr67UJ+ZHfrI+2CUMyYPVyodDI5Uf2O1q7GVyEQmgvNMjE0B1O2W3AugM4AxYifoKQBGACimOhRAPAngQABo3bowMOwHa0tMF9u2rgxdfzEFGhh1zTkQEarVtC+vatbiuILFo585AREQdfPNNHvr0cX9ZN76Yq/KdE/5Cphf+pGqrVyO8alVc7tBBSYC71atronNngfDwa3bFVx08GBFvvIErx46B7GqxjqHnuWNjq2P9+lD88ccVjxZNy5cvBwBMVZAZPuSHH1Dr2jVcj45GgZttr+fZV64MBxCJ6OgryMioOCAZ+/VDbQDZq1vuFfMAACAASURBVFYhb+ZMt+6vB4MGheGTT2pgw4ZMDBwoT1Pypr+XgAi1169H0eDByFIQLyojQ2DPnij89a+59sedkBDU6tQJtG4dMh94QLdcvc/eoQOPO6tW5aF//8AzJ0V8+y2qhoXhcpcuuscdd9772rU10L69AdWq2R93YDAAViusFosSRcFoFJyOwyrHWmKVqFAIqxWCCGQ0ghQoKoWFAoCA0Uh2n18YjRAAqKgIVNz29uZumagMSlI2gBrlztUAkFW+IBGdtPnzkBDiZQDPwo6SREQfA/gYALp370517cQ50YKrTppUDXXrOogkPXo0sHAh6kZEsBtCMuLiALO5KurWrer2taHFtlt7z2YLV787REoKEBuLuk2bena9E1y/DuzZAzz3nJP63XEHsGAB6uzfD0yZ4vY9XD33hAm8genMmbqlW+DdQPXq1XXdxyPs2gUYDKgxYQKbvNyEqzpt386Uj379ouwXqFMHaNECkdu3I1KBS/eOO4D77we2baspnZfu9fs4fBi4eBHGuXNRRcG7TUjgxfKECS7GnTffRN0qVdi6oRN6n33YMCApqSrq1KmqQg/wDikpwODBqOsmYV7Ps+fncxDfBx5wXP7c2bMAAIPFUo5VLwehoZpHSZTuKvUQVqsVBplusWLTrggJgVDgbrNaWe80GgWEo45nMEBYLBBVqgBQNL7a3k6pdDk4DiBECGHL0OsG4LCOawmAx594QgLHjHSa7mb4cP6ytm719DZOMWwYezROnnRd1qc4dYqd94pM3snJ7JevQBq2Re/evCVXkdtB82p4w0tShk2bSn2CklFYyO3vtO2F4L6fmKjEJ6ZtfVcUCss7aO51zd0uGQkJ/Fp79nRSaPhwnkkV+cRGjuQg+grolt7h3DlWUhWNOz/8wBzQMrtpy0MIPhSSt4EAdTW7sfX/t99+w4wZM1CvXj2Eh4ejY8eOSCm3I3bhwoVo2bIlwsPD0atXL2zdmmqfj2QLo9GnvKSAV5KIKAfAtwBeFkJUE0IMBDAewBflywohRgohGhT/vz2AOQA82itrtXIQw/h4Fy9s8GDeeqWIl6RtQQ24iVpTTBQNVgkJbJhzyns2Gnkm37xZyQdTvz6TiLdskS7aO2Rk8LZLRW2/ezeQleVCSQJ4os7KUpaRduhQDsH1++9KxHuOTZuATp2AZs2kiybivh8XB+dWhIED+QNRpEVq715ROCbPoY2zitItmc083rvMS62QQMyWlAAkb2u5QnTsart27RoGDhwIIsL69evx888/47333kN9G5LdV199hSeeeAIvvPAC9u/fjwEDonHnnSNx8eJZ58J9HAog4JWkYjwCoCqAPwAsB/AwER0WQjQvjoXUvLhcHICDQogcABvAytV8T2546BAnVnU5UVStyl+UImtGu3aciSDglKRNm4CWLZVswQX4eQcP5h3PTjF8OPDbb/zCFGDoUHY9KciA4jk0f4xCBdVg4C3QTqHN5Ir6vrZACKiJOidH6Y7Oo0d5w6xmxXSIKlXY3KGo7Vu25JxxATnuNGvGxCkFMJs59JVLA63RqJRArClJsnWwjIwMCCHw1ltvoU+fPggPD0fbtm3xvR5l240o22+88QYaNWqEzz//HH379kXLli0RFxeHDjbv7c0338TMmTPxwAMPoEOHDnjzzffQoEEjLFr0oXPhPo66WSmUJCK6QkS3E1E1ImpORF8Wnz9bHAvpbPHfzxBRg+JyrYjo/4jII6OlNjiUCSbmCCNG8Oh2+rQnt3IKIXiiTkpyX3GePHkyJk+eLL1OKCzkmWv4cCXExXPnOOqvSwUVUL7TJz6eA0unprp/badOndCpk6NNmF5g82bmBPXqJV82uO/37q1joqhZk7d+Kmr7Hj3YmxpQE/XWrdwhFLraAJ19f/hw3quuyBcfH89u14Bx+1gsPBAOG6Zk3MnLY6OoU1ebBsUTtSrx+/fvBwC8//77eP3113Hw4EF07doVd999d4U8n/Pnz0dkZGTpUbs2Ihs3RmStWiXnUh0MjGvWrEG/fv1w1113oX79+ujevTvef/99ULHWV1BQgL1792KYzWqgqAgYMmQYdu3a4fwhNFObj5SkykDc9gu2bOHFSpMmOgprI1pSEvDnP0uvS3w8sHgxsG8f3CIQP/LII9LrAoAd99nZOkdy9+HWRNGkCW8DTEgAnn1Wel0GDWJuZkKC+/NiH0/Y3q5AxArqkCFK4pRkZvLrff55nRcMHw7MmcMuQMkESqORH3PLFn7sgCAQJyZyhxg4UIn4hAQO6HjLLToKax1y82bg4Yel1yU+HvjPf3gDhcpwX7qxfz9w7RpbMBVg506ml+pSkmzDYxebu5csWVKhWKdOndCnTx8UFhZi2bJlFX7v3r07unfvjtzcXKxcubLkPBErbX379kavXp2RmZmJ1VoanGLM9GBXaVpaGoxGIzZs2IB27doBABYsWIDWrVvj6NGj6NGjR0nZWbNmlV1kZ2fzR1m1dBNREwcT5MmTJ7Fw4UL85S9/wfPPP4+0tDQ8/vjjAIDHHnsMGRkZsFgsaNCgQcnzWixAw4YNsHWrDn5DSIiSvKn2UCksSb5GXh4vGHXrAB07MoFFEYnSU15Sbm4uclX4iZKSeMaKjZUvGzwpNmzoRoxEk4l9YgpyiUVE8HzoiTWjsLAQhbKX4b/8wv4YRROF2ayDMG8LzSenKJhXfDw/7vHjrsv6BImJrDEo2MlaUMDN6NLVpqFNG3Y9KcofOWQIf+YBw8nT/K4u/cCewWxmvadMrjxn0MJjK+IlGQzyaTdpaWkYO3ZsiYIEAGEOduhFRUWhdevWfLRqxUe7dqXnWrdG1ar2d11brVb07NkTr732Gnr06IF7770Xs2fPxgcffFCmnLaDzWrVFkLkeFebLTRTmw/iJQUtSXawYwfvcNA9UWgKg9msZMlrSyB+4QX9140qjsKYLHsCS0ws9YVIhkaYd8uTZzIB773HJpDbbpNep6FDud1//513O+qFtnL0ZMXnENpEoUhJ2rKFUzEMGKDzgj59+ILkZGDiROn10b7BhATm5/kVGRkc5fqVV5SI37XLTQOtENz3N2zgD0eyZTEqij26CQnA//2fVNGeITGRV07ufIRuwGzm560QYd4RQkJYsy0mMzv7zkNDQ53+HhERUeH3/Hw+rFagZs2aUsaRtLQ03HXXXWXO7dmzB+Hh4WUUJ4DdbfPnO6f0bty4EYPsaJWNGjVCx44dy5zr0KED3nnnHQC8bd9oNCK9OEGjpgxmZPxRYl1yCi3qpg+UpKAlyQ62bOH+73KHgy2GDOGEo7/8oqRO8fEBQiDOzWW7tKJJ+uBBnYR5W8TE8AejaEUdUATixETOwHvrrUrEJyRwc+oO/xIayoqpIitqq1ZMIg4Ia4b2jAoVVKNRp7tHg8nEytthPRFR3Ed8PCtvWRWi0vkYeXkcaVtR2+fm8q5Ot9pe235YSXhJeXl5OHbsWIXgkm+//TamTJmCiHLW0VmzZiEtLY2PnTuRtm0b0vbvLz2XlobevXvbvdfAgQNx7NixMueOHz+OFsWxrcLCwtCrVy8kFJvotUwnW7Yk6EvlpPGSgkqSf7BlC4egcSNGW+nXpWiyGDqUFy2KwjHph+bWUjRYaYqILsK8hqgo3pKiqO179mQSs98JxFYrP6PmB5EMza3lVtsD3PePHFG2V3/oUH5sv6dqSEzkQUEF1wz8jL17u2HJAJSPO/Hx3O5+H3d27mTzvqJxZ/t2Jqi7pSQp3quvGUtkiT9UvAN4+fLlSE1NxbFjxzB9+nScOHECr71WId5yqbvt1lvR+pZb2NXWpo0ud9tf/vIX7Nq1C/PmzcOJEyfw9ddf491338Wjjz5aUuapp57CkiVLsGjRJzhy5Gc8//wTuHjxImbNmqXvgbQdhmddhAzwEkElqRwyM4G9ez34Ftu04aycigarQYN40e73PG6JiaXWAwVITma3SuPGbl5oMvFAmpcnvU7a6t7vuawOHACuXFE2UWhx3tyaKGwvUMhLun6dQ0P5FUlJbGZTkP1cs2S4TfNr0YJNbYo658CBHAbO75a8xET+EN0y7+uH2cyv1e1hTdVefZTdxCVDfFpaGtq0aYO5c+di6tSp6NGjB7KysrBnzx40bNjQ8YUaYciN8N99+vTBmjVrsHLlSnTu3BkvvvgiXnnllTKbie666y68/fbbmDfvVQwc2B07d27Dhg0bSqxNLqF9h4onRd1KkhDiASEE2Rx5QoifhBAzVFbQ19i2jfuE24OVxg9ITlbywUREAH37BoiS1L8/81Akw2LhFatHfHCTiR34igIbmkzAmTNKojzohw+Iq7Vru4gwbw89e7KFRdFErc2Lfu37586xK12RgrpjB1syPO77KSlKXA/h4cxTD4hxp08fTiqsAGYzi4+MdPNCxYENZeZzTUtLQ5cuXTBlyhScP38eubm5WLNmjcMdaiVwIz6SLUaPHo0DBw4gLy8Px48fx+zZsyuQsh955BEcO3Yaly7l48cf92Lw4MH6b6Bx8BSvXt2xJHUHkAdgQPFxB4DrAJYIIdxdewYskpN5R6dHyb1NJnY5KIrlHxvLq2m9/ICZM2fKJQ1fvcpmNkWTdFoaWww8migGDeKPRtEHo9XJnclC294rDYmJHJfCbTObPiQncwBPt/m/ISF8oaK2r1+fA1z7daJWTJhPTuaFukcGWpOJv80DB2RXCwD3fc2I6RdoiRwVtX1WFot324IK+IyXJEMHS0tLQ1e3V0DFN9eZisQTaHmC3RavXRRgStIRItpVfGwEcF/xb6PkV80/SE5mBSk83IOLFfMDYmO5Q23frq+8dCUpJYWXNQonCsBDi3rNmrw1RVHbd+zIYYD8piRpES0VKajnzgG//upFVAeTiQlNFy/KrFYJYmLYyuu3wIaJiUC9em7EpXAPyckeWjIAn4w7RJ4FVJWC1FQe+BT1fU28R0qSYl6SlibOWx2MiHDo0CH3lSQivrm3mXZdiPfYg20wKDfx61KSBNvIugIon/vhevG/7qeoD0BkZnLARo8nipYtmSOgaLAaMMA9XlJGRgYyMjLkVSApif1+/frJk2kDs5n5SI0aeSjAZGJih4ItgAYDT9TuKElS41T9+COnxFA0UXjMR9Lgg4k6J4e/T5+DiF+8yaSEMJ+Tw9ErPB53mjRhTqQiU1vfvrxo9JslLzmZt1vqjkvhHsxmHlc9DpipmJckIxyTEALXr1/HuHHj3LtQ8/Mp4OFJEe8Dl5teS1IbAJEADpY7r63590qrkR/hMR9Jg8ZLMpuV8AOqVXOPlzRx4kRMlBm7JiWFRxLd+8P1o6iIV3Rexac0mdjUoNfU5iZiY91btKxcubJMFF2voL10d3z2boqvXRvo0sVDAd26AbVq3Zy8pFOneOufItKwV3wkDSYTE/oUWDSqVPEzL0kz7zvYSeUtzGYW73F8UMUpShSniXMO7ZkUWZK8Fi8EW3gDQEnSfAZHhBAhQojaQogJAN4CcBScdLbSwys+kgaTiZ33ihKuustLkobLlzmIkaIo217xkTTcdhsPWAHES5KGlBR29UhO/aEhOZl1AI9pB9rOI4W8pI4d/dT22k0V9f3kZO62XmU6MZnYFF6cm0s2/MZLun6dzYeKFNRr17jJPLagAqUzvELytkLxzuEDPpLX4m0DOSuA3qppCV02AigEcAXACgDJAExElAcAQoivhRAef+rFypdZMLYJIVp5KssTeMVH0hBgvCRp0AgJigYrr/hIGiIjmdgRQLwkKdCsY4omaa/5SBpMJk62qihuSWysn3hJKSm8WlWUed4rPpIG7cNRaMnzCy9JM+8rGne2bmXxXnmxFSdc1eIl+TxOmI/4SF6LN5nY0qso0bM7lqTzAPoA6A2gE4CaRHQXEaUDgBCiN4AoIvJ4+iaiq0RkIk4V/CaAuZ7Kchde85E0NGvG0ZADhJckDcnJbO5WFEgvORlo394LPpIGk4m3qigwtXnCS5KCvXuZuKJYQZWiJAFKFwjZ2X7gJWlmtkDkI2lo1Ig/IEVtr/GSNO6az5CSwgOeQj5SeLiX3gOAzT1aPCEFUJgmzjECnY+kQfG4446S9CMR/UhEe4noCBHdKFfmQQBf2p4QQvxdCJEghEgVQhwp/tehv0AI8bIQQssStBbACCFELb0P4w1SU73kI9lCi1uiwD7qLi9JGlJSeKAqzngtE1L4SBpMJm73bdskCKsId3lJUhDofCQNnTsDdercXLyk06fZMqZIQd2+nfu/tL6fmqrE1BYezp+/XxZnffsqSSgMcFeNjpYwrCkOBeAXXlKg85E0tGvHGdH9pSQJIRoAaAjAlbM7DsDucud6g3e+jSGijgB+BytTjtALxSRwIioE76ZTE9q5HKTwkTRo/IC0NAnCKkIvL+nhhx/Gww8/7P0NtRgsiiYKKXwkDdHRvPIMgIm6d+/eDnMbuYWUFPb11a/vvSw78JqPpMFgUMoP8AsvSYof2Ln4kBAvdlbZwmRiU9teNftoYmP5W716VYn4isjK4mdR5Ga+fJmHNa/4SBp8xEvyqctNMR+pqEiS+PIJ5iVDT/U0PpIrJakpWAmyRW8ATxJRZvHfhwA4Y572AmBrTE8vlqscUvhIGgKEl3TXXXdVyPjsEVJTufMpJK4CkuahiAhe8ipq+06d2FiiZ6Lu3LkzOnsbV6eoiK1iiibps2fZlS9logBY0NmzvCNMAXzOS0pJ4RfeqZMS8VL4SBq07/NmiZe0fTsPdIr6vtdhL2zhA16SweBD8jYR30whH0mqeJMJ+O03JQnm9ShJ2s42V0pSLmziJQkhmgKIQlmlpz8AuxmYistbieg3m9PhAMq79aTDYuEdDtJ0AI0foCgrvV5e0rlz53Du3Dnvb5iSwma2vn29l2UHGh/JWfogt2AyMXElM9N1WTfhDi8pMzMTmd7WYd8+tg4oUlC1iUKa+JuNl+RxGHLXyM72ItKzPWjBLhXzknxmyUtJkWhmqwizmddU0miWinlJMvO4uYQH+drcgabsSaM7KRx3XH75RPQ6EQkiOu+i6EEA7W3+7gMgDEBrABBC3AmgCYCvi//+XAhxh035ElebDToAUBNr3wY5OUIeH0mDQn6AXl7S9OnTMX36dO9vKNXMVhZS+UgaTCb+yBWlLtfLS1q9ejVWr17t3c00LUYhHykqSmIg6Q4dgAYNlE3UWjP4ZKLWXrIiBXXHDol8JA0mE1tgCgokCmX4nJekmdkU5IkEuIvedpvEsG8+SlHiE16Sh/na9EJTkqTpYK1bc1BVfyhJbuAbACNt/u4N4F0AC4UQhwDMADCimGuk/W6reJVxtQkhWgIwwgdKUna2kMdH0uADfoBP4iVp3CpFE8X+/RL5SBo0hU5xvCSf7PSRbmYrC7NZEh9Jg2J+QIMGrIf5ZKLWXrDizPNSDSUmE0ec37NHotBSxMT4iJeUnc0DnKJx548/gMOHJVrxAOW8JMU6WFl4nFBNH6TxkTRo446CBPMyW2AxgGFCCM273gfAeiKKJ6IuRDSOiC4CgBCiHoALRFTyJRPR/xHR323kPQzgjeJwAEqRnS3kG0q0gVXRTOqzeEmK45Qo4cUqDhHsDi/JKyjmI505w9Qh6fOQycQ53BTwA4BSXpLyySIlRdK2P/vQNm5JNZRooQoqOy9JM7MpHnekKkk3Cy9JIwx5YUX64IMP0LVrV9SoUQM1atTAgAEDsH79+jLiX3/9HxBClDkaerMY1BLMHz3quQw7kKYkEVE2gNkAtACQPeGAf0REl4hoqAuR5wF8CgBCiCghxGohRI4Q4owQ4m5HFwkh/iKESBdCZAohPhVCuNzceeOGkPuxAMq34vgsXpKWN0mqma2seCWGEoVLXp/FS9K2/SkmrkpXkhSb2nzGS/IBH0l620dFAV27Kuuc/frxGkS5FTUlhRUOr8KQO4bZDFSvzjmxpULjJSnyifmElySBj9S0aVMsWLAA+/btw48//oghQ4bg9ttvx8GDB0uUPCGAdu3a4bfffis5DnmTqUIRL0nq109EiUR0sPj/dYnI4yD2RPQuEWk97QMABQAaAJgG4EMhRIXtJkKI4QCeB4cjuAWssOkKSKnEqqulLlewsvBZvKSUFB4ZFeRNUsJH0qCFCFYYL+n0acXxkhS7e5KT2SImPbF927bsF1M0k/okXtK5c7ztT5G7R9u4JX1xBnAD7dhRuXlJyclA796Stv1VhNkMDBqkgHKj0+X2xhsV53Kzmc87g7cpSjIyMiCEwFtvvYU+ffogPDwcbdu2xffff19aSAKrevz48Rg5ciRat26Ntm3bYt68eahevTp27txZIt5gAEJCQtCwYcOSo169eh7fEy1bAk2bSh931DgcJUIIUQ3AnQDmEFE2EW0D8D8A9hjJMwD8l4gOE9FVAK8AmOn6HooS2yte8rriJT399NN4+umnPb/B9evMqVI0Se/fz3VXMlFoS15Fo7keY8mAAQMwwJtIwcnJnN29cWPPZbgQL5WPpEEhPwDwES/JBwqqskDSsbHAjRvKeEmxsfztXrumRDyHIVdiZmNcvAgcO6Zo3NGpJPXpA0yeXKoomc38t6uddt7SnvYX5/Z7//338frrr+PgwYPo2rUr7r77bty4UbyRvKgIEALzX38dkZGRTo9UHX5Xi8WCFStWIDs7G9HR0SV8JCGAkydPokmTJmjZsiWmTJmCk96kFhGCv9eUFKnjTsArSQDaArAQ0XGbcwfAqVHKoxPKEr0PAGgghKjj7AYREaRi41bpVhzFvCRHxpKxY8di7Nixnt9g+3aJYcgrQmkgaS3XgKK218NLateuHdq1a+fZDSwWhWa2Uj6SIh2ABV+4oCyfUmwsN48yXlJKClCrFruuFEDpxq1Bg/hfheOOUl7Szp28K1ghYR5QpCTp5CWZTMDKlawY/d//8b8rV7quk8ZL8rTfp6WlwWg0YsOGDYiLi0Pbtm2xYMECXL58GUePHi3DR5o1axbS0tKcHs6C5R46dAiRkZGoUqUKZs2ahdWrV6Nz5y4ldKd+/fphyZIl2LhxIxYtWoT09HRER0fj8uXLnj0cwH3m99+B48ddl9UJNfv75CISQPlgM5kAqusoq/2/OoAyLS+EeBDF0b9r1LgFy5Ytk1LZ8hjTqBGyv/wSyQqsAfn5RhiNk7Bw4VFcuVIxuvfFixcBAI2d3DsnJwfVHIzU3VesQHujEV+fOQOLgvZZtiwWjRtHIjFxnXTZANAlKgqdt27FN4sWobBcWgNnz60XrVoNwrp1UVi27Du7v2cVm/iqV7fXVZ2j9qlTGJWZie0hITgtue1zcnKwb19nANHIzV2PZcvkmwRqZGVhLIBdr7+OXxUoekZjc2RnD8K8eZvQurX+QVXvex+7bh2ut2yJlBUrvKmmXeTlheCHHyZh7NgjWLZMzebdUc2aIW/5ciS1aFFyTkafB4CCAgNCQydj4cLjuH5dvpW869dfo5PBgK/Pn0eRpL5v++yLFvVDREQzHDmyCkePum9xaN++PQqdhHYxCAGjxYLCggKn+f5uuw148EEDXnnFiBdesOC226y6IsYIYYDFYkBBQZHLdIJEBGFTaN++fRg9ejRatWpV8gza74WFhSgsKEAoEYrA45aesctRW7Rq1Qp79uxBZmYmvv32W8yYMQObN29BmzbdARQhPj6+pGyHDh3Qq1cvtGvXDp9++imefPJJl/cFAKvVWmburn7tGsYB2O3Kb+kOiCigD3DE79xy554GsNZO2QMAJtv8XQcAAajj7B7dunUjZXjoIaIaNYiKipSIHziQqG9f+7/FxMRQTEyM0+svXbrk+Md+/Yiioz2vnBMUFhJVr0708MNKxDOSkogAonXrKvzk9Ll14t13WfypU/Z/X7x4MS1evNgz4W++ycLPnfO0eg5x6dIluvdeojp1iCwW6eIZVitR/fpE06crEZ+ezs2zYIF71+l67xcusPB//cuzyrnApk0sPiFBiXjGY48RRUQQFRSUnJLR5zXExhL17ClNXFncdhtRnz5SRdo+e6tWROPGeS5r3759zgsUFhJlZpZpe3tISiKqW5dozhz+NylJ3/0LClh8YaHrspZyH3jHjh1p7ty5Zc6tWrWKwsPDKScnhyg/n4VbLDRv3jyqVq2a02Pr1q36Kk1EcXFxNGPGnykzk4cHe4iNjaVZs2bpllnhXVitRA0aEN19N4HzzXqtg1QGd9txACFCiDY257oBOGyn7OHi32zL/U5EXtjvvERMDHN7FOZx27uXbyEViuOUaHwkReIZ/fvzzjyFbgdAETcmJQW49VYmIiqAMj6SBkX8AA1KeUnKtv0xlPKRNMTGcrykH+1uMJYiXgkvKTcX+OEHZW0vPQ2PPeggDmkcpJUrgZdfLnW96dmY5Sl5Oy8vD8eOHYO13M67t99+G1OmTEFEREQJHwlCeO1uKw+r1Yq8vHwYjfYNbHl5eTh69CgaNWrk3oPZwnbckYSAV5KIKAfAtwBeFkJUE0IMBDAewBd2in8O4D4hREchRG0ALwFY4rPK2oPirTjK4iUpzpukOG8oo2pVJnBXtnhJWrRwRY1z7pxBTXyk8oiJ4VlJ0RZAjZckPah9cjJQowbQvbvLop6Klx4fqTx8wIckUhDUftcu3pVXGflIGnTwkvbsKctB0jhKerj2WoxHd3lJ2vb65cuXIzU1FceOHcP06dNx4sQJvPbaa2XjIwmBqKgotG7d2ulR1cGu5+effx6pqak4ffo0Dh06hL/97W9ITk7GpEnTSnTIZ555BikpKTh16hR2796NiRMnIicnBzNmzHDvwcpD40NKQsArScV4BJwX7g8AywE8TESHhRDNhRDZQojmAEBEmwC8AcAM4Ezx8XcHMn2Dxo15h5KiwUpLei99opaanty+eC2DhVLExPDuQummNoXxkg4e5PhOiiaKHTtCAfhASaqs8ZK0/eEK8lZp8ZGULg4APRPwnAAAIABJREFUzuPWqZPyeEnSxZvN/GHddptkwaXi69RRFh+0FC7yuD33XEVFzWTi83rFWyzuGWnT0tLQpk0bzJ07F1OnTkWPHj2QlZWFPXv2cBBHifna0tPTcc8996Bdu3aIi4vDnj17sG7dRgwdOrLEEnb+/HlMnToV7dq1w4QJE1ClShXs2rULLWx4dB5B8sdVGYjbII63dLud82fBZG3bc28CeNNHVdOHmBjgm2+UZFWOiFBkLDGbJaYnLwstPtI990gXXRGxscCrr7JlbORIl8XdhckEfPstG0tuuUWSUMXL3e3bQ1Umti9Fx45A3brcOWfOlC7e1t0pLYTHhQscKfyhhyQJLAvNQKtcQQV43Pn8cza1hYZKFa0sXpLZzBEea9aULJjnf+lpeBzBNoeI5La3Fe9OYOy0tDR06dIFU6ZMwZQpUyoWkJh1dsmSJRXO5eWxkVCr+woFmyIAlI47GRlSxFUWS1LlRmwsO+8PHlQm3h4v6aWXXsJLL73kvsCsLOYyKJqkfcJH0qCFJvcDL2nw4MEY7El8A7OZEzY2a+ZN1Rxix45Q30wUCvgBtlAS1F6xgqrYQFsWiuO0mUySg9rn5DAfSVHbnzrF3t8hQ5SILwvFedw0PcYdl1taWhq6OgtpofGRFOZrc8RHkgohpMaVCSpJvoCf8rjFx8eX2WapG6mpCsMB+4iPpCEiQmlocltjSXm0atUKrVq1qviDM1gsTPRQ1PZnzgBnzhh90/YAv+TTp/nGCqDlcZPGS0pO5vhI3bq5LOqpeOV8JA2VjZe0fTu/SEV93yd8JA2K87hp4vXqYESEQ4cOOVaSJORrc35/9uYp8GDbx4svShMVVJJ8gaZNeaeSj/O4aTsQ3IbZzAIrOx9JQ0wMW8ays6WLdsZLSk9PR3p6unsC9+8HMjOVjeSKN25VhA8WCNnZbEmVAs0fo5CP5LO2VxyavG9fdrtJE6+Z2RTykbQm8Qlc8JK8haYk6REvhMD169cxbtw4+wUk8pHsQdMVFelgFdGzpzRRQSXJV4iJ4SWXgsSHjnhJTz75pO6gXGVgNvP2+XIBGGVAab42R9BMbTt2KBN/5kzFTVybNm3Cpk2b3BOmLXcVbj+vXdsqP1+bI3TuzElXFU3UUjePKt4f7lM+kgaF+SOl85IU8iA1PlJsrA/cPRpseUkK4G0etzJQrMVodfSZJUkigkqSrxATw857b7IcO4G0eEnXrrE142bgI2mIjuaPX3EeNynJp81moH17wJtYIU6QnAxERxeq5yNpMBjY7VMZeEk3Ex9JQ2wsf3DFObtkw2QCDhwArnicyrwYWVlKzWwnTxpw8aKPXG0aFPOSpOpgFktpzhMF8BkfSQGCSpKv4CdektvQrF2K2I0+5SNpqFaNM4orantnvCS3UFjIZjaFfKRTp1hJ8iliY9lCc+6cEvEmkyRekrY/XJGZzad8JA0+GHek8JK2bVPKg9y2LQyAj5WkAOMlOQRRqRajABofyWeuNskIKkm+QosWvEdc0WDliJfkNszm0uSwCuBzPpKG2FjeOZOTI120LS/JK/rB3r1MXFHMRxo40MdKkg8m6pwcL3lJiveH+5yPpKFhQ6Bdu8DnJWk8yIEDZVSrArZtCy0JWedTGI0Bw0tyCI0CokiL0XTEyuhqA4JKkm8RG8sThQ95SW4jKYn9AVWqyKhWGfiFj6QhJoYrsHOnEvEmk4Tg0j7gI0VFAR06qDH/O0TXrkDt2somam0Tl1fuTm1/uCIFdccOP/CRNMTElO5YlYwqVXi4kKIk9eunhAdJxLHBTCY/uHs82avva/GKtZjKzEcCgkqSbxETA1y+DBy2l3bOe5TnJc2fPx/z58/XLyAjg2M53Ux8JA0DB/JXqpiXZCs+Li4OcXFx+oWYzezqqVdPZtVKoDxfmyMYDBzBWiEvyevg0jcjH0lDbCxw/TqMP/2kRLzJxMOGx7ykzEyO5aSo7Y8cAS5dMvjW1abBR7wkr8QH+UhOEVSSfAntK5XC8K0IjZe0bRv/HR0djWh3RmVtErsZ4iOVR/XqzEtS1Pb2eEnNmjVDM70BIQsKmFCmmI/kFwUV4BufOAGcP69MvFe8JLO5lAWuAMnJvHHLp3wkDcUfXJj0BI8MjZfksQ6cmsrW9ZshPlJ5+IiX5LF4xXwkq7Vy85GAoJLkW7RoAbRqxS4tBSjPS9qxYwd2uLPt3Wxmc3efPkrq5zc+koYhQ5iXpCBekhA8Wdjyks6dO4dzesnKP/zAGdBvlvhI5aE9l6K+bzJ5kfRe8f5wv/GRNDRuDLRti9DUVCXi+/ThXNIeW/LMZvbbDRggs1plxDdtakHLlkrEu4ZiXpJX4ZgU85Equ6sNCCpJvkdcHI8mClYWERHMt9YGqxdeeAEvvPCCfgFmMwdyCwuTXje/8pE0DBnCFZGeupwRG1uWl5SYmIjExER9F5vNpWk8FEDjI/ksPlJ5dO3Kpja97eEmNF6SRxP1L79A5f7wHTu42/m178fFIWTnTomhyUvhNS9Ji8sWHi6zWgBYB0hO5s0KfnP3KOYleRUKQDEfqbKTtoGgkuR7DBnCPnhFcUs8jpf0++/svL8Z+UgaBg7kEV3RRO1VvCSzmVNhREXJrFIJ/MZH0mAwcN9KTFSyoq5XjxVAjybqm5mPpCEuDgYtN5oCaLyky5fdvPDKFU4Ap6jtDx3iW9x2m493dNoiwHhJjz76KCZMmFB6kUI+kpbppLLykYCgkuR7aIOBwonaamWrjVvQZhfFvACJeQfdR9WqPFMpcvl4HC8pL4/NDYra/vRpP/ORNMTFARcusOVGATReUkGBmxeazRy8s21bFdVCYiJvlVcQSFo/TCaQEMr6vta33OYlbd3KSrOivr9lC/87eLAflaQA4yW9+uqr+Pzzz6XxkbZu3Ypx48ahSZMmEEJgyZIlAEr5SEYjsHDhQrRs2RLh4eHo1asXUu1MUHrK+ANBJcnXaNCAl7yKlKQBAzw0lpjNTG7u1UtJvRIS+LEbNlQiXj+GDAHS0iDcXvK6hhA81icluWks2bULyM9XPlF4kutYKrQApQoXCLm5bhpLiFirVbQ//OpV5kkNHSpdtHuIioKlSxdlba/xkty2ompx2fr1U1KvLVs4gH3jxvLDrrgFjTikIPyLu+Jr166NyMjIUtOTl3yk7OxsdO7cGe+88w6qVq1acl5T2r799is88cQTeOGFF7B//35ER0dj5MiROHv2bEnZr75yXcZfCCpJ/kBcHC958/Oli65alWlF2sSoG2Yzb9NWQOC7cYMtW36fpAFuewCh2hZAyYiPZ2PJ0aNuXJSUVJq+QwESEpi767PEno7QujXQrJmyiVrTc9zq+z//zK5mhRZUqzUw+n7BoEEcJyw3V7rssDAePjxanGlucMnIz2dDVSC0vdxEaxWh1+V2/vx5CCFw7NgxaazqUaNGYf78+Zg4cSIMNm47i4W/x7fffhMzZ87EAw88gA4dOuC9995Do0aN8OGHH5aUffNN12X8haCS5A8MGcIuFkWBDYcOZV/8Sy+9j7ffftv1BRcuAMePK0tFsn07D1h+X00DvOStXl2ZkqQ9Y0ICMGLECIwYMcL1RWYzZ62uWVN6faxWnrji4wOAFyAEK6ma5iAZUVEc5SEhwY2LFPORtmxhN5siQ4lbKBw0iH2Rivr+sGGsc+qO8pCRwQOVorbftYv1wYBQkgwG7v9+Jm+npaUhIiICbdq04cLFfKT58+cjMjLS6eGO+0vz5FmtBdi7dy+GDRtW5vdhw4aV7LwuKHBdxp+oxNELKjE0Bm1SkhKiiDYopKd31jdAKJ4oEhI4NIFf+UgaQkKAwYMRpmiHW8uWwK238jPPnq3Dt5ibC+zeDTz5pJL6pKUxmTYgFFSAlaQlS7hiPXtKFz90KLBgAe+N0KVzms1s3WrVSnpdAO4HsbHc//2Nwv79uSKJiazRSIbtAuHee3VcoJgHuWULKw+xsUo29ZXBk09yl3YMAVgjWHswEv/tJbp3B7Q1sBA8tLmyJB04cABdu3aFQQguXLyTedasWZg8ebLTa5s0aaK7blpIgqtXM2CxWNCgXNyXBg0aYEuxyTcjw3UZfyJoSfIHatbkJa+iDtCjB+fp/Pzzi/o6WWIiUKsW765SgIQE5kr5lbhqi7g4GE+e5P36CjB0KI//x46dxMmTJ50X3rqVR3BFVjzt9bsT+FspFPOShg7lsV8Xeb6oSKmZ7fRpjp8ZEJYMgCNZ9u+vrO27dGHKpW5L3ubNQI0ayuKyJSQwYV6BgdYzKDblauGYnBlp09LS0L179wqutqioKLRu3drpYcs3cgXNoqV5GUW5ZyeiCuf0lPEHgpYkf2HYMGD+fGZ21q4tVbTBwJPimjVhyM9/FfHORmki4PvveXZREMwiI4O3/7/yinTRnsN2yXvffUrEf/QR8NlnR9G27R9o5cxKsXkzE1cVxUfSCPONGikR7z40clRCAvDss9LFDxjA8cK+/x4YP95F4T17gGvXgOHDpdcDKNVFAkZJAnhgmDuXzYt16kgVLQT3/c2beaJ2uquciAvGxSkxs127xq/3xReli7YLPawGWAjIyeXvXUEsupAQpjVou/rt4cCBA3j66acraDF6Ulht3LgRgwYN0lUXrQ7169eF0WhEenp6md//+OOPEstR3bquy/gTQUuSvzBiRClhRAGqVwcKCupi61YzbrkFWLbMfjnjzz9zID3JE8Ubb7Anw3aiMJv5vN/RqRMsjRoBmzYpET9kCA8Qhw83dl140yb2Q7qxStOLgCLM22LECLagKSAQV6nC7hVd1ozNm3lmV9RACQmsnCrKdOIZhg9nBcUt4pZ+DB0KXLoEHDjgouDRo8C5c8oU1OTkwCHMl0AxL0lTjByJz8nJwa+//sqWpHIJ1WbNmoW0tDSnR+/evXXVwzayQFhYGHr16oWEcv0tISGhJGWWnjJ+BREF7AEgCsBqADkAzgC420nZmQAsALJtjlg99+nWrRv5HIWFRDVrEt13n3TRS5cSVa1KxN2Vj4gIPl8e2X//Oxc4d05qHZKSiOrWJRo1ih8zIYH/TkqSehuPcePuu7lihYVK5N96K1FISBEBVmrRwn7b05kz3Pb//reSOiQksPj168uev3TpkpL76cbmzVyxDRuUiH/rLRZ/+nTF38o8e//+RH37KqmDxcL9ffp0JeI9wqVLl4iKioiioohmzFByjwsXuO0XLHBRUHtJp04pqcejjxJVq0aUn89/y+zz+/bt8/zi3FyizEwiq1VafWyRk0OUlVX6t8ViKfn/jh07yGAwUE5WFtchL0/KPbOysmj//v20f/9+qlq1Kv3973MpNXU/nThxhoiIVqxYQaGhobRo0SI6cuQIzZ49m6pVq0anbT5QPWXswdm7APAjydBDZAhRdQBYDuArAJEAbgOQCaCTg7IzAWzz5D5+UZKIiO68k6hJE+kfTIsWZRUk7WjRomLZ/MGDiTp3lnp/DYmJRAYDUbt2gaUgERFlfvIJN8r27dJlL11KFBqqQ0n9+GP+8fBh6XUgInruOaKQkLKDJlEAKEk3brAWP3u2EvE//cTNumhRxd9Knv3KFe6cc+YoqcPevVyHzz5TIt4jlDz7lClEDRqwJqcAnTsTxce7KDRyJFHbtkrub7UStWpFNHp06bmAUZIKClhBUbQ4y8tj8dqrtVWSPvzwQ2rfvj1rjpmZrDBLgNlsJgAVjhk2ivgHH3xALVq0oLCwMOrZsyelpKRUkKOnTHn8f60kAagGoABAW5tzXwB43UH5yqckLVrEr+Cnn6SKFcK+kiREuYLZ2WQNCyN6+mmp99dw+HDpvRXNRR4j45dflE2SupXUO+8katpU2aqyc2cik6nieb8rSUTKJ8nGjYkmTar4W8mzr1ypTEkmInr5Zf7efv9diXiPUPLsS5bws3sz2TvBX/5CVKUKG03sQlOSH3tMyf2PHePH++CD0nMBoyRZrayg3LghrT62KCpi8ZoFzWJPEc7JIbp+Xdm4k5VFlJ2tRHQF+EJJCmROUlsAFiI6bnPuAIBOTq7pIYTIEEIcF0LMEUIENjFd88dL5sY0b67zfHIyREEBc0QU4J13+N/Zs4EPP/Qwp5kiUK1avNNHAS/J0aa5MueLinjr2fDhSna9nD0L/PQTMHq0dNFyMGIEx+ZytfvPAwjBzfr99062fm/ezNue+vaVfn8A2LCBN23Vr69EvHdQNO7Yis/Pd7LDcNs2Jswp4iNt2MD/jhqlRLx3UJyiRKM9OQwFQKQ0oZq2u05BTGK/IZAfJRLsXrNFJoDqDspvBdAZzF3qBHbTFQF4zV5hIcSDAB4EgMaNGyMjI0NCld1E1aqo1a4drGvX4vqMGdLEPv98GJ56qjpu3Cj9CKpWJTz/fBYyMkoTW1VbswZVwsNxpX173oYmEdu2heLTT2ugRQsr5sy5CpMpFJMmVccnn2T5N9lkMTIzM1F18GBELFiAK0ePgurWlSa7SZPaOH++4k7BJk0syMi4CgAI2b0btTIzcX3AABQo6HtffRUOIBLR0VeRkVF2xMzMLP9Z+R6Gfv0QBSB71Srk6Qqq4x4GDw7D4sU1sGHDNQwcWDohZWZmAkSovXEjigYNQta1a9LvnZEhsHt3FJ59NhcZGTeky/cUJe89JAS1unQBrV2LzAcekH6fTp2AiIg6+PrrPPTpk1Ph94g1a1A1NBSXO3eWPu4AwJo1NdCunQGRkddKxMvu81YvgqEKoxGioABksXA+PckwGkVxIEfOjWRbV2G1QhCBjEaQgoCuhYUCgIDRSCX3Vw3lc7cMc5QnB4Bk2PFjFh/bAPQAkFvumqcBrNUpfwqAvXrK+s3dRkT01FNEYWHS7ZNLlxLVq5dDgJWMRgfE4TZtKN8lecAz/OMf7M16/vnSc0lJOgidPsKlS5eIdu9mu/yyZVJlL13KHCSnnKQ5c7iBrlyRem8NY8YwL8OeRT0g3G1WK1HLlkTjxikRf/0688Keeabs+UuXLpX6gT/+WMm9v/iCxe/Zo0S8xyjz3v/2NyasXbum5F63307UvLkDj44jP7AEZGU5ee+S4JW7jaiiT0wybGlPFdxt5UlLkqHYk1cBN7W7jYhiiUg4OG4DcBxAiBCijc1l3QAc1nsLyAhrqhrDh3OqAMm+qGnTgI4dR6FNm7dgsdgJbnziBPDLLyhQFMSwQwc2u9q6e0wm4LnnlNzOM/TqxbFiJLsdpk0DPv4YqF07BwChZk3+e9o0m0Lr1rG7T3KMLIA9GYmJ7G4IgFhs9iEEu9wSE5XkMKxenUMBrFtn58f//Y//HTlS+n0BYP16DqqoIKC4PIwYwS6fpCQl4seMYZfvoUPlfjh5kv3AY8YouW9iIrtYA9LVpkFxKADN1WVXfGEhu/ucBrHyDFS89V+RJ89vCFhOEhHlAPgWwMtCiGpCiIEAxoPJ2xUghBgphGhQ/P/2/4+9Lw2XrKrOfk/Vne/tkcZmhpZGpmaW0aGNAwoRbAQRFYVEQaMm8YsxGscYkc+Y6BfjQAQiBBABEWgUiKCiGFSw6QHoBpRGpoamJ7r7zkPV/n7su+quWrXWPvvcW9X3Xvqs5zlPVZ2qOnX23mt417vWOQXgcwCW7qjzHbcsXuxvRU2Ou84yd+7vAHjHXSVL/dQMNagf6bbbfPw/4YSGHL4+Uiz6QHnbbXV3WO95D/D1r/8Q+++/EQsXCoD09NP+DptLltT1N0l++UsPlKZsPxLJqacCvb3Ar37VkMO/9a3+djyPPy7eWLrUA+S99qr7b46M+HanU05pSByqn5x4or/btYoiJy4EUmoOT34u9U6f45Pbb/cA+VWvasjh6yP0HyIjIx5ZNODwattTgxuGqA/qpdSPBExhkDQqHwbQDmAD/O0A/so5txoAkiTZJ0mSniRJqB35DQAeTJKkF8Dt8AArfAvRqSCtrd6jLF3akH+IbmvbgMMOU5zVLbcARx6J8t571/03y2Xgjjt8sjrlDWbJEmDLlob96eeRRz6DBx4Ann+e7WxwoLjtNn9vygb8LWB95Q1v8H+VcfPNDTk8gUSeICQvvOD/K69Bc3/fff4m+lOayQD8Xa7//M+BH/+4IX5n9939Py/V+J2lS33T0v771/03nfMg6U1vasgNresr5BgbMPd0eP//aYzSkf8VUmdp8OEnTaY0SHLObXHOLXHOdTrn9nHOXcvee9o51+Wce3r09d875+aPfvblzrnPO+cmv0M4RpYsATZs8B62AfLnf+4xQKVHdcMG4N57GxYofv97f9fdKc9kAB7JtbV50NgAOfxw/5fodMUNAP9bBx0EvOIVdf895zwoeMMb/LCmtLS3+/lfujT8h1PjlP3392VfHqhb7rzTT1IDAWqxOIX+UDgkZ5zhDfXeexty+NNOA373O/8TAHwy8utfN2zuH3wQePbZaQBQgTEk0aB/3lVLbiMjnt5sYKmN3cT7JSNTGiTtNHLqqT6za1CgfutbfcLy05+O7vjJT7xWN6jc8+MfeztsUCWvvtLZ6SPazTc3hPree+8XsddeLFBv3erLSw0KFA895P9YtUEtH/WXM87wNNv99zfk8G99q5/u7m7/uvWOO4D99vP/xlpncQ646Sb/LzOzZ9f98PWXt7zFM9kNYvLe+tYxdgeAf1IqAaef3pDf+9GPvN+ZFrrf4JIbYaEKSGpwwxBV8hrwN3yTLjlImgoya5bvar7llroazNVXX42rr74aJ5zgG0l/9KPRN265Bdh3X+CII+r2WyTOAT/8oS/11Pn/MxsnS5b4PqGVK+t62DPOOANvf/sZOP10D1B7euADxchIw0DSD3/oneMZZzTk8PWXU0/1jruBCcLw8Gig7ulB8z33+LlvQKBYvRp47DHgHe+o+6EbIzNm+D83a1CCcNRR/v+Ml1Jn6NKlvg537LF1/y0AuPFGD1CnwH+ixklTk5/3BrCohMFKpdGlzUtt45YcJE0VWbIE+OMffadpnWTvvffG3nvvjWIROPNMz2b0buj1f27ZoEDx0EP+HoHTJlAAvi5QKNQ9UM+aNQuzZs3CO97hG6lvuw3+N+bPB44/vq6/BYwB1MWLp+hNDDWZM8cnCA0K1K96FbDbbsANNwC4804kg4MNBahJArz97Q05fGPkjDOAp56qe4IAjM3FHXcA3ZsG/VWkZGt1ljVrgEceAc46q+6HbpwQ7dLQkpu/ZxJGRsY6uhsgDbxobtLlJTikaSpEQdcxUF9//fW4/vrrAaASqFd+9U5gYKBhpTZiMqZVoNh1V+DVr647SHr44Yfx8MMP4zWv8YH6ph80NlBMOyaDZMkSj6wfeaTuhy4W/XzcfjswfONSlGfN8mvdAJl2TAbg/U6h0LCS2zvf6d3Nsq/+wlOpDSq13XijxwDThkEFqi9Da0CC4PuDHIaHq0ttH/nIR/D2OjroUumlW2oDcpA0dWTPPf1fJFRqYhOXSy65BJdccgkA4DWv8c67fMONPnt/zWvq9jskzvmM/XWvm0ZMBsmSJb7zc+3auh1y2bJlWLZsGYpFn+GWbv+pb45pkCeflgAVGGN2GhSozz4bcAMDcLfc4m950QBvvmaN36YdQKUEoUFzf9JJ3rXhB9f6Rq03vrEhv3PjjZ413GOPhhy+cdLcPNbQU2ehklsFhI3Wwi666CJcddVVdfudu+++B+ecczoWLNgTSZLgyiuvrPnMd77zHSxYsABtbW045phj8Otf/7rq/XvuuQenn3469tzTPsZkSQ6SppK8853AAw94OqDOUiwC7zm9G8c8czOGzzi7IcXjaVlqIznzTP947bXhz41Tzj4beOfw1RiYuWvDLn364Q+nIZMB+Ch64onAddc1JKM+6STgvLk/QUv/dgw2SDmJyZh2ABXwJ/3www1h8goF4NwzenHsszdj8G3v8I3idZbHHvO+Z1qV2kiCd36cuDQ3A80YhkNS+a05c+agq6urbr+xbVsPDj10Eb7xjW+gvb295v3rr78ef/u3f4tPf/rTWLFiBU466SSccsopeJr9mWVPTw8WLbKPMZmSg6SpJO96l/cqV6v3y5ywXDDvZnSgH/fs+96GHH/aMhmA//ff170OuOqqxvTGHLoVp+HH+Nku5zSEyVi9ehr2ZHB573t9oG5Ab0yhAHx09jV4Drtj8+H1Z1ABD5Je/Wrflzzt5JxzfBZVR3aBywdediu60Iu7d39P+ofHIUS+T0u/Uyj4uR8aatBVbg5NGMFI4kttzz77LJIkwWN1SsTLZeBNbzoVF110Mc466ywUlDaCr3/96zj//PNxwQUX4OCDD8Y3v/lN7L777pUqBwCceuqpuPhi+xiTKVPrbHZ22X13zzJcfXVD6NcDf38NniouwHdWnlT3Y/Or2qZdqY3kfe/zt2duwP2qCjf/CG0YxP995lxs3173w+OGGzyTQYTYtJN3vtODx0YE6i1bcOjTt+MHeBfuuLP+WeqaNdOYyQA89fiWt3i/04CbG+5/3/exrrg3vrmyMSX+a6/1d/ZvwH1xd4w0N/uBNGDuk5ERJACGXDPKZWDlypXo6OjAAQccUPW5iy++GF1dXcFNlsiAsZ5zK+8bGhrCAw88gJNPPrlq/8knn4zf/OY39Rhiw+UleMHeNJf3vhc491x/07XFi+t33OeeQ/KLn+ORV34Gt92eYPPm+l6i/7vfedr77/++fsfc4XLmmcCHP+wDdb3/T+Waa9C/9wH4zTPH4kc/Aur5x/elEnDFFb7dY7fd6nfcHSpz5/qG9muvBb761fqybT/8IQojw/j5budi8KZWfOQj9Ts0AFx+uT/dc86p73F3qJx3nr/88he/qG85eONGJD/9Kf54zN/hzp8VsGkTMG9e/Q5/332eRb300vodc1zysY9NjAUtlXyWk4VFOfJI4N//PfyZ4WG4JEHJFTEyAqxatQqHH354DVvzoQ99CGeffXbwUHvuuWfVa+c8ARa6qm3Tpk39LhQAAAAgAElEQVQolUqYL3oA5s+fj5/97Gfhc58ikjNJU02WLPE3OKxDye3GG2/EjTfe6F9cey1QLmO/z5yLwcH6J+yXXur/gm5aB4qZM31T9XXX1eVPV88++2zveJ5+GvjlL9H2/nNx4IFJ3R36nXcCzzwDXHhhfY+7w+V97/N3g6/c9bROcs01wMEH45j3H4m7727GU0/V79BkS2972zRmUAEPUGfPBv77v+t73BtuAEZGsOc/vAcjI/X3O5df7t3ltPY7gAdI9S63lcsefDU3o1BIMDzsmaQjjzyy5qNz587FwoULg5vsFaJ7MMX8BUwibjfjnKvZN2XFObfTb0cccYSbUnLeec7NnOlcX1/9jnnEEc4dd5xzzrkTTnDuwAOdK5ed27hx44QP/eKLzrW3O/fBD074UDtMzHHfcYdzgHM/+lH9fuwrX/HHXLvWff3r/unKlfU7/BlnOLfrrs4NDsZ9vh5r3hAZHHRu3jzn3vGO+h3ziSf8hH/5y+6pp5wrFMruM5+p3+Gvu84f/qc/rd8xGyWp6/7BD3pD3ratfj96/PHOHXaYc865V73KuQMOcK5Uqs+ht293rrPTub/8y/TP1lPnly9fXrdjVWR42M/70FD9jjkw4I85MuIGB/3TAw44wP3nf/5nzUe//OUvu87OzuB2zz33VH2nt9evQbk8tq+zs9NdccUVldeDg4OuWCy6G264oeq7H/7wh91rX/ta9bTlMUISWgsAy1wd8EHOJE1Fee97ge3bJ3w7gCuvvNJfSvn73wOrVvnjAvjgB31p7J576nCuAL7/fX8PpmnPZABjNasrrpjwoVauXImVDzwAfPe7vqv35S/Heef5C3y++906nCv8P3r8+MfA+edPgz/1TJOWFn/xwtKl/n++6iHf+Y6vB5x7LvbZB3jTm4Zw+eW+TFAPuewyf/P6Bl3ZvmPlvPO8IRP7PFG5/35fD3v/+wEAf/VX/n65v/hFfQ5//fVAby/wgQ/U53iTKvSnZ/VSzNFamCsWgWIRzc1Ab28v1q5dqzJJH/rQh7y/CmyvfOUrK58vl/0Fec3N4XsSt7S04JhjjsFdd91Vtf+uu+7CSSfVvze2IVIPpDXdtynHJJVKzh10kHNHHVUN0zPK4sWL3eLFi5175zs9MzWaIfb2OjdrlnPvetfEM6xy2bnDD3fumGMmdJgdLsFxf/azziWJc489NqHfuOKKK9zPP/IRTzXceGNl//ve51xXl8/CJioXX+wPn+VUpyyT5Jxzq1b5AV188cSP1d3tFf3ssyu7rr12qwOcE4ntuOTxx/2p/vM/T/xYO0JS171cdu7gg5078sgJ+Z2KvPvd3u+MKvrAgCcK3/72iR/aOU9SHXJI3KlOeSbJuSrmZ8IyNOTctm2uzOjlu+/+jSsUCq6np3fCh+en2t3d7VasWOFWrFjh2tvb3Re/+EW3YsUK99RTTznnnLvuuutcc3Ozu+yyy9yaNWvc3/zN37jOzk735JNPVo6XdgxLdgSTNOkAZSpsUw4kOefcpZf65fnFL8Z9iMWLF7uzjz/euWLRuY9/vOq9v/5r51panHv00U0TOs377vOn+d3vTugwO1yCTnP9eudaWydcP7ziiivc+gMOcG7BgirHd++99ZmzUsm5l7/cude9Ltv3pjRIcs65k092bv585/r7J3acb37TT/Rvf1vZtX79Rrfvvs69/vUTO7Rzzn3qU84VCs4988zEj7UjJGrdL7/cz9mdd07sx5591rmmJuc+9rGq3f/wD94dPfvsxA6/YoU/za9/Pe7z0wIklUoeedSjzaKnx7nt212J1Ta//e1L3CtecZAbGJjYoctln390d/vXd999twNQs5133nnst7/t9t13X9fS0uKOPvpo96tf/arqmDHH0CQHSTszSOrvd+5lL3Pu1FPH9/1rrnHPt7a6sidenfvGN6refvhhv/tTn+qZ0Gm+4x31Y0V2pKQ6zQsucK6tzbkXXhjfD1xzjeudNctP8pw5zl1zTeWtctm3aRx++MT6M264wR/++uuzfW/Kg6S77vIDu+yy8R+jVHJu4ULfgMdk48aNFfZtzZrxH37LFk+S1IsV2RESte4DA87tvrtzb3zjxH7s05/2bOzatVW71671uz//+Ykd/qyznJsxw7nNm+M+Py1AknMeIG3bNjHHMDLijzEwUAWSnPPYqbt7YkThKElV1/ap8UoOknZmkOScc1/8ol+i1auzfe+aa5zr6PDfpa2joypQO+fcaac5N3NmyW3ZMr7To2zus58d3/cnU1Kd5iOP+MF94QvZDx4x///9325CZZ+REV9qOOSQ7Oz8lAdJ5bIv+Rx44PiDxdKlfoKvu65q98aNG92GDR7YsypcZvnc5/zhV60a/zF2tESv+7/8ix/cAw+M74d6e53bZRfnlixR3z7tNOdmz3bj9jsPPpjd70wbkERs0kRYVAJa5XINSJoowOEsUj0qshOVHCTt7CBp40bPZsRcvsFl332rAzRt++5b9bGVK/3uT396fKd3+une2b344vi+P5kS5TRPO807+96MNfyI+SeQc+CB/sKWrHLtteMHWVMeJDk3NsClS7N/t1x27sQTndt775rJpbF//vP+8MuWZT/85s2eRTrzzOzfnUyJXvetWz1Nc8454/shuoTzl79U3161yrNJn/zk+A6flUVybhqBJOe8vxkFOZlFlOwkSJooyJlKLJJzOUjKQZJzvnmoUHAui2EmiR6kk6Tmo0uWDLjOzuxVpfvv94f80peyfW+qSJTT/N//deNikyLn/6ab/O7vfS/b4YeHnXvFK3zJbjxEy7QAScPDzu23n69JZvXIV1/trHIdjX3bNo9/Tz45+6l99rNu2rFIzmVc9098wvudrINcv94jyLe8JRiFzz3X539Ze5MeesjPfdbbOEwrkMTKZZmFANaoY5AgybkxoBN7yxCSqcYiOZeDpBwkOec56Ze9zF/KERsRI5kk55z77W+3uEKhpr8yKOWyc29+sw8y060XiSTaab773dThHn/wXXeNmv9y2bljj3Vun32y+cMrrvCHu+mm+O9wmRYgyTnnbr7ZD/Rf/iX+O9u2Obfbbn5iFXvhY//a1/zhf/7z+MOvX+9ZjLPOiv/OVJFM675pk9fjY4/NVs/9y790rrk51V6eeMJ/7IIL4g9dLntyd8YMf3pZZFqBJOfGwE6Wuad7LTFnooGkcrnS150J7NC9lrKCq0ZKDpJykOSFMmPlJmCqfOYztQFa6UlyzjuP97/fX4jym9/EHf6//ssf8mtfyzCGKSbRTnP9el9TfN3r4jxKX5+/zlmyScb8U4/yP/5j3On86U/+dI4/fvzZ3LQBSc75vpb29poGYFM+/nE/9/ffr77Nx97f79xee3lGLqaiWir55KCtzbesTTfJvO4/+IFXzv/3/+I+T5e6fuITUR//m7/xZJWxVDVy2WX+8P/6r3Gf5zLtQFKp5FFMT0+coRs0jwaSnBvDU7GtT1TFiz2dHSU5SMpBkpdy2bk/+zMfHZ97LvzZ3l7f7DJvnlvf2upKxGAoAdo57zy2bPFXqe+1l2+DCsmDD/og8YY31Od2HpMlmZzmd7/rTSXmLrCf+pSjRq/uXXbxVxcG5t85597/fhfVfjM05C/WmjkzHjNoMq1A0jPPeOrg5JPTvfN993m0/4EPmB+RY7/tNo+pzj03/fD/9m9+nS65JPbkp5ZkXvdy2V9d29npHLunjSrbt/u7+u+2W/Qduzdt8qax114+FwnJo4/6POMNb5j8EvOKFStM8FFXyULd9PerzUKh86T+7rRqNjFPE73ort5SKpXcihUrzPd3CpAE4KMAlgEYBHBlxOf/D4D1ALYB+B6A1pjfmfIgyTmfunZ0eABkeZTeXn8DmELBuTvuGLuZZEDIeSxb5qtKb36zbQjd3f4el7vtlu7UprpkcpqlknOvfrW/d9Jtt9mfo4bV0SB9xRVXRN1ev7/fuaOP9vc9/OMf7c998pNuXJf8S5lWIMm5sfsdffSjtnKuWuVvtbBgQRDpa2Oni0i/+U37FO6/35eHzjhjamXSWWRc6/7UUx4kHX64c88/r3+mv98nccWicz/5SabDL1/uicLXvtYO1n193j522cW5desynv+o1FPnV69e7Xp6JnbrlCjh6CR0dQeBqb6+GuUMgSQin9KqenTjyKlUZnPOuZ6eHrc6cOX3zgKS3g5gCYBL0kASgDcDeAHAoQDmAPglgK/E/M60AEnOOXf33R4oHXRQLaPU1+fTrCTx5TnnMoEk53w1D/D3fpEgaPVqf1ftJMnWwzFVJbPT3LTJe+rmZt8rI+Xb3/aTd9ZZFYcWC5Kc82W0uXOd239/58RfJLmeHuc+9CF/+AsvzHbamkw7kFQuO/f3f+8n4PzzawPGI4/4vr099/TNLgHRxl4q+V6XpiZfWZKHv+YaT2bttVe2K6qmmox73e+80wOl/fevnd+BAefe9ja/NqN+J6t8//v+62eeWet3Hn3U4zPAuVtuGd/pO1dfnd+yZYt76KGHXE9PT+MZpVJpDMlIFFkujwGk3l4VvaedH1X1tm+v1ftyeYxtMg4/KVIqlVxPT4976KGH3JbAfSTqBZKaGvJfJ3US59xNAJAkySsB7JXy8fMA/JdzbvXod74E4PsAPtXQk9yR8rrXAXfcAZx6KrBokf/H+pNPBn77W/9v288/D1x5JXDuueM6/IUX+r+M+9zngEMOAT7xCWDGDGDdOuDrXwe6uvzfyb3+9XUd1fSQXXYBfv5z4C1vAc46y8/76acDw8P+n9MfeMC/vvZaoCm7We23H3DrrX7pXvta/39Uxxzj/5vq0kv9f1594hPARRfVf2hTXpIE+OpXvTJ+4Qt+rt/2NuDQQ/3/jC1dCsyd69dnwYLMhy8UgKuv9v8k/3/+j1/OD33I/40ZmdarX+3/o3Du3AaMb6rLm94E/Oxn3u8ce6z3O3/2Z/7/IL/3PWDTJuBb3xq333n3u4FnnvF+5667gE9+Epg9G3j2WeA//gNobwduvx045ZQ6j2ucMmfOHADAU089haGhIUrSGyvDw/4P0woFvwFAqeQ7HgsF/ydq4xTn/OHpUPLwxeK4XFrDJEkStLS0YM8996ysRUOlHkir0RuAi5DOJK0C8E72eh78rc13STv+tGGSSJYt81ddzZjhU6yWFp/Nib8iz8okkTzyiHMnneSq+o5PP336l9i4jDuz3L7dsxr77z82OUcd5dy//3vNJWpZmCSSnh7fe1wojB1+n30m9O80NTLtmCQuV17p/06eJmjePOf+7u88FRchobGXy8798IeekKK5b27291Qaz72spppMeN0fftg30tOd5ItFX3+sE7X86KO+3M/9zutfP/G/MHFumuu8c76k+bGP+f8hoslZtMj/fVWg+zp23N3d/gapbW1jh99zT+d+/ON6DWDHC+rEJCVuR6DgCUqSJBcB2Ms5d37gM2sBfMQ59z+jr5sBDAFY4Jx7Uvn8hQDof+sXAXi4zqc9XWQegE2TfRKTIDvruIF87PnYdz7ZWce+s44bAA50zs2Y6EEmjURLkuSXABYbb9/rnHt1xkP2AJjJXtPzbu3DzrlLAVw6ei7LnHOvzPh7LwnZWce+s44byMeej33nk5117DvruAE/9nocZ9JAknPudXU+5GoARwC4YfT1EQBecM5trvPv5JJLLrnkkksuO4EUJvsEQpIkSVOSJG0AigCKSZK0JUliAburALw/SZJDkiSZA+CzAK7cQaeaSy655JJLLrm8xGRKgyR4oNMPf4XauaPPPwsASZLskyRJT5Ik+wDAaC/SVwHcDeCp0e0Lkb9zaZ3PezrJzjr2nXXcQD72nVXyse98srOOG6jT2KdF43YuueSSSy655JLLjpapziTlkksuueSSSy65TIrkICmXXHLJJZdccslFkRwk5ZJLLrnkkksuuSiSg6Rccskll1xyySUXRXKQlEsuueSSSy655KJIDpJyySWXXHLJJZdcFMlBUi655JJLLrnkkosiOUjKJZdccskll1xyUSQHSbnkkksuueSSSy6K5CApl1xyySWXXHLJRZEcJOWSSy655JJLLrkokoOkXHLJJZdccsklF0VykJRLLrnkkksuueSiSA6Scskll1xyySWXXBSZciApSZKPJkmyLEmSwSRJrmT790uSxCVJ0sO2z7H3kyRJ/iVJks2j21eTJEkmZRC55JJLLrnkksu0l6bJPgFFngNwEYA3A2hX3p/tnBtR9l8IYAmAIwA4AHcBeALAfzboPHPJJZdccskll5ewTDkmyTl3k3PuFgCbM371PABfc84965xbB+BrAM6v9/nlkksuueSSSy47h0w5kBQhTyVJ8mySJFckSTKP7T8UwCr2etXovlxyySWXXHLJJZfMMhXLbZZsAnAsgJUAdgHwbQDfhy/LAUAXgG3s89sAdCVJkjjnnDxYkiQXwpfo0NHRcczChQv5e6knoxwy+J7cx1/T89C+tPe0fc8//zwAYP78+Zm+Z/32RN7LMj7rs9brtP2xYq273M9fW+8lSYKREV8Vbm5uVr+jPWZ9r9Gf1x5j3wvNT9q+mPeA7HYo96c9j9HZmMd67xvPZ0KPafvkc+112v4sEqMnMXYZa3fWY8x72mcbZXtZbDP2OZes+7lk1YcY3ZqI/W3YsAHbt2+fcF/ytAFJzrkeAMtGX76QJMlHATyfJMlM59x2AD0AZrKvzATQowGk0eNdCuBSAFi4cKH70pe+hEKhEL0lSVLzKJ8DqHodMgb5XDlf81E+L5fLAIC/+Iu/gHMOl19+eWW/fCyVSlXfS9tKpVL08/E88uOMjIzAOWceW76XtvEx8jHTcbQ51F7zR9qvydatWwEAs2fPrtpfKBRUHeD7LV2izxSLxRod1HSzWCxW7afXfD9tfJ98X/teaF/oGNr5WDZG72tj5TZl2aAMRDFBJiRpdijtketa6LXUUdLHWN2WG9mOZbsh+007Ftme9Vr+LrcvzQfJ+eB2Je2Q7+PzL59bYvlbaXt8v6VX1nuWHUo75RuAoO3Ebtb3NFvTzkc7/1C8015bmzbXch2011xCoEnqxQUXXJCqDzEybUCSIjRbNKOr4Zu27x99fcTovlQpFouYPXu2qRwhRQCqDQnI5oA1g9ccLd8vnYt0OOS0hoeHAQBbtmxJBTwxQMd6X4KZkZERE+iEjjMRpyzBXhbgI+derosmIUPnTgkAWlpaUh2rdKIhgBMDYrTHJEnQ1NRU+Q495+9ZwMc6bhowshy2HHPICcfaWZqtpQEczc4se7PAQJqNWTo/nkQibZ9mb9y2LGAk7Yvv08YswU0WO0uzNbIzbb01YEM6Re9zAMLti/ZlBR7SnqRthOwvdl/a7/D3YxKNLEDHmlcJdMYDZOoR0yxb05ICin8TlSkHkpIkaYI/ryKAYpIkbQBGABwDYCuAPwKYA+A/APzSOUcltqsA/F2SJLfDA6iPA/hmzG82NTVh3rx5qmNOc8qWhJgeK3Oy2B7LcaUBnqGhIQDAc889ZzrU2OexDllz0GlOmd6jeZAgJ+R4NUfM518+J5EGz52s3GcBG/lcOqdyuYxCoYDdd9+9xtmGHF8I2FhOOYvTDYGrNHAjn9OcxWaUWe2IC+mFZktSF6SNWQ44BHI0m+KAPw3whGxDAzCh48RuWiDh+9Lsie+jz2YRri/SjqQtydchvdKATVoiwV9nYU45MIn5rmYn2vvWWKxEXJsrOadZwAuJXH+55mkJQyhOpSUSMXEsNtnQwBH/fnd3dybdtWTKgSQAnwXwBfb6XABfBPAYgIsBvAzAdvhL/N/FPvddAC8H8NDo68tH96VKf38/Vq1aFRVkNCOyjAHQyyKh4CGVXL4OoXT5+rjjjoNzDscee2xQqWMAF1fINDAlGSX5nrVpn5WslPVcA2KaYfGsRANhdFw+jzKAyHXgr/l69ff3I0kS9PT0BAMFz341wMH1TT5mAVjEFHHWSANdoc36nPztLOxT1iy43kAsS0ITyypZbEwIPKUlJnKfZl/ydRb7sYCcc67qM9xmpF3R3IVYXHrMCsC47fC11cBEGtBKs6O05MTSc+05tzXNftJsJjbR0UBaiEmSc5dmSyF7SotLWcgB/jwGNFm2dOutt2bWL00SLcve2eSwww5zS5cuTVUQS2TgTEPfPFCHULWlDNxhTQSUSEca+37ouQZaOCjRxh7LGvF92vyTWCwREO9Q0wBKU1NTDRuU1WFqzjL0fgxgiXGsIZBvBRQNpMj5TbMZ6Uhj7SUG2MfYCtdHzYay2kfILizgYjl1HhC0R24baSBEAyBpfl6umWYnmh5I/QFQBS64Xk0EdGSxizRAL5MEK6ng9pFmL3I+JgPAS/BugQvJvITAcgiojzfeSDuUwN2KJ3xfjL089thj6Ovr23katxsp/f39WLNmjYnUY7JbnuFYwYScCOCveLIkDXRpDtHKbC0knobGZRDh2WsMQAtluFYGrYGqNAAJ2MHDCiTaHPPjaUJrF5PFpgUTLbhoWwjgxJTg+HMKNNpxrUf5OzGblq1yoJUleMggQoGKr1vITmhdY8BXDBNk2Y5lQyFAlPZeTO9Q2mYBTMtv8Pnk8ydFA1x8reiqTm4zVqIifaWmQ9rzEICx2CEL/Egdj2E80xKMLDFDG79MQuT8afNuiZZEclvXbEfqRywQS0v6s24ygQiBPfn6mWeeSZ2bGMlBErwT7+rqinL6IUAE6Jm1JjIz4PtCQEhzeppDL5VK+MpXvgLnHD7+8Y8HwYkFZrSs2XpfAqhYaj9kCJrT19inGMZJc/YkWQAQfVbqiGSbCoUCNm3ahCRJsMcee6isUwzAkZm0zIJjMuW035TnlgUEceeeZg8xNhFrD5bzDulRGkCJyaBD+0OfkbYhM3bt/KxxcJvXwM6OtAeZBADhcpYEGiGgHsuWWvbAj2eVvLTftvbFgh/NHrgtNMoerBgRAi+WTYQS2onYQZqdWfbAY0esPfT19Zlzm0VykATP6uy2225RWW5IrIyWb5z9sBRZa0aTihSjtM8++yycc3j88cdrHHZIscfLCoUUWSo0zZMFdDSmxxLNqXMnrYEcek86Pc6ipDE5oefFYhGPPfYYCoUCFi1apDr7mBJBKHCkOfEQwLEyWannVtZqBVwepEOZaYwD54DHcqBZQQ7tm0ivjnTeoc0C9VLv5TxK/ddEC75NTU1V+7iuS72PZWosJiYW4HC7COkxZ3VimSB+vjGMDZ+DUGKbxd/T+vL1lP5MA7Ix+h8DatLsQsYLzVeHALql7zJG8XHKxxBwt3y71H1tTcg/UfsD7SOdqIfkIAlAX18fVqxYkRq4NGOOCUihcgMQ38sh2adQJlkqlSr3R1q8eLFpbFmyBA1cyX1ZX8eCs/EEJ96ELR+1IETP5TpodLcGNCQI27JlCwqFArZs2WIGgZhsuVgsqtlyoVBAc3Nz6ufG2+eUFqykvmu6LgNUFqZV030LeKUFnLSAYoEo/t7w8PC47cI6pvU6FpSRnpNucz3XAhK3gZDwxMNikiTgkuwRf6QgFlMmjunbi9Fpy2a0fSG950yrllhx1kgmHll0PkbfLX+v6XsIAIV8v7Y/LbGO0fU0v649cpaIQF1M4gHgpXsLgMmQzs5OHH/88RMCL/SoKTLPSNOy5jTl1ZSQFFgq8ubNm+Gcw6233lr1Hnf2/LuUXWQFLbSPj5EeLYUOsUb8uUZNS8AinRQwllnIYM6dtwUKYhyzBCj8PXpOwPuEE05QnXZaMLAy8ixs0XjZ0SzOmtaV9EfqtAVQYhs/Q0Bb0/2RkZGaY2v70vRa2m1It7Xnci5jdJv0OJYBLRaLaG5urtFvDkokm2PpWwhwh55zYMMZozSQoiUL1ibHXo+E09Jvrtca80k+NMQ0ZgETabotfXZoXywYsXQ6pNt8vjSgbSWZXKf5+7Seml7LrampqaKfGqjmn6P3Nm3apOpAVslBEoCBgQH84Q9/MKldbfE0FsHKHopFT/sRFc4lDWgBtVStzB64gvN9XV1dcM7hqKOOUg0nC/qPBWwhA5XGamU/POhaQYje1+aQzyM3Zmmk0nBDrJAsQWjgSz4+88wzKBaLaG9vD2axoWw47TvyuaXDlv7KMcYCqxDAouM65yo6n8YChbLiiTJDUv+4vmZJAkK/r9mjHE+IDZB6y/U7JJzxkbaTJEklm5asR4gVkexJrL7LR8mcakxq6HkIPMlz5c9jEgVLh2l+uFiAgDb+e845FItFU881Hbd0PqRzlu6n2UjWjbNRNBcyFln2bemxTBqkyHXhOq6t48jISGX+S6VSZT3onCcqOUiCn/zW1tYaB5EFGGUJIhIQ0XNa1BAgCoEi6dD3339/OOfQ09MzrkwmNmOJBUny3OUjYJcMJBAKBRBpVCQ8c5HrapUKQoBEZjMSzPT396NYLGKXXXYJMkgTvbxZBpSppL9WJs6dbxpgSQPoafteKvrLQa0F4uXriehvrL6+VPTXYgY18MF1Pg2oZ9Xh2PJtjP5qejyV9Vfzv1KXZeJo6W/oCvIskoMk+L+N2GuvvTJlzllpW6D2BmzjMbAspYfDDz8cpVIJDz74YNVnYowwxuhiadrYrELLIEJMTwjccAPLUnIYT1ktNmBoRi6zZS1AWJmxNkeWaE6OB4G0wJCmt1mDQ6hnJ1TyjQExdJ7yIoI0PeV2rOmqJXwNeMDnTLKms5IxkbprBYRYcBMDUPimMT0aWNHAi8bmaL5U6ii9liwwXwse3EMMjVxn+k4okUzTLe112r195PGt+/yENhojB2RST6U9h/STzzXXS97wTDqgsXRSZzWmMc3HhXyx5ROtxzTdTJIEd911V5T9pkkOkuAbt5ctWxadFcU4kSRJKv/+noXuBbI3q1qBzApcVr+GBbbofWt/qTSWrWs9IqFM3rqBmZbl0HmnOQz+nM+xnHcts9HAl9ZsagUbKyzRK48AACAASURBVMumYxDg4vtlP4e2LwagxTiWGPAVo6dpSYLUUSsjt5IDCZY0gBWjt6HPW2BNsxELgEkdlUEuK/jSsvC05EACLQ3oaMmApbsxupf2Ge21FRi1gCv1U84BnyPNxjV/YLFGcr2yJAZZEwKrhy7NZ2bpqZO6aoFIrq9ZdFTqa0hXLbZIu2lnGoiP0TH+ur+/P2oMaZKDJGRv3A4FCG54mqJmNTbL4Q8PD5tAhvbddNNNcM7hLW95ixkorCZu51yNUVrZkMx46Dmfm5DEBgSehaRlJpbhhEAJBzLyc/I9K5jw1/fccw8KhQLe+MY3mlmXDAgh0KLpoqaTIcBC60jrznWRZ8hpTl7qktQvDpqlXlqAmgcCi02K1ccQU8QfpR7KOY4B0sVibfN0CJjw5mYOmi2QrOlpSJe5HmqASAakkD6GWExtvjRgkjXJkwDBSvSkD9N8nJbgWe+F/K0FoEP6GMNcWroodTLkGy02RfqjNHBsJXPj1UVNL61zCLFEQPh/7ixdvOGGG8x5zSI5SAIwODiIxx9/PNVxWFmNRSvTMYD4rEY6Dw1wSaBlZRi33nornHM48sgjU4Oc5hBism4r2+aPlrPgwCrGcRBwA8bu6gtkdx6xZQ0ryFhBT25r165FoVDA/PnzTacQ4ySyBK80gC/nl9PoxWJtw2lals1LWqEEIEv2HSphpOlaLJi3WAQ5PzEAnz+SU+e2qoFgrod8v5VhW5l2SI84o8R/J615OrTF6F8MqNf0MY1B58lRknimvlwuo7W11QRclu8M6Y9MFKSuW8fgNqK91vSOdI5ey/HTezEiGTb+3IpXMhmmJuihoaGadQ5dWav5MHotfUyMnvENCIMkLQYDyJmkeou2uCEnQd8JOQV6TgZOz+k9LcvlTjsm+KQFGucchoaGzEzKyq5CoCotaGmliDSABKQ7A5p/mj96XS6Xq54TMOWBSQsuacxTWlaVls13dHSgUChgzpw5mXqV6ql3JDIIkfOmeddAEInGLkm2Jy0TT9O18eidDGQ7Wu80tlMCcMk40Rqnlbuk3qVl9dpjCMBrQCuL3tFcxLBJfL5j9E76E/leKIGrp96VSnqrwHTQOwme+fuNaglIK4VpemfpXj30rrW1NTi/sZKDJACtra3Yb7/9ojIhmWFqFHKa4VtAI82oqcTG94W27u5uOOdw3333mY7DOo9QZqZl31nLGFzxqXlQy0607Hg8wEYz6jTww48TAleWoa9btw4AsGjRomiKWNMzOdfDw8Mms6NlwiEgnRZkpM7xwKGVJEL9Z2kB0Aoo49ExcrA059QfGJMJk06GQK3WzC91Rgs4IeCSFTjHMDpyftL8GG1cxyQrYvm1tCQuC1OdVnYN/a5W9sqiX5qOabYqgYr0YxpTGAKpaT4u9H6opJoGiuU5Wrolxzwe/eJzzddU6hfXOc2nxejc1q1ba/zEeCQHSQB6enpw//33R6HjWCdm0YAW/axl+hrokk7AciClUgm//vWvAQBve9vbzFq8tdHN0rRNC55awCUnJxVabpSJhQIiZ+P4PHLjlZkHdwLSQVk18jSQlWV74oknUCgUsHz58hpQFpNtZdEty2lpbJIEXjEBUOqWBbDSekFCW+gYoYAZ0i8eOEP6xbN60iMrOFhZvCxJxAQzq7dDe532OQuUxeoXT1y4rqUB+zTARaDE0rEQsE8D8jG6Z/mqGHAv93GmaGhoSB1fmn7VS8csQGT5l5Bvo8/EJI5Wn1GIJdKAGB+rnIuYxJHPtebHvve979UcYzySgyQAXV1dOO6444JBh4uV5dMjGZ9WLrOcQJZgkgZiaCsWiyiXy7juuutUh5GWnUkKGQg7AikhB8ADicYcaSAmLXhoGxm9bMZOe605F8sRaCAmSRLMnTsXAHDiiSeOW5c4MKa1q7cuxeqTFnwkozSZusQ37e68WvIzHvAbq0sxyVYIDMcwROPVJe6XpE/QwIkFZi29yaJPEqRwYDwVdImDk6amJrS2tgZ1iesfPba0tEyqLsUkVhPRJQvgDg4ORiVSId0h/YzRKa5LxORPVHKQBN+4/eSTT1YUTWaEmtOysizOWABhZbMy+JER/bJ4K6O2lO+ggw6qcWj0/dD3uLOiR06POlf7VyvSWUm0T6I5Ljl2mkuNHeLrE5O5h7J3y9FZ2ZgG3ELZ+fz581EoFPD0009XnXtWllHqEVBdTmpqakKpVEJLS0tmRsjaNB3TdE7TFb6WFsvDX8tyiKU/VvBLK4XQcwAVvS2XfR/b0NCQuX5Wpq6B+Zj92rE0NicU4EIZ+ETKaxZ7zXWMHi2dylIOSfus1BEOkGSQ5udMfkgbn9QdjT2U+qQBCrJjje0h4eeRJP7O5yGGRWOJpH9L0xMrXqWBJA4YNd3RRLPNWJ2S+yRb1tTUVBX/tNho+RK+rVixInUcMZKDJNRetk4MDFDdYK2VfGICnXRMhUKh8puFQqFmwZMkqXrkQgwRPxY5fHoeyhKkQ7FAmAyMPNPTsjqu/NJRSaCkCTdgWgMOlGiMzvnb/tM8jIyMVBwJd4byuBIgaZmctVnASgNgFsUswd549MZyMlowkU6ar73FEExk09gADUhZzq8eeqMFLS3YkA7RetBxCARo5QftMZZ5StOZNBbA0pusLIAMVBqIlvrD14zWOFZ/tMxffk8eU/qiNL2RDFIj9EY+T0u+Jrpp/kbTzSys9nj0xgI4XGd4UqQlVwDUZJ7fwkZ7neZrZIyS8apUKqG3tzdVD2IkB0kA2trasHDhwmhFCqFljnRDRh+iHDW6Ue6XzzklSY/33nsvnHM4+uijVaaIOx1S6JDzofe5yGxWOhxJW2vZ9HhKIJKWtuhsK1jVI1DRuC0doVswvPWtb63oiHZ/Ii1IhdjCegGakIMZD6jR9CNNT6g0IQNUmq5IEKMFGLn2snSRRS9kMOLPrWyd6wfXkzQ2RwJOLaGRYFfTFc3XZLl/laUbUi8s/QixgpZYyRKtH59zCyxoaxhaZ+1qriw6oQHxWKBr+RBLTyTYlSAlza9oaxrSH02XtGPG+AypHyHdsNhiTUekHXI2rh6SgyQAvb29uP/++6OcaMhZNjc3o7W1NdUQLCcpEbvM4iwHZgGt1atXV4K0Bra0R63+awVVDXDRc+d8YyNQ3S/Ax02izZFkkuQmnaLMukIMAK0zB1qhfbGgizvoQqGA/v5+JMnYLQpC2R2fC1ka0PSD5jjkDEPBMpTtDw0NVQVV+XkJyLXjy/PQ2EcOtDQdkWI5ylgdsbJ+DVhZax/LIGm6GGIApL5nYRvT/IjM9LlP0QKipiuhzD+WMZJgTOqHFmDpd6QdWHoyHh3RWCLNtid6MYeWCFogTCuvSf2gRzleqR9kX1JfND3RYk4IWGkxQuqIFaNikzUZa6T/sFhoij8TlRwkAejo6MAxxxxT45yAcG1fY5GszC8N3GjMkHxPgpkQsBkZGcGmTZvgnMNNN91UdX48++BGkSVoSeOU2RIPKLyUEeN0ZLOjBDD8UWuM1AKZdEb8OXdCFkMwHn1oamqqvB4aGpp0fbAYgx2hDzIYyYRkOujDRP2DxQzx+5jtjPogS51tbW110wf6jmSapU5Mhn/Qkh2pA9o97jRgSp+LSYYpCZoO+sD9A2+Y1/RBJsDXXHMN6iE5SIK/78y6detMpsii07lhcAMCbMSuZf+h4JmG0q1MbmRkBMuXL4dzDgcccEAws6NHyQpJg4ihSKWTkE5Bq/3L7FrL5mKyf/5ezH2SLIZQ6oAMkjHBEkAlk6FLhWkugOryU6FQqAJVWbI4iyXSPmeVU7SsnmfzsTR5FvYHqL2BK+3XgkehMNZkbWXfWrZu6ZK28ew9y/rHsD3cJ3C7iGEIya4ocJDdSJu1snFtjTV7T1t7OheZtfMtTQ+kz9TKr7ycpr2mRxmkR0ZGgracttbWmmdZf27bNE45HxqrHooV0va0NSIdKRbHeu2amppqQDn5dHkM0gMNwPHz4fOtnbum95Yu8PnS5lOLu3IdLF0J/X5WyUES/IIPDAyoztW5sUbrYrFYKZ3weiig/8O1lUVoypgGlJqbm6tATnNzcyVLLBaLFbCkKQv1fJTL5coYyKhkGZBnDPXKHiTtTc951sCdFwc4IyMjFYDDnUJzc3PVvBcKBZRKpYqToPfk8WUmqVHfFIxCgCnkJGku2tvbAQAzZswwHZ8EQzKDGxnxfxMwMjJScbgEqrQ10zJFq5dNA1Ta/2ZxHdAcZmjtaf2zZo407/TXE7xviY7JGakQi8AzzBCwthgE6YxjGQQJiCTQCNm+XBtpfzGMgWScpA5oiZMFoGRADgHk2JKXZBfTkiGaf21dY/RAvtZs3wJRWoDm/keuPdeBWL9vsYga05PGJFrMkfYoY009WSPNXmL8vpUEa+tvsYr5HbfrKG1tbTjwwAPNrFAqPVd+TaHSFF5TXr5xB8e3EKWqGRMF09/+9rdBpbcyARLL0WkBLk3JLbpclky0fZajS7uBnhXkaI2t7F9ba+5EtExdC2zLly+PKo+EPmOtscUc1BPUFgoFtLS01Kw9zW0MuyfXj6+/BK1WE34aI2Rl/tYaW+vNQQ1n30L2HQpsGkANrau2zhYzJNdZ2jIHa3KtQ+vd3NyMlpaWqjXWmLu0ddcY3tBr+Sh1zUpY0thdy4dLBm94eLhyb5+Y5FVbdw5AYmxX69GyWF2N3eFgPOS/aU60uWtqaqpab4ulDa176HUaw6uxfhprJPVV+i6+zh0dHUGbj5UcJME3bv/+979XM4+YTCPUsM1Fyyy5kWpGKQ3RCqQWoKJH/tk0sCWN22IWsjAKaRllTMC1elA0cJUGsqw+FWudCWARo6EZqlzrgw46SAXUMVmkVl4NASvreaj/gK+vBu60tQ6BaW2d6blW9sgKsPhjCGRrwMyyYwmyNOecBrQsgBVabxkgLVAl+4msRwt8aSCOA3rJamnJE/ksa53ptbZxtoiDndgm6DRAHXPzRSsQczYyy1rTo8UQSlBjASzNt1u2KnVBPkobtsAWbdpFNVa5VIJOixm2AJYFsrJsWS6qoi3kp7JIDpLgG7cPP/xwM/u0GCUyDJ4JpNGnmlOLBTlaYLSyUs0pxlLnXEIskqTLubJazXSSIrXATQjwpLEMEthYNDk9pjGFNG8aPS2dHgcnaWtsbRLoyONzhyh1LpYh5FkZoJfBaJ04mNEclwZSKTPNsqba2vKAmoUxSrNZC7wMDg7WNTGZSjarlbiamsbuIB2yWVqPRtgs36TNShYkq81q7Ayt8UvRZrUEZCrZLJ2rZbMcoMXarOaLBwYGqpL3iUgOkuAbtzdu3DiuRaVH+o7FHGgGy7PJGKdsGbL1nZ///OdwzuHEE0+sUiItkwyVZTQhQ6XPA2M35SwWixgeHkZTU1NNs63GHGh9A1mzx3owBGn0PA9S9MjXn89NsVjEz372M5TLZbzqVa9SAZXmTHnAtDJFrjsaCynPNdQ/wsct2QEZrKQz5t+l8+PPKWPVsj6t/yB2DblNxq5naE01NkBzyNx2+HzyuaCxWgyCFrRlWUUCI2mbnOGRQUWWIOQmyxfcd2mP2jpQMOPnRPZurZs8tgWG5Dry8XCfQyL12gJNIXZPm3sJUOn3iXUicKkxryFblI8hUCTXU86DxtiG1ld7rdk4HZ/HhaGhITUxsfyqZo8a4JXPNZDE11krNWo2XC7ntwCoq5TLZfT09KQyEzHZq3RefNG40nHHKx8l4NGylaGhITObbW72Td0E2pqb/U3YhoeHAYzdRbdQKFQcnZaxWlSspuAyq9EyGo1qlRkJz2RKpVLlOZ1Lc/NYwzatC72m3w1lNWmAiY8rloXQaO6RkRG0tbWhVCqhs7OzZv3479ExuRPh7FUaA6EBYK5rMlMl4WuZVjaRNiGZBw5qac3kPNLca9mqxlpopRN61NhCyRRabJIVMLUkxmIcODjkx+B2GWKUQmupAV9aRx5QNZu0SiIaYAmxDGSLfH419mG8DJJkPWITUgsIcT1PW0eNEaTPhtgiuZ6kC/Q9aYv892N8K19LLXEZj2+VLKC1htIm5fdjGCMtTo6HLUpjisjOeLzk60nrlJbox0oOkuAbtw8++GA1ewHCDdsWMyQNUjM4+ZweiQ4eGhrK5Gilc+3u7oZzDg899FAqOySdapIklUa+GKdq0bR8v/U8zRg1pxoKjtba8azcyupDjjQt+EngumbNGpTLZWzcuDEV3GgZKc+MaCxcpDPlG5VQOMAJ0e+WgxxvMOQJhpVoaBkmXzceRDQwInU+LRiGyinyIgoLyFi6o61ZWkmMP3KWjgc0zspIUGqB07Tnmh5YTB4PymlMD9dRLamgtRoeHq6UQ6wkUfpRK1nQwEoMCyuBlcZQaCKZVgleyPY6Ojqq5tGac63EKdcpzSdK36gxR9qaZUkiOHjR1k4+59URKx5q62XphCQXLDvj67Z161Z1DbNKDpIw1rjN2QypxFYZgOhXi0q0gjQZrsYoWUxSCFxJoDU8PIzNmzfDOf+3JJZz4Yots9aBgYEoZ8+pXq0so5VaQsFYA1MaqAo17lrsA50frZvl7PnaaaWYmEy1qcnfwuCkk04KZqdprIMFsDRgzPVKo6s1pkU6e401spx71h4V6fytAKCVbQhwpdkbPVpBOoZp0G6joG2hBEYGCx5M+DpJ8Kc5fJIQw0BgVAPG3F9Za5DGPFgJktQTyVjxMpXG2mo2FyqPyXVLS3BiH0OBWwNYdF4ExNJ8peUvZalL2h9nbCyALBOeEMCyEhlpb0mSoLm52QTGaSBLMtly7SQLH9piGFhpY4899pi5DlkkB0kA2tvbsWjRIpUitBSBZ0VcAbRFDzESFuDR3pMZr6YwXBGJSVq5cqVpxNJpcYfb2tpqUroWyLEAjgZ2+HPLGWslFo3OBWopeQBVxkoGS5f40vxZZa3xAFP+fO3atSiVSnjyySdrgmWIio8BN1o5jAP6NEBqrROtewigakGS/7bW76CxfZpNaYCGQEuaTYXYWetRAztaZqyVMTVd09aJnsuMX0vIOECx1kquSyiR0AKlBkA1pojGEWIaQsBlYGCgZq1Cfk17JFbdAqL8uZYwpNmUlSyEWFiaZz63ZDMh+wmtkwVmrGTBShSsJIGzMVZywBva09ZHVjv456Qt8dKYxRBltSkJLiXga25urhxvopKDJPj+jxdffNFURk5XAtWOg9OstGDSeWhUpGSKNMW0AoAWIGQG9GfPP4+/HBrC7iMjeOHJJ/HtPffEHXPmRJcCuEICY03ZsheDg43m5mYMDg5G3f8oSynAchQhx07nTI+ac5dOg88HHVeyhRw0trS0mODq6EcfxemPPop5/f3Y/Pjj+MFhh+GXe+xRw2ZIXdHWhjMKPAPn4+e6qmWksmTD56hUGrujNf0+Z8NCVL8WbCWItdZHWyOL/bGYIM3x0nPq76DzIt2mtZPftdaEnxN/lOfPx8WTLF5Ss9ZKrpsEVJLRJh/ES0q0Xhr44eBVWxfOUEmRuqitk0xEJHiyWDy5ljSPxBhq/jRU2oxhc7hIX2cFYo0ll0FaY9Doc0mSVM3N4OBgzbG1NdGYX25D3DdoQFDOiVw3OifL3jQwzOeayovNzc2mnWglMXneXDQ74uulERnaHK5ZsyZKB9IkB0nwjcxbtmypyhC0YBALnDQnYjl2KyOWKN1imrTM+KQ//Qkf/cMf0DaqlLsPDeEzTz0F5xxunz27xpHJTIuLVEIAakbMAY8sw2iZsGSZZJZFBiVLClYQkeujrU2WTFiCWA20aqxSoVDAMY89hnf++tdoHfFXe+3a14cLly3D8BFH4Ofz51cCWygLls4oZm20UgsBI7lGVjmTfk86aH4ci7FIA7WhUksMmyQTDcpQZQLB14Nnp/TZUPZrlcksJkkLxjHshFa6Cq2NBEzjZSe0tdGCswxiFitB68LnK405orXlPi6NQZKMhAS4IeYoxma00jJnYvl8kr/ibLvF9IXWRZbONLClMUZpyR/NB58nLZnT1oYzt1lZWFnR4I8WU8TXhfuaNEZPJtZajNHWfzyS1OtA9ZIkST4K4HwAhwH4gXPufPbeGwB8G8A+AO4DcL5z7qnR9xIAXwHwgdGP/xeAT7qIAR7d1eV+s2gRwBEsgKovOudfjz465/xzGXzpOTmV0efjfdSOWbXROZHhADhqYAA34l34DC7G09gH++BpfBmfxln4AVa2tdFcVh7NjRzo6PMC21/I8siOVTlekgD8t/zJVD+OPq8SNs60tdDWpOZ5aM6tddDWg63D4f39uNGdEzX/cszR6yHmuWadtPUS60DfketQWRs2/1oHScU+Im0jzUZq5j303LIHwyaq9IYJH1cV0zXO9bDshe+XdiA3sLWg55VzVQAmtw2nPI5rPZS5t3yRtIuq3zLOKyRV68DXQ+ioXIeaeQ2tj/K9mk3+Npv/KP+krQH0K/RqNs3nxMQDxRb4moBeA+G1ED6Y62LNvBhblX7LuQ+sW81xUeuTYuLE8f39eOSRRwIGEydTkUl6DsBFAN4MoJ12JkkyD8BN8CDoxwC+BOB6ACeMfuRCAEsAHAG//ncBeALAf6b9IPXvpBoRxharkCRAoVC3YK4Fbxk0yuVyVHC/ceAMXIjL0IdOAMBT2A8X4jIAwIHFW8ecGdj9J9g5c5HB3AwgmvNXQJMZzLXPhBwZnVMyFuirjIYvhViPKIdlBYwQ+B19fmPf28z5P6T1djOQlP2ChJVVOAjpxKMCRhr4lc+tddB+n6/P6HlWL4YbfwBJCeJZA3qNLfLzK5f9+Wh2YQUQOR/GmownkFuAqgZcsXNAkqAAAMQIanbBxyweo8FVhjkPBnb5m3Quo99LFS2ICr3UAnvN3GYI6DXzr/inytwXCiiwc60RBbzUzMM418UCtNamrQEAfzwYdqGtBXSAE1qPVMAV8kls/oeHh9N1JkKmHJNEkiTJRQD2cqNMUpIkF8IzRyeNvu4EsAnAUc65R5Mk+Q2AK51zl46+/34AFzjnTlB/gEln59HukEN+Paq3SeXRH8d/ZmyaHHtNSmU5d6Koa59rj54mpn3V749911BqUIbgMLd/D6zD3jXj3BPPYHPbc2yciXg+qpAF/rrAXtPzrI/+uTw+P4ex12NzT/NfraKOzf3YmFVnETnvocfqea9+zc+Br8EuA/b8b2l/rmqcY/OMqjmvXoeJzbtcT/l71efTGN3XdHrsUe4bv+5LXZHQYOrpvj73E9F9XV8nrvt8DeT6j61Bta7I8UwN3dd9Tqzua7Y/Ht2v9vfavL90dJ8fe0fpfn//CVi7du1Lkkmy5FAAq+iFc643SZK1o/sfle+PPj/UOtgo6LoQAJqbF1WakqXxOBdaOIx+zrHP++/4hfKPSVKGcwnKZYck8fv9YwGFQhnl8thjtbEUgsF6bKsO1OuwlzrmddgLHYX1NP7KmKoNRwo3PodyGZVzTZJy1TwVCjSmsTGWy2Qg5YphlMuawQCWo6o6G3YusWCJz1P1Wo79XqGA0XXwn6Nz9+tH3/c35ywWZYCoPpd1A/b8z2zeCOmYNcBYu2lZk5wXvj70GTeqw/S6BK7fMXMemncNLKWtQ63Tr6wK+/3C6Dz7NfBkCP+8U85Jl9rxySDBn+trIAOKvRY0F2WmX2Xld7PN+dhrS+81G7DWRNtPc+VtwbkiAD7vtXNee558vrXx1Y5f2r6dPNX65urndB5llEq1866dly1ybPb8a+sQXhv9GOSDquedf6b23GpFAlK/Lzz/E3kcez6m8wBQQu2c156jNhZL563nIb80NNRfc/zxyHQCSV0ANop92wDMYO9vE+91JUmSOCfNGHCecboUABYuXOj++Z/vMZseraveEqEBY0GguvHUatSWl1HybXBwEIODgzWveWMqvebNdCMjI1i+/CYMDe1eM4FNTc9ht93OqWk25ePhV2/IS1/5vPDLXFtaWtDa2lrzXD5aTdpWg6l2BRvgnSMtqXZ1jXYloWzs5XMp51l7tJoW+aXptN6PPfZTjIzsWTP/xeI6zJ17ZnC+ZaM1NYjK+eNz3dzcjLa2tpr92pxrVxPy5kjZZK3pN+m4dnWSvCmjbG635prvt5p5eQMvNYRal3zXzn31lYBao67VOB2r2/Rcu62C1HFtvkm3ucgLQHgzrHVxgTaPabrN7cG63F5rlCa7kxK6TFvzKdqFHFK3tbm29JsuHklrVpf+RPpvPt+yAdqab81vy/e1i3D4sWXc0K7ki5lveTWu1G/Lp1g6zu0hdKsJ7QpLK2Zq8235cO0CpoGBgRpfPjg4iKuu2vluAdADYKbYNxNAt/H+TAA9GkCS0t7ejsMOO6zGUXGF5EFYu4Q/ZCxkKAMDAzX7aYG1YKzdH4bOgxsNlyRJMHPmV7B58/+Fcx1sfz/23PM7mDdvXpShcMdEm2Y42pU12tU0IXCpBQLrKjM+P+Rs+NzRHIeCrxZ4Y+6Hw3WDj4UH27a2NixYcBnWrv0UyuW2yhiLxQEcd9wteMUrFlcFAG1uQw5pPM6IA0p564Ph4WH09/dHOf9QIJBXlclNXjlI52A5fT9n+k0uW1tbVeAurzqS8yj3pd0Hil96LwE7SaFQqDp368o8a541p8/nVbsCzNJfeTuDrIFV3uKiubkZ7e3tNVffafOngRV+7yArIcpyFST3xfLKVO4n+Dxbc8bnUyabGvDkt1rgtysI6bCVDHGQTvNK/pZea1cHx+qt5X+tK4B5DAldlS1tmGKXNndcd635l7FUznGa/mpzzIHhwMBAzefHI9MJJK0GcB69SHxP0v6j++n9IwDcP/r6CPZeUHp7e/HAAw+Yl2zyzE/eN2f0XACkM0mag5SBhgd2ep6Wccusb+7c/0V7+5fw3HMfQam0B5qansfcuf+GpqYfY9Omsf8Jo3PPkmHz+eHshZb18eCkXRIr2QzpQFpbW9WAL52klelxIw0FfJ6JmJ7xEAAAIABJREFUWAydPC53GnydBwYGUCxej/nze7Bhw8cq8z9//jfQ2/tTrFqlByP+XHOQWuasBakQmOJOkxx2S0tLlQ6H9DiWpbMYJHlpsRaUZMCXwYhsRBMrm+aXWXOWwWKO0oJR2vzKgNTW1pbKymlBP22utXnnz+XnZeIh7w3FbWl4eLhynsznVvkKoPp2IJItknot54+zpGnBXvpgCarIRqzkIKTPcp45sJUshsaWapfAa8wbT77K5TIGBwfR36+XhDjbpemzdqsCPtecSdPm1WLvpQ7zeSZbsJIwCWJDOq3ptuYHJEiVgEquI//NCH4kSqYcSEqSpAn+vIoAikmStAEYAXAzgH9NkuRMALcB+DyAB51zj45+9SoAf5ckye3whcuPA/hmzG+2tbXhgAMOCN5rRwZnixYMZeEWAAqVGqSTk/QrLy/4/hl/vq2tN2LhwpurspempllB4CPpbM5waM9jg7PFJGkUNzcGuq8N7ZMsksXYxYJKC/TEMEjckWjBYP/9H0Vz818LCvtVNayRLCVItoOXbGTASJtXqwxpBdiQrkqGjj9qQVpj5qSuynnV2Aya046OjhqWQisPWMxnmr5SgOD2YvkA0lfSFRkc09ghy/41dk6CHsl6aj5Am9dQ+YV0kHRNm7u2trbUMkxoXiVLxNk4CyxKJpnmtr+/39RFja2Xcxtikq37lLH4BKD6nwk4aNFYt/b29qBfjS3dEvCRCY/FvskqSKiMxfW1t7c3k1+1SrUaINd8K81r6H5vkpGU86TZ/0v5ZpKfBfAF9vpcAF90zv3TKED6FoBr4O+TdA773HcBvBzAQ6OvLx/dlyrO+f8pkyg65CSB2syKFpZ/RhPeW8OpwrSbm2UNPMQYtbS01CidFXD592X2RY5L9qvwQNzU1ITh4do7/8q+Iv5dSaFr1G4aY0GGKueEzyWBSHotwa42l3LNeAmIBxwtgwZQCfAaI8Ff09xwoMifS/2werQ0FkiWfmR/XIip0DJqOjcAlXGSnklWwrIFzXbSsmUtAw5lxtyeCcwQGyV7gSzdJDuy5lQmS1xv+Txr+7k90yPpGAEXCwRJW7fAJrdtObcy2FrPtZ6eYtHfCZvsT+vzkcL9p8Y6SIAkAb7GGGjJjQYeST/a29urdFIDQDIh4nZvlSglYNEYRcuG+XPSDyoVafPI55Key7KUpq/aPmvjYIYfmzaKG21tber5kUg/JeMAn1fpE/h+0isel7VERj6vh0wqSEqS5Dh4sHOSc+63AOCc+ycA/6R93jn3MwAHGe85AP8wumWSoaEhPP/88zX/uUPOlwMMK3OXCkiBRNK1kgnhKH1gYKDyyDMiieI56g9lQevWrQMA7LbbbjTfVQZLmRDPLDWkrm2UWdJzi2XitLp0vDxIWYydLKVpbJ3W+2U9l1mlBRpiWCUOmLTy5KZNm9DU1IQFCxbUZOTaPNJr+h2aIwmqtLnUApRGfWugUyv3anpJW5IklZKXZE34MfnapQF6mZmHskepa21tbTX6yOdO60fijIcsM1j2HcrINQbZmsuBgYHK2oyMjNR8ho5FJUqNPRoZGeG+L4qRs3qILH3kZW+L8ZD2bSVH/Fw1nZQsp8XCy7nkZW6NAeXzmNW+JQCSwFxjKLV5tPwob0uQeqnppJacamyxxhDxuCHnkZg5mitu7xZbpLFEWe2bzyVvz9Dsm15LfQxVNrq6umIhQFAmm0n6E4ATMdZHNCnS2dmJI488ssqYgdoyEDk0TQFlWYKcIXeMWvAOGbMsrcmsnKNrMqa2traK8m3atAmFQgEHH3yw6gx5gJGBRnOKPLBIhygRPaCXe2QPCo3bcoRkwFqwlr1EkublWSatp8UCckdIBiv7qNLAotzuvfdeNDU14ZRTTqmZP9mDFSrvSiaov7+/CnhbZTI5XzHlXRmQ6Xf5OfFMTWM+m5ubKyUGms+00lioRMZLjnz+eCAuFouVc9IyaslO9vX1BYGNNpcSlIdKDDJw8Kyfr7NWBqMALEuMsgcwVP6SiQpnLCkIxzDmFrs7PDyMnp4es1RrlbrSSt8aMyT9nsVCSkaxtbUVXV1dNWVEDThrpUOpc6GEWeqeZrsagOnu7lbnULtiKzRvsj9HY9N43LDYW9mz19LSgq6uLlWvQraqtV9wPQ+VCK2ERJs/Podye/HFFzMiAV0mFSQ55zai9rL+HS69vb1Yvny5ika54yfnpSH6tIxdInKLQdLAQEyWSb/b19dXOaehoSEAwOOPPw6gGs3LTToQjfmQ2ZHsp6HvasZBW2trK9rb200wxTNljfFIC2hZezx4qYPKB4ODg6lgVCv5cMfa2tqKJ598sipAaU3tUt8kEJXMGx1PyyhD2bnGZsp51LJOei3LblZGznsPSGRQ4+BQOmXJmlnOWX7GKrtJMFAoFCo6HLJfzoJZ5UgJlKyeF3otWSEtqJHdUrmFs4OyF46zZTSXUof4vHLdlO/LuZNMBp8/Op4s82r9Lxq40hi4tE1jgqz+TPpsX1+fWY7UmA2N4ZA9RnJ+NXuVTJDsc+PzJ3VQ6iG3KwkUrPmU+iqBFFU5tPIl/V5fX5/acqABVKtELjc5V1rSrSXgsvJAemsBLIop3//+91Njf4xMdrltFYDlzrm/mMzzaG1txYIFC1LLQKRg0vjJAYboYQI+FjMiA7kGgtIyAw7mmpqasGHDBhQKBey7775m+UyWKyw2SQZyjVq36GAeuK1gEyqZ0ZxJ8JhWfsySTfGyAQ++GvMhWTg+b/w7v/rVr1AsFnHaaadFzRk3cC0DlcHZAtKcucxass06ZyGQrc1Ze3u7qmdaGYeXHrLomVYK6+vrS7VNPmeS6ZUAaDxzRoFSgheakxBTaTGWkjHipS/ZB5NlzqSOabYZag+QvtAqG6b5M57USSDd1taGjo4Ok9kN2WbanHGGzbJNrQ2Aj5uYtjRdC5UItUSY1s6aM1pzbkf0XLZGhOZLY9s4uOZz1tTUpPadamyuVg7kDNrQkG8eT9M1GTckI/nss8/WBvtxyKSBpCRJWgAcDP9HtJMqBIBko5dGP8vMSWtq0zb6jjw+V6xisYhSqVRROpJCoVBT9uDH0BxxS0tLxdA7OjqqDMSi6yV7JgM4KTqfs5gerbR6ubZJFkjSzJT90PwQS9XU1GSWJ7USWygTl2wGBwByv+wnoCy7XC5XHJ7mRKQupWXgEihpTIaWpdPcO+cqpSmaM96XwXUuVJLkepaWcWtZuczgkyRBqVSqrLVGxVu6pQV6rc9MYyRkaVuyY7xJleYqJsOWTAUvr3J9szJv/h0JEmlO6NzpMnIZ0K2yI58v2TcW85wfQwbuJEkqNgKgyka1ueLsGLdJPl6NUZAslwTwvB8qSfwFERRopQ+VLA6fK03XaM44a0avJQvEfRHv16GtWCyio6MDHR0dVWsoS7KyUVmCX613SX5G2rT8HS7kS7q7u2v0Sc6TZBGljWpxUpsTuS5SSLeoWZyPh4+PHl944QXzWFlkMpmkRQCaAayYxHMA4BXimWeeqQAHWTbSsn8SvuDSOcuShZbp862/v7+md0nrvdEyMq3fhkoKf/zjH6syfp6J8SxCbu3t7TX9SjIr0xy97HWgedL6Q6y+EK2fS86R/Axn+XjGz39T6w3RmBHOjnDGTZsrOU/0uZe//OVmnV72NdC6AdU3JLTKjiH2aGBgAH19fTX7aF5C2Rfv4SIHpgEAWZbQbj4q54hKrfSoNQbzueJAQoJx2bvAWV7L9jgjSXPE547mmfZpAJ3buLQ9ru8S6MgeN8niWvpULBZV20vzUVY7gMayyUxe2h+3OZpfzU/F+CguFMg58OF2Ytkd6Q+9Jh3S+iplT5s2T3KuOPChjSciMb58ZGRE9VOcWZPzZPX/WT5KJm+SEdL8kvRRvG0i5KPS+k25TsnSvTVP0peHmDXpozSdkvP0UuhJOgr+fkar0j7YaOns7MTRRx+dWiqihdN6iuTikwLIR6vEJo1FslA8OyDH29nZWeN4uaFwI6HnVkCPAYdAbWMszQ85Ti048XHzedCMRJaGZHlDo+llhsVBDjUdyj6rWKAj+4h4ANeADkm5XMahhx5aAwjpbsCxzlabFz43vA9BOhASPj8SCHZ2dlbGxMuIGhjk+/jn0hxsGrjhwGZwcBA9PT3BEo8MPBqokXojgY0UAvc8ENG4CNhZINAqVcg+KsmicYZA6k4IJPP72MSW9PnnZf+KBvpILCaRs4Stra2YMWNGTQnfKudo7HWotMol1PvJ133r1q1mqUvbJFMt54b7ZD43cn5kzw2NsbOzE7Nnz1bLplqCoTHUkj3jLIq0q1CCRfNCdqbpES998U0ysTwBpfWhuQH0PljZw0nb7Nmza6ocdAGIZlNyXqRdvec974kJ/6ky2SBprXNu+ySeA4Cxxm1SVg0s0KJ2dnZWDNfK0LjBSgeehqR5ANCcPqfHh4aGKucB1Dq0UqlUBaZ4INSyfdons36poFowTJKkAjY4IyKDYYg9Gk/PQwgo0GcpIwPs5nWtSVhz+ppTk/0NNDcDAwMoFouYOXNmxYFSGYLWLUuPg5ZlaY7ecmiyr4H2UbO/BFM8IEo2RPYeycDHQSXXGR5IJFCgteD3stEyVjlHWslRZrOyREmveSMrBwqkQ6R73d3dFf2xyo80Pj5e6dTlcxkELeaa1oZ0TpbzZenMKtNqfW3a+1opks8LrQ3NNwn3QVpjtNa4q22yL80qR8r5Id2bMWOGOUcSnMs2AN6XJRkfrWTL9VImcWSb27dvryql8dKQBIay/G+Vs7nOSZDZ3NxcU7bkPpoLL3tpJX8Z2zS90vbJ+ZVl3nK5jP7+fvT19amJC/cNciyygsFtj15v27at5pjjkckGSZNeagOAlpYW7LPPPlWKJVE6X3AZ7LWsN1RSk1kNMTAao6SxAlrWwpkgchT33HMPCoUCFi9ebJY8JDugBTUZzAiUSYOiYBRik9JKZxotbWW8fE54DV7LUqwykMYoaYwAzXEaS8JLrzfffDNKpRJOOeUUNaOPBc1ayVXOiewrstgjTq9LEMi3jo4Ok3GUZQzusPickNC80NqFytEhu7FKFxIUcj0hPQ3NCS+tcpYshm3Uys+8tDPROdEYtJCuyD4+Gdy0/jMrcYgpDWpzk5ZY8d5Ji71PK3URC2Kx9aHyjZU0WHrCfS0fT8xcxLDS1pxQLxWfE95KwHWF60l3d3fVHPDSO/+sTCCknvA54XoiwQn3BZJBnDVrVmrJTzJEdFw5JxYTzYEb1/+2tvCNLmNlUkBSkiQFAIcD+Mlk/L4UXm+VRgvo6FrSmFrWr21aGcCicZuaxu7eHapNc3DEnRop2MyZM1X6WwY8XlbjDlw6VTkfmpJyRx2iu8lo+RzS7xIYBFB55I6MbxqdK520RvfLz2gOncY6MDCgNjla8/HCCy+gVCrhD3/4Q836c9AjwaUGmJ1zlUbpYrFYkw1qQFEyQRxM8/mR45alIZm501wMDfmrULi98OZNmVDI5ILGqDGC/D2tuZgaqmkuOFsoy698TrRNy0L5uCXzJUtBdP7aXEhmSmMxtExdJktWJs5BD+k311HOWEh2R26S9bE+ZzVI07gpIPP+P84+hebE2uTnZVO01p7Q0dGBzs7OmuZe8ifkX3jiyedHe18bs9azw31Db28venp6qs5VMk7avGhjpbW2xi2FzovsWsaRLJts/rbiJq05twOeFGzfvh1bt25NnQe5yfFqzd78fHp6etQ5ySqTxSQdAKALU4RJGhwcxNNPP13TY8HBAyFbUhK58LJhlAMBnulom9ZkK4Mnd6hak7aW8fT09KBQKODBBx+synLk1tHRUdWzJPsHeMZDwIEcCmBf6s/HIHuS5Pi1Pi76Dg8qBKJ4MCbRQCSBAlkuo7HTc5oDrXertdXffZhngJJhkw3qIyMjWL16NUqlEhYtWlQFGiUjYG30GQqWBKy0bJj3ksjmYR7429ra1CyYj59vbW1tVQ3WsslTY9ZItN4RCaLT5oJsg0oX0j4k08hZRgJOVq+RVnbW5oA3mfM546yAZF41nbBKzhqLRuOW80G2RIw0fY+XhTiQlGUMDnxkSVD2nXEd4PpBvSKSIeFMgATWJBpbZLFnIb8p+xu1krPWM0OisWch5kyzDz4PUic0Nl5jRPj5SZ2I8RWSKbIY1hCTyNlO3jYgWw24b7R8pRU/JLNKSa/lK2LtQ7MV/rl6yGSBpKNGH1dO0u9XiXbHbY7oyTFJQ+ZKyp06vabnsnQiDZnTm5w14kbMHbJG73Ijpuff+c53UCgUcP7559fQm2mOnZeNyIBlw6ikuvm4tWAvnRlnCqQj01iilpaWqkZsSW/T2KVTk/QuzSFnEmRJhDtzXlfv6+vD1q1bowL88uXLMTw8jDVr1qjgl4MdzXlpLFlLSwvmzJmjlj80gBPqL5NNj1ojqAzs5Li3bdtmlgtDgNfSfwn0gOqrn2Q/xsyZM4NzoJU9tFKhLIllAXm85KPNgVY2pU1rDtYYZY0F402tc+bMUW0hpgTGWTQOZog9BVDDfPE5oF6bDRs2mGVSbfwcFIWAjNbXJHu72tra0NXVFSzpWKVimfhp5T/e6yXHT3PQ19eHLVu21PhEbd05ay77KDlwskrDHNhyEENN4Wlj51UHYk8lQ6bpv9UryS8S2LRpU3DtyXfIPi+roqLFADl2PgdkD6T3y5cvj4UAQUksmm5nkv3339994xvfMNkTDiKA6lqx5jx43VdD/TEAIi0D4BSwVmJqbW3F7373OxQKBZx++umpASQtC+JUq9VQK8ttvE6uBRKLOeM9N7JvgERezcYzY9mgLjNkCjBa71HIgVoZsVZ+HRoawl133YXh4WEcd9xxwZKjDB6y8VHSzVwPaW1kAycPJJpDkWPW+iQkGxAqNXIwLQEF79uTPSKyFK01VPMgQnMvdYDbgsYOyPKrZIv5ulvjl2UVWY7nusrnQfaQaI+SFePBg5ectNJKTGmRMwRSR7RmYDl+yRRKX0DzoF2owedBlluthnGr4Zf7Hs0f8LKz1ggtbUSCb1my10qOsuQk7UHzDbLPio9Te66VXDlbKstuch60hmfOIKY1hss5CM2DBi65XVjtCJpOSABlzYVWdub+IUkS3HPPPdi6deuE/+V2sv+7bUpIS0sL9thjj6q+HL7wXNnp0Wo4DZVPiFXiAEL2n8gmQh4EqVRCDj+tVDBv3jy0tLTguOOOq4AiAghWGU2yKVwhZR9N1nHT93k2RUwS3/i4ZZNgzLj51traWvkPLA6CpROUwS+tjBoz7vXr12NwcBDr168311uOOwR+s46blwU08KvpOWcPuZ7zCw5iyoR0h2vrwgRLz61xS4Ykbdz0GKvnGtDTGuxjyqM8Oci63lzPJVOmlb7qqee8P4yPgZczZJJnMWYT0fMQQyjHLi9A0cadtt6hBJeveVqCG1P2tNZbJnTaWvO2CKu0lXXcPIGRJcz+/n5s27atauwaO0x6TseTACZWz7WS3qxZs6rGrq03Z4UJFCdJgjPPPLMu+CBnkgAccsgh7vrrr69MrmQMtJqx7KngYCDkQHkGbTkRi2pNq5lrzzXWgDslng1odXKNISKKNY1e56BCZgXcgEg0alWO23KiVjM6BwU0XsAHZS3j0RghyX5IBoiDR9mELLNgyYLwTE+j1OWmXQ0iG66trJfrtGQ+tAyfj10yQ1x/ifnRsjvuJK3sVsvqLcYjlOVbWa1sEOVBQuql1UCuZbKc4dRKBRrTJRujtfHI1xqjxX0UL4+RPks7ttgd/pongnItibnQGBx6lI2+GpujNYTzPhU+X1J3ibnS1pXWQTaDy3Ie388/rzUGc7Eav+U4pe3xsfHP0zEsRoqfh2ze5q0g2vj4+5yBlGOk35FrqI1PY6RofHL8fA21Bm9rDeXY+Fpp+svtWWtuv/XWW7Fp06acSaqHDA4O4k9/+lNVMyLv26HGXVmvlcFF9urwvqS+vj709vZWvZYgSvaryOxLa9KmQMkzDELddBnk3nvvjc7Ozsp+mZGEWAZSaK6kvERAwE/ry+JjliAylGlzIMEdDa+pa6wCHx+NV2tK13py+Hh5gOHZtcUoaOPs7+/Hiy++WDFg2YjPDZ+E92DwDNNa3/b29qpx8v08O5VXMPL+I6AaNHFQrDVLyvWVGTZ9R45Xy6pl86wEgjKr5OOkLa1kLPttkiSpcsIWa0Tj6e3trYydj5f607Ryscym+fryBlleGtZ6DPmaWiwCJQya7ZJYiZ52YYlcY8kEa+OV7CAJL3vJXhqZ4En7lYwRfZYz6nRcPl4OLLg+a2yJNV5tzGnjlRdLWMyYxgxZaxwar9Y7KMvcXJ8lG6j1zXL7pfmS46U55uPV7NdiP7ndyr5RzVdp45UAio+XxtDf349f/OIXEwMGpMc5kwQcfvjh7ic/+UkVKOBBQy6CBAMUIHt7e2uAEKcntWZd2mSA5I2JMlBQYJSPXOlaW1vxsY99DEmS4KqrrqpyoLKcRgbAHYkG9PjYNPBDzoT3VGlAQDZjy3G2tbXVBH9tjNKweCOqDBK8lMKv/qC/ppDBX24aCCAnQnNIvwV4ALB9+3YUi0XsuuuuKtjhjlGOT6OX5ZUi3HnwfhDJ/lnATgM6siRMx5E6C1SDdtlULINgKADytSQny49pXVAgA6AG6ORYtXIBv5CAMwxpOqtdLUljJB2WpSGrNKL1/Fk+yGKvrXIQB4CcRbLWkidgnJ3VQKtcQ56I8ESTg3MuWl9nWomTxmwBGT5GzhJZV31yoGqVujTApvUxcjaFhLM+XGc5SNViiyxvcd8Tk2zxMWoAJlS61i72sEp5MrHkbKy1ltoaynFqLC6XUJLV1taGu+++G5s3b54wk5SDJAALFy503/rWt2oUXzowIN1JW4GIlJ4yU64Q0kmTAyMJZWTSefFgdMUVV6BYLOIf//EfTbTOaX0uWvkpdGWbBEuyh0UGW2nYgF5us0psEmzw5uyYK3iA9MuRZVCSzIrGmvAxrlu3Ds457LrrrjVlVDoXfo5WOVWWErnT4nrKWTENWHA20CqraeVE2WDNy1CylChZQDoXnhlqzdLEAlmN1XwdySZ5MJJZtWyol6U02TCulUqlg5ZlC/pdrXTKG2TTmqVlCVyupewtId2VJQbZHMwb4LVSqmwwTyuZko3KsqnWKC6bgCmIaeVFrSSlratWcpJrzNdZzgEfZ6ghnOZXlqE4iOTlUjlm7TE0XtniYa2vbIHgz+W4Q83f/LhaSVwri4fGKTf+mdjxyhIx2RvXZaslwFrv//mf/8GWLVvycls9pLm5udLkzMs7QC3bQgvCg6TMxDmzxLNajW3h9XQSOgfek8MzAJmVy42yga6uLhQKBRxyyCE1/SqcgZClQ5nlxLAsMqPT2IdyuWyif6tB0yonETDiV2ZpzAMvJcnsRtLu2kZrKHvKOEggHeEBrampCYODg5VgyRupQ2tHLIQsDZI+yAxO9tjQ+VklBU03OaiVgJ0HJd5vkFZOSNNNrazN2TFaOy0h4cxfPXST1k4yY3RuFisW0k0tA9fKuhy0SsZ6YGAAPT095hhjdJMDHMkuWKXrmLXTGDEKhJKNL5fLVT5PMmHbt2+vYaml3+RXgFIrguY3JXti+c25c+emjk+2IGgMUVrJh8bU3d2dWTfl2mXxm11dXWrJVpaluW5SokxxQSZUvBSnxYVt27aht7e3igyI0c1yuVzRnyy6OXPmzKBurlxZnzsM5SAJY3dn1aj9crlchVpDSmL1HMmGbQ66AFSUnx55pm3Vsq3aLg+sZNT0nAIO75nQxmY1n1s0vmTBaE5JyXk2EqLxuTFYjeea09KaTK36vGSDpCFzY+bZGAESMmb6Dz/tKhUa3xNPPIGmpiYcc8wxVUwXZ4s0Ro8CdqlUquiVRtlzFoSzmZIV0hgSyRaQDXR0dNQ03ErWQ2sm5zorM0mN3SJHKpk8K1uUTI/UOQnoyH4LhUJFryRAl4yHZHn4e1YDtWR3CKByAKRl/jLj5+sidY4zHOQvAFQCydy5c2syftkArmX2sWwVnUd3dze2bdtWxTrJNZDMRagBXI6Hn0tHRwdmzJihNrprz7VxaOwMZ1XpfKinktaB66PGOnGAxoEMF62xm8517ty52HXXXWvWSY4lK9PE2Rd6vW3bNmzevFldHzkmrWldNuSHmKWOjg7MmjWrRtcsu5E9vpLx5vMv/cLgoP9rGt4DKHsB169fb0T8bJKDJAADAwNYu3YtOjo6qmqxHKjQwtJCcnpPNi9TbxJtPT09Va/pMzxoc6PkfUoSXWt9O52dnZWtq6ursp+YjBdffLGmr0VmRrxvx+rBIhAox8PBIQdR0vkThc3HJTMinvVQNsTHJBvQNTZCAl2Z7clMlveTWWPiTIuVqctMdvv27RX2h4Nbvl5avxUvP/FeB41B0hgWjTXSxsQzQ9nLQTrB10uCIskaSX2k/bJ0KJvmZXZOzs7qwbH6AGWJlwCjBFG0VlqfkVa6Jj2kHqOuri6VNZKsiky2JJspkxHJppCvkKVt2dvIQSEPRtqFDrxfStM/+oy8xNrqaSQAoTX7awym7GfUGHbJQPMStcUSyT5GyfJJhpb7QV454D6DX6TCk2PJYMYmx3xcJLIPTDKzWq8iZ4limEurZzHUl9nT01MzTtmzaPW4WayX7N8L9ZzOnDmzinGmNdd622QvFPn43//+93XBBzlIAtDR0YEjjjiiQoPzwMoViAyfA4Xu7m4VCJFScRZDUsQyqM6YMUMFCgR++HNy2J2dnTXBh4JqU5Nf3v3226/KkdG9L6QDozHwR40etsCP7Jvq6uqqKjPxc5fPaRy8YVleoWQ11Q8MDGDr1q0q6NHWhahgraGVOzDeN0TbLrvsUtOAPGPGjKpxkMFv3boVbW1tWLBgQeUKFc15kV5wh7x9+/aaNeJj4vs4iOABlDcgS1DAm47nzp3u4qbrAAAgAElEQVRbdVEAnT93xrQ+MWCbBxgOtjdv3lyVJHD94mORgZODAZ5AWEFzxowZmD9/fuXcOciRQVMCAXLAmi+Qjah9fX148cUXTdBmldk5y8J1jcYj+w3b2/09zyQQleUUWcLkJTASrbzHwdr27dvxwgsv1CR1FqiRyZ0sqWusOPku0jutNGSBT/Izsp9QgjRuO5s2bVKBWqjpOwTQ6JxkszPpnaZn2sUXNE8E0LTyndbwLPVOsvxaMhcCMFZJsrOzE7vssgv23ntvs1wuEwPZu6s15Mux9PT0oLu7Gxs2bFDXh+uqjKFaCZnGsnnz5rrgg7xxG75x+5JLLjGDMwnvj9AW3crcZebEgzOv25NIsKHVZkPOUmYWnPKUvUhar440Sh6o+XuaQWp9Hry58/+z96YxlmZXueY6MWXGfGIeMzIjI3LOLAM2XHRb6LZsqy0hNY2YbOlauE03SBiE/KMFplvgdvOjm3/8wJIxYAEuMG2wXTY2NkKW70UyGNvgKldlVk6RkTHP84nImL/+UfWsfL8V+0SVcXAbU/FJoczKyjxnf3uvvfZa7/uutTVT0swi6j2iIDulXUmtTaRHUxRbdPbqKJXKUeiWR2FnpWR0viNVGGlDpXVi0JSqFtE1iqJyfbcUzRYh9kgRqOBY4XM93LTqLP6aEmzGtdE1UgoG6jmKp+NPSmwcaY8opNagUJ2ovpvSh/oOsYIwHgBRM3WcUDxFH6aE4XpAmuU7KafE4eXeLdKEkRYpR+fwPXynBj2R/tB3S1GGkTaMVEp8Ig0TxxrFwPH3KSpUxdZqf1EgnKINy1GhqfVK2WC0w9darxS9m6JHo56Ud8Je4jtFWlf9fMoulVqM1LWiyurTGWPcU5Gajz5Pz9cYLKoNpuQTKT+ov/J3Pve5z51IddspkmSvlIQWi0U7c+bMEeH266U19GdjY8ODphT0Gg/fVKbC4ZqiaCL9VE6P9ODBA6uoqLBr167loNYY4KVowlKp5KLR1HuwsZgf3Typ91DkK4UiReFrqqxWDyVdD4X3U+sRETECJT2A1SG81nvE9Sj3HltbW1ZdXW09PT05ygy7ItOK9MvrfY+traddrVPvwXdF/dBx7xGhfKXLIvXHe2ADMftNISyK7GkArogeDjvqwCJ0HwXU5exKaSaSBpw0B45qvvQ9sJfXopojOhmz+Nf7HsVi8cg7RCoiRasotUdgEjWUvEe5n9fzHuwPvvc4GqVYLB7xV6/3PZTKK6cFTUka1OZO6j0aGhqspaWl7D6Pfjf1HgQgJDPqc/GzpVLJFhcXj6CRESVOoSkaOOp7pM6Ptra25HtohenreY9yFB3j511e6z00odb30JYMqffgXVLocE1NjX39618/kfjgFEkysxs3bmSf+tSnclynBkeq3cC41VkSUETnH/UeGAXfoRlTqoLmtQ5h/n5Kj5Nlmb33ve+1LMvswx/+cFkoOjqWSN9ElKicBkedviJDKV79OIRIqQ7VSbEOeghrYKQHmv6owDkKR2MpfkoTELUcKTG5HrpavfTFL37Rsiyzt7/97TnYuRzCdZyAXIWZjF8FlZqhRWF86iclGtcMXHUuKcF4KovTTC6OXcW7PBH6V/Q0rof+PpaSRwSB+TGzJFqgwZgGmFHgrnNebuya4aey6nKC8JSYlc+NAuMoAD8OndL5TglyFZVShCOKcMuJ1xUBSImKy5Wtx3Gnfh8RmpRgPc75a827vpOiMoxZg31+jfYSiwX4NYUIsleYC53zlPA5ZSNq4+VQTfXzr2fc2HlMqiJlmvKRZnZk3IxL92T0LeVsPiJhscozjlt9fjm0POUj/+Ef/sHW1tZOkaSTeLa3t+3hw4eOdqhOoa6uLmeQGFNEYoik4Vf50QBKgxAWnk2lGylyq4rANDY2WkNDQ+7XlK6npqbGnUFXV5cf3BgjBqZIkuqreI84fg1EotDXLB34aSbAeMuNX7MzNqIGHlGDEIWvjFffQedeaTZ1cDx6aKcymagRU62YislramqsqanJKisr7eLFi0c0LjgCAtGoDdN51/JhRfRSwStOSB2VZsYRydPxp7LjlLA1Iqsq0lXEK5U8KKqhTvq4YoVyhQpRo6fZJNl0DADNLBdwxAIFtRWd/+MQVaXJzCwXeCi1rBlv1OYpdZ7SSunByHxpwKqayWhHUQiuyJ36oJSonTGkSsxTCVxDQ0NOg8OhrxQs+yAKbTXpSaF2Ua8SdUQEVpESP67QQN8jhcynkJVIhcfinRQSzLtEfZfOgdKRfFc58X2cc12HxsZG90PqRxUEiMxC1KYdhziil1Q/qsE5NlQoFDxAej2oULl9oEmRnsUEdalCo83NTXv55ZdPJD44DZLsFeH2rVu3rLKyMpc5b29v+yELVE3goIEQP+qI2DS6gWPmrA4lHrpNTU1HAgmqajCelOHEbPnw8NBu376dc5rr6+v+qx7CiiKxeTR7JhNQKlA3KmPlJ3UAq06KbDRF2bAR42Gl86/BW5xzNpCWS+P0z549a+3t7T4u5repqenIfKtGKh66Zkd1ajiaxcVFm5qasp2dHfvCF76Qm/soik1RmGRxOucEPMViMTfnGjTHgOG1Ak510jiY+fn5XIAZA08N2DSbS0HnHJJKpXZ0dJQNNDXYx7Fy2EXoX+ecA3ZpaelIoBkPW0XtVKulgnCQiRjkd3d3HxlzDJKVLtY512Ak6v/Yf3Nzc68ZIKfoIxIsRXUV0a2vr7fW1tbknMfgkjlP0S0xqNGAbGZmJmc3qcRKfSKPIiSK1sZij66urrI2Xg5Rj8GMHqQaQC4tLdnY2FjOF2pAw7+PgUBqzmMQ0NDQYH19fUcCsqgdxfZUlwi6qahzDL42NjZyYvs45+UCefWJGsTruBsaGqytrc0GBgZyiWwMYKqqXgkloK5JBI8DE7Dx6enp3NjL2QoBMN+nxQ2xSKOxsTGpf/uXPKd0m5ldunQp+73f+73kgULgFKNupax0waMBqINT0ZxmPRpxR7oqlS1E3YgKadW5/dzP/ZyZmX3kIx/JZc0x8y9XlaOHoDplM8sd4ikOuVz5auS/U9m+OrdYJaVOTt8jJSbf2dnJiRGVw9fNVo5mY27jAaI6iqjRUZrwb/7mb+zg4MB+5Ed+JEmxxd9HmkrtRB+FvqMA+Th6TaH9SJlohsxhnoK/OSBfD3Svn6f0BvZZTvSZoh2iCFmFvjwRuk8JcuNYy9FUKsaNCItWckbq5Lhfowg3jp93UHovJYSOFA/JjAaqerCkKCul26JoPzXfSldFmlDpE/1O9jDBeOodVIwfRdFKF6aKDXQfR4G3UkIpwbAmaRoMqh29FoUVRdDqb6LfSe3hOPeRlo3UVYq+So1f6c4UTZ6iyFUWkipciXtAqc5YwUgiqr/qGZWSJ+jY1e/Hc0ulIOWkFdjU1772NVtfXz+l207iqaysdFSHCJtDQ52oZn4xKFJqKmbbWvobjY2NrAEEAVGk1r5TVIbNTsm/ojKgSXHMEUZVgbmOOYocUzQaY1aaQTe7Ztg4UuYqwu46x2Sq0CD8aJmojpnNrLRHzJZA7yIyECseI9wLaqfZEvM8OztrOzs79o1vfCOHymiArQeFCnw1O9WA8zj0SJFGpWp0zKypjjlq61Ioo9qGBhXaziLSS5o1g9Lp2FOZaQqp4wBNZdNx7ymKoSJkDeDMngYLrKsiRmobOseMOaKLGvwwHymEjjGp/TLXmqhokK8+gzGztuWo7DjXmqxEGpUkkMBek0DQlEhfH+fnCBRSPoMf1lzH2NzcfKyf04rjcuhz3H/4uZWVlRzyrLSpUqYpn6Eoi46ZsXZ2dib3H0mgUqVR7hApxjjHoEMpP6djVp+h1FY5P9fY2OhI+kn4uTjuhYWF3P7b2trKBcHRZzDmiMKlzhJaeyjarMkqz+7urr3zne88kfjgFEkys5s3b2af/vSnraqqKpeNxs23tbWVpNv4b4VpNTCKVQgpaPa1qJNUNQiP6nSUG3/++edte3vburq6jvRAej2QrGZnKX3RcRVqwN/w+WR9qSoi5jdVEZVqnUBQEeFjFZErh69jVlQrpf3QDF8PjxgoK38fkSz0Equrq7a/v2/V1dU5BC6VicX2Aepsoz4iVWWmCFYqcywnDldBpyJX6sj00FCEJ6JrrLk6Ww3glbrRTDcl1GTMsR2A0texwifqmXSvacuC1FymkDVdf91jUbxeboyRKmCcikRFwX0KOVBKAyopUmHMRUSYUmPVxE9RKuYSGzXLawzjfKbGqXOp8xkFxuXslD2WQlV13Xl03aMImjWPv6bGWs4+FfGKiLYW9OjaxwBGxxkR4Lh/UoUhKeo2hfxG1IWARnWAuq/4+zpWtU+ltSLKrolCuR5dSnvqnCoKp8yAnrMpZkD9vorKIyNAgPj5z3/e5ufnT5Gkk3iePHli9+/f9y6faAoaGhrKBk4qztvY2LD19fXkjx72GGipVMppIFQrozw82p6mpib/UcRD9Q91dXW5wGl/f9+uXLni4mCN/Blv/DXqZDY3N3OOHsesUKrqejQjVE0VGZaOVzdQFEDy3VEIz1gjKlNO66BBiApPNUOJGirNrlJansi3a2Cn2VQKqdNgD6dmZjk7AFVUJCYK9VNjVf0Oc6u6I5xnFMWqPRyHHCnVqsFIqrggBvwRNYKSVeqGeeBQVCd/HGIbbTZSZ9isipA1WFYEVOcVm01l2BHJSGXYihQdh9aWS6QISFKie91XOl5FiggAtFoXkWuk3OMY1TbwYehymFvWizXUxKQcghE1OYq4qB2YPe28ji2ojiilkyuVSo6Yg1xQZq4aP0WIInpYDuksFovHJqkanGoCFRFa9V3o5/i7qQBAkU72jDINqYS6sbHROjs7j6D2+ENNUiOavLX19B69qPtcWVk5gr5BJ+ITUoGK2mxK98n9a9qeA1uICKcib6l5VVtgvCfxnCJJZnbr1q3s85//fA7p0EXRxVhbW/MAaG1tzdbW1o44QD0ENTLH2Sr6ogFQc3Oz/z4eLArd8miFFIauAcVLL71kpVLJGhsb/dDGuLQyhCxUHR4OTIOIONYYBKmQj75GcUNGKFx/mEs9qPXg49HAUh0HAaT+MG51zjh0RWJU8K4OWedzbW0tF6xFSkczSGypuvppJ3V1GHE+NQjSCkV1yIpq7e7u5hA3xhnHGC/WLCfIjyJl7E+D3miX6tw0G1dNgR4YSuXxEysoywVmBH/l9lAco1K9UdjLowdxrM5LJRKxwAGbJoFKCZA1idDkgd8rWqz0gVLRMSjXICyOTw+2GJTv7z/tZ6YHWkSz4ziZTxV4k3yYPb0QFdvUJEvXPJXoKPKeoupeK4FMJWSMsba21oOiFK2fSnDYQzEgxzYU2dL9g22Vo4s0yaX6DMqIYEYDcAIQ1V4eN5dx/0Tta6oYIfqjOE6VHej+gVaOfd5UxB9tU5NGfJHS4Fqdi8+LVYjq33XdtdhDRfA/+ZM/aS+++OJ3jSR9TwZJhULhv5jZD5sZljqVZdmVV//f28zsw2Y2YGb/aGb/c5ZlY8d93qVLl7I/+IM/SB5MhUIhp4nQAzQ6U91UbD42Y6ygUS5Ws68U5cbGr6+vz8GwwLcKZWJ4W1tb9mu/9mt2cHBgv/Irv+KbSB09zj6W0prlSzg1k4ml2FpRECkshVz5TIXZVeOlUGtKPB5L3mM1klIWKVg49paKlJBWfSkCo1UxkWqLAmylBw4ODmxsbMyyLLNz584lezKpTqqcyFFpAdXqcDDpIUpgEmmLKGqM9EWE2qMw+ThhdYoOimJMHhVlpnrnxJ8o5FURLDYV6YEoQo7iaR2jCl/5LJ6UgDoKweOPinwVwSjX60dtNyWe5icKplW4HPvN6HhV+B0F7KlxRwpTx6wUcUrkrT9x7CmBbuynFGlCDk+lhdReyvX0UTF33GtxXx0nho40tq5tpAhTPzputQc+kzlR1DBSb/hcflW6UGlsAivGqZRwFDvjt6L/iggsTznaLbZTiBKGWPmma6/rHNGs2H4gShmUIoxnQESx/vZv/9YWFxff0HTbL2dZ9gf6B4VCod3MPm1m/6uZ/ZWZ/ZaZ/b/2SkBV9iHz0zJMs6dVGlo6qsFGRBgUBWFh9VBnEysFpBlcKmuPImJ1PpoZAUtrlskGHRsbO5K1Y3SqoYg9dmImzJhAvDSQIzvRjcH86YFNxqHzp6iCwtFapq2HuFInEXrWMWomp9QJ42ONdcMqmqBZED8p2iQGlzjnjY0NzxAValbEUDNLgjgteecwUWfC95ebw3KojNqgOmOlRsqhMmqDGlhqQKlIB+OLc6hrrELUaIMqQE0hcPy+ubk5h2aWs8GIZip6sLKy4ns47uO4R74TZFjHqQiHCtTVBplDTWKYs8XFxSOIkR4e6mdigYWiBnF9U6iBIjBqg3GPxH2sY4x+RoPylABd6SIdIy0XmMNyNoifJjAoZ4MLCws5gX9E21J+RpPYlKxA20KoDTLGSBupDSo9z8/i4qKPkeQxFh5wTmGDUVQebZD2CapvjSh1TLKjDeoeZq3L2SBzSLCkwvdyNtjR0ZFLagnAow1qYUG5s2R9fT2H9n03z/cykvRsIkj6BXsFOfqPr/53vZktmtn3Z1l2t9zn3bx5M/vsZz+b42zjoaRQLJTb6upqDlFSoWmsOlB9gW56DTgUPkRfAF+vtBDRtGoK1Hmura3Z5uamPffcc3ZwcGA/8AM/cKT6Sx0BzrQcVBwpFt1ksdpEM6yo21L4XfuQKJJlls+0cKJR6xB/9ADSTItHs5dYVRJbIMT2B2TbWsoe0aCYWdXV1dkLL7xgNTU19va3v/0IXB2pKc1Ooxg86lhitZbu4ZSIUQWWGiymROsRmVBkR1GzlFBdEQClTWLLBc3oNfPUthDHCdSjiFoRM/1RIbhmyaly5ij21bGlWj9ExCGW7EfRvP4ahb4RadDycNWkRJQ0jiuWVUcxvwaKERlVu9JSdj6DPanFEdi+ogtxLfnRUvU4Lp2riHQofRSRWp7UGkZ717YqamOKdrOP4lwdh3bH8nNFt1S7p3sxVnEp9aa+lQQrVsZh23y/Bv2RJeAHlJlH0Xedq6jNUh+r/hVbw/ZThU7lqk8jZa26PEWFUyxLpFePoy7f+c532u3bt9/QSNL/XSgU/h8zu2dm/0eWZf/FzG6Y2Qv8hSzLNguFwsirf142SEK4rRk9ok4CJ4yAzUJETdC0urpqq6urOc2SGmqpVDIz88BJ9QAEJQRMxWLRmpub/UeDlJqaVzo5q3BQKwXUGL/yla/YwcGBveUtb7HV1dVcVoUhl0qlI1m8HhAaKOmYGBfGijE3Nze7o1YnyLxFBE7nS3UKmj1xeGh1DVmbZuvMm+o9tPUAG1sDgCi+jhmJooQpBC4KQnE4DQ0NNj8/b2fPnrVvfetbR5AFRW9At3CqBBnqoBXhUNRDkSN4ftVMZFnmB3sMhGNGp06H+VXhOp+nQnDN2FV7oKib7oPj5k2Ry6iJifodDjotsU6VV8cSZcYVUUsNWnSfaiUqh4jOVRSixhYMOm+KJKiAXtGEeKCkNFqvhfgqopXS6qhAlnlL6TA1S49ygrW1tSP2FltYpDSDiiTEgw57IzFU/6YoG/OGj40aotXVVfc3jE0DYw2GUyiqrmtTU5P19PTkqjfxb3oupIoMos6yVHpa1q8tHgiAVIKBHaX0QuqPe3p6ciJtpY6VcsPfpxgQ/pux6T49ODjwJEcTCA2mFBlXZLe7uzunDcOPKHrP2CIQEdHx2dnZI1raaG+cQeyDubm5f3Fwoc/3KpL0H8zsjpntmtm7zOx3zez7zOx/N7OFLMs+IH/3q2b2+1mW/VH4jF8ws18wM+vt7X3zV7/6VausrMxlEar450BfWVnxgEgXUR0thmVmOW5ZD6VisWgtLS1WLBatWCzmHJoaFgeAirQJJGKApuMqlUr2T//0T3Z4eGj9/f2eRXAQxyAII9dxaSCkDlZL5MnuFPpMjUsDRzIghY+1sgvnoGNiPMyVipw5kF5d19whjrNnTDGg1cMS54Dz4yBh/Ti8cQaMh9+rQLympsa+8IUvWKFQsJ/6qZ+yLMtyULYeNoxJf9XARynH1zNXMdBWKkWrXRRijwE249AxqXiZuQIBiWJ6tas4V4xVAx6CMaiJFJLLPtSxaWCh9Czvpy0smAcdE+NS6g5Hy7/F1sniyd7VqeuYmCtdvyiWx2ajKDVlU+w/1lsLI0B8YrEBSLeOSYs3lGLiUcQj0pu6B9WPKc0OoqPJEUGNJpUp/wkK8lo+VBH41BoSJCqtCRqjPlQDrWhXSq2zhpFKUqRD5yquHzYF5c+7aJsJ1kFpdGUrdK60YlqRSKVYtegm5RfUh5KkqQ81s1yAReDHmKK9q18gMFUfGudKfehxgIBqb5VSjXRlyoeur6/bl7/85RNpJvk9GSTFp1AofMnMvmBmw2ZWnWXZ++T/vWhm/2eWZZ8q9+8vX76cfexjH7P6+voc180BACyofZJAZsptKgxMHwwxVmlE2k11PlErpULHVOWDZnxjY2P25MkTO3v27LF0mwqzNbNSB64CPc1acGSxjF+DzCjESzWfi5qeCFFrRUYUCmqpvgrFyd6Va48ixkhjqYBdBc0Ehq9Ff2jQtrW1ZVVVVdbW1uaHmllewBxFoCpmTFFYShcxX1FQe1wPG9W0lRPTqkg1JaxmnVWEHUutX917ufVMiX61mkWFyfwa6b84xjhWfq/l1LFnjT6sq1JcUUwd/4x3SgnUtWeR7tU4nzpufs+/iUJvrUgrJ5rm96lxxv4/qV5F2L1SWnoI6/gYlwr8o7C/XG8inVc+RynUVB+dSFcSCCu6FilUnRsNshSlUcpS9wbj0jVS6vS4nj66XzWBUC1qqrWD/qrj0/OIR5E1RWL0HFDfq9RzpHVVd6XtBWLhUDlNnQrbWa9yeiYNpkqlUs7H4RfVfynNFvWwelZqZbWifD/90z/9hqfb9MnMrGBmt83sPfzhq5qkoVf/vOyDA4oHR2yAFfuLKAUSS1pxMipE1QxEKbaYZWOobGCM0cw8yFG0K0bSBG4Y48LCQq4igA3LZ585cyYXqGnWSDaiG0QrKnBWbMatra0jWZC2StA2CTigFGIDaqPzo60H2EB6wOIsVXPEOsUMNrZuwDErDUQQFCnHcpmiBmzYlWobYiuJmPlgR4qI4IAitUJ5Pg4D9E+pAhXhAsNz6KkWK5Uhsn56GCgti4NVarqhoSGH+B2XHSpqpLB71P7pHOkBwJxqpZsKqXHsOg5sKO6xs2fP5jJ7HHcqWwVJ1sxeD8hUBh3tOWb0ihapYFWpV6VMIvqxuLh4ZI/RK0jF3JWVlblkQ2lqEGTmijlUnQc+SAMYpQp1vWZnZ3MokQY3+Fa1adYjokOdnZ05raYWX4Aw8J6qg4no//r6uk1NTR2RQahNM+/aQyuFhPb19blPRDuqhQJRmpFCQdfX1215edkeP37sdqa6JhU+s2bYbUSLi8Witbe3uw1xbvCAVmHTx/mgubm5HM27u7ubqyJTpDF1hikr0tvbmxPaq5wAjaNSzjpHuscmJyfd9qN4Xc9T5B6s2erq6ncXVRAffK8hSYVCoWhm/8HM/qu90gLgnWb2UTP7ATNbNrOHZvZz9gqy9CEz+09Zlh1b3Xbr1i0XbrPhlJdX2mhlZcU3HVAjm0HLJ8nclXqIxqSHHMEB8KJqespRf5Fy2NjYyDmj9fV1MzNrbW3NCYy134QalVZosBn08FDBeOwnk6r+SvXBUMGn8sfxR8XYKsYze9rf5rVEgirE1hJis6cXPGrWkmp+luphoxUhseJCofzJyUnb2tqympoap120nxK9ddjssRpJs0vNMFOC/qiP0CAxtlVIZbxmlkMF+I5Y4ptq80AFCnMSha8pIXpEBVT/oEGq6pWiEFftVOkes3zzxHIogAqD1T5YVy0e0CoiRQEiKqHzAUryeuajXKsL1jciI7p/on1oVh2rl6JOBfo+CpIVVdWiBUUfVOOGX1E0RBtxavJJcqRromhzREJYQ5IY+o8p7ZwSHcfxMBbtRxQPbGULVIulqIz2bcte7S+HHzmuLxZoTENDg8+hIsz4S62ijvocgjz0kaBcr56NubWJNK76epUGKJLG+aUi+nIBTExc4lgqKytzmjSl2FTSUSwWfayKPGoRlSbgjIOzmMSFgiW0S+Pj47azs/PGo9sKhUKHmf21mV01swN7RZD9G1mW/e2r///t9opG6bw97ZP0+LjPHBoayj784Q97Z1Xl2AuFQu4QZNOwMMvLy7aysmLLy8s5hAIHpBQEjlT1Gi0tLbkfzS4VRnz13TyL07FgMDGI+7u/+zvb39+3CxcueJbL52h5NRtZM0rGolArWZ9mlBx2Kc2BbiiQtogAqFBcBbtRh6RBHHoR5oWDSEWnKa0BDkc3EoFC5PTV0aXQLHXAsZSWJnDPPfecbW9v2w//8A/ndD6MS/UFKoAtJwSPeihFRQio9JBWuriccFPnSYMHHJ4ioQRIajOKFvFnBBAplE8dXswgo4A/ttFQ+k4DNz2QlLaOgmAOVaU6NchOlRGrtkjHYpbX92nAn6paBXlQJCS28Yi6ItUW6Z/FsagmTIPJcvOiY1FRLQEIVP5xlb0pHQoBldKooD+x5Fs1RIp6alVqubHERJGxadAb9UMEdGfPnj0iNFb0A0QY+2UvavKsVJJqv7Q4RvVozCsBd1VVVRKJSen29AzA3vBZOpbjdKBacKLJGXtCWwfgd/VHx8JeYi+q/pPgUsfBeaTapYiUw2xEfRdnkJ6PKd0ZD4gr59HP//zP28OHD994QdK/xnPr1q3sr/7qr3KLHgMiAiF+VAjJZuAB1j579mzuUGttbfUfpUcaGxtz6ACQraISanSMgf9WMaZm5HNzc1YoFOyZZ57xcbS0tPj3t7a25ugIHKyWxHKoKUS7srJiS0tLuTHw/8iI6ZM0/48AACAASURBVDYenaVuvjgXwPzQaBzQOCiF0jVIJVBlc5IRqlhWxYNKeegYoBvimmgJrgYbOEcdA+Pa3Ny08fFx29/ft7q6uhwFo1mvOgLGotkVh5mK5ck4VROn38+6cJipU+JzlHZh/XUMjEkzYAIw7FP1B0pFLS8v29LSktsshykUEMG69rpKJQ2siwYZBPcaqHOIMhfMgWaaSv2ABKCBqKyszCEA2GdbW1suYSgWi7n2HCBNKgjW3kv6oz5D14T30GIKPTjVNrFXpXcVASAg5uCMPkOLTjQQJRjRyi+tXNL1wD5TPsPslQRMxe0c1PgM7EPtUwXbiqwe5zM0iVKfge9iTWI/rHI+Q9t+kOgoHfh6fQZIiCYpSvu/ls+AmiQAVtpWKeTUmhznMzSQ+U59BmeaIofM/Wv5DNaERHZvby+Hkql9qs9QWlR9hiJ2zLmuB79vamqy97///fbyyy+fBkkn8Vy5ciX74z/+Y6urq/NoX7luLYdVI49aEhVH80RRXKqUXqFPdXzHVa+osWsZuBr37duvSLHe9KY35SL1FMWl4myFX7WnCYczmYmWV6sei8BKe4ZA5eh8RIpLs1uQIi1p1aqZFKWkSIh2Zo2Zv3bgjb16IoUT+xppPxzNouLP7u6uffvb37aDgwMbGhpyPRjvoshVSvSqf656OWgts7wA/DiRdRR9q4A5CnBVqKzfnRIt61hUWM33RHF1FHozJ/zEMek8aedmfh//TEXOfI6KveN8pX5fbp54tFUDgTwZuQqoVayu1FsUnutaRaF+qsM82hDtOxW7SfN7taMollbxduxTpC0UUj2deFICaXxM7E+kWjTtrabUG1S+onuxJxHjZS5AISKKlurzo6gMcxIbBmtFbKqVhfZvUoSIeVAxdkQSlcbXpJhH+x/F6ldFz/C9UVOFLarmTNH4iJgxV+z/WD2mSKYGcwTZBFNQoXyOtrbhOxUg0GAKv8/4tTGtnnMElYuLi372chazHpw1ZmZ1dXU2MTFhW1tbp8Ltk3xiVY3+xGoVrQpR/hZnotVjCvuq+FbLazVI0lJaHIKZ5bRSkVpjM4PiENiQ2anQVjMARZK09Jnxa18XnGfMlJeXlz14U72LblwtjQdKJfpXAXTMgrRqgk27tbV1BFHT/jdsWkVvtB8Vm1ezQZA2RSt4oBUVrSAL0g2rpcz7+/u2tLRkhULBlpeXHTEhO4/ZoELsqm/hMNAqFoXW9ftBFVkrFcZjdwTEEcFSmF+LBsyedoRW4bB+vzrPtbU1P+CiFg2NE4mCoiMRWdV2BUobcnBqFgqqmRIva2DBARkpXd6fDJQxaEM/ghql5nTeFxYWcrSuImdmT4XKKTQTxKq3t/dY9E7RkRR6Nz8/fwQNYA7wb6nycEXMent73R+hEWEPaVNRbRWhvmB+fv5IW4bd3d2cP4TOUopJ7bC3t9eRQyhkggn2tgYROgbmYWRkxNbX14/4I+ZAD/GIsBeLRevp6clVdEWfjB9MzcHy8rItLCzYo0ePjvhkPkf7gTGGiE51d3fnqGuSAK0ig8ZSNIhfFxcXbXR09EghETalYvmUP2pubraOjg4PPrWgiSBa6U9FC/lZWFiw0dHRXJCd8snRFyiCPDg4aNeuXSuLIGuRkAZzn/nMZ04mLjhFkl6h2z73uc/lDBCoeG1tzZ3w4uKiG6OiR2R8Co1q5UiEAoFnVahGEIYT1INgdXX1CBSpkKiWv2oTsq9//etWVVVl73nPe3KUlvLL5QIidQCxZw6HpR7EKrLV6is9/GMwyGblM1QjUk4foodPyvkqLK0HQdRWAYmbPa3+UPooNh/kYFJBK4+K9LWS6f79+3b27Fl761vf6u+slTmgY9ooLyJ0ESGEJtIAUIXw2sJBK7hAyAhcFUUhwI8tG1TLoGutyBPvpN+dEpur4F31SdqEjwxa9UhRnGr2FPqP1X4RNcC+Y7kyh2Z8Z+YbG4xif9Y6inSj0F/FsFpwENc6tu6InZK1DxXvoBWgWnWZKtPWtiGxqjEKk0k+UlVEsZJRvzfur1hxqho9peBi+5QYVEG9MedRfxZ7SUU6VnsjaXKnPk21RKo70+SCOVetGUEMQZ1W34Gqsd6vJRNQ7ZAi8Ly7apeUBo60YyrBVVE876toDD8696Cq2K4Gsy0tLdbe3m7t7e3W1tZmbW1tuZ5L7AutvmbOodMWFxdtcXHRFhYWcnol7EMLBPBj+s6dnZ3W3t5uHR0dPgdqd7z7kydP7Ed/9EftpZdeOqXbTuIZGhrKPvKRj+SE21BuWZblqh5Y2MXFRV90jE8FwQQu0Dkasbe2trqRsdAqYMTBaEWMcvsxYCOb4Ltxbhh7e3t7jsuO0CdOBvRAe0RpR+XUBlcIGLqRR4MWnJlm7SmdBxvkOG1YRE4IGFX4rK0NdO61skJLr1UgTxASWwho5p7SlyjcyyFVX19vbW1tyYZ8Eb2DhuBwVnpVWxdoJYdq0SLsniqL1dYOzA3OzSwvgi9Xkq/Bo747iKmW4qv4XZvaaQm1vruiFVFMrWOIvVbQ1EQNnL67isxjmw1Fa+OhGitJVQzLoRobamqgHkX/oHUE7Lw7a6jVXlGQGytZ9d2Z+1TLgZTNEzjzaPFDrGhS/Y7SYByIWhWpQVTc6xpQqYaIw1nbLsS9HhNEfXczy9H3keZRxDr17gTN2sIkotUpLRfrTnCmCWHUoGq7EJIb1epgV9BLJOfqawmmKH6JAQVrHc8YbJB314ayvLeiYpxtjAFfy57nbNRCFw3iNJBqa2vzgFqRQapxt7e3c3amZyvnK++ObzR7WvGpbVD47vb2dvvd3/1dGx0dPQ2STuLRFgDKS2sfkmg0Sq1wqCqSolB+R0eHGysGg6PiUFMUh+/V71RKge/VnixkmByCGKcaq2Y8iiigQ1AKRb93aWkpF4jxvZrR4wRaWlr8fTXbwDmQlSqSEDcImQ4bhO/V7BanpFlOW1tbbq6VulD+X+k7nB/fq45Je7zwvRyqSpV2dHTk5lnFtYreqG2x8fV71bagCrAt5lnpumhbBMNQdjgjtS2+N9rW4uJizqbJprXRqNr067EtDt2dnZ2clk/XWG0Lik5tC2pKbau9vd3HAEXX0NCQtC2+V22Ld07ZFjoxtS0OOrVrpQb1HitNatR/6PdS6AA1XM62UvOsFT6gkooCa4Vpyn9gW9qg9CRsSwO66DuwLZD3crbV3Nzsa6vfrYGdUn/od/he/T71l2pbIETRtlL+koCagx2BvB7sGtD8S20rBhTRtkBBCYo1YYvf+53YVvxeKC66cGNbsVBEz0S+n+9V21Lxu9oW39ve3u7o1HdqW/r9qo9CZlFdXU1/pdMg6SSeK1euZB//+MdduE1WR1ZFBqPl/jGziLSXNgCLnK+W4yo0HqFpzaJjJsnhTUZhlq/gamhosOXlZaurq7Pv//7vz214RQ/4XrQ/ZPARjlcaBAcFx64tBcpRPlriC92kV1BwmJYTbKpIURs+AguX66WjAkn9XjJgMhMVqvKr9jTS7tva/TiKwFUsu7y8bBUVFdbb2+vzrLSq6kxSYmsVG/OQOWrPGi0tjoJmsuzYuZrPVhExv9c/j52udRxRXK1ji7+PYm90fmaWHEsUVbPuKqjGDlTIrJoL7bWk4vOoM9QijSigVspN14Q50EIArYZiP0C7IebW8nal/rRnUOzfBNWp1Bvvpu0ztHdTbEarGj/d66BWSinjz7D/FOUXWx4o5QfFqkUP2iWaQDxFpyu9GxupqiA7tuWILRaUblNUVIXI6lO1hxi2pUlvrACNFbmg0diGXiGlVJeyD4rKbW9v+xqpXkhRIRAS1e/h81gjbVqrQczi4qJr1gho8HnYper1+K6Ojg7r6Oiwzs7OXLUptqXUmgYy8/PztrCwYPPz8zY3N5dDw7BpDRo5H9vb262rq8s6Ojqsu7vburq6fA6amppylB4+mqR2YWHB5ubmbHZ21ubm5uzTn/60ra2tnQq3T+KJ7fa1WoFoGEenDlZF3ApBqrNWGiIKt7U/CBUnHAbQRvR+gMsH9gZ2hd7is83MD4Xbt29bVVWVve1tb0tmKLwfRqtCx/X1dc/GlpaWcj1jlLPnQCgUCv4uiqhEzpxDUoV3QLma/epmVufFxsJBt7S05L6zvb0911eJTagBCe9CZs07gmqA1mnmqaXAii4oQqg9iyoqKuyzn/2s7e/v25vf/GZHcdRZIvZFfI+j5qH7rwrNNdtlbjXY1iaKBLcqbsZhak8vFbYS8HHwvBZqpNQw9Kgikooo8J4ckuwX1XOlIHvtkYKNaQM+qDelRxYWFnLJjHZ9VkoSCkAzakUS+P/4ANXOcbjre87OznrPtIicqHidA17fc2BgIFf1g18AKeLQVYEwGfX09LQtLy/nGjBqHzL8DQedrufAwIDvUeycyjvVlagN8SsCaW3YqnoW7Cgirb29vXblyhX3gawJ/ielo+H7x8bGcp3XsSOtIsZmIlJy6dIln1u6iGdZ5nPLeypKgx3RHJb9EjWoikiB1OAnLly4kKtcZq60YSQ2q353dnbWHj58mOu3FGUMis4oin7u3Dm7evVqrs+eaj6jdERbAjx69Mju3LljOzs7nlhgQ1p2j/2q2PvKlSt269Ytq66uznVDZ7+oT1CEc3V11WZnZ+355593hJOAUfeL+tzW1larr6+3jo4O6+vr86B8e3vbvva1r51IfHCKJFleuA16gfPD4S4sLLgxrayseHYCH19RUZFryKUCN4wIh6HiPoXnIyerYm2cn6IoOAStjFAH/1u/9VtWWVlpv//7v5/jv9XxqcPl9yqQ1sAPJxT1PfpuKozWrDXVvTxmU9BamsWBxsUGk7w3GY2KROHY+T7VlRDoIf4GSeAQVAQsaliAoclUcVpaeabo3ze+8Q3b2dmxnp6e3GGpCBgCXG0WGTufqzPXUu2I9rFu0KdKA7N+HBApwa2WaWs3cYJZgi5FF1VczVxi39oPKiUw1k7qOpeqAVT9U9RAoclg7XgPbWypc6lZKOiRVshgJ4hstfs036fl7gQzKiBWKkwrFLWpZ7wKJjYWBb1kLtF3YSOxyZ/aShRLq9ZEqUetBlU6iO9TYbYiJwTloDZmT5vcMpeqI1JNjeomNcggKNeq15ReE8Qd3Z4iNZF64s/07jIQE5pbRpoNf81aKvWjFD1+knNBgxsSAJJXzoW6urqc8Lmzs9OpYwI6FZ0TXGjhjqIznEfYrDaJZG0I1Nrb2627u9u6u7tzaBQImJn5Xsd3KQo0OzvrwT9JuiZW+GnWrKury1EgfpTKI3jnXFhfX/d3mp2dtZmZGZuZmXGBtyaQ2AyJY0tLi3V3d1tvb6+/Y09Pj7W2ttp73vMe+/a3v31Kt53EMzQ0lH30ox+11tbWHP2F88CZUV4KnEgUjGGpeBbIWYXSnZ2dDlsSeauzwVBxXpoRRxGbirSVY1fdyF//9V9bdXW1ffCDH3RnwyHJk8rENcrXsu7o3DRTjI4NXQ6BBVCyVjOpPkbnkgo6bYDI5qf/SGw4qNoU5dPJhrVaMVKmCM/R4pg9LZeODQa1VJwASoXPZOBbW1v2uc99zra3t+3GjRu5g0MrmWIHXEVxlKblINZ+L2Z25LBPdd7VAAo6SSkbbYuggaiK6uMBpWiKCj61WkcpIxxjbGKpa6nCVrJIUCoEzRy0+n1adq+tBw4PD3O9e+KcaqDBAaVdsFPl/ipijkFGDGq0gasGGSpa1/5bqZJmbbFAkI9Nm5kH3Nh+bLqIrRJAql4MZC1qXDSwUTrI7CmSi12A0ESRLrZKUAnSqIhC1EupDpDilyzL3N41YOOw1wIU7fGmlWXse3yoauL4fxqYYqONjY3u06CdFMkA/dQGwJp8EshAAynFx3xUVDy9Dw07gebq6uqytrY26+zsdF1nQ0OD+25aIYB+EbDNzc3lzidE/js7O86QgNq2tLT4udTV1eW/Z2+qBEXRNt6NIIpzcXFx0ZMZ/T5skyCqs7PTAynej+ak0MMkKQAHGkQRVC0sLPg+RctYWVlpTU1NNjExYaVS6TRIOonn5s2b2XPPPWdVVVW5iiY2sC4IC6bl9xgCh2lLS4vzqio05QDQ8m/0AARCmiloVYEGKBzebCo4YxUQNzQ02Pve9z6rqKiwZ5991p0+TkGzIH74fzRhxEEhrNRMgVJMAhU0A9rtNpUFsbmgQEqlkq9DVVWVz1GKE+cAoDKmuro6V8oNyscczs/P5zI8EBYCBM3MmT91hor8oe3h0OBAWVpasrm5OVtYWPDvX11dte3tbVtaWjIzs7a2Ng9gVWDOHLa3t+c6B5s9rTrBCWIb/GhPHBWlKhSvZbN8F8EswTmU8P7+vmel9LvR9VJht+rCOHz53OM0DDU1NS58hWZlP2l2DL2LBk6rl0A3dK340fYWOEsNPDg0sA0Vj6f2s1Krul4dHR25ogDdzyCmUQsyNzeXq8hM7WfmS+dQAw+CAK08JYBTu4iUqu5ngjNFUbAPpWrift7d3c3t5/hd5fYzdCV7infr6urKVbeyn7W4Af/HekXEBn0kCJFW1EX/m9rPWrmsSJT6DsTfGuyDJKqOBi0N76jUu+5ngii1DdZOE9K4nzVAjP63vb3d9zPFOJrcq0/EjyB41v1cWVnpiaiiUJrcE+TrfiYgiggbWiR8R2o/gyDyPnqOaVf3cvuZs4WgkD2+vb1tDx48sO3t7dMg6SSeq1evZs8++6zV19fnaIaUAI7faykqaIA2asMBqK6CzJJsRwWNwI6KdIAIoBUie9S7tGJGro0ZP/ShD1lFRYX9zu/8Tk7ECL2nZb4cSmSN0Avah0fLqrWMWjUbKojl4NE+NOV6wZhZjoYiOMOpaw8UFcGqwDc29lQxNgLgg4ODnO4MvZF2LAYZUpE5j/acUaGvintxOnQ8v379erKrtf6q4mbWOIqmVUytYmcVIJuZxT2tYmmzp4JjFT3H3+uPvrs2VS0ntI4ibz6bd1XBe+zqnWremupIrYJmXVczy302NqNd1VVIzXzrWmI/sXBA9wX2SpFGqnu8lltrSxFQMYIBDhB+rwgc64lWTHuwcehr5+LUdSlKXeJf2PtUQaHtwUZTjQYj3UYAYGY+bwQc2ipENVoEAGg98St8fkpzp1Qi66Tvogc/lVb4GYotqqqqfN40WNMEDL+mFceg7DHgJQCgkS76UHyIolBQQBq4gQpVVVX5fGilLRQXPxpEse9BaAgyoJt6e3utt7fXgxwNavCPoEHz8/M2PT1t09PTNjU15e+HTSoapFWPfX191tfXZ729vdbT02Pd3d1uK0hJDg4OHKWcm5vz75mYmLDp6Wmbm5vz9VI9G9Qy89bX12f9/f3W39/v6BPBdW1trVe1afI/MTFhH/rQh2x6evo0SDqJ5/Lly9mf/MmfWFNTU666jcM9ZoVaJUAkvre35+WsCCQjYgA0rJsex6zBkUbGCwsLuQ6+ZuYQLQER2S3fg4MplUpWW1tr3d3dZma5nj8xkyHzJCDDoatwmHcgk9H+HyoEx8lrNQebT3VPBDCqz8FZkr0oxM1BoA0o1VlqtsQakS1h58c5y46OjlwvF2wBug5nyXzxXdiClqACGzc3N1tXV5cjOnwPc6edrTn8maNUlqTUIM4cjRHtCDQba29vz9GQrGsKheCHJAAKGe0PAUC0beaNgwanqs1ByfAiUorYGGpVS4aZI0XCSDhwkBUVFW4LKhiPSBj0WKlU8mAKarOhocHnS9dItSnYAj4BuijuIWxRb4pX6l2RIuwC6l1F0xFt03nT0ucUOsphpe+iCDOHM4EoAUzU2YCck0iZmR/O2AKfiy1owIGwv7q6Olexi19QRA/fp41LSYjwCRGxoUiDZEo1l6lKK0XYQKIIyrUTPOiazh0BqVYE6x150RaQKmhJfEQN4x7C3lX7iE/Y3Nx0CpR5U6Sc80GbYJKUR98DAkUgTFJC8Kl0miLYilyriBxEnjWJe6hYLOaSW2xBRfnYN35idXXV6WAoXt1DbW1tfg4p2kpyfXBwYD/xEz9ht2/fPg2STuKhT1JFRYUbiQYrZA5a/UT/iRRFxOJ1dXXlGpipowU5UiPESPQ79IBicykVpRlXfX29Z+yqNYqloByCysWbWQ6K12ZksTIuFUhq5sj3aaNHNhXfoRUoqivAiYO80McpCkI12NIWDIiwceKqsyGg4/9RtVFOD6KVUarL4DtACqKOR8uR6TINtK+BquppgKK3trZypcC6JvEHZES7SStql0INokaI91ABsjZg1C7hIDvaFVzfQa/GAfXEYad0T6AhegUJiJ92hU418OTw4DuiLkepgebm5pxwWy+kReejOjWlqbhOgkA+pfvRbvIgPvod2iNKDwESEvYg2b7S9tpPhnXhPbSUP4qXNXgnkNrf3/fDV4NQpZgJSDjUzJ4Wl0B/6aEW0ZTt7e3kd3R3d+foZU0WqUYiQeDgpJSb94H22t7edgQYXRSIDUgDwYA2LMVmtIoWkbAGHfgTpdbwu6AbPT09Of0OHfW1gla/Y3p62mZmZlwvBKpOYUV1dbX72q6uLuvt7bW+vj5/J+2Ppd3T8bVTU1OO1PBOJDp6RRRoXWdnZw4N6u3ttc7OTg8G8Ys7Oztus6BNfNfU1FSOGgcxRoAP4tTf3+/f1dPT47YNunpwcOD+fHZ21iYnJ21qasomJydtcnLSg/XV1VVPdmtra/384PNBm/r7+62trc3e9a532QsvvHAaJJ3EMzQ0lP3hH/6htbW15a4JoRqFwGJ+ft7hT6Xd2HwKh2tU3d3d7ZkC+hQyEja9OrgovNNy/5iNqF4CrQQO6Ctf+YpVVFTY2972NqfYUhV7OFMEoRq5Rx2NVmNAs8WMJ9WgUdEWKDYORu1Qq+W6HKQ4XaiX7e3tnJheD54o+uaQ1u7Hqr3gICXgIBsFpQJtiZ13tYoMekOvbGhpabHd3V1ramqy69eve8YLv05Ah7PUbt58j16RQVCjWbzSIPxKwMMBZ2ZOHcUAEFGnUhNojZRa1SAzVaJOUI42TKuoWHuQK+27o1fopAIPgh7t0Ku9y3RdEFMTCLKGMfBQLQ7/rUioNu3ToDwWMpRKpZwIXu8jU9RD22Cw77VRngZoIMcaRGnbAj2w9bDje9iTvDN7UosyNBkDwWFOzcyDIxVIk/Bpxo64mUBSNUT4L1AikA7oSgJOfBXBhyKGihCB1IOmLC8v29zcnAcd+DKtWsW3ME8dHR1OCxFEtbW15VAo0Dtt5xCDKJDcnZ0dD1ZB89vb2z3g4H06Ojpy7U8IOPGNs7OzHnQwb/iXra2tXDEH693b22v9/f1O4XV2duZaKeBjFcEn6EBovby87L5FK1/5PL6jr6/P/4yEprq62hE/NJlzc3Me1Oi6KOrNHmEN+A5oNAJOgmfOX2xqdnbWxsfHc/M1Nzfnf0/b0HR3d9udO3dsZWXlNEg6iefmzZvZpz/9aauqqvIsXGFNFohsZnV11R09Dripqck3OQsOkoRTIdvXg1c5Z5wXYzB72iASh6JVAcpvkynjfNfW1ux973uf7e3t2bvf/W4vqVQOmAMewwK61IwMZ6ndbjlUVVTJHGnbgu9kjuDNmSMNgk5yjqCBYkaJQ0zNkbY/KDdHBKYc6FtbW/bJT37SNjc37U1vetPrnqPGxsYcEslPnCMzy3X8jXNEdp+ao2KxeCTr/m7maHt7O6dbg4LhO1QUzOdXVFT4oaaHt84RgQ9zBDqhdqRzRCBEpSFzBGLLZ2u1EXSs3v/3WnPEXgA10AQnzpGiH3ptx3FzpG0mCMa1ekkTte9mjkALtNrsuDlSDVa5Oers7PRAXTVEGqSVm6Otra0cCnWSc0QVYqSIdI5IZknavtM52tzcdBRY9VwpO1I0+7g5UslFao5IxuMcacKv/ZH+W82RMjDfzRwhe2COCMRTc0RFLXM0Pj5uW1tbp0HSSTxXr17NPvGJT1hdXZ0LcLXMOFVdwQFq9vR+Nu3xoZwsdFtU6EPrKRqC6DDqaLQUPfYt0bvHlK74wAc+YAcHB/be977XM1MtsdXeKCA68a4tRSYw7tirB1pEL2OFCmH8bGwMXwXZKt5FJ0HGlRLQahds7bbMd/BTTqSL0FdF12Td6FtUCK3fwY+KsPWzzZ42J/3mN79pWZbZ933f9+UE1wSCqUeF0sf9t3a7Tomu41j4fhWw83vGo59ZTmCtdkBFF3oG7IJ10w7iUZxPUKVd0nW9tEM4n6e2oD2M+A4OcaVCY4EBAQ6dr7VwQvtcqQ4QlIUgJLYU0H5afC86DypYtZ+PloKDeu7t7fm49MobRb1AVkFJoNsQS6sGkOADNAp5APOgpeaqx2O+mHttK6FBuKI3oJDYiPpAqCmt4mQOQQDW1tb8M5WWovoQ5JmeQ83NzY52dHd3O2UEwlZfX+/2Bnq+uLhok5OTOUpqbm7O19zMHD0nKIiC4e7u7hw9TJAPpQZaMz4+bjMzM47Ug2qCOhWLRevv77dz5875DxQRTAP+gbWcmZmx8fFxGx8ft4mJCZucnLT5+XkPTKAfm5qaPJg5f/68XbhwwQYGBhzhIhEiSdnc3PSxT0xM2OPHj+3x48c+P8vLyx6gMPft7e12/vx5GxgY8O9g/O3t7Y7I7u3tOTXLZ4+Njdn4+LiNjY25tKRUKjkqB23a19dng4ODNjg4aOfOnfM15szQ6k4QrPHxcXv06JGP//bt27a7u3saJJ3Eg3C7WCzmGl3hZIi4gUNR/9M7CANVJANoNzoHKgygPwjAYhUDvTSoOMEhK0oC4qCcOAa9trZmv/iLv2h7e3v2sz/7s7lNu7S05NQaARjiVTIGoFycMxmJWT47Z16YG8Tsm5ubuR4ZOGU4fRURMi9m5geTIguKLnB4cbBrM03mnblB7N3Q0OAdt4GIgYnVIS8uLuaa+bEhtewWZ8nBBX9PtgbSODs7a1/60pdsbW3N7gdb8AAAIABJREFUWltbbWFhweeFqkFsRmlZDhRoDeaGeaFSRDPBFGdPCXtTU5Pbi6J3eskoNqPl3THbX11d9SCIaymoqkGEjE4DmyHYOTw8dJtRtIjf4yjRmiA61SwZm2FuQGVICLa2to6gmrOzs76XtBiBvYR+UPu10MG3oaHhCM0DBRO1MpoUkLCokBlfoIc4NqO3w0c/g81QsME7q3YlNgnUCjcQFtVXzszMON0Cdci8EFTGzJ65YV4oOCGzx2a0CSBl31CX2koEe+E7VPhPAI6uhwCKecHe0cHgw6qqqnI0J3OOzeDD0KRR8RdtBhRW/W+hUPC9pPaoSDiJHwG7tpBhzpV+BIEFya+vrz+C2HR3d+cq/LAZ7VzNPtWSe5JskjgttiC47Orq8rnSJpbsJRXWR5shgaewh15gajPQptrbCVG9lu7z2cwLCfze3p7Pi7a5wb+zpnpmI8NYXV21ubk5+83f/E2bmpo6DZJO4kG4bWae6WlpJAa+srLimSWVbMCRGDYCOBAfRV9wSlraiSHqbe5k2Ko3UmoHgTNBC4encreLi4v2mc98xvb29uzKlSteJcABxwGkzlyFoRFCVUeLY9QKNTYa/Ty0okJFh2TY2t1VG7wx/9rygPnQviQcFFQn6S32ZId66KsIG9SDzFG7xfIDxciBRpDFmBGwQncgwsWx1tbW2vz8vNXW1toP/dAPeQBNYAVyoo0S44WZKlTWlgwgIip6j1170b0wZu2ATMM31o95I1jW5qDMuQZs2hhQPxfYHlQGpFJ72KjuTJFQtDogRvG6D+aDii72lTY0jVVCxWLREUYE5wQ8WrnDfGAbIFKMT3sJaaNUkBMgfqqpYp8u5hr9Fnubyj2CV9Usoikj209RGKDDdCImmdKrP3p7e3O92kDqtO2IBiAcWsy16vpYN/xcX1+fJzsqKNfWKQRO+FItZcc26urq3BehU+Eg7+rqco2Kmflnz87O+meCJKj+jc9m/bq7ux21AYHq6Ohw36IXqWqp+sTEhE1NTTniTwPIqqqqnA5pYGDABgYGvGRdEX5dQ4TPICoaIPAwH21tbXb+/Hk7f/68oyl9fX2OjGrDRUWDxsbGbGxszD97fX3dgxqdD5Ag5gU6trGxMaf9RAMEEjQxMeE2SCVioVBwLWl/f39u3AMDA+6r6+vrHRleWlryuR4dHbXHjx/b1NSUn4skKfh+5oNxDwwM2Llz53yfQoHv7OzY7Oys/czP/Iw9ePDgNEg6iWd4eDj72Mc+5lAhk80BxUZns8/Ozjq9pHcVaVBD/whFBVTTg2BTkRiyUxyBijW1/F6dKvw8VVSMa2lpyX7jN37D9vb27B3veIdDs0DE9AFCZBybylFNw5jJqGMZPD+IjHGqOCjtraEZo96srcJipQy062+pVHJKKd4tFgWyWplDxqaCaO1Ey7i1wRni6ygijxcs6mcT3ECZbmxs2P37921/f99bMPDZeo0Mn40QmsooqlcIlFScjqAc2oyDRis+9PMRp0LRaGdgnQ8OXb23jkAyViMS5EDfxDJiDSRV9A7qQjWXNsrjc0FeEYib5ZvxaTCivXhwvmaWSzS0aSIHOmJ9Eg0ds4qOOWRIjlh/qln5XLQU0ClUTtLUDxSNA4DMn/2IyBzansCGA53MWZFLTY7wS/r52jYEilDRLXwUc6/zATI5Pz/vhyRBFHvyyZMnnmQo+gRNBZpLsmhmuQtJCW74/JmZGR8zSC4iXAI+DnT1q5xh2PLc3JzTUhzC8/PzHsxS/VosFp3GUeqLMdPSZG9vz+lFDXCo8NK766Aru7q6nFLjUFeNp+5DfL/SUZw3JA3KJjBODRh0DfEf2Nvk5KTTaFNTUzYzM+MB387OTu4qLf3M8+fPW29vr/tVklsouunpaRsbG/NxYx+I583M/bIGTnw+rRUaGho8KV5cXPR51cAJdJV508R2cHDQP5fAqbGx0X7sx37MvvWtb50GSSfx3LhxI/vUpz5lVVVVOSiTjat31+DUtBMqGQ+ORykHUAgiZ7IrPptgAxTCzFwTpMEWn68XbmqFARuCz+UQKZVKHjBoUEEmqJUe2vWWEnI2KxsAxw7kyiGqdBQODYegonXtBcVm4MAAnt/b2/M7uLQZGz/a7wUId3Nz0w83HCPw88rKirc5ICNuaWnJHRQqOqUsGQQAKoH5nZ2ddcTkyZMnjrAoldDb2+sHPo7GzPxwY1x6AHG4cdhTldfQ0OD0gc4DDpk1Uw2AZtoEQYi4syzzw5zsncy9s7Mz13FZ7xpTpAFUgERCy6UVGmd+cWhUdR4eHvpBTPWNUk1aHg9ypgJVPluv3dBGgxxq2iSPgA1dGLePNzc3+xwojQJFZPYU0VFqnOwdWkjpQtUlgo5ocz+QDoI+SqyxBVqOEJyBICLsV1ugkR+fS5+lWEGlVBh3CUI/Rltg/WgDUVdX5/QddDINAvFjUM3o9Orq6ty3EIxoeT5zRY+r9fX1XELKd0A57u/vH6EzFdXSCk+avuLLCXDwC+wJkgEQSfwC80pCCqqML9/c3HRbVX8OYr23t+fFANqiIKJ7mjwTqGvbAGwNf0OQB7pC0Kvrxjzo2UPyGc8e/A1tQUCSdJ8xF4rak+wQlKo/VyE4QbeKzLUCEPtAZ6eIXursAUDQs0fpW+yspaXFfumXfsnu3LlzGiSdxHPt2jUXbusdWCy+VgGQgQO5Kt9LsKQZpdJtUf2P015ZWcmJK9mQeicSmQKHeFVVVa6MON71pJ179RZnFYUi1OQQwqD18kgOAIyT3h4gRppZ8/mgC3qRr9JKUXyLaFqF2GwAaE3tpYPo1sw8a2Iu+Cy9i8nsqfhahcja9Tp2esbJQnMhftaHfxe7VJtZrlu2dqBmPCq25sBMddxWkTU/2tmav8u7MEesi4rW+Ty9t0tF0YjZeTeoqtj9XDuSMzeIcNF6aPd2aFMz889Tmgz6gOIGqllU2Ku9mGi7wUFEFQ8BD4iXomipppNalaZ9pLAX3SuxIawiodorjICfYFXF2XoxLqiWVvTwfQi+t7e3PYnSAF2RMzNzPQ40CrQMwR5JVWVlZQ7VUhRneno6lwQSjKDvUaExWsWWlhYPTNfX152yg+4hOAUB4F40gkUV/VJuDg2dZZmvk1JIIC2gt7u7u26XPT09jiYg+gXNam5u9r1D0jM5OWkjIyP26NEjm5iY8MCBPcf1N93d3TY0NGQXL17MCaFBf5FSLC8v2+joqI2OjtqjR4/s0aNHNjY2lqtkZawgHxcuXLBLly7Z0NCQJym1tbXubwg8xsbG7MGDB/bw4UNHyFZWVnz+mdP+/n4bHh624eFhu3Dhgl24cME1iPX19Z78zs/P26NHj2x0dNRGRkZsZGTEJiYmPIml0hPk6vz587nPxW4RaW9vb3sANjo6ag8fPrSRkRGbnJzkDjXb3d21LMt8rAMDAzY0NGRDQ0M+VhLg2tpaDxpnZmbs8ePHPq8jIyM2MzPj0gwKdVpbW91Gh4aG7GMf+5g9evToNEg6iefSpUvZxz/+cWttbc015SPrgkfW6JuKHTrCtrS0eEajfDoHPaX/0Hex8Ze27GcjaXUIjbjInGtraz0ooG8IGePU1JQtLi7agwcP7MmTJw53s+nZTAqJt7W1+cEN0rG8vOwVG2R1CwsLHohQTUEmzruThejlvVpJoe8/PT3tmwE91pkzZ3ICcjJxoFs6qu7v73vJLJ85OTmZQ9Gwb232iUMmS2xpaXGRORRGfHcEo9rMDuoMBEIzpPr6epuYmDAzs8HBQdva2vLP0WsAtCsvQY22SWCsWkKvlS+sPQceaJSW82rvHu19QpM6aB4CnJhx0ieGYIAml42NjZ5lKjIJqmH2NOCO2SaHMo6erBv7xObJujs6Orwqx+ypdpB3VzROm00SiClKwp4iiKqvr3f6iHYKoETaMG9jY8MDUBDkzs7O3JySxZuZNylVVIs+MqAbBLtVVVWeFPHuaH5AtdDdRdRFEW8QLUXgFNXjB51ZXV1d7u4tbJQ5BdVbW1vzwFmDPN1L6AVJFLTnkDYhZD5IcBDqp3wofoQLnaMmSd+fZE51nQQ5+rl0OEdoTfk9mi/2knafJwFC86WNEglyW1tbPenSbunMpQq49cJZbFTXh32l9zmShGj/JuYUm8CP1NTU5KhV7AkUHbTt8PDQNXoEzjRwpGADW0aakfKhigZpHzP2u747FdCKjvK+eoaojpVkjnNI96f2+QO86OjosOeff96Wl5dPg6STeG7dupV95jOfsSzLPAucnp72kksOXfh9M3OdjQoCqVJiYQuFQq5DLVE1ToLKEg4yvRdHoWnVONCfSMtOORihlfb3962+vt7u379vVVVV9u53vztX0cBmRg9ExqYHImJCUBqoxebm5hwMrdVGaJc0GIxN5XBKBG04XIXhdcMdHh7maK94NQM0E5et0l9FM38qONBtQRdCUzK2paUlP2ConKNrMJ+piB7VW3yeXuuwvLxszz//vO3u7lpHR4cdHBx4BYxqtAh+tBWCdrPWXizoqKBTQO20+gNUU4XFdK/W3l/aqJTDiuomqF5dX71uBG0Q44tXMRCggJpwNYuuMZQe1Z5kuFopyWfyeVRKUrGnDQg7OjpywnLmUCF7DkGoAkrKI51JlvxahzOHFejL4eGhfx6Uqx56rJf2Y1N6GKfPYa8oCSXp2oCPQAda9MmTJ7lAFOExB32pVPLgob293Q9OkAK9lke7bHMQoz/BflZXV30Ni8Wiv+uFCxdcz8LeAQVdX1/3+QMd4FBGvMz1KqzFhQsX7OLFizYwMODrDTL65MkTp9AeP37siAjBHshoTU1NrjQeVIi14eqZg4MDt7+xsTFHW5jTUqnkiDI+YWBgwC5evGgXL170tUG/WFlZ6cn25OSkI0xoprTBLnu3t7fXhoaGclobKhfPnj3rAcjMzIyjK6oLIpnRvQeyBmJDO4DGxka/aWBxcdHXBD3Q5OSkB8m0cGlpafHP0TJ92AkuU1cUkM8cGxvzBNzMXO+Idou1BlVqa2uzqqoqD5I4Q3VtOF+54qq6utr6+vrs7t27p80kT+oZHh7O/uiP/sg6OztzWRDGjbPRLEjhYxwBmYWWuMPPUy67vLzswQ1BBBklWZU2aovcrV5DAeSslKBqDl566SUzM3vLW97iGVCqZBjqiWAEdEI7825ubjrFodVVetjDsXNYKXVJIMKvZLyUlOvN9fpDB9ba2lqvSlNNj3ZCBg2AgtGKOBwQXYnJzgg4QHRSVVpUb2jWR9bf2trqGQ/UIqjhF7/4Rdva2rKhoSGnatfW1mxnZ8d1MQROID0qQI/NKeHq9Z0ZJ8Enc6VVTjpOoGntcwUipZ8L1UXZuTaJpBKJjLSurs7nUa9K0ICMoEAvZ9au9Eo5kWRAC2LTauc4bqV9tfkg+4a55BoP6BalsBTVgp6uqKjwPaJiZOYSfcrh4WGubULMxHnnQqGQS6wQshLgNTc3O0KoXau1CopqOR5oQO2Jg34RZNjMcqJxBLx0X15YWHD0XBOWixcv2uDgoCPYXV1dTtsS3M3NzTn9oWgJ81JXV+fBMUEEwVNvb6/7RhACxjcyMuJjnJmZyXXoJ9AZHBx06odqOLrZ7+3tOeIA9fPo0SNHhtFjVVZWeqB98eJFGx4etqGhIQ90aPZ4eHjoSP3jx4/t4cOH9vDhQ19rEhczc3RxcHDQLl26ZMPDw05VNjU1eX813nd8fDxHo01MTNjq6qonu7zvuXPn7NKlS3bp0iVPykHAKysrPZmamJiwhw8f2oMHD2x8fNyr/jhfkIX09fX55yF2RidWXV3tZ8D09LSPj8BOK8+KxaIj/tBxfB50HFpf9FuPHj2yhw8f2tjYmCP1+K+Ghga3Qai4gYEBX2OCIG0Tgw2qWB+x/d7e3mmQdBLPjRs3sr/4i79w41AKQ8sdS6WSOwAOnxSSRF8IM8s5UDaB9oQAIdKeLbHKor6+3ps/aldfxqdVYJopPv/881ZTU2Mf+MAHPPOkQSQiTBW38qveX0RvFBAuvesHUXZtbW1ZrQOiQ0ozObQ6Ojq8iRqBm1bTwfHPz8/7ZldRL+vAZ3V2dvq8aRUTeiBFZfg8DvCNjQ0/GLX/DOMjIGhoaPB10P4nOEsa+JGxl0olq6+vtx/8wR90XQe0IaJr7UuiazA5OZkThytKpmXSZOusu4o/9ZoANAysgwq3ceIgKOp8tf0D9qvXJzC/BHkdHR3uJGMAwDrQtZjP4/BXSgfnS0Bx7tw5XweQjv39fVtZWclRD+wv2j0wL1A5eqs4Qb2uQ7yjCrqNIFULClgD9gRJDGuPaJjmf4xTEVUoW/aDrgM0eXV1de4+SRrnaY8s5lcvvNbPI0Hg79EjKLUOKysruX5VjIfPYx2KxWLuPjT8EmXz2uyQREgRJygbOplT5YS+K94TphV7WhXJemoRhlLI0OfsB/wSySmInVJJ586dywWc0Dh64TnzxmctLS35/GrfOV0HEmr8FwGn7i98JtogKE4CuqgNKxaLuYvS9XzQppOgQcwddqsaNtaBXn6pdaBSkPNGq+7iOiCfgLFYXl52+1CKnKCLCl3VF1FM0dHR4eugyRjzpnIYM7NHjx7ZkydPToOkk3iuXbuW/fmf/7mdPXs2J67msMKZoJvZ39/PHfZseoIa7VhL5k9wQ4O72N8E4aXeooxmQK8bIXjAQJaXl73zMKWRCO6+9KUvWXV1tf3qr/6qN7OsqqrKNQ2j7FlL7YGozcxRI6Bb4H2yNhA1KAc+Q5EnNBdaqg50DDJGt23Kw9H+qBhbRdgqwFZBN5+jImVE3KlO2vwbgiQVhfNuIBCp7tQq/kYkzRi+/vWv2+HhoT3zzDN+KGq3cPoekb0xH4iioQfNnl4yqgJp5hfBO43pQHcIcMnmKioqcndt0QKBX9EwmVkOkUK8DDxPYH9wcJCjBkkwaHGhaxXvTSP4r6urMzNz+8autfEpwmcqprSUnb0CRcteUdFnbBjK/IMCc9CCPjFG1rpUKnnWqokTmhPtdswBAX3Q1dXlc0jlzsrKik1MTHjmy+exJ0EW29ranH6gT05XV5fbOxT03NycC5C12zN2TmDS19dnQ0NDNjw87J8FFba3t2dLS0veDXlkZMTREtYVu1L6BvSFSrjm5uYcfQ8KAc3Ee6KVJEi/fPmyXblyxWkcUPiqqipP5MbGxuzu3bt29+5dDzyXlpa8cALh9uDgoF29etWuXr3quplisej7Z2pqytGHe/fu2b179zx43djYcDRzYGDA0RZ+Ojuf3mHGvDx+/Nju3btnd+/edeRqeXnZaTSCzMHBQbt27ZpdvnzZ35PinoqKCg9qRkZG/D1JOEulkvsZSumHhob8PVX0DYIPKgeydP/+/VxjWIIbUC+Qm6GhId/vFRUVvg9HR0ft/v379vDhQ6f3oIUrKipc/3ThwgW7fPmy21l/f7/T1vv7+x50qcCbYImWOmfPnvVk68KFC04/sl/Rzm5ubjoNPD4+7nRhqVSykZGR047bJ/Ug3C7HnaNNojvw6uqqOwwc7MDAgBsEEXR9fX1Ou4ED0syew5uACw6evhr0jOHvqaiWcVFFs7W15YdbV1eX/fM//7OdOXPGPvjBDzpnTHWcXjXA5/C+BEoHBwe5Jm80B2NDnj171qswVEwHRUA2pEJKskec/sDAQK4rK4cbh4heJaDXN4Dk0XND2xkQXKjQDx4bpGxlZcUKhYLfgUQ2RRapnD2XhK6uruauBWC+COI0Gx0YGLC5uTkrFov2rne9yzo6OtzetDeK0ikEvRz2KmTl0NUeQgSj6N20d4t2goYe1AZy2tEXwb52waWSSGkjggEVg+LI6FlTW1vrzgv0lMCCcnnaXZBoqG2hC6LSLcuyZI8adAh0vK+urnZBNlTWuXPnclWmBCEcRjo2MmMSFq3qgULv6enxgHN3d9fnhn2tNBud8rnQUyuvtPcMVWzr6+s+LsY0OzvrSKJenM3nQIV1dnb6uGhauLCw4GuoV0wgfIfO7+npcd0LKGJzc7MHFCAwqnHCHjY3N3NIGLagZdjYsnYsZ760xJ8HehObOH/+vAettbW17gP00lZ8BeLlvb09RwiZd1Bc9jUdpvFR2BYB6+zsrFc6VlZWOgXU39/v848P0krj2IhyYmLCfZqZua8DJcHn9PX1uY8EIQV1wW+BXKvUg0SBfd3b2+s0MT2eSqVSblz4G5J17RsFGqTCbFpAgPAtLS3lbIvWEnqWwYywhtpZnb8HCpw6y9bX1716UNE9foXqr62t9bMMloBE4Ytf/KKVSqXTIOkknlu3bmWf+tSn7ODgwKNcvceGzQeM2tTU5IcWzqqvr88zPByoBkYcXIuLi17izuGszcwo2a2vr3ekB0gRxwIcy9ohtFVIHOif0n40VgpzUlkGdaFaKK2GaG9vz2k6Yn+Q6enpXMNHxh/7YXR2djpigwPW/jDaa4WybvrC8EMVDf8fnY5+jiIaZGmqn0IQipC2uro61xU36rvQEFVUVHjvJ726BQG6mbkuBwGtNiXUaz04vDl8oTAoq6bbOQceDpKu71tbW47EUBGFg+RySBAftFsqEtZb5lOVQARRWp2pglGcNsjP4eGhd5MmiMaG9BoUhP1ra2t+yOnVBFR7qQBfaVQqk6KwmM+CpgRtQ+OlASK2RGCO5kSD/MePHzvCSiFEbW2t73kO3nPnzvn8oXfiEEGoChKtgRMOH/FrX1+fI3Y7Ozu+7xG8EpzPzs465at6EBAieuW0tbV58M7hgdh1ZGTEKyC3tra8aKSvr8+RIfyb9nTCdihHf/DggVNFOzs7jjAjEL548aKXztPqIcsytxkQDkrwJyYmHHmura31+R0eHnb0hT3M1SLLy8v+TvxMTU35fm5sbPQ9f/nyZbt8+bIH0J2dnWZmHojj8+/fv28PHjxwqnVtbc3tV+eI9yOAq6mp8fngc+7fv++2qZ3AscWLFy866oKmrqKiwotf0D+NjIy4SBuUnYSlu7vbLl265HPEvtPmriQXzDmi8ZWVldwVQ0NDQ3bp0iVHurq7u12XSPXj5OSkrz/9zVZWVjxQ10QfZAr5Q11dnfuR2IJhfHzc1tfXc/ouupkPDw/bxYsX/Ryhp9L29rY9fvw4d14/fvzYSqWSPXz40HZ2dk6DpJN4hoeHs49//OPW2dmZ6/uB8wXGw7EoJEsGRUv6np6eHISqNxYT5YJkwK1T8q6lqj09PbnLCPUeOW1HsLi4mGtcpmJTDnHtvwLnj+PmV60iIjihki3elQMNwRUPKjRV2k6vKmFccPI0JSNAobpJK//oMYMAFXiYz+BzCHT05nrtxLy8vOwdcRGU0rEV6id135heyYK4GXoKCpEWENrZWbVaXIcBhw50jY6MwJPLLylXpzUBf8bf43PQpyiKiD3V1tbmAmfmisZ1iG9VHwSSxZxDdXB1BoexBpfaPR6kRxtCbm9ve3DJ3KieiqCQPmJcr6CaoOnpaacPqKJqaWnJoTIUIvBsbGzkhMAkFzhngsuOjg7vewNCQAf7mpoaD1Cnpqb8kIIW4FLOqqoqP8y1MocqsYqKChfY05NmdHTUUaydnR1vbMm8DA8P+0HF2lGxt7KykgsI+CwODLQrfX19TmGBMBBcEDhBgT148MDu3buXu0mAfXH+/Hm7evWqf86FCxdyXb85fEdGRuzll1+2u3fv5kr88T0c4oiZh4eHnXJXBObu3bv28ssv24MHD9wOKisr7cyZM9bS0uL/Vt+NPbK+vu7rxVio/JqdnXU/wiE+ODho169fd1qORp8gE9B7UEwjIyO2srLifoRgZHh42K5fv25XrlzxxJAq0K2tLQ90CL4ePHjg50hNTY3vd6gzxMoDAwPua1dXVz14hyLkLFlaWvJ57unp8bkeHBy0ixcvelBBEMeZplQja0/Af+7cOf/3BIVQcFqxp5QZe3ZnZ8eBABKcwcFB74dEMk7H8dXVVQ8AQfMmJydzF+tyPlIAoA05kWSo/m9sbMz+7M/+zNbW1k6DpJN4rl+/nn3yk5+0qqoqP3impqY8KiUQ0JJcbWMPZKpIEhkFARYZLhE31AwZm4oYVbys9AeBGkJS1QiQHSmk/Jd/+ZdmZvbWt77V9RRE3VqGyXUZmiGDcBEIaGClBw8HK2Npb28/8k4IgdEXsFGZX6BlnV8OLb2ziOtGXu/8FovFI++Uml89TLmjr1Qq5eY3ooYNDQ1O6/FOoH04r+npacuyzEvyy80vQkTNQnHs/9L51fuQUvOrVBmfw51vBwcHOVEvNhPnt7m5+cg7dXZ2+liovIz2S7D5nc4v9quJBnfO1dfX58bCPig3v1AFzA1jiXQWBQ98DlV2ijhRDbmzs5OcX95JdXxxfkEt6Fj9ncyvVq0dN7/qq5jfiooKD2rZk5RpHze/vBNjof2A2m+5+Y32qy1JmF+9K4ykQOcXhEHtNzW/2Az+YXl52SorK62+vt4ZAShVUE/GQoNMquSYX+5rBDGH7lL71YSEeVH7Za339/e9wpX30c9JzS92x/Uiy8vLuauDoE9Za+3fpIiy2i8aqtraWmtoaMjNL2Opra31ar945UmcX66qea355XNAS9H6peZX15r5raysTNrv0tKSffOb3zwVbp/Ug3D7zJkzHoTAk2pfB7Q1QMGxeSIC4J2dHUd5lEqCx0evBN2iYlFurNdKIDJaqjvIZODdEUDy76F+fv3Xf90ODw/tl3/5lx25UOrw4ODAtQ70v1DdkgrG6b4NRM9t9mSUGDRZunZPPjw8dHoE0Sb9jfSuN8r2Y7doKka0mzZZhtnT7tXaSZt/a2bewZh/x4/+W74PyJg2C0DWfDdZC5AwuiaqRlSM/o//+I92eHhob37zm30OaVPA5zBWnFPsWM4D2kaVCq0EQBHRooFqcTN5Y2Ojo1A0S9Qfypdx1lA+VNJoewNgfijJ2P8qyzKrrq7O3cRFSCcNAAAVJUlEQVQOmslBC5UE3K6l6Fx6CiKqlWj9/f1+RUZVVZWjewhAOXig7GJnY73Ik/XV5qZaJk4wqH1mgPv5HCrFsizLBfzQGTq3UKDDw8N25cqVnKDVzLwbPxk5Itvx8XFHIdFuDQwM2PXr1+3atWs2MDDgFVMI6MfGxhwhALWgVYKZuUbkypUrduPGDbt69apTWDQWXFpastu3b9udO3fs/v37XpLPXm9ra3NE5/r163bjxg3XPLa0tOQCixdffNFeeuklP7yodKupqXHEAzTn+vXrfu3S/v6+28ft27ft29/+tj148MC1gMgVOjs77fr1647kXL582c6dO+d7maTu4cOH9tJLL9lLL73k2rGNjQ2nz4eHh+3GjRt27do1p4goKtjZ2XHk7u7du3bnzh27d++eI5Nog3p6enw+KIXv6+tz3zE7O+trwhrR12hra8vt/eLFi3bt2jW7du2a7yG9KBf7wGZHRkbcFyhNqWgb5wvyhLm5OR8L6M3i4qL7G9pKKGVGI2P23/Lyso2NjbmQm6o6mizX1tbm7lRjPmAYaNisgTUFFuvr6+7TtPkxsgIQrUKh4GetVrhubm7aV7/61VNNUuopFAqtZvaHZvY/mNmimf16lmV/dty/uXTpUvbss89aZ2dn7iB4/KpSHr5U7/qBGjt37pxDgNAjlZWVThc9fvw4B/nOzMyYmXnHZjUimnxxbxZIx/T0tHO3GNHW1laOJwfm12qC97znPZZlmX3oQx/KXXI4Pj7uTvXs2bN+SSccO40n29rafD40A2dTkQnt7+/nLqCkEgEKy8w8QNNmahxQHIyaOcOJIxA/e/as3xM0PT2d01ZQAaWiSBqxIXqmW/HOzo47X9Z3cnLS5wO0Bn4eaJdyW+38ynyAZlEhlmWZ9fT02JMnT6ytrc1+/Md/3CkYmq3hYHkP+nvMzs76IV4sFnPXF0C/oP1QhAUnDpKwu7vrVCTUAhldf3+/V/FtbW25ZoL5mJ6e9uaD2l8L+0As3NTU5BWImplCK29ubnoQh3PTxoBavoyuDPvgMJybm/P5QBCv80H7gzNnzuTQRW00R+CEfVCZpfNBALCxseHzwb6lmKFUKnkgwJ67ePGiz4de1Ml8Yh9jY2O5wgMOPyp3QKyKxaKXSy8tLfm+h8pAfE1X4Wgf2l8LVFwb7xFIghBR3TQ4OJjrR8N8qHYMSoSu5qVSyQsVaIB48eJF13xxK8DOzs6Ra0XGxsb8YD979qzLA6B49C42ErSFhQW3D6jYlZUVnw8V7aOLohKtUCjk5mNkZCRXmAC61NzcnLNRWpRoc2C1j9HR0Zy+C5qyv7/f7YP5oHHq9vZ2bj6YE6pfEfvrnkPs39LS4hQVPYIePXrk1CT3pVVVVeUahWIfFFccHh564g0NTKC0sLBg1dXVnjgzl6A5HR0dnlxSeTgxMeH2QfNSGmS2trZ6w010TiBKJM74DlDryclJb1eCAL+3tze35zhvVYqi8/HlL3/ZNjc3T4Ok+BQKhU+YWYWZ/S9m9n1m9gUz+49Zlt0u929u3bqVffKTn7Td3V2HVfl1bGzMEQhoIEppWSwydvRHBDUjIyNeVfLkyRNHfjAYNBU4pf39fdvc3PRDRqvqCEa4KLGvr88Nnx4+2udpcnLSfvu3f9t2dnbs5s2btrKyYmbm1ShaDYGmg+8ASdO+Imtra45w6XUMZOdk7lwVQmWGQqjoQerr653W6O/v9woNGkbu7+/nWu4TXFJxR/UeTdGg0Qim0BEhxoRe5LoKrXyhkg2Bupl5B2jmQO/uQ9dAtZiK3Omyi/N47rnnbG1tzTo6OtyBId4kQNDGa+iZ0OBoDxCycxV/6tUdHPboZLj3Sys+9IJI7jjr6OhwJ4hwlKyVSjCCYoJaWhdE2pkKKdA4hJ6gPaOjo74OOzs7nt1y35RqcMzMs17VK2APoJQEChwEg4ODLuyvrq52JBc0YWRkxJE4+pO1tbU5qkGFXXd3t1e8zc/Pu65E6WqQStUjUUrd2NjoYm7sGAQBCn95edkPop6eHtf+kHG3tLR4yweaDiJ0Hh0ddTS3urra15D3GBoa8qCQ7vdodV5++WWvJFpeXvYiggsXLrguBv9QV1fnCCrfz8/Dhw8dZW1sbPSADWRIA2Gliu7cuWO3b9/26lBN+C5duuTzQMAEhbaxseEl7YiiCT6fPHni7QQuXLjgaBu+rba21gXjIyMjjpJxINMSolgsusAbLc7AwIBrTNHhjI6O2t27dx0NKpVKtr297cGeludrjx+SZ/6tNkEE/Wpvb/fvRpDf0dHhNk1FHyX59+7d8+75ZubIILaggRqVugsLC456asUo76nnC4EaNp1lmWsHaQwJrQ+y1traap2dnY44UinX2trqdzrSWJI9NTY2Zk+ePDGzVyoK8Svs7fPnzzuTA30OVffo0aOc7rBYLNq9e/dsfX39NEjSp1Ao1JvZipndzLLs/qt/9nEzm8qy7APl/t3w8HD27LPPWldXlzdR1ChdrwugJwfQNz/nz5/3CgdKVDmgyEYpBdWyWRwLIjQu2d3Z2cndI8TnaZt9InKibDYSVV/vf//7bW9vz975znf6HTdc4qk3q0P5IaStqqryJmugFTReXFpayvWZ4d9o40BgUigivQ6DOaAfRnV1tQuxyTpaW1sdSt3b23PDJwvWbt1U8CEOp4qCDU12xr/ncEJjlmVZLmhRgTlB1+7urguUoYaoDtvc3LSqqiq/YFU7kD///PNWXV1t73jHO3JUBsEL4mbucKLPDL122traXKwOTbmyspKjcNGnETjpDfF8jl6ITPao/X4InBoaGvwdcEzamR3qB3pJ2zNw/5oGTiAcPT09TqFqdZyiX+vr67a1tZW7/JSAgyC6UCi4wB8HD9KLwBeERauPqBzC3lRszGFFNlpVVeV76tq1a3b9+nW/KLWrq8s2NjYcTXj55Zftzp07nkytrKy4zXBIXrlyxQ8rrr5AzIvY+fbt297bpVQqOWI2ODhot27dslu3bnnVWkNDg+/Je/fu2Ysvvmh37tzxQJjKwJaWFrt165bdvHnTx3D+/Hmn2WdmZrwXz+3bt+327dt+tcPe3p4Lba9evWrPPPOM3bx50/fm7u6uIxcvvPCCU2EEgyAe58+ft2eeecaeeeYZRwBAY3d2dpy6unPnjo+Ffm+NjY0u8r5x44bdvHnTBgcHnf5FzzUyMmIvvPCCvfjii15NSsDU3t7u47927ZrbJEUnS0tL9tJLL9mdO3fcniYnJz1Q6O3t9bW7evWqXbt2zZOIiooKR4Hu379vd+7csZdfftl91ZkzZ9xmbty44XaEz6YoZHJy0t+dz9N2G9iv9jAiEdKEWqvNKC6hMW5/f7+/B1cNnTlzxqn7x9LlnMCVMwbkVZEkroVBwI0tQHtTvLO/v5+7KxSqWhF1zgPeQxtzMge0QtF7F6neNTNHv2AZKPb4+7//e9vY2DgNkvQpFArfb2Z/n2VZrfzZ/2Zm/ynLsv+x3L+7fv169olPfMKqq6t9o4+Pj3uEio6kurrajVxV9jSL4yBWjQNCYG3eBcRO5k1VHY3mUpAy1QlA9BcuXPDD6MyZM2b2CgKCsY2Njdmf/umf2s7Ojg0PD3vJMHQcGYLeG0ZlEgfI48ePfTPX1NQ42kLFA92eW1tbnU6bmZnJUYMEdmhuyA4VRSMgW1lZyVGciIWBuhHDEpReuHDBNSo7OzsezCptRNaC/ktpAXr7UMm0uLjo1ATZ/tLSkpfst7W1uaPi0GppaXGtAGJpvvvFF190alZLo3E2vb29rmtZWVnxtcPZaB8t0CqohMHBQafd0Dso3ExQvrGxcYQSoWy5tbXVqqurfe1wluPj4y585UqS1tZWt3nNCtFW8e6gThMTE95kjsZwrBsIJkENZfyImEGc6KdE4IfNQhs0Njb6tSULCwsedCFoJaPEyUIDE3TpvXZKORDEUSjQ1tbma4d4FCefZZmjbbr3qOyjwofMfHBw0DUmSiFrg8OVlRW/oFi7mKNzQUdDc9nFxcXcuxPEg85AU/Dd9HuC6gA1py8ZASf6I1238+fPe/VuZWWlo83Y7ejoaO66IRIwbIfqyIaGBkd7Qd5pwsilxyRg6i9JQnZ2drwLuVKSMzMz7q/oywYKwdpVV1e7Ro/1Yv1mZmZcwwfKCX2PhODMmTNWUVHhCLf6DBJDrlBRGQOVz83Nza57UykFCcz+/r7PEfOG/qynp8eDTJWEsA4I/ysrKz0wYf0GBgYc6X7y5MkR2cHs7KzTgSTPShtypROI2sLCQs5nKJIESo2/6enp8R5vULVQuKOjo15hh+63vr4+RwNT8c3aaK8m1pC7ULMss76+PvvGN75hCwsLp0GSPoVC4UfM7C+yLOuWP/t5M/vPWZb99+Hv/oKZ/cKr/3nTzF76bzXOf2NPu72i3XqjPW/U9zY7fffTd3/jPW/Ud3+jvreZ2ZUsyxq/2w+peu2/8j31lMysKfxZk5ltxL+YZdlHzeyjZmaFQuGbWZa95V9/eP/2njfqu79R39vs9N1P3/2N97xR3/2N+t5mr7z7SXxOxUl8yL+h576ZVRUKhUvyZ28ys7Ki7dPn9Dl9Tp/T5/Q5fU6f1PPvKkjKsmzTzD5tZv9XoVCoLxQK/52Z/U9m9vH/f0d2+pw+p8/pc/qcPqfP99rz7ypIevV5n5nVmtm8mX3CzH7xuPL/V5+P/quP6t/u80Z99zfqe5udvvsb9Tl99zfe80Z9b7MTevd/V8Lt0+f0OX1On9Pn9Dl9Tp+Tev49Ikmnz+lz+pw+p8/pc/qcPt/1cxok/X/t3W+oHFcZx/HvL42YF0HT1lJNxIRGbG2llVz8g2IU4h8sLYoplDaY+KaKJVRQRBASg1XEQME/LdUXNU0kSBO11RaLKDWYRqxGpcVIfNHW2jQiNsGYe9PbP+nji3M2TCazuyHdvXP3zO8Dw73MmYXn3OfMznPn7OwxMzMza9DpIknSBZLulTQj6SlJN7Yd0zhIerWku3Ifj0v6i6SP5rYVkkLSdGXb1HbMoyRpj6TZSv/+XmlbI+mgpBOSfiNpeZuxjlItp9OSTkr6bm4rKu+SNkraL+l5SXfX2vrmWMk3JR3J21b11p6ZEP36Lundkn4l6aik/0jaLekNlfYtkl6sjYFLWunEORjQ74Fju/Ccr6v1+0T+W0zl9onOOQy+nuX2kZ7vnS6SgDuAF4CLgXXAnZKuaDeksVgIPA28H3gtsAnYJWlF5ZglEbE4b7fOfYhjt7HSv0sBJL2O9DTkJuACYD9wT4sxjlSlv4tJY/w5YHftsFLyfhj4GvCD6s6zyPGngY+TvirkSuAa4DNzEO8oNfYdOJ/04dUVwHLS98Vtqx1zT3WcRMQT4w52hPr1u6ff2C425xGxs3be3ww8Afy5ctgk5xwGXM/Gcb53tkhSWudtLbApIqYj4mHg58An241s9CJiJiK2RMQ/IuLliHgAeBKYaju2ln0COBARuyNiFtgCXCXpsnbDGovrSE987m07kHGIiJ9GxH3AkVrTsBxvAG6LiEMR8QxwG/CpOQp7JPr1PSIezP3+X0ScAG4H3ttKkGMwIOfDFJvzBhuAHVHQE1pDrmcjP987WyQBbwFO9hbCzR4FSryTdBpJF5P6X/1qhKckHZK0LVfjpfmGpGcl7ZP0gbzvClLOgVPfs/U4ZY6Bfm+Wped9WI5Pa6fs94DVnPnFutfm6bgDkj7bRlBj1G9sdyLneZppNbCj1lRUzmvXs5Gf710ukhYDx2r7jgGveK2X+UzSq4CdwPaIOEha1+cdpNvxU6T+72wvwrH4EnAJsIw0/XC/pJV0ZAxIehPp1vT2yu4u5B2G57jefgxYPGmfURlG0pXAZuCLld27gLcCFwE3AZsl3dBCeKM2bGx3IufAemBvRDxZ2VdUzhuuZyM/37tcJJ31Om+lkLSA9O3jLwAbAfJU4/6IeCki/p33f1hS/W8zsSLikYg4HhHPR8R2YB9wNd0ZA+uBh6tvll3IezYsx/X21wDTJU1PSHoz8CDwuYg4Nd0aEX+LiMMRcTIifgd8mzQtO9HOYmwXn/NsPaf/Y1RUzpuuZ4zhfO9ykdSpdd5ypXwX6QO8ayPixT6H9gZLaf9VVQWpfwdIOQdOfU5tJeWNgTPeLBuUmvdhOT6tncLeA/KUy6+BWyNi2PJMvfOiNPWxXXTOAZSW5FoK/HjIoROZ8wHXs5Gf750tkjq4ztudpNus10bEc72dkt4l6VJJCyRdCHwH2BMR9VuWE0nSEkkfkbRI0kJJ60jz9L8E7gXeJmmtpEWk6YjH8m3bIkh6D2macXdtf1F5z7ldBJwHnNfLN8NzvAP4vKRlkpYCXwDubqEL56xf3yUtAx4C7oiI7zW87mOSzs+PRb8TuAX42dxGf+4G9HvY2C4255VDNgA/iYjjtddNdM4rGq9njON8j4jObqRHBO8DZoB/Aje2HdOY+rmc9B/DLOl2Y29bB9xAejJgBvhXHkSvbzvmEfb9IuCPpNut/wV+D3yo0v5B4CDp8fg9wIq2Yx5x/78P/LBhf1F5Jz3FErVty7Ack/6L3goczdtW8nJNk7L16zvwlfx79ZyfrrzuR6Sno6bz3+eWtvsyon4PHNsl5zy3LcrvdWsaXjfROc996Hs9y+0jPd+9dpuZmZlZg85Ot5mZmZkN4iLJzMzMrIGLJDMzM7MGLpLMzMzMGrhIMjMzM2vgIsnMzMysgYskMzMzswYukszMzMwauEgys6JJ+oOkXZK+KulxSbOSHpO0pu3YzGx+8zdum1mx8npWx4GXgUeAb5HWu/o6aU27lRHxbHsRmtl8tnD4IWZmE+ty0lpWvyWt2XcSQNJR0rpO7yMtimlmdgZPt5lZyabyzy/3CqSstyr4hXMcj5lNEBdJZlayVcDhiNhX2780/zw0x/GY2QRxkWRmJVsFPNOw/3rgBLB3bsMxs0nizySZWZEkLQCuAmYkLYyIl/L+pcDNwO0RMdNmjGY2v/npNjMrkqTLgQPA06QPbm8D3ghsBo4AqyNitr0IzWy+83SbmZVqVf55NbAEuB/YCvwCWOMCycyG8XSbmZVqCjgUEX8Frmk7GDObPL6TZGalWgX8qe0gzGxyuUgys+JIEvB2XCSZ2SvgD26bmZmZNfCdJDMzM7MGLpLMzMzMGrhIMjMzM2vgIsnMzMysgYskMzMzswYukszMzMwauEgyMzMza+AiyczMzKzB/wHdRN3AtLxoHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 648x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "i1, i2, crop_i = 100, 101, 150\n",
    "p1, p2, p3 = 22, 60, 35\n",
    "fig, (ax1, ax2) = plt.subplots(nrows=2, ncols=1, sharex=True, figsize=(9, 5))\n",
    "ax1.plot([p1, p1], [-1, 1], \"k--\", label=\"$p = {}$\".format(p1))\n",
    "ax1.plot([p2, p2], [-1, 1], \"k--\", label=\"$p = {}$\".format(p2), alpha=0.5)\n",
    "ax1.plot(p3, PE[p3, i1], \"bx\", label=\"$p = {}$\".format(p3))\n",
    "ax1.plot(PE[:,i1], \"b-\", label=\"$i = {}$\".format(i1))\n",
    "ax1.plot(PE[:,i2], \"r-\", label=\"$i = {}$\".format(i2))\n",
    "ax1.plot([p1, p2], [PE[p1, i1], PE[p2, i1]], \"bo\")\n",
    "ax1.plot([p1, p2], [PE[p1, i2], PE[p2, i2]], \"ro\")\n",
    "ax1.legend(loc=\"center right\", fontsize=14, framealpha=0.95)\n",
    "ax1.set_ylabel(\"$P_{(p,i)}$\", rotation=0, fontsize=16)\n",
    "ax1.grid(True, alpha=0.3)\n",
    "ax1.hlines(0, 0, max_steps - 1, color=\"k\", linewidth=1, alpha=0.3)\n",
    "ax1.axis([0, max_steps - 1, -1, 1])\n",
    "ax2.imshow(PE.T[:crop_i], cmap=\"gray\", interpolation=\"bilinear\", aspect=\"auto\")\n",
    "ax2.hlines(i1, 0, max_steps - 1, color=\"b\")\n",
    "cheat = 2 # need to raise the red line a bit, or else it hides the blue one\n",
    "ax2.hlines(i2+cheat, 0, max_steps - 1, color=\"r\")\n",
    "ax2.plot([p1, p1], [0, crop_i], \"k--\")\n",
    "ax2.plot([p2, p2], [0, crop_i], \"k--\", alpha=0.5)\n",
    "ax2.plot([p1, p2], [i2+cheat, i2+cheat], \"ro\")\n",
    "ax2.plot([p1, p2], [i1, i1], \"bo\")\n",
    "ax2.axis([0, max_steps - 1, 0, crop_i])\n",
    "ax2.set_xlabel(\"$p$\", fontsize=16)\n",
    "ax2.set_ylabel(\"$i$\", rotation=0, fontsize=16)\n",
    "plt.savefig(\"positional_embedding_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_size = 512; max_steps = 500; vocab_size = 10000\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "embeddings = keras.layers.Embedding(vocab_size, embed_size)\n",
    "encoder_embeddings = embeddings(encoder_inputs)\n",
    "decoder_embeddings = embeddings(decoder_inputs)\n",
    "positional_encoding = PositionalEncoding(max_steps, max_dims=embed_size)\n",
    "encoder_in = positional_encoding(encoder_embeddings)\n",
    "decoder_in = positional_encoding(decoder_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a (very) simplified Transformer (the actual architecture has skip connections, layer norm, dense nets, and most importantly it uses Multi-Head Attention instead of regular Attention):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "Z = encoder_in\n",
    "for N in range(6):\n",
    "    Z = keras.layers.Attention(use_scale=True)([Z, Z])\n",
    "\n",
    "encoder_outputs = Z\n",
    "Z = decoder_in\n",
    "for N in range(6):\n",
    "    Z = keras.layers.Attention(use_scale=True, causal=True)([Z, Z])\n",
    "    Z = keras.layers.Attention(use_scale=True)([Z, encoder_outputs])\n",
    "\n",
    "outputs = keras.layers.TimeDistributed(\n",
    "    keras.layers.Dense(vocab_size, activation=\"softmax\"))(Z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a basic implementation of the `MultiHeadAttention` layer. One will likely be added to `keras.layers` in the near future. Note that `Conv1D` layers with `kernel_size=1` (and the default `padding=\"valid\"` and `strides=1`) is equivalent to a `TimeDistributed(Dense(...))` layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class MultiHeadAttention(keras.layers.Layer):\n",
    "    def __init__(self, n_heads, causal=False, use_scale=False, **kwargs):\n",
    "        self.n_heads = n_heads\n",
    "        self.causal = causal\n",
    "        self.use_scale = use_scale\n",
    "        super().__init__(**kwargs)\n",
    "    def build(self, batch_input_shape):\n",
    "        self.dims = batch_input_shape[0][-1]\n",
    "        self.q_dims, self.v_dims, self.k_dims = [self.dims // self.n_heads] * 3 # could be hyperparameters instead\n",
    "        self.q_linear = keras.layers.Conv1D(self.n_heads * self.q_dims, kernel_size=1, use_bias=False)\n",
    "        self.v_linear = keras.layers.Conv1D(self.n_heads * self.v_dims, kernel_size=1, use_bias=False)\n",
    "        self.k_linear = keras.layers.Conv1D(self.n_heads * self.k_dims, kernel_size=1, use_bias=False)\n",
    "        self.attention = keras.layers.Attention(causal=self.causal, use_scale=self.use_scale)\n",
    "        self.out_linear = keras.layers.Conv1D(self.dims, kernel_size=1, use_bias=False)\n",
    "        super().build(batch_input_shape)\n",
    "    def _multi_head_linear(self, inputs, linear):\n",
    "        shape = K.concatenate([K.shape(inputs)[:-1], [self.n_heads, -1]])\n",
    "        projected = K.reshape(linear(inputs), shape)\n",
    "        perm = K.permute_dimensions(projected, [0, 2, 1, 3])\n",
    "        return K.reshape(perm, [shape[0] * self.n_heads, shape[1], -1])\n",
    "    def call(self, inputs):\n",
    "        q = inputs[0]\n",
    "        v = inputs[1]\n",
    "        k = inputs[2] if len(inputs) > 2 else v\n",
    "        shape = K.shape(q)\n",
    "        q_proj = self._multi_head_linear(q, self.q_linear)\n",
    "        v_proj = self._multi_head_linear(v, self.v_linear)\n",
    "        k_proj = self._multi_head_linear(k, self.k_linear)\n",
    "        multi_attended = self.attention([q_proj, v_proj, k_proj])\n",
    "        shape_attended = K.shape(multi_attended)\n",
    "        reshaped_attended = K.reshape(multi_attended, [shape[0], self.n_heads, shape_attended[1], shape_attended[2]])\n",
    "        perm = K.permute_dimensions(reshaped_attended, [0, 2, 1, 3])\n",
    "        concat = K.reshape(perm, [shape[0], shape_attended[1], -1])\n",
    "        return self.out_linear(concat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer multi_head_attention is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer multi_head_attention is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 50, 512])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q = np.random.rand(2, 50, 512)\n",
    "V = np.random.rand(2, 80, 512)\n",
    "multi_attn = MultiHeadAttention(8)\n",
    "multi_attn([Q, V]).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See Appendix A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.\n",
    "_Exercise:_ Embedded Reber grammars _were used by Hochreiter and Schmidhuber in [their paper](https://homl.info/93) about LSTMs. They are artificial grammars that produce strings such as \"BPBTSXXVPSEPE.\" Check out Jenny Orr's [nice introduction](https://homl.info/108) to this topic. Choose a particular embedded Reber grammar (such as the one represented on Jenny Orr's page), then train an RNN to identify whether a string respects that grammar or not. You will first need to write a function capable of generating a training batch containing about 50% strings that respect the grammar, and 50% that don't._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to build a function that generates strings based on a grammar. The grammar will be represented as a list of possible transitions for each state. A transition specifies the string to output (or a grammar to generate it) and the next state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_reber_grammar = [\n",
    "    [(\"B\", 1)],           # (state 0) =B=>(state 1)\n",
    "    [(\"T\", 2), (\"P\", 3)], # (state 1) =T=>(state 2) or =P=>(state 3)\n",
    "    [(\"S\", 2), (\"X\", 4)], # (state 2) =S=>(state 2) or =X=>(state 4)\n",
    "    [(\"T\", 3), (\"V\", 5)], # and so on...\n",
    "    [(\"X\", 3), (\"S\", 6)],\n",
    "    [(\"P\", 4), (\"V\", 6)],\n",
    "    [(\"E\", None)]]        # (state 6) =E=>(terminal state)\n",
    "\n",
    "embedded_reber_grammar = [\n",
    "    [(\"B\", 1)],\n",
    "    [(\"T\", 2), (\"P\", 3)],\n",
    "    [(default_reber_grammar, 4)],\n",
    "    [(default_reber_grammar, 5)],\n",
    "    [(\"T\", 6)],\n",
    "    [(\"P\", 6)],\n",
    "    [(\"E\", None)]]\n",
    "\n",
    "def generate_string(grammar):\n",
    "    state = 0\n",
    "    output = []\n",
    "    while state is not None:\n",
    "        index = np.random.randint(len(grammar[state]))\n",
    "        production, state = grammar[state][index]\n",
    "        if isinstance(production, list):\n",
    "            production = generate_string(grammar=production)\n",
    "        output.append(production)\n",
    "    return \"\".join(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's generate a few strings based on the default Reber grammar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BTXXTTVPXTVPXTTVPSE BPVPSE BTXSE BPVVE BPVVE BTSXSE BPTVPXTTTVVE BPVVE BTXSE BTXXVPSE BPTTTTTTTTVVE BTXSE BPVPSE BTXSE BPTVPSE BTXXTVPSE BPVVE BPVVE BPVVE BPTTVVE BPVVE BPVVE BTXXVVE BTXXVVE BTXXVPXVVE "
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "for _ in range(25):\n",
    "    print(generate_string(default_reber_grammar), end=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good. Now let's generate a few strings based on the embedded Reber grammar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BTBPTTTVPXTVPXTTVPSETE BPBPTVPSEPE BPBPVVEPE BPBPVPXVVEPE BPBTXXTTTTVVEPE BPBPVPSEPE BPBTXXVPSEPE BPBTSSSSSSSXSEPE BTBPVVETE BPBTXXVVEPE BPBTXXVPSEPE BTBTXXVVETE BPBPVVEPE BPBPVVEPE BPBTSXSEPE BPBPVVEPE BPBPTVPSEPE BPBTXXVVEPE BTBPTVPXVVETE BTBPVVETE BTBTSSSSSSSXXVVETE BPBTSSSXXTTTTVPSEPE BTBPTTVVETE BPBTXXTVVEPE BTBTXSETE "
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "for _ in range(25):\n",
    "    print(generate_string(embedded_reber_grammar), end=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now we need a function to generate strings that do not respect the grammar. We could generate a random string, but the task would be a bit too easy, so instead we will generate a string that respects the grammar, and we will corrupt it by changing just one character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "POSSIBLE_CHARS = \"BEPSTVX\"\n",
    "\n",
    "def generate_corrupted_string(grammar, chars=POSSIBLE_CHARS):\n",
    "    good_string = generate_string(grammar)\n",
    "    index = np.random.randint(len(good_string))\n",
    "    good_char = good_string[index]\n",
    "    bad_char = np.random.choice(sorted(set(chars) - set(good_char)))\n",
    "    return good_string[:index] + bad_char + good_string[index + 1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a few corrupted strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BTBPTTTPPXTVPXTTVPSETE BPBTXEEPE BPBPTVVVEPE BPBTSSSSXSETE BPTTXSEPE BTBPVPXTTTTTTEVETE BPBTXXSVEPE BSBPTTVPSETE BPBXVVEPE BEBTXSETE BPBPVPSXPE BTBPVVVETE BPBTSXSETE BPBPTTTPTTTTTVPSEPE BTBTXXTTSTVPSETE BBBTXSETE BPBTPXSEPE BPBPVPXTTTTVPXTVPXVPXTTTVVEVE BTBXXXTVPSETE BEBTSSSSSXXVPXTVVETE BTBXTTVVETE BPBTXSTPE BTBTXXTTTVPSBTE BTBTXSETX BTBTSXSSTE "
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "for _ in range(25):\n",
    "    print(generate_corrupted_string(embedded_reber_grammar), end=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cannot feed strings directly to an RNN, so we need to encode them somehow. One option would be to one-hot encode each character. Another option is to use embeddings. Let's go for the second option (but since there are just a handful of characters, one-hot encoding would probably be a good option as well). For embeddings to work, we need to convert each string into a sequence of character IDs. Let's write a function for that, using each character's index in the string of possible characters \"BEPSTVX\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "def string_to_ids(s, chars=POSSIBLE_CHARS):\n",
    "    return [POSSIBLE_CHARS.index(c) for c in s]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 4, 4, 4, 6, 6, 5, 5, 1, 4, 1]"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string_to_ids(\"BTTTXXVVETE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now generate the dataset, with 50% good strings, and 50% bad strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_dataset(size):\n",
    "    good_strings = [string_to_ids(generate_string(embedded_reber_grammar))\n",
    "                    for _ in range(size // 2)]\n",
    "    bad_strings = [string_to_ids(generate_corrupted_string(embedded_reber_grammar))\n",
    "                   for _ in range(size - size // 2)]\n",
    "    all_strings = good_strings + bad_strings\n",
    "    X = tf.ragged.constant(all_strings, ragged_rank=1)\n",
    "    y = np.array([[1.] for _ in range(len(good_strings))] +\n",
    "                 [[0.] for _ in range(len(bad_strings))])\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "X_train, y_train = generate_dataset(10000)\n",
    "X_valid, y_valid = generate_dataset(2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the first training sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(22,), dtype=int32, numpy=\n",
       "array([0, 4, 0, 2, 4, 4, 4, 5, 2, 6, 4, 5, 2, 6, 4, 4, 5, 2, 3, 1, 4, 1],\n",
       "      dtype=int32)>"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What classes does it belong to?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.])"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect! We are ready to create the RNN to identify good strings. We build a simple sequence binary classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 10000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ageron/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n",
      "/Users/ageron/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 5s 42us/sample - loss: 0.6847 - accuracy: 0.5138 - val_loss: 8.1518 - val_accuracy: 0.6115\n",
      "Epoch 2/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 28us/sample - loss: 0.6524 - accuracy: 0.5571 - val_loss: 7.9259 - val_accuracy: 0.6085\n",
      "Epoch 3/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 28us/sample - loss: 0.6686 - accuracy: 0.5783 - val_loss: 7.7483 - val_accuracy: 0.6110\n",
      "Epoch 4/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 28us/sample - loss: 0.6201 - accuracy: 0.5969 - val_loss: 7.5567 - val_accuracy: 0.6110\n",
      "Epoch 5/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 28us/sample - loss: 0.5705 - accuracy: 0.6428 - val_loss: 6.9117 - val_accuracy: 0.7075\n",
      "Epoch 6/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 0.5660 - accuracy: 0.7008 - val_loss: 5.7277 - val_accuracy: 0.7580\n",
      "Epoch 7/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 28us/sample - loss: 0.3997 - accuracy: 0.8336 - val_loss: 4.3641 - val_accuracy: 0.8550\n",
      "Epoch 8/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 0.1771 - accuracy: 0.8958 - val_loss: 1.5009 - val_accuracy: 0.9605\n",
      "Epoch 9/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 0.2710 - accuracy: 0.9566 - val_loss: 3.2648 - val_accuracy: 0.9005\n",
      "Epoch 10/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 0.2574 - accuracy: 0.9620 - val_loss: 1.0385 - val_accuracy: 0.9790\n",
      "Epoch 11/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 0.0356 - accuracy: 0.9845 - val_loss: 0.1081 - val_accuracy: 1.0000\n",
      "Epoch 12/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 4s 29us/sample - loss: 0.0029 - accuracy: 1.0000 - val_loss: 0.0261 - val_accuracy: 1.0000\n",
      "Epoch 13/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 0.0019 - accuracy: 1.0000 - val_loss: 0.0144 - val_accuracy: 1.0000\n",
      "Epoch 14/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 28us/sample - loss: 8.1710e-04 - accuracy: 1.0000 - val_loss: 0.0101 - val_accuracy: 1.0000\n",
      "Epoch 15/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 5.8225e-04 - accuracy: 1.0000 - val_loss: 0.0079 - val_accuracy: 1.0000\n",
      "Epoch 16/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 5.8369e-04 - accuracy: 1.0000 - val_loss: 0.0064 - val_accuracy: 1.0000\n",
      "Epoch 17/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 4s 30us/sample - loss: 3.8744e-04 - accuracy: 1.0000 - val_loss: 0.0054 - val_accuracy: 1.0000\n",
      "Epoch 18/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 4s 29us/sample - loss: 4.2988e-04 - accuracy: 1.0000 - val_loss: 0.0047 - val_accuracy: 1.0000\n",
      "Epoch 19/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 4s 29us/sample - loss: 2.7449e-04 - accuracy: 1.0000 - val_loss: 0.0041 - val_accuracy: 1.0000\n",
      "Epoch 20/20\n",
      "120278/10000 [========================================================================================================================================================================================================================================================================================================================================================================] - 3s 29us/sample - loss: 2.9469e-04 - accuracy: 1.0000 - val_loss: 0.0037 - val_accuracy: 1.0000\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "embedding_size = 5\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.InputLayer(input_shape=[None], dtype=tf.int32, ragged=True),\n",
    "    keras.layers.Embedding(input_dim=len(POSSIBLE_CHARS), output_dim=embedding_size),\n",
    "    keras.layers.GRU(30),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(lr=0.02, momentum = 0.95, nesterov=True)\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "history = model.fit(X_train, y_train, epochs=20, validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's test our RNN on two tricky strings: the first one is bad while the second one is good. They only differ by the second to last character. If the RNN gets this right, it shows that it managed to notice the pattern that the second letter should always be equal to the second to last letter. That requires a fairly long short-term memory (which is the reason why we used a GRU cell)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Estimated probability that these are Reber strings:\n",
      "BPBTSSSSSSSXXTTVPXVPXTTTTTVVETE: 0.40%\n",
      "BPBTSSSSSSSXXTTVPXVPXTTTTTVVEPE: 99.96%\n"
     ]
    }
   ],
   "source": [
    "test_strings = [\"BPBTSSSSSSSXXTTVPXVPXTTTTTVVETE\",\n",
    "                \"BPBTSSSSSSSXXTTVPXVPXTTTTTVVEPE\"]\n",
    "X_test = tf.ragged.constant([string_to_ids(s) for s in test_strings], ragged_rank=1)\n",
    "\n",
    "y_proba = model.predict(X_test)\n",
    "print()\n",
    "print(\"Estimated probability that these are Reber strings:\")\n",
    "for index, string in enumerate(test_strings):\n",
    "    print(\"{}: {:.2f}%\".format(string, 100 * y_proba[index][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ta-da! It worked fine. The RNN found the correct answers with very high confidence. :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.\n",
    "_Exercise: Train an Encoder–Decoder model that can convert a date string from one format to another (e.g., from \"April 22, 2019\" to \"2019-04-22\")._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by creating the dataset. We will use random days between 1000-01-01 and 9999-12-31:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import date\n",
    "\n",
    "# cannot use strftime()'s %B format since it depends on the locale\n",
    "MONTHS = [\"January\", \"February\", \"March\", \"April\", \"May\", \"June\",\n",
    "          \"July\", \"August\", \"September\", \"October\", \"November\", \"December\"]\n",
    "\n",
    "def random_dates(n_dates):\n",
    "    min_date = date(1000, 1, 1).toordinal()\n",
    "    max_date = date(9999, 12, 31).toordinal()\n",
    "\n",
    "    ordinals = np.random.randint(max_date - min_date, size=n_dates) + min_date\n",
    "    dates = [date.fromordinal(ordinal) for ordinal in ordinals]\n",
    "\n",
    "    x = [MONTHS[dt.month - 1] + \" \" + dt.strftime(\"%d, %Y\") for dt in dates]\n",
    "    y = [dt.isoformat() for dt in dates]\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are a few random dates, displayed in both the input format and the target format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input                    Target                   \n",
      "--------------------------------------------------\n",
      "September 20, 7075       7075-09-20               \n",
      "May 15, 8579             8579-05-15               \n",
      "January 11, 7103         7103-01-11               \n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "n_dates = 3\n",
    "x_example, y_example = random_dates(n_dates)\n",
    "print(\"{:25s}{:25s}\".format(\"Input\", \"Target\"))\n",
    "print(\"-\" * 50)\n",
    "for idx in range(n_dates):\n",
    "    print(\"{:25s}{:25s}\".format(x_example[idx], y_example[idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the list of all possible characters in the inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ADFJMNOSabceghilmnoprstuvy01234567890, '"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "INPUT_CHARS = \"\".join(sorted(set(\"\".join(MONTHS)))) + \"01234567890, \"\n",
    "INPUT_CHARS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's the list of possible characters in the outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTPUT_CHARS = \"0123456789-\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a function to convert a string to a list of character IDs, as we did in the previous exercise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "def date_str_to_ids(date_str, chars=INPUT_CHARS):\n",
    "    return [chars.index(c) for c in date_str]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7, 11, 19, 22, 11, 16, 9, 11, 20, 38, 28, 26, 37, 38, 33, 26, 33, 31]"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_str_to_ids(x_example[0], INPUT_CHARS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7, 0, 7, 5, 10, 0, 9, 10, 2, 0]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_str_to_ids(y_example[0], OUTPUT_CHARS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_date_strs(date_strs, chars=INPUT_CHARS):\n",
    "    X_ids = [date_str_to_ids(dt, chars) for dt in date_strs]\n",
    "    X = tf.ragged.constant(X_ids, ragged_rank=1)\n",
    "    return (X + 1).to_tensor() # using 0 as the padding token ID\n",
    "\n",
    "def create_dataset(n_dates):\n",
    "    x, y = random_dates(n_dates)\n",
    "    return prepare_date_strs(x, INPUT_CHARS), prepare_date_strs(y, OUTPUT_CHARS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "X_train, Y_train = create_dataset(10000)\n",
    "X_valid, Y_valid = create_dataset(2000)\n",
    "X_test, Y_test = create_dataset(2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(10,), dtype=int32, numpy=array([ 8,  1,  8,  6, 11,  1, 10, 11,  3,  1], dtype=int32)>"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First version: a very basic seq2seq model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first try the simplest possible model: we feed in the input sequence, which first goes through the encoder (an embedding layer followed by a single LSTM layer), which outputs a vector, then it goes through a decoder (a single LSTM layer, followed by a dense output layer), which outputs a sequence of vectors, each representing the estimated probabilities for all possible output character.\n",
    "\n",
    "Since the decoder expects a sequence as input, we repeat the vector (which is output by the decoder) as many times as the longest possible output sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 1.8111 - accuracy: 0.3533 - val_loss: 1.3581 - val_accuracy: 0.4965\n",
      "Epoch 2/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.3518 - accuracy: 0.5103 - val_loss: 1.1915 - val_accuracy: 0.5694\n",
      "Epoch 3/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.1706 - accuracy: 0.5908 - val_loss: 0.9983 - val_accuracy: 0.6398\n",
      "Epoch 4/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.9158 - accuracy: 0.6686 - val_loss: 0.8012 - val_accuracy: 0.6987\n",
      "Epoch 5/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.7058 - accuracy: 0.7308 - val_loss: 0.6224 - val_accuracy: 0.7599\n",
      "Epoch 6/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.7756 - accuracy: 0.7203 - val_loss: 0.6541 - val_accuracy: 0.7599\n",
      "Epoch 7/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.5379 - accuracy: 0.8034 - val_loss: 0.4174 - val_accuracy: 0.8440\n",
      "Epoch 8/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.4867 - accuracy: 0.8262 - val_loss: 0.4188 - val_accuracy: 0.8480\n",
      "Epoch 9/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.2979 - accuracy: 0.8951 - val_loss: 0.2549 - val_accuracy: 0.9126\n",
      "Epoch 10/20\n",
      "313/313 [==============================] - 5s 14ms/step - loss: 0.1785 - accuracy: 0.9479 - val_loss: 0.1461 - val_accuracy: 0.9594\n",
      "Epoch 11/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.1830 - accuracy: 0.9557 - val_loss: 0.1644 - val_accuracy: 0.9550\n",
      "Epoch 12/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0775 - accuracy: 0.9857 - val_loss: 0.0595 - val_accuracy: 0.9901\n",
      "Epoch 13/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0400 - accuracy: 0.9953 - val_loss: 0.0342 - val_accuracy: 0.9957\n",
      "Epoch 14/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0248 - accuracy: 0.9979 - val_loss: 0.0231 - val_accuracy: 0.9983\n",
      "Epoch 15/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0161 - accuracy: 0.9991 - val_loss: 0.0149 - val_accuracy: 0.9995\n",
      "Epoch 16/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0108 - accuracy: 0.9997 - val_loss: 0.0106 - val_accuracy: 0.9996\n",
      "Epoch 17/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0074 - accuracy: 0.9999 - val_loss: 0.0077 - val_accuracy: 0.9999\n",
      "Epoch 18/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0053 - accuracy: 1.0000 - val_loss: 0.0054 - val_accuracy: 0.9999\n",
      "Epoch 19/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0039 - accuracy: 1.0000 - val_loss: 0.0041 - val_accuracy: 1.0000\n",
      "Epoch 20/20\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0029 - accuracy: 1.0000 - val_loss: 0.0032 - val_accuracy: 1.0000\n"
     ]
    }
   ],
   "source": [
    "embedding_size = 32\n",
    "max_output_length = Y_train.shape[1]\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "encoder = keras.models.Sequential([\n",
    "    keras.layers.Embedding(input_dim=len(INPUT_CHARS) + 1,\n",
    "                           output_dim=embedding_size,\n",
    "                           input_shape=[None]),\n",
    "    keras.layers.LSTM(128)\n",
    "])\n",
    "\n",
    "decoder = keras.models.Sequential([\n",
    "    keras.layers.LSTM(128, return_sequences=True),\n",
    "    keras.layers.Dense(len(OUTPUT_CHARS) + 1, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    encoder,\n",
    "    keras.layers.RepeatVector(max_output_length),\n",
    "    decoder\n",
    "])\n",
    "\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks great, we reach 100% validation accuracy! Let's use the model to make some predictions. We will need to be able to convert a sequence of character IDs to a readable string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ids_to_date_strs(ids, chars=OUTPUT_CHARS):\n",
    "    return [\"\".join([(\"?\" + chars)[index] for index in sequence])\n",
    "            for sequence in ids]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use the model to convert some dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = prepare_date_strs([\"September 17, 2009\", \"July 14, 1789\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-15-472ea7c41409>:1: Sequential.predict_classes (from tensorflow.python.keras.engine.sequential) is deprecated and will be removed after 2021-01-01.\n",
      "Instructions for updating:\n",
      "Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype(\"int32\")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).\n",
      "2009-09-17\n",
      "1789-07-14\n"
     ]
    }
   ],
   "source": [
    "ids = model.predict_classes(X_new)\n",
    "for date_str in ids_to_date_strs(ids):\n",
    "    print(date_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect! :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, since the model was only trained on input strings of length 18 (which is the length of the longest date), it does not perform well if we try to use it to make predictions on shorter sequences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = prepare_date_strs([\"May 02, 2020\", \"July 14, 1789\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-01-02\n",
      "1789-02-14\n"
     ]
    }
   ],
   "source": [
    "ids = model.predict_classes(X_new)\n",
    "for date_str in ids_to_date_strs(ids):\n",
    "    print(date_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops! We need to ensure that we always pass sequences of the same length as during training, using padding if necessary. Let's write a little helper function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_input_length = X_train.shape[1]\n",
    "\n",
    "def prepare_date_strs_padded(date_strs):\n",
    "    X = prepare_date_strs(date_strs)\n",
    "    if X.shape[1] < max_input_length:\n",
    "        X = tf.pad(X, [[0, 0], [0, max_input_length - X.shape[1]]])\n",
    "    return X\n",
    "\n",
    "def convert_date_strs(date_strs):\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    ids = model.predict_classes(X)\n",
    "    return ids_to_date_strs(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2020-05-02', '1789-07-14']"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "convert_date_strs([\"May 02, 2020\", \"July 14, 1789\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool! Granted, there are certainly much easier ways to write a date conversion tool (e.g., using regular expressions or even basic string manipulation), but you have to admit that using neural networks is way cooler. ;-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, real-life sequence-to-sequence problems will usually be harder, so for the sake of completeness, let's build a more powerful model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Second version: feeding the shifted targets to the decoder (teacher forcing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of feeding the decoder a simple repetition of the encoder's output vector, we can feed it the target sequence, shifted by one time step to the right. This way, at each time step the decoder will know what the previous target character was. This should help is tackle more complex sequence-to-sequence problems.\n",
    "\n",
    "Since the first output character of each target sequence has no previous character, we will need a new token to represent the start-of-sequence (sos).\n",
    "\n",
    "During inference, we won't know the target, so what will we feed the decoder? We can just predict one character at a time, starting with an sos token, then feeding the decoder all the characters that were predicted so far (we will look at this in more details later in this notebook).\n",
    "\n",
    "But if the decoder's LSTM expects to get the previous target as input at each step, how shall we pass it it the vector output by the encoder? Well, one option is to ignore the output vector, and instead use the encoder's LSTM state as the initial state of the decoder's LSTM (which requires that encoder's LSTM must have the same number of units as the decoder's LSTM).\n",
    "\n",
    "Now let's create the decoder's inputs (for training, validation and testing). The sos token will be represented using the last possible output character's ID + 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "sos_id = len(OUTPUT_CHARS) + 1\n",
    "\n",
    "def shifted_output_sequences(Y):\n",
    "    sos_tokens = tf.fill(dims=(len(Y), 1), value=sos_id)\n",
    "    return tf.concat([sos_tokens, Y[:, :-1]], axis=1)\n",
    "\n",
    "X_train_decoder = shifted_output_sequences(Y_train)\n",
    "X_valid_decoder = shifted_output_sequences(Y_valid)\n",
    "X_test_decoder = shifted_output_sequences(Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the decoder's training inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(10000, 10), dtype=int32, numpy=\n",
       "array([[12,  8,  1, ..., 10, 11,  3],\n",
       "       [12,  9,  6, ...,  6, 11,  2],\n",
       "       [12,  8,  2, ...,  2, 11,  2],\n",
       "       ...,\n",
       "       [12, 10,  8, ...,  2, 11,  4],\n",
       "       [12,  2,  2, ...,  3, 11,  3],\n",
       "       [12,  8,  9, ...,  8, 11,  3]], dtype=int32)>"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's build the model. It's not a simple sequential model anymore, so let's use the functional API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 1.6898 - accuracy: 0.3714 - val_loss: 1.4141 - val_accuracy: 0.4603\n",
      "Epoch 2/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.2118 - accuracy: 0.5541 - val_loss: 0.9360 - val_accuracy: 0.6653\n",
      "Epoch 3/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.6399 - accuracy: 0.7766 - val_loss: 0.4054 - val_accuracy: 0.8631\n",
      "Epoch 4/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.2207 - accuracy: 0.9463 - val_loss: 0.1069 - val_accuracy: 0.9869\n",
      "Epoch 5/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0805 - accuracy: 0.9910 - val_loss: 0.0445 - val_accuracy: 0.9976\n",
      "Epoch 6/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0297 - accuracy: 0.9993 - val_loss: 0.0237 - val_accuracy: 0.9992\n",
      "Epoch 7/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0743 - accuracy: 0.9857 - val_loss: 0.0702 - val_accuracy: 0.9889\n",
      "Epoch 8/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0187 - accuracy: 0.9995 - val_loss: 0.0112 - val_accuracy: 0.9999\n",
      "Epoch 9/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0084 - accuracy: 1.0000 - val_loss: 0.0072 - val_accuracy: 1.0000\n",
      "Epoch 10/10\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0057 - accuracy: 1.0000 - val_loss: 0.0053 - val_accuracy: 1.0000\n"
     ]
    }
   ],
   "source": [
    "encoder_embedding_size = 32\n",
    "decoder_embedding_size = 32\n",
    "lstm_units = 128\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "encoder_input = keras.layers.Input(shape=[None], dtype=tf.int32)\n",
    "encoder_embedding = keras.layers.Embedding(\n",
    "    input_dim=len(INPUT_CHARS) + 1,\n",
    "    output_dim=encoder_embedding_size)(encoder_input)\n",
    "_, encoder_state_h, encoder_state_c = keras.layers.LSTM(\n",
    "    lstm_units, return_state=True)(encoder_embedding)\n",
    "encoder_state = [encoder_state_h, encoder_state_c]\n",
    "\n",
    "decoder_input = keras.layers.Input(shape=[None], dtype=tf.int32)\n",
    "decoder_embedding = keras.layers.Embedding(\n",
    "    input_dim=len(OUTPUT_CHARS) + 2,\n",
    "    output_dim=decoder_embedding_size)(decoder_input)\n",
    "decoder_lstm_output = keras.layers.LSTM(lstm_units, return_sequences=True)(\n",
    "    decoder_embedding, initial_state=encoder_state)\n",
    "decoder_output = keras.layers.Dense(len(OUTPUT_CHARS) + 1,\n",
    "                                    activation=\"softmax\")(decoder_lstm_output)\n",
    "\n",
    "model = keras.models.Model(inputs=[encoder_input, decoder_input],\n",
    "                           outputs=[decoder_output])\n",
    "\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit([X_train, X_train_decoder], Y_train, epochs=10,\n",
    "                    validation_data=([X_valid, X_valid_decoder], Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model also reaches 100% validation accuracy, but it does so even faster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's once again use the model to make some predictions. This time we need to predict characters one by one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "sos_id = len(OUTPUT_CHARS) + 1\n",
    "\n",
    "def predict_date_strs(date_strs):\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    Y_pred = tf.fill(dims=(len(X), 1), value=sos_id)\n",
    "    for index in range(max_output_length):\n",
    "        pad_size = max_output_length - Y_pred.shape[1]\n",
    "        X_decoder = tf.pad(Y_pred, [[0, 0], [0, pad_size]])\n",
    "        Y_probas_next = model.predict([X, X_decoder])[:, index:index+1]\n",
    "        Y_pred_next = tf.argmax(Y_probas_next, axis=-1, output_type=tf.int32)\n",
    "        Y_pred = tf.concat([Y_pred, Y_pred_next], axis=1)\n",
    "    return ids_to_date_strs(Y_pred[:, 1:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-01']"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Works fine! :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Third version: using TF-Addons's seq2seq implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build exactly the same model, but using TF-Addon's seq2seq API. The implementation below is almost very similar to the TFA example higher in this notebook, except without the model input to specify the output sequence length, for simplicity (but you can easily add it back in if you need it for your projects, when the output sequences have very different lengths)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 1.6757 - accuracy: 0.3683 - val_loss: 1.4602 - val_accuracy: 0.4214\n",
      "Epoch 2/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.3873 - accuracy: 0.4566 - val_loss: 1.2904 - val_accuracy: 0.4957\n",
      "Epoch 3/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.0471 - accuracy: 0.6109 - val_loss: 0.7737 - val_accuracy: 0.7276\n",
      "Epoch 4/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.5056 - accuracy: 0.8296 - val_loss: 0.2695 - val_accuracy: 0.9305\n",
      "Epoch 5/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.1677 - accuracy: 0.9657 - val_loss: 0.0870 - val_accuracy: 0.9912\n",
      "Epoch 6/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.1007 - accuracy: 0.9850 - val_loss: 0.0492 - val_accuracy: 0.9975\n",
      "Epoch 7/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0308 - accuracy: 0.9993 - val_loss: 0.0228 - val_accuracy: 0.9996\n",
      "Epoch 8/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0168 - accuracy: 0.9999 - val_loss: 0.0144 - val_accuracy: 0.9999\n",
      "Epoch 9/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0107 - accuracy: 1.0000 - val_loss: 0.0095 - val_accuracy: 0.9999\n",
      "Epoch 10/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0074 - accuracy: 1.0000 - val_loss: 0.0066 - val_accuracy: 0.9999\n",
      "Epoch 11/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0053 - accuracy: 1.0000 - val_loss: 0.0051 - val_accuracy: 0.9999\n",
      "Epoch 12/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0039 - accuracy: 1.0000 - val_loss: 0.0037 - val_accuracy: 1.0000\n",
      "Epoch 13/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0029 - accuracy: 1.0000 - val_loss: 0.0030 - val_accuracy: 1.0000\n",
      "Epoch 14/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0023 - accuracy: 1.0000 - val_loss: 0.0022 - val_accuracy: 1.0000\n",
      "Epoch 15/15\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 0.0018 - accuracy: 1.0000 - val_loss: 0.0018 - val_accuracy: 1.0000\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_addons as tfa\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "encoder_embedding_size = 32\n",
    "decoder_embedding_size = 32\n",
    "units = 128\n",
    "\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "sequence_lengths = keras.layers.Input(shape=[], dtype=np.int32)\n",
    "\n",
    "encoder_embeddings = keras.layers.Embedding(\n",
    "    len(INPUT_CHARS) + 1, encoder_embedding_size)(encoder_inputs)\n",
    "\n",
    "decoder_embedding_layer = keras.layers.Embedding(\n",
    "    len(INPUT_CHARS) + 2, decoder_embedding_size)\n",
    "decoder_embeddings = decoder_embedding_layer(decoder_inputs)\n",
    "\n",
    "encoder = keras.layers.LSTM(units, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_embeddings)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "sampler = tfa.seq2seq.sampler.TrainingSampler()\n",
    "\n",
    "decoder_cell = keras.layers.LSTMCell(units)\n",
    "output_layer = keras.layers.Dense(len(OUTPUT_CHARS) + 1)\n",
    "\n",
    "decoder = tfa.seq2seq.basic_decoder.BasicDecoder(decoder_cell,\n",
    "                                                 sampler,\n",
    "                                                 output_layer=output_layer)\n",
    "final_outputs, final_state, final_sequence_lengths = decoder(\n",
    "    decoder_embeddings,\n",
    "    initial_state=encoder_state)\n",
    "Y_proba = keras.layers.Activation(\"softmax\")(final_outputs.rnn_output)\n",
    "\n",
    "model = keras.models.Model(inputs=[encoder_inputs, decoder_inputs],\n",
    "                           outputs=[Y_proba])\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit([X_train, X_train_decoder], Y_train, epochs=15,\n",
    "                    validation_data=([X_valid, X_valid_decoder], Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And once again, 100% validation accuracy! To use the model, we can just reuse the `predict_date_strs()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-01']"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there's a much more efficient way to perform inference. Until now, during inference, we've run the model once for each new character. Instead, we can create a new decoder, based on the previously trained layers, but using a `GreedyEmbeddingSampler` instead of a `TrainingSampler`.\n",
    "\n",
    "At each time step, the `GreedyEmbeddingSampler` will compute the argmax of the decoder's outputs, and run the resulting token IDs through the decoder's embedding layer. Then it will feed the resulting embeddings to the decoder's LSTM cell at the next time step. This way, we only need to run the decoder once to get the full prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_sampler = tfa.seq2seq.sampler.GreedyEmbeddingSampler(\n",
    "    embedding_fn=decoder_embedding_layer)\n",
    "inference_decoder = tfa.seq2seq.basic_decoder.BasicDecoder(\n",
    "    decoder_cell, inference_sampler, output_layer=output_layer,\n",
    "    maximum_iterations=max_output_length)\n",
    "batch_size = tf.shape(encoder_inputs)[:1]\n",
    "start_tokens = tf.fill(dims=batch_size, value=sos_id)\n",
    "final_outputs, final_state, final_sequence_lengths = inference_decoder(\n",
    "    start_tokens,\n",
    "    initial_state=encoder_state,\n",
    "    start_tokens=start_tokens,\n",
    "    end_token=0)\n",
    "\n",
    "inference_model = keras.models.Model(inputs=[encoder_inputs],\n",
    "                                     outputs=[final_outputs.sample_id])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few notes:\n",
    "* The `GreedyEmbeddingSampler` needs the `start_tokens` (a vector containing the start-of-sequence ID for each decoder sequence), and the `end_token` (the decoder will stop decoding a sequence once the model outputs this token).\n",
    "* We must set `maximum_iterations` when creating the `BasicDecoder`, or else it may run into an infinite loop (if the model never outputs the end token for at least one of the sequences). This would force you would to restart the Jupyter kernel.\n",
    "* The decoder inputs are not needed anymore, since all the decoder inputs are generated dynamically based on the outputs from the previous time step.\n",
    "* The model's outputs are `final_outputs.sample_id` instead of the softmax of `final_outputs.rnn_outputs`. This allows us to directly get the argmax of the model's outputs. If you prefer to have access to the logits, you can replace `final_outputs.sample_id` with `final_outputs.rnn_outputs`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can write a simple function that uses the model to perform the date format conversion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fast_predict_date_strs(date_strs):\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    Y_pred = inference_model.predict(X)\n",
    "    return ids_to_date_strs(Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-01']"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fast_predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check that it really is faster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "199 ms ± 3.94 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18.3 ms ± 366 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit fast_predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's more than a 10x speedup! And it would be even more if we were handling longer sequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fourth version: using TF-Addons's seq2seq implementation with a scheduled sampler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: due to a TF bug, this version only works using TensorFlow 2.2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we trained the previous model, at each time step _t_ we gave the model the target token for time step _t_ - 1. However, at inference time, the model did not get the previous target at each time step. Instead, it got the previous prediction. So there is a discrepancy between training and inference, which may lead to disappointing performance. To alleviate this, we can gradually replace the targets with the predictions, during training. For this, we just need to replace the `TrainingSampler` with a `ScheduledEmbeddingTrainingSampler`, and use a Keras callback to gradually increase the `sampling_probability` (i.e., the probability that the decoder will use the prediction from the previous time step rather than the target for the previous time step)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ageron/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:434: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 6s 19ms/step - loss: 1.6759 - accuracy: 0.3681 - val_loss: 1.4611 - val_accuracy: 0.4198\n",
      "Epoch 2/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 1.3872 - accuracy: 0.4583 - val_loss: 1.2827 - val_accuracy: 0.5021\n",
      "Epoch 3/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 1.0425 - accuracy: 0.6152 - val_loss: 0.8165 - val_accuracy: 0.7000\n",
      "Epoch 4/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 0.6353 - accuracy: 0.7673 - val_loss: 0.4365 - val_accuracy: 0.8464\n",
      "Epoch 5/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 0.3764 - accuracy: 0.8765 - val_loss: 0.2795 - val_accuracy: 0.9166\n",
      "Epoch 6/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 0.2506 - accuracy: 0.9269 - val_loss: 0.1805 - val_accuracy: 0.9489\n",
      "Epoch 7/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 0.1427 - accuracy: 0.9625 - val_loss: 0.1115 - val_accuracy: 0.9718\n",
      "Epoch 8/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 0.0853 - accuracy: 0.9804 - val_loss: 0.0785 - val_accuracy: 0.9809\n",
      "Epoch 9/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 0.1010 - accuracy: 0.9797 - val_loss: 0.1198 - val_accuracy: 0.9746\n",
      "Epoch 10/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 0.0447 - accuracy: 0.9917 - val_loss: 0.0306 - val_accuracy: 0.9949\n",
      "Epoch 11/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0241 - accuracy: 0.9961 - val_loss: 0.0205 - val_accuracy: 0.9968\n",
      "Epoch 12/20\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 0.0705 - accuracy: 0.9861 - val_loss: 0.0823 - val_accuracy: 0.9860\n",
      "Epoch 13/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0182 - accuracy: 0.9977 - val_loss: 0.0117 - val_accuracy: 0.9980\n",
      "Epoch 14/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0088 - accuracy: 0.9990 - val_loss: 0.0085 - val_accuracy: 0.9990\n",
      "Epoch 15/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0059 - accuracy: 0.9994 - val_loss: 0.0061 - val_accuracy: 0.9993\n",
      "Epoch 16/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0045 - accuracy: 0.9996 - val_loss: 0.0048 - val_accuracy: 0.9996\n",
      "Epoch 17/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0038 - accuracy: 0.9997 - val_loss: 0.0039 - val_accuracy: 0.9995\n",
      "Epoch 18/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0029 - accuracy: 0.9997 - val_loss: 0.0024 - val_accuracy: 0.9999\n",
      "Epoch 19/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0020 - accuracy: 0.9999 - val_loss: 0.0031 - val_accuracy: 0.9992\n",
      "Epoch 20/20\n",
      "313/313 [==============================] - 5s 16ms/step - loss: 0.0018 - accuracy: 0.9999 - val_loss: 0.0022 - val_accuracy: 0.9999\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_addons as tfa\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "n_epochs = 20\n",
    "encoder_embedding_size = 32\n",
    "decoder_embedding_size = 32\n",
    "units = 128\n",
    "\n",
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "sequence_lengths = keras.layers.Input(shape=[], dtype=np.int32)\n",
    "\n",
    "encoder_embeddings = keras.layers.Embedding(\n",
    "    len(INPUT_CHARS) + 1, encoder_embedding_size)(encoder_inputs)\n",
    "\n",
    "decoder_embedding_layer = keras.layers.Embedding(\n",
    "    len(INPUT_CHARS) + 2, decoder_embedding_size)\n",
    "decoder_embeddings = decoder_embedding_layer(decoder_inputs)\n",
    "\n",
    "encoder = keras.layers.LSTM(units, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_embeddings)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "sampler = tfa.seq2seq.sampler.ScheduledEmbeddingTrainingSampler(\n",
    "    sampling_probability=0.,\n",
    "    embedding_fn=decoder_embedding_layer)\n",
    "# we must set the sampling_probability after creating the sampler\n",
    "# (see https://github.com/tensorflow/addons/pull/1714)\n",
    "sampler.sampling_probability = tf.Variable(0.)\n",
    "\n",
    "decoder_cell = keras.layers.LSTMCell(units)\n",
    "output_layer = keras.layers.Dense(len(OUTPUT_CHARS) + 1)\n",
    "\n",
    "decoder = tfa.seq2seq.basic_decoder.BasicDecoder(decoder_cell,\n",
    "                                                 sampler,\n",
    "                                                 output_layer=output_layer)\n",
    "final_outputs, final_state, final_sequence_lengths = decoder(\n",
    "    decoder_embeddings,\n",
    "    initial_state=encoder_state)\n",
    "Y_proba = keras.layers.Activation(\"softmax\")(final_outputs.rnn_output)\n",
    "\n",
    "model = keras.models.Model(inputs=[encoder_inputs, decoder_inputs],\n",
    "                           outputs=[Y_proba])\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "def update_sampling_probability(epoch, logs):\n",
    "    proba = min(1.0, epoch / (n_epochs - 10))\n",
    "    sampler.sampling_probability.assign(proba)\n",
    "\n",
    "sampling_probability_cb = keras.callbacks.LambdaCallback(\n",
    "    on_epoch_begin=update_sampling_probability)\n",
    "history = model.fit([X_train, X_train_decoder], Y_train, epochs=n_epochs,\n",
    "                    validation_data=([X_valid, X_valid_decoder], Y_valid),\n",
    "                    callbacks=[sampling_probability_cb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not quite 100% validation accuracy, but close enough!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For inference, we could do the exact same thing as earlier, using a `GreedyEmbeddingSampler`. However, just for the sake of completeness, let's use a `SampleEmbeddingSampler` instead. It's almost the same thing, except that instead of using the argmax of the model's output to find the token ID, it treats the outputs as logits and uses them to sample a token ID randomly. This can be useful when you want to generate text. The `softmax_temperature` argument serves the \n",
    "same purpose as when we generated Shakespeare-like text (the higher this argument, the more random the generated text will be)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax_temperature = tf.Variable(1.)\n",
    "\n",
    "inference_sampler = tfa.seq2seq.sampler.SampleEmbeddingSampler(\n",
    "    embedding_fn=decoder_embedding_layer,\n",
    "    softmax_temperature=softmax_temperature)\n",
    "inference_decoder = tfa.seq2seq.basic_decoder.BasicDecoder(\n",
    "    decoder_cell, inference_sampler, output_layer=output_layer,\n",
    "    maximum_iterations=max_output_length)\n",
    "batch_size = tf.shape(encoder_inputs)[:1]\n",
    "start_tokens = tf.fill(dims=batch_size, value=sos_id)\n",
    "final_outputs, final_state, final_sequence_lengths = inference_decoder(\n",
    "    start_tokens,\n",
    "    initial_state=encoder_state,\n",
    "    start_tokens=start_tokens,\n",
    "    end_token=0)\n",
    "\n",
    "inference_model = keras.models.Model(inputs=[encoder_inputs],\n",
    "                                     outputs=[final_outputs.sample_id])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "def creative_predict_date_strs(date_strs, temperature=1.0):\n",
    "    softmax_temperature.assign(temperature)\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    Y_pred = inference_model.predict(X)\n",
    "    return ids_to_date_strs(Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-01']"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "creative_predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dates look good at room temperature. Now let's heat things up a bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2289607-12', '9272-03-01']"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "creative_predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"],\n",
    "                           temperature=5.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops, the dates are overcooked, now. Let's call them \"creative\" dates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fifth version: using TFA seq2seq, the Keras subclassing API and attention mechanisms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequences in this problem are pretty short, but if we wanted to tackle longer sequences, we would probably have to use attention mechanisms. While it's possible to code our own implementation, it's simpler and more efficient to use TF-Addons's implementation instead. Let's do that now, this time using Keras' subclassing API.\n",
    "\n",
    "**Warning**: due to a TensorFlow bug (see [this issue](https://github.com/tensorflow/addons/issues/1153) for details), the `get_initial_state()` method fails in eager mode, so for now we have to use the subclassing API, as Keras automatically calls `tf.function()` on the `call()` method (so it runs in graph mode)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this implementation, we've reverted back to using the `TrainingSampler`, for simplicity (but you can easily tweak it to use a `ScheduledEmbeddingTrainingSampler` instead). We also use a `GreedyEmbeddingSampler` during inference, so this class is pretty easy to use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DateTranslation(keras.models.Model):\n",
    "    def __init__(self, units=128, encoder_embedding_size=32,\n",
    "                 decoder_embedding_size=32, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.encoder_embedding = keras.layers.Embedding(\n",
    "            input_dim=len(INPUT_CHARS) + 1,\n",
    "            output_dim=encoder_embedding_size)\n",
    "        self.encoder = keras.layers.LSTM(units,\n",
    "                                         return_sequences=True,\n",
    "                                         return_state=True)\n",
    "        self.decoder_embedding = keras.layers.Embedding(\n",
    "            input_dim=len(OUTPUT_CHARS) + 2,\n",
    "            output_dim=decoder_embedding_size)\n",
    "        self.attention = tfa.seq2seq.LuongAttention(units)\n",
    "        decoder_inner_cell = keras.layers.LSTMCell(units)\n",
    "        self.decoder_cell = tfa.seq2seq.AttentionWrapper(\n",
    "            cell=decoder_inner_cell,\n",
    "            attention_mechanism=self.attention)\n",
    "        output_layer = keras.layers.Dense(len(OUTPUT_CHARS) + 1)\n",
    "        self.decoder = tfa.seq2seq.BasicDecoder(\n",
    "            cell=self.decoder_cell,\n",
    "            sampler=tfa.seq2seq.sampler.TrainingSampler(),\n",
    "            output_layer=output_layer)\n",
    "        self.inference_decoder = tfa.seq2seq.BasicDecoder(\n",
    "            cell=self.decoder_cell,\n",
    "            sampler=tfa.seq2seq.sampler.GreedyEmbeddingSampler(\n",
    "                embedding_fn=self.decoder_embedding),\n",
    "            output_layer=output_layer,\n",
    "            maximum_iterations=max_output_length)\n",
    "\n",
    "    def call(self, inputs, training=None):\n",
    "        encoder_input, decoder_input = inputs\n",
    "        encoder_embeddings = self.encoder_embedding(encoder_input)\n",
    "        encoder_outputs, encoder_state_h, encoder_state_c = self.encoder(\n",
    "            encoder_embeddings,\n",
    "            training=training)\n",
    "        encoder_state = [encoder_state_h, encoder_state_c]\n",
    "\n",
    "        self.attention(encoder_outputs,\n",
    "                       setup_memory=True)\n",
    "        \n",
    "        decoder_embeddings = self.decoder_embedding(decoder_input)\n",
    "\n",
    "        decoder_initial_state = self.decoder_cell.get_initial_state(\n",
    "            decoder_embeddings)\n",
    "        decoder_initial_state = decoder_initial_state.clone(\n",
    "            cell_state=encoder_state)\n",
    "        \n",
    "        if training:\n",
    "            decoder_outputs, _, _ = self.decoder(\n",
    "                decoder_embeddings,\n",
    "                initial_state=decoder_initial_state,\n",
    "                training=training)\n",
    "        else:\n",
    "            start_tokens = tf.zeros_like(encoder_input[:, 0]) + sos_id\n",
    "            decoder_outputs, _, _ = self.inference_decoder(\n",
    "                decoder_embeddings,\n",
    "                initial_state=decoder_initial_state,\n",
    "                start_tokens=start_tokens,\n",
    "                end_token=0)\n",
    "\n",
    "        return tf.nn.softmax(decoder_outputs.rnn_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "313/313 [==============================] - 7s 21ms/step - loss: 2.1549 - accuracy: 0.2295 - val_loss: 2.1450 - val_accuracy: 0.2239\n",
      "Epoch 2/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 1.8147 - accuracy: 0.3492 - val_loss: 1.4931 - val_accuracy: 0.4476\n",
      "Epoch 3/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 1.3585 - accuracy: 0.4909 - val_loss: 1.3168 - val_accuracy: 0.5100\n",
      "Epoch 4/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 1.2787 - accuracy: 0.5293 - val_loss: 1.1767 - val_accuracy: 0.5624\n",
      "Epoch 5/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 1.1236 - accuracy: 0.5776 - val_loss: 1.0769 - val_accuracy: 0.5907\n",
      "Epoch 6/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 1.0369 - accuracy: 0.6073 - val_loss: 1.0159 - val_accuracy: 0.6199\n",
      "Epoch 7/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 0.9752 - accuracy: 0.6295 - val_loss: 0.9723 - val_accuracy: 0.6346\n",
      "Epoch 8/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 0.9794 - accuracy: 0.6315 - val_loss: 0.9444 - val_accuracy: 0.6371\n",
      "Epoch 9/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 0.9338 - accuracy: 0.6415 - val_loss: 0.9296 - val_accuracy: 0.6381\n",
      "Epoch 10/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.9439 - accuracy: 0.6418 - val_loss: 0.9028 - val_accuracy: 0.6574\n",
      "Epoch 11/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.8807 - accuracy: 0.6637 - val_loss: 0.9835 - val_accuracy: 0.6369\n",
      "Epoch 12/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.7307 - accuracy: 0.6953 - val_loss: 0.8942 - val_accuracy: 0.6873\n",
      "Epoch 13/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.5833 - accuracy: 0.7327 - val_loss: 0.6944 - val_accuracy: 0.7391\n",
      "Epoch 14/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.4664 - accuracy: 0.7940 - val_loss: 0.6228 - val_accuracy: 0.7885\n",
      "Epoch 15/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.3205 - accuracy: 0.8740 - val_loss: 0.4825 - val_accuracy: 0.8780\n",
      "Epoch 16/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.2329 - accuracy: 0.9216 - val_loss: 0.3851 - val_accuracy: 0.9118\n",
      "Epoch 17/25\n",
      "313/313 [==============================] - 7s 21ms/step - loss: 0.2480 - accuracy: 0.9372 - val_loss: 0.2785 - val_accuracy: 0.9111\n",
      "Epoch 18/25\n",
      "313/313 [==============================] - 7s 22ms/step - loss: 0.1182 - accuracy: 0.9801 - val_loss: 0.1372 - val_accuracy: 0.9786\n",
      "Epoch 19/25\n",
      "313/313 [==============================] - 7s 22ms/step - loss: 0.0643 - accuracy: 0.9937 - val_loss: 0.0681 - val_accuracy: 0.9909\n",
      "Epoch 20/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 0.0446 - accuracy: 0.9952 - val_loss: 0.0487 - val_accuracy: 0.9934\n",
      "Epoch 21/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 0.0247 - accuracy: 0.9987 - val_loss: 0.0228 - val_accuracy: 0.9987\n",
      "Epoch 22/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 0.0456 - accuracy: 0.9918 - val_loss: 0.0207 - val_accuracy: 0.9985\n",
      "Epoch 23/25\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 0.0131 - accuracy: 0.9997 - val_loss: 0.0127 - val_accuracy: 0.9993\n",
      "Epoch 24/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.0360 - accuracy: 0.9933 - val_loss: 0.0146 - val_accuracy: 0.9990\n",
      "Epoch 25/25\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 0.0092 - accuracy: 0.9998 - val_loss: 0.0089 - val_accuracy: 0.9992\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = DateTranslation()\n",
    "optimizer = keras.optimizers.Nadam()\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "history = model.fit([X_train, X_train_decoder], Y_train, epochs=25,\n",
    "                    validation_data=([X_valid, X_valid_decoder], Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not quite 100% validation accuracy, but close. It took a bit longer to converge this time, but there were also more parameters and more computations per iteration. And we did not use a scheduled sampler.\n",
    "\n",
    "To use the model, we can write yet another little function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fast_predict_date_strs_v2(date_strs):\n",
    "    X = prepare_date_strs_padded(date_strs)\n",
    "    X_decoder = tf.zeros(shape=(len(X), max_output_length), dtype=tf.int32)\n",
    "    Y_probas = model.predict([X, X_decoder])\n",
    "    Y_pred = tf.argmax(Y_probas, axis=-1)\n",
    "    return ids_to_date_strs(Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1789-07-14', '2020-05-01']"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fast_predict_date_strs_v2([\"July 14, 1789\", \"May 01, 2020\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are still a few interesting features from TF-Addons that you may want to look at:\n",
    "* Using a `BeamSearchDecoder` rather than a `BasicDecoder` for inference. Instead of outputing the character with the highest probability, this decoder keeps track of the several candidates, and keeps only the most likely sequences of candidates (see chapter 16 in the book for more details).\n",
    "* Setting masks or specifying `sequence_length` if the input or target sequences may have very different lengths.\n",
    "* Using a `ScheduledOutputTrainingSampler`, which gives you more flexibility than the `ScheduledEmbeddingTrainingSampler` to decide how to feed the output at time _t_ to the cell at time _t_+1. By default it feeds the outputs directly to cell, without computing the argmax ID and passing it through an embedding layer. Alternatively, you specify a `next_inputs_fn` function that will be used to convert the cell outputs to inputs at the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.\n",
    "_Exercise: Go through TensorFlow's [Neural Machine Translation with Attention tutorial](https://homl.info/nmttuto)._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simply open the Colab and follow its instructions. Alternatively, if you want a simpler example of using TF-Addons's seq2seq implementation for Neural Machine Translation (NMT), look at the solution to the previous question. The last model implementation will give you a simpler example of using TF-Addons to build an NMT model using attention mechanisms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.\n",
    "_Exercise: Use one of the recent language models (e.g., GPT) to generate more convincing Shakespearean text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest way to use recent language models is to use the excellent [transformers library](https://huggingface.co/transformers/), open sourced by Hugging Face. It provides many modern neural net architectures (including BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet and more) for Natural Language Processing (NLP), including many pretrained models. It relies on either TensorFlow or PyTorch. Best of all: it's amazingly simple to use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's load a pretrained model. In this example, we will use OpenAI's GPT model, with an additional Language Model on top (just a linear layer with weights tied to the input embeddings). Let's import it and load the pretrained weights (this will download about 445MB of data to `~/.cache/torch/transformers`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TFOpenAIGPTLMHeadModel\n",
    "\n",
    "model = TFOpenAIGPTLMHeadModel.from_pretrained(\"openai-gpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will need a specialized tokenizer for this model. This one will try to use the [spaCy](https://spacy.io/) and [ftfy](https://pypi.org/project/ftfy/) libraries if they are installed, or else it will fall back to BERT's `BasicTokenizer` followed by Byte-Pair Encoding (which should be fine for most use cases)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import OpenAIGPTTokenizer\n",
    "\n",
    "tokenizer = OpenAIGPTTokenizer.from_pretrained(\"openai-gpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use the tokenizer to tokenize and encode the prompt text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 10), dtype=int32, numpy=\n",
       "array([[  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187]], dtype=int32)>"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt_text = \"This royal throne of kings, this sceptred isle\"\n",
    "encoded_prompt = tokenizer.encode(prompt_text,\n",
    "                                  add_special_tokens=False,\n",
    "                                  return_tensors=\"tf\")\n",
    "encoded_prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Easy! Next, let's use the model to generate text after the prompt. We will generate 5 different sentences, each starting with the prompt text, followed by 40 additional tokens. For an explanation of what all the hyperparameters do, make sure to check out this great [blog post](https://huggingface.co/blog/how-to-generate) by Patrick von Platen (from Hugging Face). You can play around with the hyperparameters to try to obtain better results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(5, 50), dtype=int32, numpy=\n",
       "array([[  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   239,   784,   645,  1184,   558,  1886,   688,  6437,\n",
       "          240,   784,   645,   507,   641,  5486,   240,   600,   636,\n",
       "          868,   604,   694,  2816,   485,  1894,   822,   481,  1491,\n",
       "          600,   880,  6061,   239,   256, 40477,   256,   600,   635,\n",
       "          538,   604,  1816,   525,   239],\n",
       "       [  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   488,  1288,   989,   640, 16605,   239,   256, 40477,\n",
       "          674,   481, 12744,  3912,   488,  3912,  5936,  2441,   811,\n",
       "          488,  1040,   485,   754,  3952,   239, 40477,   481,  1375,\n",
       "         1981,   833,  1210,   481, 17384,   488,   481,  3089,   488,\n",
       "          481,  4815,   509,   498,  1424],\n",
       "       [  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   980,   987,  1074, 13138,   240,   531,   501,   517,\n",
       "          836,   525, 12659,   485,  2642,   512,   239,   500,   616,\n",
       "         7339,   704,   989,  1259, 38752,   481,  9606,   498,   481,\n",
       "         6903,   239,   500,   616,  7339,   704,  3064,   994,   580,\n",
       "         3953,   617,   616,  4741,   488],\n",
       "       [  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187, 10595,   485,   510,   239,   244, 40477,   244,   481,\n",
       "         1424,  6404,   498,  1922,    23, 37492,   257,   244, 40477,\n",
       "          244,  3491,   240,   244,   603,   481,   618,   556,   246,\n",
       "         3386,   498,   524,   756,   239,   244,   616,  1276,   509,\n",
       "         1098, 10945,   498,   246,  6785],\n",
       "       [  616,  5751,  6404,   498,  9606,   240,   616, 26271,  7428,\n",
       "        16187,   544,  2203,   239,   616,   544,   246,  6460,   260,\n",
       "          850,   629,  4844,  3064,  3766,   240,   246,  1082,   806,\n",
       "         9606,   640, 32581,   240,   595,  7914,  1243,   488, 18535,\n",
       "          239,   249,   587,   538,   788,   775,  2319,   498,  1013,\n",
       "          525,   544,   595,   754,  1074]], dtype=int32)>"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_sequences = 5\n",
    "length = 40\n",
    "\n",
    "generated_sequences = model.generate(\n",
    "    input_ids=encoded_prompt,\n",
    "    do_sample=True,\n",
    "    max_length=length + len(encoded_prompt[0]),\n",
    "    temperature=1.0,\n",
    "    top_k=0,\n",
    "    top_p=0.9,\n",
    "    repetition_penalty=1.0,\n",
    "    num_return_sequences=num_sequences,\n",
    ")\n",
    "\n",
    "generated_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's decode the generated sequences and print them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this royal throne of kings, this sceptred isle. even if someone had given them permission, even if it were required, they would never have been allowed to live through the hell they've survived.'\n",
      "'they couldn't have known that.\n",
      "--------------------------------------------------------------------------------\n",
      "this royal throne of kings, this sceptred isle and these people are royalty.'\n",
      " then the mute prince and prince edward broke off and went to their rooms. \n",
      " the talk passed again between the princes and the guards and the princess was of great\n",
      "--------------------------------------------------------------------------------\n",
      "this royal throne of kings, this sceptred isle has its own highness, an alatte that waits to save you. in this kingdom your people must emulate the kings of the realm. in this kingdom your kin should be saved from this pit and\n",
      "--------------------------------------------------------------------------------\n",
      "this royal throne of kings, this sceptred isle belongs to me. \" \n",
      " \" the great throne of penvynne? \" \n",
      " \" indeed, \" said the king with a nod of his head. \" this world was once composed of a magical\n",
      "--------------------------------------------------------------------------------\n",
      "this royal throne of kings, this sceptred isle is empty. this is a modern - day fedaykin court, a place where kings are governed, not emperors and judges. i don't see any sign of life that is not their own\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for sequence in generated_sequences:\n",
    "    text = tokenizer.decode(sequence, clean_up_tokenization_spaces=True)\n",
    "    print(text)\n",
    "    print(\"-\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can try more recent (and larger) models, such as GPT-2, CTRL, Transformer-XL or XLNet, which are all available as pretrained models in the transformers library, including variants with Language Models on top. The preprocessing steps vary slightly between models, so make sure to check out this [generation example](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py) from the transformers documentation (this example uses PyTorch, but it will work with very little tweaks, such as adding `TF` at the beginning of the model class name, removing the `.to()` method calls, and using `return_tensors=\"tf\"` instead of `\"pt\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hope you enjoyed this chapter! :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}